Subject: Re: [storm] Proposed text changes for the connection allocation problem

From: "Black, David" <[email-masked]>
Date: Fri, 14 Sep 2012 20:25:53 -0400

Mike,

I apologize for the delay in getting to this.  I think your latest text is close to what's
needed, but I want to rewrite the latter part of your second paragraph for clarity and to
remove a "MUST" that appears easy to misinterpret (IMHO):

OLD
   To prevent the potential problem caused by the target sending the number of
   unexpected iSCSI control-type PDUs as determined by the MaxOustandingUnexpectedPDUs
   key allowed in the FullFeaturePhase when "late" connection allocation is practised,
   the iSER Hello exchanges MUST be used.  This allows the initiator to allocate the
   connection resources before sending the iSER Hello Messages.  The iSERHelloRequired
   key is used by the initiator to determine if it is dealing with a target that
   supports the iSER Hello exchanges.
NEW
   An initiator that employs "late" connection allocation may encounter problems (e.g.,
   RCaP connection closure) with a target that sends unexpected iSCSI PDUs immediately
   upon transitioning to Full Feature Phase, as allowed by the negotiated value of the
   the MaxOustandingUnexpectedPDUs key.  The only way to prevent this situation in
   full generality is to use iSER Hello Messages, as they enable the initiator to allocate
   its connection resources before sending its iSER Hello Message.  The iSERHelloRequired
   key is used by the initiator to determine if it is dealing with a target that
   supports the iSER Hello exchanges.  Fortunately, known iSER target implementations
   do not take full advantage of the number of allowed unexpected PDUs immediately upon
   transition into full feature phase, enabling an initiator workaround that involves
   a smaller quantity of connection resources prior to full-feature phase, as explained
   further below.
END

Would you please go ahead and submit a revised version of the draft with the text changes
that we've agreed to.  I'll want to review the changes one more time in the context of an
entire draft before (finally) sending it to our AD with a request to publish it as an RFC.

Thank you for your patience with this.

Thanks,
--David

From: Michael Ko [mailto:[email-masked]]
Sent: Sunday, August 19, 2012 9:49 PM
To: Black, David
Cc: [email-masked]
Subject: Re: [storm] Proposed text changes for the connection allocation problem

Changed "must be able to deal with" to "needs to be able to deal with" as suggested.

For the rest of the suggestions, I have created new text that replaces the "new" text in section 5.1.3:

An initiator that conforms to [RFC5046] allocates connection resources before seding the Login Request with the T (Transit) bit set to 1 and the NSG (Next Stage) field set to FullFeaturePhase.  (For brevity, this is referred to as "early" connection allocation.)  The current iSER specification relaxes this requirement to allow an initiator to allocate connection resources after it receives the final Login Response PDU from the target.  (For brevity, this is referred to as "late" connection allocation.)  To prevent the potential problem caused by the target sending the number of unexpected iSCSI control-type PDUs as determined by the MaxOustandingUnexpectedPDUs key allowed in the FullFeaturePhase when "late" connection allocation is practised, the iSER Hello exchanges MUST be used.  This allows the initiator to allocate the connection resources before sending the iSER Hello Messages.  The iSERHelloRequired key is used by the initiator to determine if it is dealing with a target that supports the iSER Hello exchanges.

In the following summary where "late" connection allocation is practised, an initiator that follows [RFC5046] is referred to as an "old" initiator; otherwise it is referred to as a "new" initiator.  Similarly, a target that does not support the iSERHelloRequired key (and responds with "NotUnderstood" when negotiating the iSERHelloRequired key) is referred to as an "old" target; otherwise it is referred to as a "new" target.  Note that an "old" target can still support the iSER Hello exchanges but this fact is not known by the initiator.  A "new" target can also respond with "No" when negotiating the iSERHelloREquired key.  In this case its beahvior with respect to "late" connection allocation is similar to an "old" target.

A "new" initiator will work fine with a "new" target.

For an "old" initiator and an "old" target, the failure by the initiator to handle the number of unexpected iSCSI control-type PDUs that are sent by the target before the buffer resources are allocated at the initiator can result in the failure of the iSER session caused by closure of the underlying RCaP connection.  For the "old" target, there is known implementation that sends one unexpected iSCSI control-type PDU after sending the final Login Response and then waits awhile before sending the next one.  This tends to alleviate somewhat the buffer allocation problem at the initiator.

For a "new" initiator and an "old" target, the failure by the initiator to handle the number of unexpected iSCSI control-type PDUs that are sent by the target before the buffer resources are allocated at the initiator can result in the failure of the iSER session caused by closure of the underlying RCaP connection.  A "new" initiator MAY choose to terminate the connection; otherwise it SHOULD do one of the following:

1. Allocate the connection resources before sending the final Login Request PDU.

2. Allocate one or more buffers for receiving unexpected control-type PDUs from the target before sending the final Login Request PDU.  This reduces the possibility of the unexpected control-type PDUs causing the RCaP connection to close before the connection resources have been allocated.

For an "old" initiator and a "new" target, if the iSERHelloRequired key is not negotiated, a "new" target MUST still respond with the iSER HelloReply Message when it receives the iSER Hello Message.  If the iSERHelloRequired key is negotiated to "No" or "NotUnderstood", a "new" target MAY choose to terminate the connection; otherwise it SHOULD delay sending any unexpected control-type PDUs until one of the following events has occurred:

1. A PDU is received from the initiator after it sends the final Login Response PDU.

2. A system configurable timeout period, say one second, has expired.

Mike
On Mon, Aug 13, 2012 at 3:27 PM, Black, David <[email-masked]<mailto:[email-masked]>> wrote:
Mike,

Thanks for posting this new text, and I apologize for the delayed response -
too much "stuff" going on in Vancouver ...

I think we could also use some additional text that summarizes the new/old, old/new
and new/new implementation interactions.  Some comments on your posted text ...

> If the iSCSI Layer on the initiator side allocates the connection resources
> to support RCaP only after it receives the final Login Response PDU from the
> target, then it may not be able to handle the number of unexpected iSCSI
> control-type PDUs (as declared by the MaxOutstandingUnexpectedPDUs key from
> the initiator) that can be sent by the target before the buffer resources
> are allocated at the initiator side.  In this case the iSERHelloRequired key
> MUST be negotiated to Yes so that the initiator can allocate the connection
> resources before sending the iSER Hello Message.  See section 5.1.3 for more
> details.

That "MUST be negotiated to Yes" seems wrong, as that can't be done if the target
responds with NotUnderstood.  Also, that text doesn't provide a sufficiently blunt
warning, IMHO - it ought to state that the consequences of "may not be able to
handle" include failure of the iSER session caused by closure of the underlying
RCaP connection.

> As the iSERHelloRequired key is not defined in [RFC5046], an initiator/target
> must be able to deal with a target/initiator that does not understand the key
> or support the iSER Hello exchanges.
That's correct, although I would change:
"must be able to deal with" ->"needs to be able to deal with".

> For a target, if the iSERHelloRequired key is not negotiated, it MUST still
> respond with the iSER HelloReply Message when it receives the iSER Hello Message.
> If the iSERHelloRequired key is negotiated to No or NotUnderstood, a target MAY
> choose to terminate the connection; otherwise it SHOULD delay sending any
> unexpected control-type PDUs until one of the following events has occurred:
>
> 1. A PDU is received from the initiator after it sends the final Login Response PDU.
>
> 2. A system configurable timeout period, say one second, has expired.

That's ok, I think additional text is needed somewhere to point out that there is
known target code that sends one PDU and then delays - that might go in the additional
summary text that I suggest above, and also should be included as part of the
rationale for:

> For an initiator, if the iSERHelloRequired key is negotiated to No or NotUnderstood,
> it MAY choose to terminate the connection; otherwise it SHOULD do one of the following:
>
> 1. Allocate the connection resources before sending the final Login Request PDU.
>
> 2. Allocate one or more buffers for receiving unexpected control-type PDUs from
> the target before sending the final Login Request PDU.  This reduces the possibility
> of the unexpected control-type PDUs causing the RCaP connection to close before
> the connection resources have been allocated.

Thanks,
--David

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Michael Ko
Sent: Tuesday, July 24, 2012 8:27 PM
To: [email-masked]<mailto:[email-masked]>
Subject: [storm] Proposed text changes for the connection allocation problem

These are the changes I intend to incorporate into the existing draft to solve the connection allocation problem by using the iSER Hello exchanges.  Please comment.

5.1.1  Initiator Behavior

original:

If the outcome of the iSCSI negotiation is to enable iSER-assisted mode, then on the initiator side, prior to sending the Login Request with the T (Transit) bit set to 1 and the NSG (Next Stage) field set to FullFeaturePhase, the iSCSI Layer MUST request the iSER Layer to allocate the connection resources necessary to support RCaP by invoking the Allocate_Connection_Resources Operational Primitive.

replacement:

If the outcome of the iSCSI negotiation is to enable iSER-assisted mode, then on the initiator side, prior to sending the Login Request with the T (Transit) bit set to 1 and the NSG (Next Stage) field set to FullFeaturePhase, the iSCSI Layer SHOULD request the iSER Layer to allocate the connection resources necessary to support RCaP by invoking the Allocate_Connection_Resources Operational Primitive.

deleted:

If the iSER Layer at the initiator is successful in allocating the connection resources necessary to support RCaP, the following events MUST occur in the specified sequence:  [The numbered items will be retained but without the numbering.]

new:

If the iSCSI Layer on the initiator side allocates the connection resources to support RCaP only after it receives the final Login Response PDU from the target, then it may not be able to handle the number of unexpected iSCSI control-type PDUs (as declared by the MaxOutstandingUnexpectedPDUs key from the initiator) that can be sent by the target before the buffer resources are allocated at the initiator side.  In this case the iSERHelloRequired key MUST be negotiated to Yes so that the initiator can allocate the connection resources before sending the iSER Hello Message.  See section 5.1.3 for more details.

5.1.3  iSER Hello Exchange

new:

The connection resources can be allocated at the initiator side after it has received the final Login Response PDU from the target.  To prevent the potential problem caused by the target sending the number of unexpected iSCSI control-type PDUs as determined by the MaxOustandingUnexpectedPDUs key allowed in the FullFeaturePhase, the iSER Hello exchanges can be used.  This allows the initiator to allocate the connection resources before sending the iSER Hello Message.  The iSERHelloRequired key is used by the initiator to determine if it is dealing with a target that supports the iSER Hello exchanges.

As the iSERHelloRequired key is not defined in [RFC5046], an initiator/target must be able to deal with a target/initiator that does not understand the key or support the iSER Hello exchanges.

For a target, if the iSERHelloRequired key is not negotiated, it MUST still respond with the iSER HelloReply Message when it recieves the iSER Hello Message.  If the iSERHelloRequired key is negotiated to No or NotUnderstood, a target MAY choose to terminate the connection; otherwise it SHOULD delay sending any unexpected control-type PDUs until one of the following events has occurred:

1. A PDU is received from the initiator after it sends the final Login Response PDU.

2. A system configurable timeout period, say one second, has expired.

For an initiator, if the iSERHelloRequired key is negotiated to No or NotUnderstood, it MAY choose to terminate the connection; otherwise it SHOULD do one of the following:

1. Allocate the connection resources before sending the final Login Request PDU.

2. Allocate one or more buffers for receiving unexpected control-type PDUs from the target before sending the final Login Request PDU.  This reduces the possibility of the unexpected control-type PDUs causing the RCaP connection to close before the connection resources have been allocated.

10.1.3.4 iSER Protocol Errors

original:

Conversely, it is a protocol error if the iSER Hello Message is sent by the iSER Layer at the initiator when iSERHelloRequired is negotiated to "No".

new:

Conversely, if the iSER Hello Message is sent by the iSER Layer at the initiator when iSERHelloRequired is negotiated to "No", the iSER Layer at the target MAY treat this as a protocol error or respond with an iSER HelloReply Message.

Mike

---

From: Michael Ko <[email-masked]>
Date: Sun, 16 Sep 2012 10:19:39 -0700

David,

I have submiited a new version of the iSER draft incorporating all the
latest changes so far, including your recommendations below.  I have also
reword item #7 in the summary of changes to reflect the new use of the
iSERHelloRequired key.

Mike

On Fri, Sep 14, 2012 at 5:25 PM, Black, David <[email-masked]> wrote:

> Mike,****
>
> ** **
>
> I apologize for the delay in getting to this.  I think your latest text is
> close to what’s****
>
> needed, but I want to rewrite the latter part of your second paragraph for
> clarity and to****
>
> remove a “MUST” that appears easy to misinterpret (IMHO):****
>
> ** **
>
> OLD****
>
>    To prevent the potential problem caused by the target sending the
> number of****
>
>    unexpected iSCSI control-type PDUs as determined by the
> MaxOustandingUnexpectedPDUs****
>
>    key allowed in the FullFeaturePhase when "late" connection allocation
> is practised,****
>
>    the iSER Hello exchanges MUST be used.  This allows the initiator to
> allocate the****
>
>    connection resources before sending the iSER Hello Messages.  The
> iSERHelloRequired****
>
>    key is used by the initiator to determine if it is dealing with a
> target that****
>
>    supports the iSER Hello exchanges.****
>
> NEW
>    An initiator that employs “late” connection allocation may encounter
> problems (e.g.,****
>
>    RCaP connection closure) with a target that sends unexpected iSCSI PDUs
> immediately****
>
>    upon transitioning to Full Feature Phase, as allowed by the negotiated
> value of the****
>
>    the MaxOustandingUnexpectedPDUs key.  The only way to prevent this
> situation in ****
>
>    full generality is to use iSER Hello Messages, as they enable the
> initiator to allocate****
>
>    its connection resources before sending its iSER Hello Message.  The
> iSERHelloRequired****
>
>    key is used by the initiator to determine if it is dealing with a
> target that****
>
>    supports the iSER Hello exchanges.  Fortunately, known iSER target
> implementations****
>
>    do not take full advantage of the number of allowed unexpected PDUs
> immediately upon****
>
>    transition into full feature phase, enabling an initiator workaround
> that involves****
>
>    a smaller quantity of connection resources prior to full-feature phase,
> as explained****
>
>    further below.****
>
> END****
>
> ** **
>
> Would you please go ahead and submit a revised version of the draft with
> the text changes****
>
> that we’ve agreed to.  I’ll want to review the changes one more time in
> the context of an****
>
> entire draft before (finally) sending it to our AD with a request to
> publish it as an RFC.****
>
> ** **
>
> Thank you for your patience with this.****
>
> ** **
>
> Thanks,
> --David****
>

---

From: "Black, David" <[email-masked]>
Date: Mon, 17 Sep 2012 11:29:49 -0400

Mike,

That new -12 version of the iSER draft looks fine, so I've sent it onward to our AD (Martin) for further review, along with a minor change to the PROTO writeup (new writeup is attached).  I do hope that this third attempt to complete this stage of storm WG technical work on iSER turns out to be the charm :).  Thank you for the quick turn-around on the new draft version.

Many thanks to everyone who's contributed to working through the iSER issues since the start of the year.  Although the delays to the iSER draft have not been ideal, they've occurred for good reasons, as we do need to make sure that the technical content of the draft is actually correct and matches the implementations.

Also - RDMAP draft authors - this is your heads-up that I will get to taking a look at the RDMAP draft in the next week or so ...

Thanks,
--David (storm WG co-chair)

From: Michael Ko [mailto:[email-masked]]
Sent: Sunday, September 16, 2012 1:20 PM
To: Black, David
Cc: [email-masked]
Subject: Re: [storm] Proposed text changes for the connection allocation problem

David,

I have submiited a new version of the iSER draft incorporating all the latest changes so far, including your recommendations below.  I have also reword item #7 in the summary of changes to reflect the new use of the iSERHelloRequired key.

Mike
On Fri, Sep 14, 2012 at 5:25 PM, Black, David <[email-masked]<mailto:[email-masked]>> wrote:
Mike,

OLD

Thanks,
--David