Subject: [storm] iSER - what to do

From: <[email-masked]>
Date: Tue, 26 Jun 2012 09:16:38 -0400

<WG chair hat on>

The discussion about whether the buffer behavior of the Linux iSER
implementation complies with RFC 5046 is missing at least one
important point - to my knowledge, there are *no* iSER
implementations that comply with RFC 5046.

After RFC 5046 was issued, implementers looked at its requirements and
decided to do a number of things differently - the important difference
for this discussion is that the iSER Hello exchange is not implemented,
even though RFC 5046 requires that it be used.  There are a number of
other "running code" changes in the current iSER draft that reflect
implementations and that are incompatible with RFC 5046, see Appendix A
of the draft for details.

The iSER work item was added to the storm WG's charter in order to produce
an RFC specification of iSER that matches the "running code" and hence will
interoperate with existing implementations.  Here's the text from the
WG charter:

	(6) iSER: Experience with Infiniband implementations suggest
	a few minor updates to reflect what has been done in practice.

As WG chair, I'm going to insist on the following:
1) The iSER draft has to match the implementations in order to be
	published as an RFC.  Replacing RFC 5046 with another RFC that
	won't interoperate with the implementations is pointless.
2) The issue encountered by the Linux iSER implementation cannot be
	ignored by the WG.  The workaround (1 extra buffer, at most
	one unsolicited PDU until initiator has time to deploy
	resources) at least needs to be explained in the draft.
3) While requiring the iSER Hello exchange may be the proverbial
	"right thing to do" going forward, it is necessary to provide
	at least guidance on how to interoperate with current iSER
	implementations that don't support the Hello exchange.

In contrast, as WG chair, I'm not going to insist that we adopt the
workaround as part of the standard.  For example, it's possible to
recommend (SHOULD) use of the iSER Hello exchange (explicitly negotiated
using the new key) and explain the workaround as what should be
done in order to interoperate if that key results in a NotUnderstood
response.  For this approach to the workaround, I think a lower case
"should" may be acceptable (e.g., as opposed to a "SHOULD" or "MUST",
but that's subject to further discussion.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: Michael Ko <[email-masked]>
Date: Tue, 26 Jun 2012 11:45:12 -0700

The last minute problem as related by Alex stems from the fact that the
iSER initiator posts its RX buffers for that connection only after the
final Login Response arrives.  During that time (after the target had sent
the Last Login Response but before
the Full Featured phase related RX-buffers are posted on the initiator
side) the target may send the first NOP-IN as it considers the connection
in Full Featured phase already and MaxOutstandingUnexpectedPDUs has been
negotiated to a non-zero value.

The current solution is for the initiator to post an extra buffer before
sending the last Login Request as the target most probably sends the next
NOP-IN only after some timeout period measuring in seconds or hundreds of
milliseconds.  But this is not a foolproof solution as the target MAY
theoretically send up to MaxOustandingUnexpectedPUDs to the initiator.

The new proposed solution is to require iSER Hello exchanges in order to
delay the onset of the iSER assisted Full Featured phase until the
iSERHelloReply PDU has arrived at the target.  This allows the initiator to
delay posting its RX buffers until after receiving the final LoginResponse
returns but before sending the iSER Hello PDU.  To support this new
proposed solution, the iSERHelloRequired key must be negotiated to Yes.
This require both sides support the new iSERHelloRequired key, leading to
the need for clarifying texts suggested by David in the spec in order to
specify what should be done otherwise.

But if we look at the current solution, the initiator is already posting an
extra buffer before sending the last Login Request.  Given that the
existing implementation is broken and has to be fixed anyway, how difficult
would it be for it to post MaxOustandingUnexpectedPDU instead of just an
extra one before sending the last Login Request?  This complies with the
spec and is guaranteed to work.  The alternative is to require the use of
and implement iSER Hello exchanges.  If the partner does not support the
new iSERHelloRequired key, the solution fails anyway.  It seems to me it
makes more sense (and a cleaner soluton besides) for the initiator to
allocate the iSER resources, i.e., MaxOustandingUnexpectedPDUs, before
sending the last Login Request as stated in the spec than requiring both
sides to support iSER Hello exchanges and negotiate the new key.

Mike

On Tue, Jun 26, 2012 at 6:16 AM, <[email-masked]> wrote:

> _______________________________________________
> storm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/storm
>

---

From: <[email-masked]>
Date: Tue, 26 Jun 2012 15:02:08 -0400

Mike,

<WG chair hat off>

Thanks for responding - we will eventually get to a conclusion in this discussion :-).

> theoretically send up to MaxOustandingUnexpectedPUDs to the initiator.

Right - my thought about how to make it fool-resistant is to require that
the target MUST NOT send the next unsolicited PDU until at least 200ms
after the first one or after receipt of a full-phase PDU from the initiator
(that requirement could be a "SHOULD NOT" with a warning that sending
sooner risks the initiator closing the RCaP connection).  I pulled 200ms
out of my proverbial "hat" and would like to see comments on what a
good number would be for both IB and iWarp.

> But if we look at the current solution, the initiator is already
> posting an extra buffer before sending the last Login Request.  Given
> that the existing implementation is broken and has to be fixed anyway,
> how difficult would it be for it to post MaxOustandingUnexpectedPDU
> instead of just an extra one before sending the last Login Request?

That's a question for Alexander and other implementers.  My guess is that
we're talking about quite a few buffers, and there may be resource management
advantages to not allocating that full set of resources until the initiator
is committed to the connection.

Thanks,
--David

From: Michael Ko [mailto:[email-masked]]
Sent: Tuesday, June 26, 2012 2:45 PM
To: Black, David
Cc: [email-masked]
Subject: Re: [storm] iSER - what to do

The last minute problem as related by Alex stems from the fact that the iSER initiator posts its RX buffers for that connection only after the final Login Response arrives.  During that time (after the target had sent the Last Login Response but before
the Full Featured phase related RX-buffers are posted on the initiator side) the target may send the first NOP-IN as it considers the connection in Full Featured phase already and MaxOutstandingUnexpectedPDUs has been negotiated to a non-zero value.

The current solution is for the initiator to post an extra buffer before sending the last Login Request as the target most probably sends the next NOP-IN only after some timeout period measuring in seconds or hundreds of milliseconds.  But this is not a foolproof solution as the target MAY theoretically send up to MaxOustandingUnexpectedPUDs to the initiator.

The new proposed solution is to require iSER Hello exchanges in order to delay the onset of the iSER assisted Full Featured phase until the iSERHelloReply PDU has arrived at the target.  This allows the initiator to delay posting its RX buffers until after receiving the final LoginResponse returns but before sending the iSER Hello PDU.  To support this new proposed solution, the iSERHelloRequired key must be negotiated to Yes.  This require both sides support the new iSERHelloRequired key, leading to the need for clarifying texts suggested by David in the spec in order to specify what should be done otherwise.

But if we look at the current solution, the initiator is already posting an extra buffer before sending the last Login Request.  Given that the existing implementation is broken and has to be fixed anyway, how difficult would it be for it to post MaxOustandingUnexpectedPDU instead of just an extra one before sending the last Login Request?  This complies with the spec and is guaranteed to work.  The alternative is to require the use of and implement iSER Hello exchanges.  If the partner does not support the new iSERHelloRequired key, the solution fails anyway.  It seems to me it makes more sense (and a cleaner soluton besides) for the initiator to allocate the iSER resources, i.e., MaxOustandingUnexpectedPDUs, before sending the last Login Request as stated in the spec than requiring both sides to support iSER Hello exchanges and negotiate the new key.

Mike
On Tue, Jun 26, 2012 at 6:16 AM, <[email-masked]<mailto:[email-masked]>> wrote:
<WG chair hat on>

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953<tel:%2B1%20%28508%29%20293-7953>             FAX: +1 (508) 293-7786<tel:%2B1%20%28508%29%20293-7786>
[email-masked]<mailto:[email-masked]>        Mobile: +1 (978) 394-7754<tel:%2B1%20%28978%29%20394-7754>
----------------------------------------------------

_______________________________________________
storm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/storm