Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response

From: <[email-masked]>
Date: Mon, 11 Jun 2012 11:34:57 -0400

Hi Mallikarjun,

IMHO, the reason that this is a protocol issue is that two implementations that comply with the RFC can nonetheless reliably and repeatedly fail to interoperate because the lack of additional buffers causes the RCaP connection to close before full feature phase is reached by the initiator.

I believe that we have a responsibility to tell implementers what can go wrong here and how to avoid it - the technique you describe (post all unsolicited buffers before sending final negotiation message) could be mentioned as part of this, and we should also describe what's possible with use of a single buffer, as that approach is being pursued as Alexander describes.

Thanks,
--David

From: Mallikarjun Chadalapaka [mailto:[email-masked]]
Sent: Friday, May 25, 2012 9:02 PM
To: Black, David; [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response

I am curious to understand a bit more on why this is a protocol issue per se.

Seems like one way to address this problem is via an implementation approach with the initiator posting in advance the negotiated number of unsolicited PDU buffers, at the same time it makes the (final) negotiation offer. As the in-bound unsolicited PDUs can technically arrive any time after the offer, due to standard network latency mechanics that Alexander summarized. Has that approach been considered?

Mallikarjun

From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Thursday, May 24, 2012 12:03 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response
Importance: High

Mike (Ko) and Alexander,

Mike is of course correct that iSER Hello usage can be forced by negotiating iSERHelloRequired to "Yes".  However, existing implementations are likely to reply with iSERHelloRequired=NotUnderstood, so we do need to specify what should be done in order to interoperate with an implementation that refuses to deal with the iSER Hello exchange.

I think the situation that Alexander described should be documented in a new section 5.1.4 of the iSER draft.  My general rule of thumb on this sort of surprise found by implementers in the "running code" is that it indicates that something is missing in the spec.  I believe that Alexander has described the solution - more below.

The new section 5.1.4 (suggested section title: Omission of the iSER Hello Exchange) should describe default omission of the exchange, use of iSERHelloRequired key to omit the iSER Hello exchange, and the consequences of target use of unsolicited PDUs after login when the exchange is omitted, including IB's use of NOP-IN (as a keep-alive measure, right?)

The crucial requirements points that I take away from Alexander's description are that if the iSER Hello exchange is omitted, then:

1) The target MAY send *one* unsolicited PDU immediately after sending the Login Response.

2) The target MUST wait at least 200ms (use some other number if 200ms isn't a good choice)
or until it receives a full feature mode PDU from the initiator before sending a
second unsolicited PDU in order to ensure that initiator has sufficient
      time to allocate the full feature buffer resources for the connection.
3) The initiator SHOULD allocate at least one additional buffer for use during login (so that
at least two buffers are in use during login) in order to receive an unsolicited PDU
that may follow login completion.  Failure to allocate this second buffer may
cause connection termination if no buffer is available when an unsolicited PDU arrives.

Both Mike and I are on vacation, so it may be a few weeks until we can agree on the new text and get a -12 version of the draft with that new text submitted.  In the interim, I've asked our AD (Martin Stiemerling) to hold off on further processing of the iSER draft until a -12 version with this new text is submitted.  I'd prefer to work this text out now rather than deal with it as an IETF Last Call comment - as the problem turned up in actual implementations, I think it's worth the extra month that it's likely to take to get correct text on how to avoid the problem into the draft.

I'd suggest that Mike Ko post an initial draft of the text for the new section 5.1.4 to the list when he resurfaces ...

Thanks,
--David

From: Michael Ko [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Monday, May 21, 2012 10:23 AM
To: Alexander Nezhinsky
Cc: [email-masked]<mailto:[email-masked]>; Black, David; Or Gerlitz; Mike Christie
Subject: Re: iSER - problem with unsolicited NOP-IN right after final Login Response

Alex,

The iSER Hello support has never been removed in the latest spec.  Only its use is made optional.  So during login negotiation, just negotiate iSERHelloRequired to Yes.

Mike
On Mon, May 21, 2012 at 6:15 AM, Alexander Nezhinsky <[email-masked]<mailto:[email-masked]>> wrote:
Hi

I understand that it is a bad timing for sending this kind of mail, now that iSER draft was submitted,
but actually we still have a small problem.
It is related to the final Login Response handling and the transition to Full-Featured phase on the initiator side in
Infiniband setups.

When the target receives the final Login Request it send the final Login Response and from its perspective
the connection is now in Full Featured Phase (assuming that it agreed to transition in the Login Response being sent).

This means that the target is ready to accept SCSI commands, Text Requests etc. sent by the initiator.
It also means that the target is eligible to send some unsolicited PDUs, notably unsolicited NOP-INs.

With IB sending NOP-IN periodically is the easiest (an almost only feasible) way to determine closed connections
reliably, because this kind of error is delivered to user only in response to a previously initiated TX operation.

This leaves the initiator in a dubious position. It posts its RX buffers for that connection only when the final
Login Response arrives. But during that time (after the target had sent the Last Login Response but before
the Full Featured phase related RX-buffers are posted on the initiator side) the target may send
the first NOP-IN as it considers the connection in Full Featured phase already and NumOfUnsolicited PDUs
accounting for NOP-INs has been negotiated to a non-zero value.

If the initiator works with a single RX-buffer posted during the entire login phase (which is a logical thing to do
judging by the login exchange protocol) then an error occurs, as no buffers are posted when the NOP-IN arrives
and the connection is shut down.

Posting a single extra buffer before sending the last Login Request only alleviates the problem. Although this
often solves it in practical terms (as the target most probably sends the next NOP-IN only after some timeout
period measuring seconds or hundreds of milliseconds), it does not solves it in terms of protocol completeness,
as the target MAY theoretically send more than one NOP-IN until FF buffers are posted.

This issue was encountered recently in linux iscsi/iser initiator and the above solution has been applied to solve
it against the existing target implementation (STGT), but the initiator remains exposed to this kind of errors.

The solution is actually quite simple (theoretically) - if we bring back the requirement for iSER Hello exchange
then the iSER assisted Full Featured phase does not commence until HelloReply PDU arrives at the target
and the initiator has a definitive point in time when it can safely post its RX buffers - after the final LoginResponse
returns but before it sends iSER Hello PDU.

In practical terms it means that iSER Hello support requirement should be brought back to spec, which is a hassle.

Should we decide on this now?

Alexander

P.S. : Thanx to Mike Christie and Or Gerlitz, the maintainers of linux iSCSI and iSER initiator for raising the issue.

---

From: <[email-masked]>
Date: Mon, 11 Jun 2012 17:59:14 +0000

That sounds like the right way to look at it.  The whole point of protocol standards is to specify what is needed to interoperate.  Not more, but also not less.  Whenever two implementations both comply with a standard yet do not interoperate, that's a bug in the standard.

paul

On Jun 11, 2012, at 11:34 AM, <[email-masked]<mailto:[email-masked]>>
 <[email-masked]<mailto:[email-masked]>> wrote:

Hi Mallikarjun,

I believe that we have a responsibility to tell implementers what can go wrong here and how to avoid it - the technique you describe (post all unsolicited buffers before sending final negotiation message) could be mentioned as part of this, and we should also describe what’s possible with use of a single buffer, as that approach is being pursued as Alexander describes.

Thanks,
--David

From: Mallikarjun Chadalapaka [mailto:[email-masked]]
Sent: Friday, May 25, 2012 9:02 PM
To: Black, David; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response

Mallikarjun

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Thursday, May 24, 2012 12:03 AM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response
Importance: High

Mike (Ko) and Alexander,

Mike is of course correct that iSER Hello usage can be forced by negotiating iSERHelloRequired to “Yes”.  However, existing implementations are likely to reply with iSERHelloRequired=NotUnderstood, so we do need to specify what should be done in order to interoperate with an implementation that refuses to deal with the iSER Hello exchange.

I think the situation that Alexander described should be documented in a new section 5.1.4 of the iSER draft.  My general rule of thumb on this sort of surprise found by implementers in the “running code” is that it indicates that something is missing in the spec.  I believe that Alexander has described the solution - more below.

The new section 5.1.4 (suggested section title: Omission of the iSER Hello Exchange) should describe default omission of the exchange, use of iSERHelloRequired key to omit the iSER Hello exchange, and the consequences of target use of unsolicited PDUs after login when the exchange is omitted, including IB’s use of NOP-IN (as a keep-alive measure, right?)

The crucial requirements points that I take away from Alexander’s description are that if the iSER Hello exchange is omitted, then:

2) The target MUST wait at least 200ms (use some other number if 200ms isn’t a good choice)

Both Mike and I are on vacation, so it may be a few weeks until we can agree on the new text and get a -12 version of the draft with that new text submitted.  In the interim, I’ve asked our AD (Martin Stiemerling) to hold off on further processing of the iSER draft until a -12 version with this new text is submitted.  I’d prefer to work this text out now rather than deal with it as an IETF Last Call comment - as the problem turned up in actual implementations, I think it’s worth the extra month that it’s likely to take to get correct text on how to avoid the problem into the draft.

I’d suggest that Mike Ko post an initial draft of the text for the new section 5.1.4 to the list when he resurfaces ...

Thanks,
--David

Alex,

Mike

Alexander

_______________________________________________
storm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/storm

---

From: Mallikarjun Chadalapaka <[email-masked]>
Date: Mon, 11 Jun 2012 18:32:54 +0000

Sorry, if the implementation is promising x number of unsolicited buffers but it is has <x buffers ready at the end of negotiation, I would consider the implementation to not comply with the standard anymore. Standard on its part should clearly define the semantics of the promise - which I presume it does in this case.

I am all for providing helpful implementation guidance in the standard, but we have to be cautious and re-confirm the need for the guidance, when we start providing guidance down to latency numbers (see #2) - that's what got me concerned when I read the original note.

Thanks.

Mallikarjun

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Monday, June 11, 2012 10:59 AM
To: [email-masked]
Cc: Mallikarjun Chadalapaka; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response

paul

Hi Mallikarjun,

Thanks,
--David

Mallikarjun

Mike (Ko) and Alexander,

2) The target MUST wait at least 200ms (use some other number if 200ms isn't a good choice) or until it receives a full feature mode PDU from the initiator before sending a second unsolicited PDU in order to ensure that initiator has sufficient
      time to allocate the full feature buffer resources for the connection.
3) The initiator SHOULD allocate at least one additional buffer for use during login (so that at least two buffers are in use during login) in order to receive an unsolicited PDU that may follow login completion.  Failure to allocate this second buffer may cause connection termination if no buffer is available when an unsolicited PDU arrives.

Thanks,
--David

Alex,

Mike

I understand that it is a bad timing for sending this kind of mail, now that iSER draft was submitted, but actually we still have a small problem.
It is related to the final Login Response handling and the transition to Full-Featured phase on the initiator side in Infiniband setups.

When the target receives the final Login Request it send the final Login Response and from its perspective the connection is now in Full Featured Phase (assuming that it agreed to transition in the Login Response being sent).

With IB sending NOP-IN periodically is the easiest (an almost only feasible) way to determine closed connections reliably, because this kind of error is delivered to user only in response to a previously initiated TX operation.

This leaves the initiator in a dubious position. It posts its RX buffers for that connection only when the final Login Response arrives. But during that time (after the target had sent the Last Login Response but before the Full Featured phase related RX-buffers are posted on the initiator side) the target may send the first NOP-IN as it considers the connection in Full Featured phase already and NumOfUnsolicited PDUs accounting for NOP-INs has been negotiated to a non-zero value.

If the initiator works with a single RX-buffer posted during the entire login phase (which is a logical thing to do judging by the login exchange protocol) then an error occurs, as no buffers are posted when the NOP-IN arrives and the connection is shut down.

Posting a single extra buffer before sending the last Login Request only alleviates the problem. Although this often solves it in practical terms (as the target most probably sends the next NOP-IN only after some timeout period measuring seconds or hundreds of milliseconds), it does not solves it in terms of protocol completeness, as the target MAY theoretically send more than one NOP-IN until FF buffers are posted.

This issue was encountered recently in linux iscsi/iser initiator and the above solution has been applied to solve it against the existing target implementation (STGT), but the initiator remains exposed to this kind of errors.

The solution is actually quite simple (theoretically) - if we bring back the requirement for iSER Hello exchange then the iSER assisted Full Featured phase does not commence until HelloReply PDU arrives at the target and the initiator has a definitive point in time when it can safely post its RX buffers - after the final LoginResponse returns but before it sends iSER Hello PDU.

Alexander

---

From: <[email-masked]>
Date: Mon, 11 Jun 2012 18:41:06 +0000

If the standard says that the expected semantics is that the buffers promised are all there at end of negotiation, then yes, it's an implementation issue.  In that case, the original premise (both ends compliant) would not be true -- one end is in violation.  On the other hand, if the specification of the negotiation process doesn't say when the promised resources have to be available, that's a hole in the standard.

	paul

---

From: Mallikarjun Chadalapaka <[email-masked]>
Date: Mon, 11 Jun 2012 18:56:32 +0000

Paul,

I completely agree. This promise is conceptually similar to the CmdSN window promise. FWIW, I did not believe both sides were compliant in this case.

Mallikarjun

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Monday, June 11, 2012 11:41 AM
To: Mallikarjun Chadalapaka
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response

	paul

---

From: <[email-masked]>
Date: Mon, 11 Jun 2012 21:44:24 -0400

> FWIW, I did not believe both sides were compliant in this case.

There's an additional consideration here - one of the goals of the iSER
update is to match "running code", and we have already made a number of
changes that diverge from RFC 5046 for this reason, including in this
area of iSER start-up (see Appendix A of the iSER draft for details).
As a result, we are  already well into non-compliance territory, as
omission of the Hello messages does not comply with RFC 5046, but those
messages are simply not implemented in practice.

The implementation under discussion is (I believe) the predominant iSER
implementation and hence I think the new iSER RFC needs to explain what
that implementation has done (even if it may not be what should have been
done), and (more importantly) explain how to interoperate.  The key
requirement is the following (IMHO, a fine example of "be conservative
in what you send"):

> > 2) The target MUST wait at least 200ms (use some other number if 200ms
> > isn't a good choice) or until it receives a full feature mode PDU from
> > the initiator before sending a second unsolicited PDU in order to ensure
> > that initiator has sufficient time to allocate the full feature buffer
> > resources for the connection.

Not putting this into the iSER update requires either that Hello messages
be implemented (not done, primarily for optimization reasons), or that the
initiator effectively commit the buffer resources when starting negotiation.

Thanks,
--David

> Sent: Monday, June 11, 2012 2:57 PM
> To: [email-masked]
> Cc: Black, David; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked]
> I completely agree. This promise is conceptually similar to the CmdSN window
> promise. FWIW, I did not believe both sides were compliant in this case.
> 
> Mallikarjun
> 
> 
> Cc: [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked]
> If the standard says that the expected semantics is that the buffers promised
> are all there at end of negotiation, then yes, it's an implementation issue.
> In that case, the original premise (both ends compliant) would not be true --
> one end is in violation.  On the other hand, if the specification of the
> negotiation process doesn't say when the promised resources have to be
> available, that's a hole in the standard.
> 
> 	paul
> 
> > Sorry, if the implementation is promising x number of unsolicited buffers
> but it is has <x buffers ready at the end of negotiation, I would consider the
> implementation to not comply with the standard anymore. Standard on its part
> should clearly define the semantics of the promise - which I presume it does
> in this case.
> >
> > I am all for providing helpful implementation guidance in the standard, but
> we have to be cautious and re-confirm the need for the guidance, when we start
> providing guidance down to latency numbers (see #2) - that's what got me
> concerned when I read the original note.
> >
> > Thanks.
> >
> > Mallikarjun
> >
> >
> > Cc: Mallikarjun Chadalapaka; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked]
> > That sounds like the right way to look at it.  The whole point of protocol
> standards is to specify what is needed to interoperate.  Not more, but also
> not less.  Whenever two implementations both comply with a standard yet do not
> interoperate, that's a bug in the standard.
> >
> > paul
> >
> > On Jun 11, 2012, at 11:34 AM,
> <[email-masked]<mailto:[email-masked]>>
> > IMHO, the reason that this is a protocol issue is that two implementations
> that comply with the RFC can nonetheless reliably and repeatedly fail to
> interoperate because the lack of additional buffers causes the RCaP connection
> to close before full feature phase is reached by the initiator.
> >
> > I believe that we have a responsibility to tell implementers what can go
> wrong here and how to avoid it - the technique you describe (post all
> unsolicited buffers before sending final negotiation message) could be
> mentioned as part of this, and we should also describe what's possible with
> use of a single buffer, as that approach is being pursued as Alexander
> describes.
> >
> > Thanks,
> > --David
> >
> > To: Black, David; [email-masked]<mailto:[email-masked]>;
> [email-masked]<mailto:[email-masked]>
> > Cc: [email-masked]<mailto:[email-masked]>;
> [email-masked]<mailto:[email-masked]>;
> > Seems like one way to address this problem is via an implementation approach
> with the initiator posting in advance the negotiated number of unsolicited PDU
> buffers, at the same time it makes the (final) negotiation offer. As the in-
> bound unsolicited PDUs can technically arrive any time after the offer, due to
> standard network latency mechanics that Alexander summarized. Has that
> approach been considered?
> >
> > Mallikarjun
> >
> >
> >
> >
> > From: [email-masked]<mailto:[email-masked]> [mailto:storm-
> [email-masked]] On Behalf Of [email-masked]<mailto:[email-masked]>
> > To: [email-masked]<mailto:[email-masked]>;
> [email-masked]<mailto:[email-masked]>
> > Cc: [email-masked]<mailto:[email-masked]>;
> [email-masked]<mailto:[email-masked]>;
> > Mike is of course correct that iSER Hello usage can be forced by negotiating
> iSERHelloRequired to "Yes".  However, existing implementations are likely to
> reply with iSERHelloRequired=NotUnderstood, so we do need to specify what
> should be done in order to interoperate with an implementation that refuses to
> deal with the iSER Hello exchange.
> >
> > I think the situation that Alexander described should be documented in a new
> section 5.1.4 of the iSER draft.  My general rule of thumb on this sort of
> surprise found by implementers in the "running code" is that it indicates that
> something is missing in the spec.  I believe that Alexander has described the
> solution - more below.
> >
> > The new section 5.1.4 (suggested section title: Omission of the iSER Hello
> Exchange) should describe default omission of the exchange, use of
> iSERHelloRequired key to omit the iSER Hello exchange, and the consequences of
> target use of unsolicited PDUs after login when the exchange is omitted,
> including IB's use of NOP-IN (as a keep-alive measure, right?)
> >
> > The crucial requirements points that I take away from Alexander's
> description are that if the iSER Hello exchange is omitted, then:
> >
> > 2) The target MUST wait at least 200ms (use some other number if 200ms isn't
> a good choice) or until it receives a full feature mode PDU from the initiator
> before sending a second unsolicited PDU in order to ensure that initiator has
> sufficient
> > 3) The initiator SHOULD allocate at least one additional buffer for use
> during login (so that at least two buffers are in use during login) in order
> to receive an unsolicited PDU that may follow login completion.  Failure to
> allocate this second buffer may cause connection termination if no buffer is
> available when an unsolicited PDU arrives.
> >
> > Both Mike and I are on vacation, so it may be a few weeks until we can agree
> on the new text and get a -12 version of the draft with that new text
> submitted.  In the interim, I've asked our AD (Martin Stiemerling) to hold off
> on further processing of the iSER draft until a -12 version with this new text
> is submitted.  I'd prefer to work this text out now rather than deal with it
> as an IETF Last Call comment - as the problem turned up in actual
> implementations, I think it's worth the extra month that it's likely to take
> to get correct text on how to avoid the problem into the draft.
> >
> > I'd suggest that Mike Ko post an initial draft of the text for the new
> section 5.1.4 to the list when he resurfaces ...
> >
> > Thanks,
> > --David
> >
> > The iSER Hello support has never been removed in the latest spec.  Only its
> use is made optional.  So during login negotiation, just negotiate
> iSERHelloRequired to Yes.
> >
> > Mike
> > On Mon, May 21, 2012 at 6:15 AM, Alexander Nezhinsky
> <[email-masked]<mailto:[email-masked]>> wrote:
> > Hi
> >
> > I understand that it is a bad timing for sending this kind of mail, now that
> iSER draft was submitted, but actually we still have a small problem.
> > It is related to the final Login Response handling and the transition to
> Full-Featured phase on the initiator side in Infiniband setups.
> >
> > When the target receives the final Login Request it send the final Login
> Response and from its perspective the connection is now in Full Featured Phase
> (assuming that it agreed to transition in the Login Response being sent).
> >
> > With IB sending NOP-IN periodically is the easiest (an almost only feasible)
> way to determine closed connections reliably, because this kind of error is
> delivered to user only in response to a previously initiated TX operation.
> >
> > This leaves the initiator in a dubious position. It posts its RX buffers for
> that connection only when the final Login Response arrives. But during that
> time (after the target had sent the Last Login Response but before the Full
> Featured phase related RX-buffers are posted on the initiator side) the target
> may send the first NOP-IN as it considers the connection in Full Featured
> phase already and NumOfUnsolicited PDUs accounting for NOP-INs has been
> negotiated to a non-zero value.
> >
> > If the initiator works with a single RX-buffer posted during the entire
> login phase (which is a logical thing to do judging by the login exchange
> protocol) then an error occurs, as no buffers are posted when the NOP-IN
> arrives and the connection is shut down.
> >
> > Posting a single extra buffer before sending the last Login Request only
> alleviates the problem. Although this often solves it in practical terms (as
> the target most probably sends the next NOP-IN only after some timeout period
> measuring seconds or hundreds of milliseconds), it does not solves it in terms
> of protocol completeness, as the target MAY theoretically send more than one
> NOP-IN until FF buffers are posted.
> >
> > This issue was encountered recently in linux iscsi/iser initiator and the
> above solution has been applied to solve it against the existing target
> implementation (STGT), but the initiator remains exposed to this kind of
> errors.
> >
> > The solution is actually quite simple (theoretically) - if we bring back the
> requirement for iSER Hello exchange then the iSER assisted Full Featured phase
> does not commence until HelloReply PDU arrives at the target and the initiator
> has a definitive point in time when it can safely post its RX buffers - after
> the final LoginResponse returns but before it sends iSER Hello PDU.
> >
> > P.S. : Thanx to Mike Christie and Or Gerlitz, the maintainers of linux iSCSI
> and iSER initiator for raising the issue.
> >

---

From: Michael Ko <[email-masked]>
Date: Mon, 11 Jun 2012 19:00:35 -0700

I agree with Mallikarjun.  In the scenario described by Alex, the initiator
side was not compliant with the standard.

>From section 5.1.1 of the iSER draft on the initiator behavior during
connection setup:

"If the outcome of the iSCSI negotiation is to enable iSER-assisted mode,
then on the initiator side, prior to sending the Login Request with the T
(Transit) bit set to 1 and the NSG (Next Stage) field set to
FullFeaturePhase, the iSCSI Layer MUST request the iSER Layer to allocate
the connection resources necessary to support RCaP by invoking the
Allocate_Connection_Resources Operational Primitive."

It clearly states that the initiator must "allocate the connection
resources necessary to support RCaP" before sending the Login Request to
transition to FullFeaturePhase.  Had the initiator been compliant with this
requirement, it would not run into the insufficient Rx buffer problem.
Negotiating iSERHelloRequired to "Yes" is a way to shortcut this
requirement and buy time for the initiator if both sides support it, but it
should not be the modus operandi, and I don't think clarifying text is
needed to justify its usage to solve this particular problem caused by
non-compliance..

Mike
On Mon, Jun 11, 2012 at 11:56 AM, Mallikarjun Chadalapaka <
[email-masked]> wrote:

> Paul,
>
> I completely agree. This promise is conceptually similar to the CmdSN
> window promise. FWIW, I did not believe both sides were compliant in this
> case.
>
> Mallikarjun
>
>
> If the standard says that the expected semantics is that the buffers
> promised are all there at end of negotiation, then yes, it's an
> implementation issue.  In that case, the original premise (both ends
> compliant) would not be true -- one end is in violation.  On the other
> hand, if the specification of the negotiation process doesn't say when the
> promised resources have to be available, that's a hole in the standard.
>
>        paul
>
> > Sorry, if the implementation is promising x number of unsolicited
> buffers but it is has <x buffers ready at the end of negotiation, I would
> consider the implementation to not comply with the standard anymore.
> Standard on its part should clearly define the semantics of the promise -
> which I presume it does in this case.
> >
> > I am all for providing helpful implementation guidance in the standard,
> but we have to be cautious and re-confirm the need for the guidance, when
> we start providing guidance down to latency numbers (see #2) - that's what
> got me concerned when I read the original note.
> >
> > Thanks.
> >
> > Mallikarjun
> >
> >
> > That sounds like the right way to look at it.  The whole point of
> protocol standards is to specify what is needed to interoperate.  Not more,
> but also not less.  Whenever two implementations both comply with a
> standard yet do not interoperate, that's a bug in the standard.
> >
> > paul
> >
> > IMHO, the reason that this is a protocol issue is that two
> implementations that comply with the RFC can nonetheless reliably and
> repeatedly fail to interoperate because the lack of additional buffers
> causes the RCaP connection to close before full feature phase is reached by
> the initiator.
> >
> [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:
> [email-masked]>
> > Seems like one way to address this problem is via an implementation
> approach with the initiator posting in advance the negotiated number of
> unsolicited PDU buffers, at the same time it makes the (final) negotiation
> offer. As the in-bound unsolicited PDUs can technically arrive any time
> after the offer, due to standard network latency mechanics that Alexander
> summarized. Has that approach been considered?
> >
> > Mallikarjun
> >
> >
> >
> >
> > From: [email-masked]<mailto:[email-masked]> [mailto:
> [email-masked]] On Behalf Of [email-masked]<mailto:
> [email-masked]>
> [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:
> [email-masked]>
> > Mike is of course correct that iSER Hello usage can be forced by
> negotiating iSERHelloRequired to "Yes".  However, existing implementations
> are likely to reply with iSERHelloRequired=NotUnderstood, so we do need to
> specify what should be done in order to interoperate with an implementation
> that refuses to deal with the iSER Hello exchange.
> >
> > I think the situation that Alexander described should be documented in a
> new section 5.1.4 of the iSER draft.  My general rule of thumb on this sort
> of surprise found by implementers in the "running code" is that it
> indicates that something is missing in the spec.  I believe that Alexander
> has described the solution - more below.
> >
> > The new section 5.1.4 (suggested section title: Omission of the iSER
> Hello Exchange) should describe default omission of the exchange, use of
> iSERHelloRequired key to omit the iSER Hello exchange, and the consequences
> of target use of unsolicited PDUs after login when the exchange is omitted,
> isn't a good choice) or until it receives a full feature mode PDU from the
> initiator before sending a second unsolicited PDU in order to ensure that
> initiator has sufficient
> during login (so that at least two buffers are in use during login) in
> order to receive an unsolicited PDU that may follow login completion.
>  Failure to allocate this second buffer may cause connection termination if
> no buffer is available when an unsolicited PDU arrives.
> >
> > Both Mike and I are on vacation, so it may be a few weeks until we can
> agree on the new text and get a -12 version of the draft with that new text
> submitted.  In the interim, I've asked our AD (Martin Stiemerling) to hold
> off on further processing of the iSER draft until a -12 version with this
> new text is submitted.  I'd prefer to work this text out now rather than
> deal with it as an IETF Last Call comment - as the problem turned up in
> actual implementations, I think it's worth the extra month that it's likely
> to take to get correct text on how to avoid the problem into the draft.
> >
> > The iSER Hello support has never been removed in the latest spec.  Only
> its use is made optional.  So during login negotiation, just negotiate
> > On Mon, May 21, 2012 at 6:15 AM, Alexander Nezhinsky <
> [email-masked]<mailto:[email-masked]>> wrote:
> > Hi
> >
> > I understand that it is a bad timing for sending this kind of mail, now
> that iSER draft was submitted, but actually we still have a small problem.
> Response and from its perspective the connection is now in Full Featured
> Phase (assuming that it agreed to transition in the Login Response being
> sent).
> >
> > With IB sending NOP-IN periodically is the easiest (an almost only
> feasible) way to determine closed connections reliably, because this kind
> of error is delivered to user only in response to a previously initiated TX
> operation.
> >
> > This leaves the initiator in a dubious position. It posts its RX buffers
> for that connection only when the final Login Response arrives. But during
> that time (after the target had sent the Last Login Response but before the
> Full Featured phase related RX-buffers are posted on the initiator side)
> the target may send the first NOP-IN as it considers the connection in Full
> Featured phase already and NumOfUnsolicited PDUs accounting for NOP-INs has
> been negotiated to a non-zero value.
> >
> alleviates the problem. Although this often solves it in practical terms
> (as the target most probably sends the next NOP-IN only after some timeout
> period measuring seconds or hundreds of milliseconds), it does not solves
> it in terms of protocol completeness, as the target MAY theoretically send
> more than one NOP-IN until FF buffers are posted.
> >
> > This issue was encountered recently in linux iscsi/iser initiator and
> the above solution has been applied to solve it against the existing target
> > The solution is actually quite simple (theoretically) - if we bring back
> the requirement for iSER Hello exchange then the iSER assisted Full
> Featured phase does not commence until HelloReply PDU arrives at the target
> and the initiator has a definitive point in time when it can safely post
> its RX buffers - after the final LoginResponse returns but before it sends
> iSER Hello PDU.
> >
> > In practical terms it means that iSER Hello support requirement should
> be brought back to spec, which is a hassle.
> >
> > P.S. : Thanx to Mike Christie and Or Gerlitz, the maintainers of linux
> iSCSI and iSER initiator for raising the issue.
> >

---

From: Michael Ko <[email-masked]>
Date: Mon, 11 Jun 2012 19:08:33 -0700

David,

The current implemenation must be changed to correct this problem.  There
are two possible solutions.

1. Implement the exchange of iSERHelloRequired key and negotiate its use to
Yes.  This require both sides support iSERHelloRequired, leading to the
need for clarifying texts (as you suggested) in the spec in order to
specify what should be done otherwise.

2. Fix the non-compliance and adhere to the spec as currently defined.

I agree with changing the spec to comply with the implementation when the
current implementation works fine as is.  But this is not the case here.

Mike

On Mon, Jun 11, 2012 at 6:44 PM, <[email-masked]> wrote:

> > Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right after
> final
> > Login Response
> >
> > Paul,
> >
> window
> > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after
> final
> > Login Response
> >
> promised
> buffers
> protocol
> implementations
> > > I am curious to understand a bit more on why this is a protocol issue
> per
> > se.
> > >
> approach
> storm-
> negotiating
> Hello
> > > 1) The target MAY send *one* unsolicited PDU immediately after sending
> the
> > Login Response.
> > >
> isn't
> order
> agree
> > > Cc: [email-masked]<mailto:[email-masked]>; Black, David; Or Gerlitz;
> Mike
> > Christie
> > > Subject: Re: iSER - problem with unsolicited NOP-IN right after final
> Login
> > Response
> > >
> > > Alex,
> > >
> Phase
> > > This means that the target is ready to accept SCSI commands, Text
> Requests
> > etc. sent by the initiator.
> > > It also means that the target is eligible to send some unsolicited
> PDUs,
> > notably unsolicited NOP-INs.
> > >
> feasible)
> (as
> the
> iSCSI
> [email-masked]

---

From: <[email-masked]>
Date: Tue, 12 Jun 2012 13:56:56 +0000

>>> resources for the connection.

I don't understand how a protocol standard can have such language.  "200 ms (use some other number if...") is not a testable property.  If you delete the "(use...)" then you have  standard, or if you delete "at least... or" but as written, an implementer has no clue what is expected.

	paul

---

From: <[email-masked]>
Date: Tue, 12 Jun 2012 14:01:11 -0400

Hi Paul,

We're in violent agreement, and I wasn't clear in what I wrote.

I'm proposing 200ms unless someone on the list has a better idea ... 

Thanks,
--David

> Sent: Tuesday, June 12, 2012 9:57 AM
> To: Black, David
> Cc: [email-masked]; [email-masked]
> I don't understand how a protocol standard can have such language.  "200 ms
> (use some other number if...") is not a testable property.  If you delete the
> "(use...)" then you have  standard, or if you delete "at least... or" but as
> written, an implementer has no clue what is expected.
> 
> 	paul
>