From david.black@emc.com Mon Jun 11 08:36:17 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2D3D021F85CC for ; Mon, 11 Jun 2012 08:36:17 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -100.184 X-Spam-Level: X-Spam-Status: No, score=-100.184 tagged_above=-999 required=5 tests=[BAYES_40=-0.185, HTML_MESSAGE=0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MohBzclyPw7s for ; Mon, 11 Jun 2012 08:36:04 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id A37A121F855E for ; Mon, 11 Jun 2012 08:36:03 -0700 (PDT) Received: from hop04-l1d11-si03.isus.emc.com (HOP04-L1D11-SI03.isus.emc.com [10.254.111.23]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5BFZncg023224 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 11 Jun 2012 11:35:50 -0400 Received: from mailhub.lss.emc.com (mailhubhoprd04.lss.emc.com [10.254.222.226]) by hop04-l1d11-si03.isus.emc.com (RSA Interceptor); Mon, 11 Jun 2012 11:35:37 -0400 Received: from mxhub16.corp.emc.com (mxhub16.corp.emc.com [128.222.70.237]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5BFYPB9010046; Mon, 11 Jun 2012 11:35:12 -0400 Received: from mx15a.corp.emc.com ([169.254.1.101]) by mxhub16.corp.emc.com ([128.222.70.237]) with mapi; Mon, 11 Jun 2012 11:34:58 -0400 From: To: , , Date: Mon, 11 Jun 2012 11:34:57 -0400 Thread-Topic: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response Thread-Index: AQHNOXtOIIaCinNGykWqG66+HjtgdZbbOw+ggAecuIA= Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120707E7F1@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71205813C92@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_8D3D17ACE214DC429325B2B98F3AE7120707E7F1MX15Acorpemccom_" MIME-Version: 1.0 X-EMM-MHVC: 1 Cc: ogerlitz@mellanox.com, michaelc@cs.wisc.edu, storm@ietf.org Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jun 2012 15:36:17 -0000 --_000_8D3D17ACE214DC429325B2B98F3AE7120707E7F1MX15Acorpemccom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Mallikarjun, IMHO, the reason that this is a protocol issue is that two implementations = that comply with the RFC can nonetheless reliably and repeatedly fail to in= teroperate because the lack of additional buffers causes the RCaP connectio= n to close before full feature phase is reached by the initiator. I believe that we have a responsibility to tell implementers what can go wr= ong here and how to avoid it - the technique you describe (post all unsolic= ited buffers before sending final negotiation message) could be mentioned a= s part of this, and we should also describe what's possible with use of a s= ingle buffer, as that approach is being pursued as Alexander describes. Thanks, --David From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] Sent: Friday, May 25, 2012 9:02 PM To: Black, David; mkosjc@gmail.com; nezhinsky@gmail.com Cc: ogerlitz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right after fin= al Login Response I am curious to understand a bit more on why this is a protocol issue per s= e. Seems like one way to address this problem is via an implementation approac= h with the initiator posting in advance the negotiated number of unsolicite= d PDU buffers, at the same time it makes the (final) negotiation offer. As = the in-bound unsolicited PDUs can technically arrive any time after the off= er, due to standard network latency mechanics that Alexander summarized. Ha= s that approach been considered? Mallikarjun From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of d= avid.black@emc.com Sent: Thursday, May 24, 2012 12:03 AM To: mkosjc@gmail.com; nezhinsky@gmail.com Cc: ogerlitz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after fin= al Login Response Importance: High Mike (Ko) and Alexander, Mike is of course correct that iSER Hello usage can be forced by negotiatin= g iSERHelloRequired to "Yes". However, existing implementations are likely= to reply with iSERHelloRequired=3DNotUnderstood, so we do need to specify = what should be done in order to interoperate with an implementation that re= fuses to deal with the iSER Hello exchange. I think the situation that Alexander described should be documented in a ne= w section 5.1.4 of the iSER draft. My general rule of thumb on this sort o= f surprise found by implementers in the "running code" is that it indicates= that something is missing in the spec. I believe that Alexander has descr= ibed the solution - more below. The new section 5.1.4 (suggested section title: Omission of the iSER Hello = Exchange) should describe default omission of the exchange, use of iSERHell= oRequired key to omit the iSER Hello exchange, and the consequences of targ= et use of unsolicited PDUs after login when the exchange is omitted, includ= ing IB's use of NOP-IN (as a keep-alive measure, right?) The crucial requirements points that I take away from Alexander's descripti= on are that if the iSER Hello exchange is omitted, then: 1) The target MAY send *one* unsolicited PDU immediately after sending the = Login Response. 2) The target MUST wait at least 200ms (use some other number if 200ms isn'= t a good choice) or until it receives a full feature mode PDU from the initiator before send= ing a second unsolicited PDU in order to ensure that initiator has sufficient time to allocate the full feature buffer resources for the connection= . 3) The initiator SHOULD allocate at least one additional buffer for use dur= ing login (so that at least two buffers are in use during login) in order to receive an unsoli= cited PDU that may follow login completion. Failure to allocate this second buffer m= ay cause connection termination if no buffer is available when an unsolicited = PDU arrives. Both Mike and I are on vacation, so it may be a few weeks until we can agre= e on the new text and get a -12 version of the draft with that new text sub= mitted. In the interim, I've asked our AD (Martin Stiemerling) to hold off= on further processing of the iSER draft until a -12 version with this new = text is submitted. I'd prefer to work this text out now rather than deal w= ith it as an IETF Last Call comment - as the problem turned up in actual im= plementations, I think it's worth the extra month that it's likely to take = to get correct text on how to avoid the problem into the draft. I'd suggest that Mike Ko post an initial draft of the text for the new sect= ion 5.1.4 to the list when he resurfaces ... Thanks, --David From: Michael Ko [mailto:mkosjc@gmail.com] Sent: Monday, May 21, 2012 10:23 AM To: Alexander Nezhinsky Cc: storm@ietf.org; Black, David; Or Gerlitz; Mike C= hristie Subject: Re: iSER - problem with unsolicited NOP-IN right after final Login= Response Alex, The iSER Hello support has never been removed in the latest spec. Only its= use is made optional. So during login negotiation, just negotiate iSERHel= loRequired to Yes. Mike On Mon, May 21, 2012 at 6:15 AM, Alexander Nezhinsky > wrote: Hi I understand that it is a bad timing for sending this kind of mail, now tha= t iSER draft was submitted, but actually we still have a small problem. It is related to the final Login Response handling and the transition to Fu= ll-Featured phase on the initiator side in Infiniband setups. When the target receives the final Login Request it send the final Login Re= sponse and from its perspective the connection is now in Full Featured Phase (assuming that it agreed to tr= ansition in the Login Response being sent). This means that the target is ready to accept SCSI commands, Text Requests = etc. sent by the initiator. It also means that the target is eligible to send some unsolicited PDUs, no= tably unsolicited NOP-INs. With IB sending NOP-IN periodically is the easiest (an almost only feasible= ) way to determine closed connections reliably, because this kind of error is delivered to user only in response = to a previously initiated TX operation. This leaves the initiator in a dubious position. It posts its RX buffers fo= r that connection only when the final Login Response arrives. But during that time (after the target had sent the= Last Login Response but before the Full Featured phase related RX-buffers are posted on the initiator side= ) the target may send the first NOP-IN as it considers the connection in Full Featured phase alre= ady and NumOfUnsolicited PDUs accounting for NOP-INs has been negotiated to a non-zero value. If the initiator works with a single RX-buffer posted during the entire log= in phase (which is a logical thing to do judging by the login exchange protocol) then an error occurs, as no buffers= are posted when the NOP-IN arrives and the connection is shut down. Posting a single extra buffer before sending the last Login Request only al= leviates the problem. Although this often solves it in practical terms (as the target most probably sends the n= ext NOP-IN only after some timeout period measuring seconds or hundreds of milliseconds), it does not solves i= t in terms of protocol completeness, as the target MAY theoretically send more than one NOP-IN until FF buffers = are posted. This issue was encountered recently in linux iscsi/iser initiator and the a= bove solution has been applied to solve it against the existing target implementation (STGT), but the initiator rem= ains exposed to this kind of errors. The solution is actually quite simple (theoretically) - if we bring back th= e requirement for iSER Hello exchange then the iSER assisted Full Featured phase does not commence until HelloRep= ly PDU arrives at the target and the initiator has a definitive point in time when it can safely post it= s RX buffers - after the final LoginResponse returns but before it sends iSER Hello PDU. In practical terms it means that iSER Hello support requirement should be b= rought back to spec, which is a hassle. Should we decide on this now? Alexander P.S. : Thanx to Mike Christie and Or Gerlitz, the maintainers of linux iSCS= I and iSER initiator for raising the issue. --_000_8D3D17ACE214DC429325B2B98F3AE7120707E7F1MX15Acorpemccom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi Mallikarjun,=

 

IMHO, the reason that this is a protocol issue is that two implementati= ons that comply with the RFC can nonetheless reliably and repeatedly fail t= o interoperate because the lack of additional buffers causes the RCaP conne= ction to close before full feature phase is reached by the initiator.<= /o:p>

 

I believe that we have a responsibility to tell implementers what can go= wrong here and how to avoid it - the technique you describe (post all unso= licited buffers before sending final negotiation message) could be mentione= d as part of this, and we should also describe what’s possible with u= se of a single buffer, as that approach is being pursued as Alexander descr= ibes.

 

Thanks,
--David

 

From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] Sent: Friday, May 25, 2012 9:02 PM
To: Black, David; mko= sjc@gmail.com; nezhinsky@gmail.com
Cc: ogerlitz@mellanox.com; mic= haelc@cs.wisc.edu; storm@ietf.org
Subject: RE: [storm] iSER - pro= blem with unsolicited NOP-IN right after final Login Response

 

I am curious to understand a bit more on why this is a proto= col issue per se.

&nb= sp;

Seems like one way to addre= ss this problem is via an implementation approach with the initiator postin= g in advance the negotiated number of unsolicited PDU buffers, at the same = time it makes the (final) negotiation offer. As the in-bound unsolicited PD= Us can technically arrive any time after the offer, due to standard network= latency mechanics that Alexander summarized. Has that approach been consid= ered?

 

Mallikarjun

 

 

 

 

From: storm-bounces@ietf.org [mailto:storm-bounce= s@ietf.org] On Behalf Of david.black@emc.com
Sent: Thursda= y, May 24, 2012 12:03 AM
To: mkosjc@gmail.com; nezhinsky@gmail.co= m
Cc: ogerlitz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org=
Subject: Re: [storm] iSER - problem with unsolicited NOP-IN righ= t after final Login Response
Importance: High

 

Mi= ke (Ko) and Alexander,

 

Mike is of course correct that iSER Hello us= age can be forced by negotiating iSERHelloRequired to “Yes”.&nb= sp; However, existing implementations are likely to reply with iSERHelloReq= uired=3DNotUnderstood, so we do need to specify what should be done in orde= r to interoperate with an implementation that refuses to deal with the iSER= Hello exchange.

 

I think the situation that Alexander described sho= uld be documented in a new section 5.1.4 of the iSER draft.  My genera= l rule of thumb on this sort of surprise found by implementers in the ̶= 0;running code” is that it indicates that something is missing in the= spec.  I believe that Alexander has described the solution - more bel= ow.

 

The new section 5.1.4 (suggested section title: Omission of the= iSER Hello Exchange) should describe default omission of the exchange, use= of iSERHelloRequired key to omit the iSER Hello exchange, and the conseque= nces of target use of unsolicited PDUs after login when the exchange is omi= tted, including IB’s use of NOP-IN (as a keep-alive measure, right?)<= o:p>

 

The crucial requirements points that I take away from Alexander= 217;s description are that if the iSER Hello exchange is omitted, then:

 

1) The target MAY send *one* unsolicited PDU immediately after = sending the Login Response.

&nbs= p;

2) The target MUST wait at least 200ms = (use some other number if 200ms isn’t a good choice)

or until it receives a ful= l feature mode PDU from the initiator before sending a

second unsolicited PDU in orde= r to ensure that initiator has sufficient

      time to allocate the full feature buffe= r resources for the connection.

<= span style=3D'font-size:10.0pt;font-family:"Courier New";color:black'>3) Th= e initiator SHOULD allocate at least one additional buffer for use during l= ogin (so that

at least two buffers are in use during login) in order to receive an un= solicited PDU

that may follow login completion.  Failure to allocate this second= buffer may

 

Both Mike and I are on vacation, so it may be a few = weeks until we can agree on the new text and get a -12 version of the draft= with that new text submitted.  In the interim, I’ve asked our A= D (Martin Stiemerling) to hold off on further processing of the iSER draft = until a -12 version with this new text is submitted.  I’d prefer= to work this text out now rather than deal with it as an IETF Last Call co= mment - as the problem turned up in actual implementations, I think it̵= 7;s worth the extra month that it’s likely to take to get correct tex= t on how to avoid the problem into the draft.

 

I’d suggest tha= t Mike Ko post an initial draft of the text for the new section 5.1.4 to th= e list when he resurfaces ...

&n= bsp;

Thanks,
--David=

 

<= div style=3D'border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0= in 0in'>

From: Michael Ko [mailto:mkosjc@gmail.com]
Sent: Monday, May 21= , 2012 10:23 AM
To: Alexander Nezhinsky
Cc: storm@ietf.org; Black, David; Or Gerlitz; Mike C= hristie
Subject: Re: iSER - problem with unsolicited NOP-IN right= after final Login Response

 

Alex,

The iSER Hello support has never been removed in the late= st spec.  Only its use is made optional.  So during login negotia= tion, just negotiate iSERHelloRequired to Yes.

Mike

On Mon, May 21, 2012 at 6:15 AM, Alexander Nezhinsk= y <nezhinsky@gm= ail.com> wrote:

Hi

I understand that it is a bad timing fo= r sending this kind of mail, now that iSER draft was submitted,
but actu= ally we still have a small problem.

It is related to the final Login= Response handling and the transition to Full-Featured phase on the initiat= or side in
Infiniband setups.

When the target receives = ;the final Login Request it send the final Login Response and from its= perspective
the connection is now in Full Featured Phase (assuming that= it agreed to transition in the Login Response being sent).

This mea= ns that the target is ready to accept SCSI commands, Text Requests etc. sen= t by the initiator.
It also means that the target is eligible to send s= ome unsolicited PDUs, notably unsolicited NOP-INs.

With IB send= ing NOP-IN periodically is the easiest (an almost only feasible) way to det= ermine closed connections
reliably, because this kind of error is delive= red to user only in response to a previously initiated TX operation.
This leaves the initiator in a dubious position. It posts its RX buffers f= or that connection only when the final
Login Response arrives. But durin= g that time (after the target had sent the Last Login Response but beforethe Full Featured phase related RX-buffers are posted on the initiator si= de) the target may send
the first NOP-IN as it considers the connection= in Full Featured phase already and NumOfUnsolicited PDUs
accounting fo= r NOP-INs has been negotiated to a non-zero value.

If the initiator= works with a single RX-buffer posted during the entire login phase (which = is a logical thing to do
judging by the login exchange protocol) then a= n error occurs, as no buffers are posted when the NOP-IN arrives
and the= connection is shut down.

Posting a single extra buffer before sendi= ng the last Login Request only alleviates the problem. Although this
oft= en solves it in practical terms (as the target most probably sends the next= NOP-IN only after some timeout
period measuring seconds or hundreds of = milliseconds), it does not solves it in terms of protocol completeness,
= as the target MAY theoretically send more than one NOP-IN until FF buffers = are posted.

This issue was encountered recently in linux iscsi/iser = initiator and the above solution has been applied to solve
it against th= e existing target implementation (STGT), but the initiator remains exposed = to this kind of errors.

The solution is actually quite simple (theor= etically) - if we bring back the requirement for iSER Hello exchange
the= n the iSER assisted Full Featured phase does not commence until HelloReply = PDU arrives at the target
and the initiator has a definitive point in t= ime when it can safely post its RX buffers - after the final LoginResponse =
returns but before it sends iSER Hello PDU.

In practical terms i= t means that iSER Hello support requirement should be brought back to spec,= which is a hassle.

Should we decide on this now?

Alexander

P.S.= : Thanx to Mike Christie and Or Gerlitz, the maintainers of linux iSCSI an= d iSER initiator for raising the issue.

 

= --_000_8D3D17ACE214DC429325B2B98F3AE7120707E7F1MX15Acorpemccom_-- From Paul_Koning@Dell.com Mon Jun 11 11:00:16 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1D08321F8568 for ; Mon, 11 Jun 2012 11:00:16 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -110.599 X-Spam-Level: X-Spam-Status: No, score=-110.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CWL58VSaFFf7 for ; Mon, 11 Jun 2012 11:00:11 -0700 (PDT) Received: from ausxippc101.us.dell.com (ausxippc101.us.dell.com [143.166.85.207]) by ietfa.amsl.com (Postfix) with ESMTP id 7C4AA21F8565 for ; Mon, 11 Jun 2012 11:00:10 -0700 (PDT) X-Loopcount0: from 10.170.28.39 From: To: Thread-Topic: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response Thread-Index: AQHNOXtOIIaCinNGykWqG66+HjtgdZbbOw+ggAecuICAEwDkAA== Date: Mon, 11 Jun 2012 17:59:14 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE71205813C92@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120707E7F1@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120707E7F1@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.175.217.61] Content-Type: text/plain; charset="Windows-1252" Content-ID: <7A4EA1F73AFF5440831167DBA7B6D7E3@dell.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: michaelc@cs.wisc.edu, storm@ietf.org, ogerlitz@mellanox.com Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jun 2012 18:00:16 -0000 That sounds like the right way to look at it. The whole point of protocol = standards is to specify what is needed to interoperate. Not more, but also= not less. Whenever two implementations both comply with a standard yet do= not interoperate, that's a bug in the standard. paul On Jun 11, 2012, at 11:34 AM, > > wrote: Hi Mallikarjun, IMHO, the reason that this is a protocol issue is that two implementations = that comply with the RFC can nonetheless reliably and repeatedly fail to in= teroperate because the lack of additional buffers causes the RCaP connectio= n to close before full feature phase is reached by the initiator. I believe that we have a responsibility to tell implementers what can go wr= ong here and how to avoid it - the technique you describe (post all unsolic= ited buffers before sending final negotiation message) could be mentioned a= s part of this, and we should also describe what=92s possible with use of a= single buffer, as that approach is being pursued as Alexander describes. Thanks, --David From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] Sent: Friday, May 25, 2012 9:02 PM To: Black, David; mkosjc@gmail.com; nezhinsky@gmai= l.com Cc: ogerlitz@mellanox.com; michaelc@cs.wisc.e= du; storm@ietf.org Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right after fin= al Login Response I am curious to understand a bit more on why this is a protocol issue per s= e. Seems like one way to address this problem is via an implementation approac= h with the initiator posting in advance the negotiated number of unsolicite= d PDU buffers, at the same time it makes the (final) negotiation offer. As = the in-bound unsolicited PDUs can technically arrive any time after the off= er, due to standard network latency mechanics that Alexander summarized. Ha= s that approach been considered? Mallikarjun From: storm-bounces@ietf.org [mailto:storm-b= ounces@ietf.org] On Behalf Of david.black@emc.com Sent: Thursday, May 24, 2012 12:03 AM To: mkosjc@gmail.com; nezhinsky@gmail.com Cc: ogerlitz@mellanox.com; michaelc@cs.wisc.e= du; storm@ietf.org Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after fin= al Login Response Importance: High Mike (Ko) and Alexander, Mike is of course correct that iSER Hello usage can be forced by negotiatin= g iSERHelloRequired to =93Yes=94. However, existing implementations are li= kely to reply with iSERHelloRequired=3DNotUnderstood, so we do need to spec= ify what should be done in order to interoperate with an implementation tha= t refuses to deal with the iSER Hello exchange. I think the situation that Alexander described should be documented in a ne= w section 5.1.4 of the iSER draft. My general rule of thumb on this sort o= f surprise found by implementers in the =93running code=94 is that it indic= ates that something is missing in the spec. I believe that Alexander has d= escribed the solution - more below. The new section 5.1.4 (suggested section title: Omission of the iSER Hello = Exchange) should describe default omission of the exchange, use of iSERHell= oRequired key to omit the iSER Hello exchange, and the consequences of targ= et use of unsolicited PDUs after login when the exchange is omitted, includ= ing IB=92s use of NOP-IN (as a keep-alive measure, right?) The crucial requirements points that I take away from Alexander=92s descrip= tion are that if the iSER Hello exchange is omitted, then: 1) The target MAY send *one* unsolicited PDU immediately after sending the = Login Response. 2) The target MUST wait at least 200ms (use some other number if 200ms isn= =92t a good choice) or until it receives a full feature mode PDU from the initiator before send= ing a second unsolicited PDU in order to ensure that initiator has sufficient time to allocate the full feature buffer resources for the connection= . 3) The initiator SHOULD allocate at least one additional buffer for use dur= ing login (so that at least two buffers are in use during login) in order to receive an unsoli= cited PDU that may follow login completion. Failure to allocate this second buffer m= ay cause connection termination if no buffer is available when an unsolicited = PDU arrives. Both Mike and I are on vacation, so it may be a few weeks until we can agre= e on the new text and get a -12 version of the draft with that new text sub= mitted. In the interim, I=92ve asked our AD (Martin Stiemerling) to hold o= ff on further processing of the iSER draft until a -12 version with this ne= w text is submitted. I=92d prefer to work this text out now rather than de= al with it as an IETF Last Call comment - as the problem turned up in actua= l implementations, I think it=92s worth the extra month that it=92s likely = to take to get correct text on how to avoid the problem into the draft. I=92d suggest that Mike Ko post an initial draft of the text for the new se= ction 5.1.4 to the list when he resurfaces ... Thanks, --David From: Michael Ko [mailto:mkosjc@gmail.com] Sent: Monday, May 21, 2012 10:23 AM To: Alexander Nezhinsky Cc: storm@ietf.org; Black, David; Or Gerlitz; Mike C= hristie Subject: Re: iSER - problem with unsolicited NOP-IN right after final Login= Response Alex, The iSER Hello support has never been removed in the latest spec. Only its= use is made optional. So during login negotiation, just negotiate iSERHel= loRequired to Yes. Mike On Mon, May 21, 2012 at 6:15 AM, Alexander Nezhinsky > wrote: Hi I understand that it is a bad timing for sending this kind of mail, now tha= t iSER draft was submitted, but actually we still have a small problem. It is related to the final Login Response handling and the transition to Fu= ll-Featured phase on the initiator side in Infiniband setups. When the target receives the final Login Request it send the final Login Re= sponse and from its perspective the connection is now in Full Featured Phase (assuming that it agreed to tr= ansition in the Login Response being sent). This means that the target is ready to accept SCSI commands, Text Requests = etc. sent by the initiator. It also means that the target is eligible to send some unsolicited PDUs, no= tably unsolicited NOP-INs. With IB sending NOP-IN periodically is the easiest (an almost only feasible= ) way to determine closed connections reliably, because this kind of error is delivered to user only in response = to a previously initiated TX operation. This leaves the initiator in a dubious position. It posts its RX buffers fo= r that connection only when the final Login Response arrives. But during that time (after the target had sent the= Last Login Response but before the Full Featured phase related RX-buffers are posted on the initiator side= ) the target may send the first NOP-IN as it considers the connection in Full Featured phase alre= ady and NumOfUnsolicited PDUs accounting for NOP-INs has been negotiated to a non-zero value. If the initiator works with a single RX-buffer posted during the entire log= in phase (which is a logical thing to do judging by the login exchange protocol) then an error occurs, as no buffers= are posted when the NOP-IN arrives and the connection is shut down. Posting a single extra buffer before sending the last Login Request only al= leviates the problem. Although this often solves it in practical terms (as the target most probably sends the n= ext NOP-IN only after some timeout period measuring seconds or hundreds of milliseconds), it does not solves i= t in terms of protocol completeness, as the target MAY theoretically send more than one NOP-IN until FF buffers = are posted. This issue was encountered recently in linux iscsi/iser initiator and the a= bove solution has been applied to solve it against the existing target implementation (STGT), but the initiator rem= ains exposed to this kind of errors. The solution is actually quite simple (theoretically) - if we bring back th= e requirement for iSER Hello exchange then the iSER assisted Full Featured phase does not commence until HelloRep= ly PDU arrives at the target and the initiator has a definitive point in time when it can safely post it= s RX buffers - after the final LoginResponse returns but before it sends iSER Hello PDU. In practical terms it means that iSER Hello support requirement should be b= rought back to spec, which is a hassle. Should we decide on this now? Alexander P.S. : Thanx to Mike Christie and Or Gerlitz, the maintainers of linux iSCS= I and iSER initiator for raising the issue. _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From cbm@chadalapaka.com Mon Jun 11 11:33:06 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6BF2321F856D for ; Mon, 11 Jun 2012 11:33:06 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bWDiCOhlzh9m for ; Mon, 11 Jun 2012 11:33:05 -0700 (PDT) Received: from ch1outboundpool.messaging.microsoft.com (ch1ehsobe006.messaging.microsoft.com [216.32.181.186]) by ietfa.amsl.com (Postfix) with ESMTP id 0226E21F859A for ; Mon, 11 Jun 2012 11:33:04 -0700 (PDT) Received: from mail120-ch1-R.bigfish.com (10.43.68.246) by CH1EHSOBE002.bigfish.com (10.43.70.52) with Microsoft SMTP Server id 14.1.225.23; Mon, 11 Jun 2012 18:32:03 +0000 Received: from mail120-ch1 (localhost [127.0.0.1]) by mail120-ch1-R.bigfish.com (Postfix) with ESMTP id 95D931E0311; Mon, 11 Jun 2012 18:32:03 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.240.117; KIP:(null); UIP:(null); IPV:NLI; H:BL2PRD0610HT003.namprd06.prod.outlook.com; RD:none; EFVD:NLI X-SpamScore: -28 X-BigFish: PS-28(zz98dI9371I542M4015Izz1202hzz1033IL8275bh8275dhz2fh2a8h668h839h944hd25hf0ah) Received-SPF: pass (mail120-ch1: domain of chadalapaka.com designates 157.56.240.117 as permitted sender) client-ip=157.56.240.117; envelope-from=cbm@chadalapaka.com; helo=BL2PRD0610HT003.namprd06.prod.outlook.com ; .outlook.com ; Received: from mail120-ch1 (localhost.localdomain [127.0.0.1]) by mail120-ch1 (MessageSwitch) id 1339439521100244_24399; Mon, 11 Jun 2012 18:32:01 +0000 (UTC) Received: from CH1EHSMHS014.bigfish.com (snatpool2.int.messaging.microsoft.com [10.43.68.234]) by mail120-ch1.bigfish.com (Postfix) with ESMTP id 0BF3E320043; Mon, 11 Jun 2012 18:32:01 +0000 (UTC) Received: from BL2PRD0610HT003.namprd06.prod.outlook.com (157.56.240.117) by CH1EHSMHS014.bigfish.com (10.43.70.14) with Microsoft SMTP Server (TLS) id 14.1.225.23; Mon, 11 Jun 2012 18:32:00 +0000 Received: from BL2PRD0610MB361.namprd06.prod.outlook.com ([169.254.10.144]) by BL2PRD0610HT003.namprd06.prod.outlook.com ([10.255.101.38]) with mapi id 14.16.0164.004; Mon, 11 Jun 2012 18:32:56 +0000 From: Mallikarjun Chadalapaka To: "Paul_Koning@Dell.com" , "david.black@emc.com" Thread-Topic: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response Thread-Index: AQHNOXtOIIaCinNGykWqG66+HjtgdZbbOw+ggAecuICAEwDkAP//sNUg Date: Mon, 11 Jun 2012 18:32:54 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE71205813C92@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120707E7F1@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [207.190.44.200] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: chadalapaka.com Cc: "ogerlitz@mellanox.com" , "storm@ietf.org" , "michaelc@cs.wisc.edu" Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jun 2012 18:33:06 -0000 Sorry, if the implementation is promising x number of unsolicited buffers b= ut it is has > > wrote: Hi Mallikarjun, IMHO, the reason that this is a protocol issue is that two implementations = that comply with the RFC can nonetheless reliably and repeatedly fail to in= teroperate because the lack of additional buffers causes the RCaP connectio= n to close before full feature phase is reached by the initiator. I believe that we have a responsibility to tell implementers what can go wr= ong here and how to avoid it - the technique you describe (post all unsolic= ited buffers before sending final negotiation message) could be mentioned a= s part of this, and we should also describe what's possible with use of a s= ingle buffer, as that approach is being pursued as Alexander describes. Thanks, --David From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] Sent: Friday, May 25, 2012 9:02 PM To: Black, David; mkosjc@gmail.com; nezhinsky@gmai= l.com Cc: ogerlitz@mellanox.com; michaelc@cs.wisc.e= du; storm@ietf.org Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right after fin= al Login Response I am curious to understand a bit more on why this is a protocol issue per s= e. Seems like one way to address this problem is via an implementation approac= h with the initiator posting in advance the negotiated number of unsolicite= d PDU buffers, at the same time it makes the (final) negotiation offer. As = the in-bound unsolicited PDUs can technically arrive any time after the off= er, due to standard network latency mechanics that Alexander summarized. Ha= s that approach been considered? Mallikarjun From: storm-bounces@ietf.org [mailto:storm-b= ounces@ietf.org] On Behalf Of david.black@emc.com Sent: Thursday, May 24, 2012 12:03 AM To: mkosjc@gmail.com; nezhinsky@gmail.com Cc: ogerlitz@mellanox.com; michaelc@cs.wisc.e= du; storm@ietf.org Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after fin= al Login Response Importance: High Mike (Ko) and Alexander, Mike is of course correct that iSER Hello usage can be forced by negotiatin= g iSERHelloRequired to "Yes". However, existing implementations are likely= to reply with iSERHelloRequired=3DNotUnderstood, so we do need to specify = what should be done in order to interoperate with an implementation that re= fuses to deal with the iSER Hello exchange. I think the situation that Alexander described should be documented in a ne= w section 5.1.4 of the iSER draft. My general rule of thumb on this sort o= f surprise found by implementers in the "running code" is that it indicates= that something is missing in the spec. I believe that Alexander has descr= ibed the solution - more below. The new section 5.1.4 (suggested section title: Omission of the iSER Hello = Exchange) should describe default omission of the exchange, use of iSERHell= oRequired key to omit the iSER Hello exchange, and the consequences of targ= et use of unsolicited PDUs after login when the exchange is omitted, includ= ing IB's use of NOP-IN (as a keep-alive measure, right?) The crucial requirements points that I take away from Alexander's descripti= on are that if the iSER Hello exchange is omitted, then: 1) The target MAY send *one* unsolicited PDU immediately after sending the = Login Response. 2) The target MUST wait at least 200ms (use some other number if 200ms isn'= t a good choice) or until it receives a full feature mode PDU from the init= iator before sending a second unsolicited PDU in order to ensure that initi= ator has sufficient time to allocate the full feature buffer resources for the connection= . 3) The initiator SHOULD allocate at least one additional buffer for use dur= ing login (so that at least two buffers are in use during login) in order t= o receive an unsolicited PDU that may follow login completion. Failure to = allocate this second buffer may cause connection termination if no buffer i= s available when an unsolicited PDU arrives. Both Mike and I are on vacation, so it may be a few weeks until we can agre= e on the new text and get a -12 version of the draft with that new text sub= mitted. In the interim, I've asked our AD (Martin Stiemerling) to hold off= on further processing of the iSER draft until a -12 version with this new = text is submitted. I'd prefer to work this text out now rather than deal w= ith it as an IETF Last Call comment - as the problem turned up in actual im= plementations, I think it's worth the extra month that it's likely to take = to get correct text on how to avoid the problem into the draft. I'd suggest that Mike Ko post an initial draft of the text for the new sect= ion 5.1.4 to the list when he resurfaces ... Thanks, --David From: Michael Ko [mailto:mkosjc@gmail.com] Sent: Monday, May 21, 2012 10:23 AM To: Alexander Nezhinsky Cc: storm@ietf.org; Black, David; Or Gerlitz; Mike C= hristie Subject: Re: iSER - problem with unsolicited NOP-IN right after final Login= Response Alex, The iSER Hello support has never been removed in the latest spec. Only its= use is made optional. So during login negotiation, just negotiate iSERHel= loRequired to Yes. Mike On Mon, May 21, 2012 at 6:15 AM, Alexander Nezhinsky > wrote: Hi I understand that it is a bad timing for sending this kind of mail, now tha= t iSER draft was submitted, but actually we still have a small problem. It is related to the final Login Response handling and the transition to Fu= ll-Featured phase on the initiator side in Infiniband setups. When the target receives the final Login Request it send the final Login Re= sponse and from its perspective the connection is now in Full Featured Phas= e (assuming that it agreed to transition in the Login Response being sent). This means that the target is ready to accept SCSI commands, Text Requests = etc. sent by the initiator. It also means that the target is eligible to send some unsolicited PDUs, no= tably unsolicited NOP-INs. With IB sending NOP-IN periodically is the easiest (an almost only feasible= ) way to determine closed connections reliably, because this kind of error = is delivered to user only in response to a previously initiated TX operatio= n. This leaves the initiator in a dubious position. It posts its RX buffers fo= r that connection only when the final Login Response arrives. But during th= at time (after the target had sent the Last Login Response but before the F= ull Featured phase related RX-buffers are posted on the initiator side) the= target may send the first NOP-IN as it considers the connection in Full Fe= atured phase already and NumOfUnsolicited PDUs accounting for NOP-INs has b= een negotiated to a non-zero value. If the initiator works with a single RX-buffer posted during the entire log= in phase (which is a logical thing to do judging by the login exchange prot= ocol) then an error occurs, as no buffers are posted when the NOP-IN arrive= s and the connection is shut down. Posting a single extra buffer before sending the last Login Request only al= leviates the problem. Although this often solves it in practical terms (as = the target most probably sends the next NOP-IN only after some timeout peri= od measuring seconds or hundreds of milliseconds), it does not solves it in= terms of protocol completeness, as the target MAY theoretically send more = than one NOP-IN until FF buffers are posted. This issue was encountered recently in linux iscsi/iser initiator and the a= bove solution has been applied to solve it against the existing target impl= ementation (STGT), but the initiator remains exposed to this kind of errors= . The solution is actually quite simple (theoretically) - if we bring back th= e requirement for iSER Hello exchange then the iSER assisted Full Featured = phase does not commence until HelloReply PDU arrives at the target and the = initiator has a definitive point in time when it can safely post its RX buf= fers - after the final LoginResponse returns but before it sends iSER Hello= PDU. In practical terms it means that iSER Hello support requirement should be b= rought back to spec, which is a hassle. Should we decide on this now? Alexander P.S. : Thanx to Mike Christie and Or Gerlitz, the maintainers of linux iSCS= I and iSER initiator for raising the issue. _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From Paul_Koning@Dell.com Mon Jun 11 11:41:12 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D490821F856C for ; Mon, 11 Jun 2012 11:41:12 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -108.599 X-Spam-Level: X-Spam-Status: No, score=-108.599 tagged_above=-999 required=5 tests=[AWL=-2.000, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aOeoHyrpbYnL for ; Mon, 11 Jun 2012 11:41:11 -0700 (PDT) Received: from ausc60ps301.us.dell.com (ausc60ps301.us.dell.com [143.166.148.206]) by ietfa.amsl.com (Postfix) with ESMTP id 4C6D221F8595 for ; Mon, 11 Jun 2012 11:41:11 -0700 (PDT) X-Loopcount0: from 10.170.28.41 From: To: Thread-Topic: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response Thread-Index: AQHNOXtOIIaCinNGykWqG66+HjtgdZbbOw+ggAecuICAEwDkAP//sNUggABa4AA= Date: Mon, 11 Jun 2012 18:41:06 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE71205813C92@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120707E7F1@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.175.217.61] Content-Type: text/plain; charset="us-ascii" Content-ID: <7732D235BC685846B7CDCC3E6895A43E@dell.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: michaelc@cs.wisc.edu, storm@ietf.org, ogerlitz@mellanox.com Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jun 2012 18:41:13 -0000 If the standard says that the expected semantics is that the buffers promis= ed are all there at end of negotiation, then yes, it's an implementation is= sue. In that case, the original premise (both ends compliant) would not be= true -- one end is in violation. On the other hand, if the specification = of the negotiation process doesn't say when the promised resources have to = be available, that's a hole in the standard. paul On Jun 11, 2012, at 2:32 PM, Mallikarjun Chadalapaka wrote: > Sorry, if the implementation is promising x number of unsolicited buffers= but it is has =20 > I am all for providing helpful implementation guidance in the standard, b= ut we have to be cautious and re-confirm the need for the guidance, when we= start providing guidance down to latency numbers (see #2) - that's what go= t me concerned when I read the original note. >=20 > Thanks. >=20 > Mallikarjun >=20 >=20 > -----Original Message----- > From: Paul_Koning@Dell.com [mailto:Paul_Koning@Dell.com]=20 > Sent: Monday, June 11, 2012 10:59 AM > To: david.black@emc.com > Cc: Mallikarjun Chadalapaka; mkosjc@gmail.com; nezhinsky@gmail.com; ogerl= itz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after f= inal Login Response >=20 > That sounds like the right way to look at it. The whole point of protoco= l standards is to specify what is needed to interoperate. Not more, but al= so not less. Whenever two implementations both comply with a standard yet = do not interoperate, that's a bug in the standard. >=20 > paul >=20 > On Jun 11, 2012, at 11:34 AM, > > > wrote: >=20 > Hi Mallikarjun, >=20 > IMHO, the reason that this is a protocol issue is that two implementation= s that comply with the RFC can nonetheless reliably and repeatedly fail to = interoperate because the lack of additional buffers causes the RCaP connect= ion to close before full feature phase is reached by the initiator. >=20 > I believe that we have a responsibility to tell implementers what can go = wrong here and how to avoid it - the technique you describe (post all unsol= icited buffers before sending final negotiation message) could be mentioned= as part of this, and we should also describe what's possible with use of a= single buffer, as that approach is being pursued as Alexander describes. >=20 > Thanks, > --David >=20 > From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] > Sent: Friday, May 25, 2012 9:02 PM > To: Black, David; mkosjc@gmail.com; nezhinsky@gm= ail.com > Cc: ogerlitz@mellanox.com; michaelc@cs.wisc= .edu; storm@ietf.org > Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right after f= inal Login Response >=20 > I am curious to understand a bit more on why this is a protocol issue per= se. >=20 > Seems like one way to address this problem is via an implementation appro= ach with the initiator posting in advance the negotiated number of unsolici= ted PDU buffers, at the same time it makes the (final) negotiation offer. A= s the in-bound unsolicited PDUs can technically arrive any time after the o= ffer, due to standard network latency mechanics that Alexander summarized. = Has that approach been considered? >=20 > Mallikarjun >=20 >=20 >=20 >=20 > From: storm-bounces@ietf.org [mailto:storm= -bounces@ietf.org] On Behalf Of david.black@emc.com > Sent: Thursday, May 24, 2012 12:03 AM > To: mkosjc@gmail.com; nezhinsky@gmail.com > Cc: ogerlitz@mellanox.com; michaelc@cs.wisc= .edu; storm@ietf.org > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after f= inal Login Response > Importance: High >=20 > Mike (Ko) and Alexander, >=20 > Mike is of course correct that iSER Hello usage can be forced by negotiat= ing iSERHelloRequired to "Yes". However, existing implementations are like= ly to reply with iSERHelloRequired=3DNotUnderstood, so we do need to specif= y what should be done in order to interoperate with an implementation that = refuses to deal with the iSER Hello exchange. >=20 > I think the situation that Alexander described should be documented in a = new section 5.1.4 of the iSER draft. My general rule of thumb on this sort= of surprise found by implementers in the "running code" is that it indicat= es that something is missing in the spec. I believe that Alexander has des= cribed the solution - more below. >=20 > The new section 5.1.4 (suggested section title: Omission of the iSER Hell= o Exchange) should describe default omission of the exchange, use of iSERHe= lloRequired key to omit the iSER Hello exchange, and the consequences of ta= rget use of unsolicited PDUs after login when the exchange is omitted, incl= uding IB's use of NOP-IN (as a keep-alive measure, right?) >=20 > The crucial requirements points that I take away from Alexander's descrip= tion are that if the iSER Hello exchange is omitted, then: >=20 > 1) The target MAY send *one* unsolicited PDU immediately after sending th= e Login Response. >=20 > 2) The target MUST wait at least 200ms (use some other number if 200ms is= n't a good choice) or until it receives a full feature mode PDU from the in= itiator before sending a second unsolicited PDU in order to ensure that ini= tiator has sufficient > time to allocate the full feature buffer resources for the connectio= n. > 3) The initiator SHOULD allocate at least one additional buffer for use d= uring login (so that at least two buffers are in use during login) in order= to receive an unsolicited PDU that may follow login completion. Failure t= o allocate this second buffer may cause connection termination if no buffer= is available when an unsolicited PDU arrives. >=20 > Both Mike and I are on vacation, so it may be a few weeks until we can ag= ree on the new text and get a -12 version of the draft with that new text s= ubmitted. In the interim, I've asked our AD (Martin Stiemerling) to hold o= ff on further processing of the iSER draft until a -12 version with this ne= w text is submitted. I'd prefer to work this text out now rather than deal= with it as an IETF Last Call comment - as the problem turned up in actual = implementations, I think it's worth the extra month that it's likely to tak= e to get correct text on how to avoid the problem into the draft. >=20 > I'd suggest that Mike Ko post an initial draft of the text for the new se= ction 5.1.4 to the list when he resurfaces ... >=20 > Thanks, > --David >=20 > From: Michael Ko [mailto:mkosjc@gmail.com] > Sent: Monday, May 21, 2012 10:23 AM > To: Alexander Nezhinsky > Cc: storm@ietf.org; Black, David; Or Gerlitz; Mike= Christie > Subject: Re: iSER - problem with unsolicited NOP-IN right after final Log= in Response >=20 > Alex, >=20 > The iSER Hello support has never been removed in the latest spec. Only i= ts use is made optional. So during login negotiation, just negotiate iSERH= elloRequired to Yes. >=20 > Mike > On Mon, May 21, 2012 at 6:15 AM, Alexander Nezhinsky > wrote: > Hi >=20 > I understand that it is a bad timing for sending this kind of mail, now t= hat iSER draft was submitted, but actually we still have a small problem. > It is related to the final Login Response handling and the transition to = Full-Featured phase on the initiator side in Infiniband setups. >=20 > When the target receives the final Login Request it send the final Login = Response and from its perspective the connection is now in Full Featured Ph= ase (assuming that it agreed to transition in the Login Response being sent= ). >=20 > This means that the target is ready to accept SCSI commands, Text Request= s etc. sent by the initiator. > It also means that the target is eligible to send some unsolicited PDUs, = notably unsolicited NOP-INs. >=20 > With IB sending NOP-IN periodically is the easiest (an almost only feasib= le) way to determine closed connections reliably, because this kind of erro= r is delivered to user only in response to a previously initiated TX operat= ion. >=20 > This leaves the initiator in a dubious position. It posts its RX buffers = for that connection only when the final Login Response arrives. But during = that time (after the target had sent the Last Login Response but before the= Full Featured phase related RX-buffers are posted on the initiator side) t= he target may send the first NOP-IN as it considers the connection in Full = Featured phase already and NumOfUnsolicited PDUs accounting for NOP-INs has= been negotiated to a non-zero value. >=20 > If the initiator works with a single RX-buffer posted during the entire l= ogin phase (which is a logical thing to do judging by the login exchange pr= otocol) then an error occurs, as no buffers are posted when the NOP-IN arri= ves and the connection is shut down. >=20 > Posting a single extra buffer before sending the last Login Request only = alleviates the problem. Although this often solves it in practical terms (a= s the target most probably sends the next NOP-IN only after some timeout pe= riod measuring seconds or hundreds of milliseconds), it does not solves it = in terms of protocol completeness, as the target MAY theoretically send mor= e than one NOP-IN until FF buffers are posted. >=20 > This issue was encountered recently in linux iscsi/iser initiator and the= above solution has been applied to solve it against the existing target im= plementation (STGT), but the initiator remains exposed to this kind of erro= rs. >=20 > The solution is actually quite simple (theoretically) - if we bring back = the requirement for iSER Hello exchange then the iSER assisted Full Feature= d phase does not commence until HelloReply PDU arrives at the target and th= e initiator has a definitive point in time when it can safely post its RX b= uffers - after the final LoginResponse returns but before it sends iSER Hel= lo PDU. >=20 > In practical terms it means that iSER Hello support requirement should be= brought back to spec, which is a hassle. >=20 > Should we decide on this now? >=20 > Alexander >=20 > P.S. : Thanx to Mike Christie and Or Gerlitz, the maintainers of linux iS= CSI and iSER initiator for raising the issue. >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm >=20 >=20 >=20 From cbm@chadalapaka.com Mon Jun 11 11:56:41 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 800F521F8471 for ; Mon, 11 Jun 2012 11:56:41 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EwYTol287coA for ; Mon, 11 Jun 2012 11:56:40 -0700 (PDT) Received: from ch1outboundpool.messaging.microsoft.com (ch1ehsobe003.messaging.microsoft.com [216.32.181.183]) by ietfa.amsl.com (Postfix) with ESMTP id 1D79921F846E for ; Mon, 11 Jun 2012 11:56:40 -0700 (PDT) Received: from mail14-ch1-R.bigfish.com (10.43.68.243) by CH1EHSOBE013.bigfish.com (10.43.70.63) with Microsoft SMTP Server id 14.1.225.23; Mon, 11 Jun 2012 18:55:40 +0000 Received: from mail14-ch1 (localhost [127.0.0.1]) by mail14-ch1-R.bigfish.com (Postfix) with ESMTP id 0B1FE320058; Mon, 11 Jun 2012 18:55:40 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.240.117; KIP:(null); UIP:(null); IPV:NLI; H:BL2PRD0610HT002.namprd06.prod.outlook.com; RD:none; EFVD:NLI X-SpamScore: -29 X-BigFish: PS-29(zz98dI9371I542M1432I4015Izz1202hzz1033IL8275bh8275dhz2fh2a8h668h839h944hd25hf0ah) Received-SPF: pass (mail14-ch1: domain of chadalapaka.com designates 157.56.240.117 as permitted sender) client-ip=157.56.240.117; envelope-from=cbm@chadalapaka.com; helo=BL2PRD0610HT002.namprd06.prod.outlook.com ; .outlook.com ; Received: from mail14-ch1 (localhost.localdomain [127.0.0.1]) by mail14-ch1 (MessageSwitch) id 1339440937717079_2301; Mon, 11 Jun 2012 18:55:37 +0000 (UTC) Received: from CH1EHSMHS008.bigfish.com (snatpool1.int.messaging.microsoft.com [10.43.68.248]) by mail14-ch1.bigfish.com (Postfix) with ESMTP id A2710300046; Mon, 11 Jun 2012 18:55:37 +0000 (UTC) Received: from BL2PRD0610HT002.namprd06.prod.outlook.com (157.56.240.117) by CH1EHSMHS008.bigfish.com (10.43.70.8) with Microsoft SMTP Server (TLS) id 14.1.225.23; Mon, 11 Jun 2012 18:55:37 +0000 Received: from BL2PRD0610MB361.namprd06.prod.outlook.com ([169.254.10.144]) by BL2PRD0610HT002.namprd06.prod.outlook.com ([10.255.101.37]) with mapi id 14.16.0164.004; Mon, 11 Jun 2012 18:56:33 +0000 From: Mallikarjun Chadalapaka To: "Paul_Koning@Dell.com" Thread-Topic: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response Thread-Index: AQHNOXtOIIaCinNGykWqG66+HjtgdZbbOw+ggAecuICAEwDkAP//sNUggABa4AD//62tIA== Date: Mon, 11 Jun 2012 18:56:32 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE71205813C92@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120707E7F1@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [207.190.44.200] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: chadalapaka.com Cc: "michaelc@cs.wisc.edu" , "storm@ietf.org" , "ogerlitz@mellanox.com" Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jun 2012 18:56:41 -0000 Paul, I completely agree. This promise is conceptually similar to the CmdSN windo= w promise. FWIW, I did not believe both sides were compliant in this case. Mallikarjun -----Original Message----- From: Paul_Koning@Dell.com [mailto:Paul_Koning@Dell.com]=20 Sent: Monday, June 11, 2012 11:41 AM To: Mallikarjun Chadalapaka Cc: david.black@emc.com; mkosjc@gmail.com; nezhinsky@gmail.com; ogerlitz@me= llanox.com; michaelc@cs.wisc.edu; storm@ietf.org Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after fin= al Login Response If the standard says that the expected semantics is that the buffers promis= ed are all there at end of negotiation, then yes, it's an implementation is= sue. In that case, the original premise (both ends compliant) would not be= true -- one end is in violation. On the other hand, if the specification = of the negotiation process doesn't say when the promised resources have to = be available, that's a hole in the standard. paul On Jun 11, 2012, at 2:32 PM, Mallikarjun Chadalapaka wrote: > Sorry, if the implementation is promising x number of unsolicited buffers= but it is has =20 > I am all for providing helpful implementation guidance in the standard, b= ut we have to be cautious and re-confirm the need for the guidance, when we= start providing guidance down to latency numbers (see #2) - that's what go= t me concerned when I read the original note. >=20 > Thanks. >=20 > Mallikarjun >=20 >=20 > -----Original Message----- > From: Paul_Koning@Dell.com [mailto:Paul_Koning@Dell.com]=20 > Sent: Monday, June 11, 2012 10:59 AM > To: david.black@emc.com > Cc: Mallikarjun Chadalapaka; mkosjc@gmail.com; nezhinsky@gmail.com; ogerl= itz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after f= inal Login Response >=20 > That sounds like the right way to look at it. The whole point of protoco= l standards is to specify what is needed to interoperate. Not more, but al= so not less. Whenever two implementations both comply with a standard yet = do not interoperate, that's a bug in the standard. >=20 > paul >=20 > On Jun 11, 2012, at 11:34 AM, > > > wrote: >=20 > Hi Mallikarjun, >=20 > IMHO, the reason that this is a protocol issue is that two implementation= s that comply with the RFC can nonetheless reliably and repeatedly fail to = interoperate because the lack of additional buffers causes the RCaP connect= ion to close before full feature phase is reached by the initiator. >=20 > I believe that we have a responsibility to tell implementers what can go = wrong here and how to avoid it - the technique you describe (post all unsol= icited buffers before sending final negotiation message) could be mentioned= as part of this, and we should also describe what's possible with use of a= single buffer, as that approach is being pursued as Alexander describes. >=20 > Thanks, > --David >=20 > From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] > Sent: Friday, May 25, 2012 9:02 PM > To: Black, David; mkosjc@gmail.com; nezhinsky@gm= ail.com > Cc: ogerlitz@mellanox.com; michaelc@cs.wisc= .edu; storm@ietf.org > Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right after f= inal Login Response >=20 > I am curious to understand a bit more on why this is a protocol issue per= se. >=20 > Seems like one way to address this problem is via an implementation appro= ach with the initiator posting in advance the negotiated number of unsolici= ted PDU buffers, at the same time it makes the (final) negotiation offer. A= s the in-bound unsolicited PDUs can technically arrive any time after the o= ffer, due to standard network latency mechanics that Alexander summarized. = Has that approach been considered? >=20 > Mallikarjun >=20 >=20 >=20 >=20 > From: storm-bounces@ietf.org [mailto:storm= -bounces@ietf.org] On Behalf Of david.black@emc.com > Sent: Thursday, May 24, 2012 12:03 AM > To: mkosjc@gmail.com; nezhinsky@gmail.com > Cc: ogerlitz@mellanox.com; michaelc@cs.wisc= .edu; storm@ietf.org > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after f= inal Login Response > Importance: High >=20 > Mike (Ko) and Alexander, >=20 > Mike is of course correct that iSER Hello usage can be forced by negotiat= ing iSERHelloRequired to "Yes". However, existing implementations are like= ly to reply with iSERHelloRequired=3DNotUnderstood, so we do need to specif= y what should be done in order to interoperate with an implementation that = refuses to deal with the iSER Hello exchange. >=20 > I think the situation that Alexander described should be documented in a = new section 5.1.4 of the iSER draft. My general rule of thumb on this sort= of surprise found by implementers in the "running code" is that it indicat= es that something is missing in the spec. I believe that Alexander has des= cribed the solution - more below. >=20 > The new section 5.1.4 (suggested section title: Omission of the iSER Hell= o Exchange) should describe default omission of the exchange, use of iSERHe= lloRequired key to omit the iSER Hello exchange, and the consequences of ta= rget use of unsolicited PDUs after login when the exchange is omitted, incl= uding IB's use of NOP-IN (as a keep-alive measure, right?) >=20 > The crucial requirements points that I take away from Alexander's descrip= tion are that if the iSER Hello exchange is omitted, then: >=20 > 1) The target MAY send *one* unsolicited PDU immediately after sending th= e Login Response. >=20 > 2) The target MUST wait at least 200ms (use some other number if 200ms is= n't a good choice) or until it receives a full feature mode PDU from the in= itiator before sending a second unsolicited PDU in order to ensure that ini= tiator has sufficient > time to allocate the full feature buffer resources for the connectio= n. > 3) The initiator SHOULD allocate at least one additional buffer for use d= uring login (so that at least two buffers are in use during login) in order= to receive an unsolicited PDU that may follow login completion. Failure t= o allocate this second buffer may cause connection termination if no buffer= is available when an unsolicited PDU arrives. >=20 > Both Mike and I are on vacation, so it may be a few weeks until we can ag= ree on the new text and get a -12 version of the draft with that new text s= ubmitted. In the interim, I've asked our AD (Martin Stiemerling) to hold o= ff on further processing of the iSER draft until a -12 version with this ne= w text is submitted. I'd prefer to work this text out now rather than deal= with it as an IETF Last Call comment - as the problem turned up in actual = implementations, I think it's worth the extra month that it's likely to tak= e to get correct text on how to avoid the problem into the draft. >=20 > I'd suggest that Mike Ko post an initial draft of the text for the new se= ction 5.1.4 to the list when he resurfaces ... >=20 > Thanks, > --David >=20 > From: Michael Ko [mailto:mkosjc@gmail.com] > Sent: Monday, May 21, 2012 10:23 AM > To: Alexander Nezhinsky > Cc: storm@ietf.org; Black, David; Or Gerlitz; Mike= Christie > Subject: Re: iSER - problem with unsolicited NOP-IN right after final Log= in Response >=20 > Alex, >=20 > The iSER Hello support has never been removed in the latest spec. Only i= ts use is made optional. So during login negotiation, just negotiate iSERH= elloRequired to Yes. >=20 > Mike > On Mon, May 21, 2012 at 6:15 AM, Alexander Nezhinsky > wrote: > Hi >=20 > I understand that it is a bad timing for sending this kind of mail, now t= hat iSER draft was submitted, but actually we still have a small problem. > It is related to the final Login Response handling and the transition to = Full-Featured phase on the initiator side in Infiniband setups. >=20 > When the target receives the final Login Request it send the final Login = Response and from its perspective the connection is now in Full Featured Ph= ase (assuming that it agreed to transition in the Login Response being sent= ). >=20 > This means that the target is ready to accept SCSI commands, Text Request= s etc. sent by the initiator. > It also means that the target is eligible to send some unsolicited PDUs, = notably unsolicited NOP-INs. >=20 > With IB sending NOP-IN periodically is the easiest (an almost only feasib= le) way to determine closed connections reliably, because this kind of erro= r is delivered to user only in response to a previously initiated TX operat= ion. >=20 > This leaves the initiator in a dubious position. It posts its RX buffers = for that connection only when the final Login Response arrives. But during = that time (after the target had sent the Last Login Response but before the= Full Featured phase related RX-buffers are posted on the initiator side) t= he target may send the first NOP-IN as it considers the connection in Full = Featured phase already and NumOfUnsolicited PDUs accounting for NOP-INs has= been negotiated to a non-zero value. >=20 > If the initiator works with a single RX-buffer posted during the entire l= ogin phase (which is a logical thing to do judging by the login exchange pr= otocol) then an error occurs, as no buffers are posted when the NOP-IN arri= ves and the connection is shut down. >=20 > Posting a single extra buffer before sending the last Login Request only = alleviates the problem. Although this often solves it in practical terms (a= s the target most probably sends the next NOP-IN only after some timeout pe= riod measuring seconds or hundreds of milliseconds), it does not solves it = in terms of protocol completeness, as the target MAY theoretically send mor= e than one NOP-IN until FF buffers are posted. >=20 > This issue was encountered recently in linux iscsi/iser initiator and the= above solution has been applied to solve it against the existing target im= plementation (STGT), but the initiator remains exposed to this kind of erro= rs. >=20 > The solution is actually quite simple (theoretically) - if we bring back = the requirement for iSER Hello exchange then the iSER assisted Full Feature= d phase does not commence until HelloReply PDU arrives at the target and th= e initiator has a definitive point in time when it can safely post its RX b= uffers - after the final LoginResponse returns but before it sends iSER Hel= lo PDU. >=20 > In practical terms it means that iSER Hello support requirement should be= brought back to spec, which is a hassle. >=20 > Should we decide on this now? >=20 > Alexander >=20 > P.S. : Thanx to Mike Christie and Or Gerlitz, the maintainers of linux iS= CSI and iSER initiator for raising the issue. >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm >=20 >=20 >=20 From david.black@emc.com Mon Jun 11 18:44:46 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0810021F84DF for ; Mon, 11 Jun 2012 18:44:46 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eumAFhSSKDKj for ; Mon, 11 Jun 2012 18:44:44 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 7285D21F84D8 for ; Mon, 11 Jun 2012 18:44:44 -0700 (PDT) Received: from hop04-l1d11-si01.isus.emc.com (HOP04-L1D11-SI01.isus.emc.com [10.254.111.54]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5C1icem030232 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 11 Jun 2012 21:44:38 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.222.129]) by hop04-l1d11-si01.isus.emc.com (RSA Interceptor); Mon, 11 Jun 2012 21:44:27 -0400 Received: from mxhub11.corp.emc.com (mxhub11.corp.emc.com [10.254.92.106]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5C1iQoE015835; Mon, 11 Jun 2012 21:44:26 -0400 Received: from mx15a.corp.emc.com ([169.254.1.101]) by mxhub11.corp.emc.com ([10.254.92.106]) with mapi; Mon, 11 Jun 2012 21:44:26 -0400 From: To: , Date: Mon, 11 Jun 2012 21:44:24 -0400 Thread-Topic: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response Thread-Index: AQHNOXtOIIaCinNGykWqG66+HjtgdZbbOw+ggAecuICAEwDkAP//sNUggABa4AD//62tIIAAcLpA Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120707E8D5@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71205813C92@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120707E7F1@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Cc: storm@ietf.org Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2012 01:44:46 -0000 > FWIW, I did not believe both sides were compliant in this case. There's an additional consideration here - one of the goals of the iSER update is to match "running code", and we have already made a number of changes that diverge from RFC 5046 for this reason, including in this area of iSER start-up (see Appendix A of the iSER draft for details). As a result, we are already well into non-compliance territory, as omission of the Hello messages does not comply with RFC 5046, but those messages are simply not implemented in practice. The implementation under discussion is (I believe) the predominant iSER implementation and hence I think the new iSER RFC needs to explain what that implementation has done (even if it may not be what should have been done), and (more importantly) explain how to interoperate. The key requirement is the following (IMHO, a fine example of "be conservative in what you send"): > > 2) The target MUST wait at least 200ms (use some other number if 200ms > > isn't a good choice) or until it receives a full feature mode PDU from > > the initiator before sending a second unsolicited PDU in order to ensur= e > > that initiator has sufficient time to allocate the full feature buffer > > resources for the connection. Not putting this into the iSER update requires either that Hello messages be implemented (not done, primarily for optimization reasons), or that the initiator effectively commit the buffer resources when starting negotiation= . Thanks, --David > -----Original Message----- > From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] > Sent: Monday, June 11, 2012 2:57 PM > To: Paul_Koning@Dell.com > Cc: Black, David; mkosjc@gmail.com; nezhinsky@gmail.com; > ogerlitz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org > Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right after f= inal > Login Response >=20 > Paul, >=20 > I completely agree. This promise is conceptually similar to the CmdSN win= dow > promise. FWIW, I did not believe both sides were compliant in this case. >=20 > Mallikarjun >=20 >=20 > -----Original Message----- > From: Paul_Koning@Dell.com [mailto:Paul_Koning@Dell.com] > Sent: Monday, June 11, 2012 11:41 AM > To: Mallikarjun Chadalapaka > Cc: david.black@emc.com; mkosjc@gmail.com; nezhinsky@gmail.com; > ogerlitz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after f= inal > Login Response >=20 > If the standard says that the expected semantics is that the buffers prom= ised > are all there at end of negotiation, then yes, it's an implementation iss= ue. > In that case, the original premise (both ends compliant) would not be tru= e -- > one end is in violation. On the other hand, if the specification of the > negotiation process doesn't say when the promised resources have to be > available, that's a hole in the standard. >=20 > paul >=20 > On Jun 11, 2012, at 2:32 PM, Mallikarjun Chadalapaka wrote: >=20 > > Sorry, if the implementation is promising x number of unsolicited buffe= rs > but it is has implementation to not comply with the standard anymore. Standard on its p= art > should clearly define the semantics of the promise - which I presume it d= oes > in this case. > > > > I am all for providing helpful implementation guidance in the standard,= but > we have to be cautious and re-confirm the need for the guidance, when we = start > providing guidance down to latency numbers (see #2) - that's what got me > concerned when I read the original note. > > > > Thanks. > > > > Mallikarjun > > > > > > -----Original Message----- > > From: Paul_Koning@Dell.com [mailto:Paul_Koning@Dell.com] > > Sent: Monday, June 11, 2012 10:59 AM > > To: david.black@emc.com > > Cc: Mallikarjun Chadalapaka; mkosjc@gmail.com; nezhinsky@gmail.com; > ogerlitz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org > > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after > final Login Response > > > > That sounds like the right way to look at it. The whole point of proto= col > standards is to specify what is needed to interoperate. Not more, but al= so > not less. Whenever two implementations both comply with a standard yet d= o not > interoperate, that's a bug in the standard. > > > > paul > > > > On Jun 11, 2012, at 11:34 AM, > > > > > wrote: > > > > Hi Mallikarjun, > > > > IMHO, the reason that this is a protocol issue is that two implementati= ons > that comply with the RFC can nonetheless reliably and repeatedly fail to > interoperate because the lack of additional buffers causes the RCaP conne= ction > to close before full feature phase is reached by the initiator. > > > > I believe that we have a responsibility to tell implementers what can g= o > wrong here and how to avoid it - the technique you describe (post all > unsolicited buffers before sending final negotiation message) could be > mentioned as part of this, and we should also describe what's possible wi= th > use of a single buffer, as that approach is being pursued as Alexander > describes. > > > > Thanks, > > --David > > > > From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] > > Sent: Friday, May 25, 2012 9:02 PM > > To: Black, David; mkosjc@gmail.com; > nezhinsky@gmail.com > > Cc: ogerlitz@mellanox.com; > michaelc@cs.wisc.edu; > storm@ietf.org > > Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right after > final Login Response > > > > I am curious to understand a bit more on why this is a protocol issue p= er > se. > > > > Seems like one way to address this problem is via an implementation app= roach > with the initiator posting in advance the negotiated number of unsolicite= d PDU > buffers, at the same time it makes the (final) negotiation offer. As the = in- > bound unsolicited PDUs can technically arrive any time after the offer, d= ue to > standard network latency mechanics that Alexander summarized. Has that > approach been considered? > > > > Mallikarjun > > > > > > > > > > From: storm-bounces@ietf.org [mailto:sto= rm- > bounces@ietf.org] On Behalf Of david.black@emc.com > > Sent: Thursday, May 24, 2012 12:03 AM > > To: mkosjc@gmail.com; > nezhinsky@gmail.com > > Cc: ogerlitz@mellanox.com; > michaelc@cs.wisc.edu; > storm@ietf.org > > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after > final Login Response > > Importance: High > > > > Mike (Ko) and Alexander, > > > > Mike is of course correct that iSER Hello usage can be forced by negoti= ating > iSERHelloRequired to "Yes". However, existing implementations are likely= to > reply with iSERHelloRequired=3DNotUnderstood, so we do need to specify wh= at > should be done in order to interoperate with an implementation that refus= es to > deal with the iSER Hello exchange. > > > > I think the situation that Alexander described should be documented in = a new > section 5.1.4 of the iSER draft. My general rule of thumb on this sort o= f > surprise found by implementers in the "running code" is that it indicates= that > something is missing in the spec. I believe that Alexander has described= the > solution - more below. > > > > The new section 5.1.4 (suggested section title: Omission of the iSER He= llo > Exchange) should describe default omission of the exchange, use of > iSERHelloRequired key to omit the iSER Hello exchange, and the consequenc= es of > target use of unsolicited PDUs after login when the exchange is omitted, > including IB's use of NOP-IN (as a keep-alive measure, right?) > > > > The crucial requirements points that I take away from Alexander's > description are that if the iSER Hello exchange is omitted, then: > > > > 1) The target MAY send *one* unsolicited PDU immediately after sending = the > Login Response. > > > > 2) The target MUST wait at least 200ms (use some other number if 200ms = isn't > a good choice) or until it receives a full feature mode PDU from the init= iator > before sending a second unsolicited PDU in order to ensure that initiator= has > sufficient > > time to allocate the full feature buffer resources for the connect= ion. > > 3) The initiator SHOULD allocate at least one additional buffer for use > during login (so that at least two buffers are in use during login) in or= der > to receive an unsolicited PDU that may follow login completion. Failure = to > allocate this second buffer may cause connection termination if no buffer= is > available when an unsolicited PDU arrives. > > > > Both Mike and I are on vacation, so it may be a few weeks until we can = agree > on the new text and get a -12 version of the draft with that new text > submitted. In the interim, I've asked our AD (Martin Stiemerling) to hol= d off > on further processing of the iSER draft until a -12 version with this new= text > is submitted. I'd prefer to work this text out now rather than deal with= it > as an IETF Last Call comment - as the problem turned up in actual > implementations, I think it's worth the extra month that it's likely to t= ake > to get correct text on how to avoid the problem into the draft. > > > > I'd suggest that Mike Ko post an initial draft of the text for the new > section 5.1.4 to the list when he resurfaces ... > > > > Thanks, > > --David > > > > From: Michael Ko [mailto:mkosjc@gmail.com] > > Sent: Monday, May 21, 2012 10:23 AM > > To: Alexander Nezhinsky > > Cc: storm@ietf.org; Black, David; Or Gerlitz; Mi= ke > Christie > > Subject: Re: iSER - problem with unsolicited NOP-IN right after final L= ogin > Response > > > > Alex, > > > > The iSER Hello support has never been removed in the latest spec. Only= its > use is made optional. So during login negotiation, just negotiate > iSERHelloRequired to Yes. > > > > Mike > > On Mon, May 21, 2012 at 6:15 AM, Alexander Nezhinsky > > wrote: > > Hi > > > > I understand that it is a bad timing for sending this kind of mail, now= that > iSER draft was submitted, but actually we still have a small problem. > > It is related to the final Login Response handling and the transition t= o > Full-Featured phase on the initiator side in Infiniband setups. > > > > When the target receives the final Login Request it send the final Logi= n > Response and from its perspective the connection is now in Full Featured = Phase > (assuming that it agreed to transition in the Login Response being sent). > > > > This means that the target is ready to accept SCSI commands, Text Reque= sts > etc. sent by the initiator. > > It also means that the target is eligible to send some unsolicited PDUs= , > notably unsolicited NOP-INs. > > > > With IB sending NOP-IN periodically is the easiest (an almost only feas= ible) > way to determine closed connections reliably, because this kind of error = is > delivered to user only in response to a previously initiated TX operation= . > > > > This leaves the initiator in a dubious position. It posts its RX buffer= s for > that connection only when the final Login Response arrives. But during th= at > time (after the target had sent the Last Login Response but before the Fu= ll > Featured phase related RX-buffers are posted on the initiator side) the t= arget > may send the first NOP-IN as it considers the connection in Full Featured > phase already and NumOfUnsolicited PDUs accounting for NOP-INs has been > negotiated to a non-zero value. > > > > If the initiator works with a single RX-buffer posted during the entire > login phase (which is a logical thing to do judging by the login exchange > protocol) then an error occurs, as no buffers are posted when the NOP-IN > arrives and the connection is shut down. > > > > Posting a single extra buffer before sending the last Login Request onl= y > alleviates the problem. Although this often solves it in practical terms = (as > the target most probably sends the next NOP-IN only after some timeout pe= riod > measuring seconds or hundreds of milliseconds), it does not solves it in = terms > of protocol completeness, as the target MAY theoretically send more than = one > NOP-IN until FF buffers are posted. > > > > This issue was encountered recently in linux iscsi/iser initiator and t= he > above solution has been applied to solve it against the existing target > implementation (STGT), but the initiator remains exposed to this kind of > errors. > > > > The solution is actually quite simple (theoretically) - if we bring bac= k the > requirement for iSER Hello exchange then the iSER assisted Full Featured = phase > does not commence until HelloReply PDU arrives at the target and the init= iator > has a definitive point in time when it can safely post its RX buffers - a= fter > the final LoginResponse returns but before it sends iSER Hello PDU. > > > > In practical terms it means that iSER Hello support requirement should = be > brought back to spec, which is a hassle. > > > > Should we decide on this now? > > > > Alexander > > > > P.S. : Thanx to Mike Christie and Or Gerlitz, the maintainers of linux = iSCSI > and iSER initiator for raising the issue. > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > > > > > > >=20 >=20 From mkosjc@gmail.com Mon Jun 11 19:00:38 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E553D21F8559 for ; Mon, 11 Jun 2012 19:00:38 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.598 X-Spam-Level: X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KlJrfuG+gqGc for ; Mon, 11 Jun 2012 19:00:37 -0700 (PDT) Received: from mail-qc0-f172.google.com (mail-qc0-f172.google.com [209.85.216.172]) by ietfa.amsl.com (Postfix) with ESMTP id C37CF21F8512 for ; Mon, 11 Jun 2012 19:00:36 -0700 (PDT) Received: by qcsq13 with SMTP id q13so2810453qcs.31 for ; Mon, 11 Jun 2012 19:00:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=gx45U9hkTAp/32UvNCX29kNgo9+bUVKrh8c7augxDpk=; b=cJoNEpvTO7D9TKC42qz6egr5GEEBo78oonidxYKYhREYqY4V+Fk+HU1Sg9ZSxr3Yh3 8DEqUKb+uZFlpO2ajPBkC9X3+xJOAZmKbc5rwl/ufj6H89c/L5iyTmI0jXhSG6psyMks rGKULyMaO4t7Tz/776VTiewi9+0OvSNDUu1N1JWQPUpFU0JvQFGSR3sYYdYyb3R9etm+ uuHbRJ7OAtxmUpNBnfDuSoZV78WUcgGX4vE1ZNfzsRIW2/QPtleSpYjVdEC03EcdawIh UwVI4jlgllUt21MTNNxyBxMjWVsdz5V4MuXHApr0CjtFjQ3/js8mb9Xcdr6VnTbLV7jH /new== MIME-Version: 1.0 Received: by 10.224.191.74 with SMTP id dl10mr17235920qab.65.1339466436040; Mon, 11 Jun 2012 19:00:36 -0700 (PDT) Received: by 10.229.130.233 with HTTP; Mon, 11 Jun 2012 19:00:35 -0700 (PDT) In-Reply-To: References: <8D3D17ACE214DC429325B2B98F3AE71205813C92@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120707E7F1@MX15A.corp.emc.com> Date: Mon, 11 Jun 2012 19:00:35 -0700 Message-ID: From: Michael Ko To: Mallikarjun Chadalapaka Content-Type: multipart/alternative; boundary=20cf3005dcb21bd6de04c23cd589 Cc: "michaelc@cs.wisc.edu" , "storm@ietf.org" , "ogerlitz@mellanox.com" Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2012 02:00:39 -0000 --20cf3005dcb21bd6de04c23cd589 Content-Type: text/plain; charset=ISO-8859-1 I agree with Mallikarjun. In the scenario described by Alex, the initiator side was not compliant with the standard. >From section 5.1.1 of the iSER draft on the initiator behavior during connection setup: "If the outcome of the iSCSI negotiation is to enable iSER-assisted mode, then on the initiator side, prior to sending the Login Request with the T (Transit) bit set to 1 and the NSG (Next Stage) field set to FullFeaturePhase, the iSCSI Layer MUST request the iSER Layer to allocate the connection resources necessary to support RCaP by invoking the Allocate_Connection_Resources Operational Primitive." It clearly states that the initiator must "allocate the connection resources necessary to support RCaP" before sending the Login Request to transition to FullFeaturePhase. Had the initiator been compliant with this requirement, it would not run into the insufficient Rx buffer problem. Negotiating iSERHelloRequired to "Yes" is a way to shortcut this requirement and buy time for the initiator if both sides support it, but it should not be the modus operandi, and I don't think clarifying text is needed to justify its usage to solve this particular problem caused by non-compliance.. Mike On Mon, Jun 11, 2012 at 11:56 AM, Mallikarjun Chadalapaka < cbm@chadalapaka.com> wrote: > Paul, > > I completely agree. This promise is conceptually similar to the CmdSN > window promise. FWIW, I did not believe both sides were compliant in this > case. > > Mallikarjun > > > -----Original Message----- > From: Paul_Koning@Dell.com [mailto:Paul_Koning@Dell.com] > Sent: Monday, June 11, 2012 11:41 AM > To: Mallikarjun Chadalapaka > Cc: david.black@emc.com; mkosjc@gmail.com; nezhinsky@gmail.com; > ogerlitz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after > final Login Response > > If the standard says that the expected semantics is that the buffers > promised are all there at end of negotiation, then yes, it's an > implementation issue. In that case, the original premise (both ends > compliant) would not be true -- one end is in violation. On the other > hand, if the specification of the negotiation process doesn't say when the > promised resources have to be available, that's a hole in the standard. > > paul > > On Jun 11, 2012, at 2:32 PM, Mallikarjun Chadalapaka wrote: > > > Sorry, if the implementation is promising x number of unsolicited > buffers but it is has consider the implementation to not comply with the standard anymore. > Standard on its part should clearly define the semantics of the promise - > which I presume it does in this case. > > > > I am all for providing helpful implementation guidance in the standard, > but we have to be cautious and re-confirm the need for the guidance, when > we start providing guidance down to latency numbers (see #2) - that's what > got me concerned when I read the original note. > > > > Thanks. > > > > Mallikarjun > > > > > > -----Original Message----- > > From: Paul_Koning@Dell.com [mailto:Paul_Koning@Dell.com] > > Sent: Monday, June 11, 2012 10:59 AM > > To: david.black@emc.com > > Cc: Mallikarjun Chadalapaka; mkosjc@gmail.com; nezhinsky@gmail.com; > ogerlitz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org > > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after > final Login Response > > > > That sounds like the right way to look at it. The whole point of > protocol standards is to specify what is needed to interoperate. Not more, > but also not less. Whenever two implementations both comply with a > standard yet do not interoperate, that's a bug in the standard. > > > > paul > > > > On Jun 11, 2012, at 11:34 AM, david.black@emc.com>> > > > wrote: > > > > Hi Mallikarjun, > > > > IMHO, the reason that this is a protocol issue is that two > implementations that comply with the RFC can nonetheless reliably and > repeatedly fail to interoperate because the lack of additional buffers > causes the RCaP connection to close before full feature phase is reached by > the initiator. > > > > I believe that we have a responsibility to tell implementers what can go > wrong here and how to avoid it - the technique you describe (post all > unsolicited buffers before sending final negotiation message) could be > mentioned as part of this, and we should also describe what's possible with > use of a single buffer, as that approach is being pursued as Alexander > describes. > > > > Thanks, > > --David > > > > From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] > > Sent: Friday, May 25, 2012 9:02 PM > > To: Black, David; mkosjc@gmail.com; > nezhinsky@gmail.com > > Cc: ogerlitz@mellanox.com; > michaelc@cs.wisc.edu; storm@ietf.org storm@ietf.org> > > Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right after > final Login Response > > > > I am curious to understand a bit more on why this is a protocol issue > per se. > > > > Seems like one way to address this problem is via an implementation > approach with the initiator posting in advance the negotiated number of > unsolicited PDU buffers, at the same time it makes the (final) negotiation > offer. As the in-bound unsolicited PDUs can technically arrive any time > after the offer, due to standard network latency mechanics that Alexander > summarized. Has that approach been considered? > > > > Mallikarjun > > > > > > > > > > From: storm-bounces@ietf.org [mailto: > storm-bounces@ietf.org] On Behalf Of david.black@emc.com david.black@emc.com> > > Sent: Thursday, May 24, 2012 12:03 AM > > To: mkosjc@gmail.com; nezhinsky@gmail.com > > > Cc: ogerlitz@mellanox.com; > michaelc@cs.wisc.edu; storm@ietf.org storm@ietf.org> > > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after > final Login Response > > Importance: High > > > > Mike (Ko) and Alexander, > > > > Mike is of course correct that iSER Hello usage can be forced by > negotiating iSERHelloRequired to "Yes". However, existing implementations > are likely to reply with iSERHelloRequired=NotUnderstood, so we do need to > specify what should be done in order to interoperate with an implementation > that refuses to deal with the iSER Hello exchange. > > > > I think the situation that Alexander described should be documented in a > new section 5.1.4 of the iSER draft. My general rule of thumb on this sort > of surprise found by implementers in the "running code" is that it > indicates that something is missing in the spec. I believe that Alexander > has described the solution - more below. > > > > The new section 5.1.4 (suggested section title: Omission of the iSER > Hello Exchange) should describe default omission of the exchange, use of > iSERHelloRequired key to omit the iSER Hello exchange, and the consequences > of target use of unsolicited PDUs after login when the exchange is omitted, > including IB's use of NOP-IN (as a keep-alive measure, right?) > > > > The crucial requirements points that I take away from Alexander's > description are that if the iSER Hello exchange is omitted, then: > > > > 1) The target MAY send *one* unsolicited PDU immediately after sending > the Login Response. > > > > 2) The target MUST wait at least 200ms (use some other number if 200ms > isn't a good choice) or until it receives a full feature mode PDU from the > initiator before sending a second unsolicited PDU in order to ensure that > initiator has sufficient > > time to allocate the full feature buffer resources for the > connection. > > 3) The initiator SHOULD allocate at least one additional buffer for use > during login (so that at least two buffers are in use during login) in > order to receive an unsolicited PDU that may follow login completion. > Failure to allocate this second buffer may cause connection termination if > no buffer is available when an unsolicited PDU arrives. > > > > Both Mike and I are on vacation, so it may be a few weeks until we can > agree on the new text and get a -12 version of the draft with that new text > submitted. In the interim, I've asked our AD (Martin Stiemerling) to hold > off on further processing of the iSER draft until a -12 version with this > new text is submitted. I'd prefer to work this text out now rather than > deal with it as an IETF Last Call comment - as the problem turned up in > actual implementations, I think it's worth the extra month that it's likely > to take to get correct text on how to avoid the problem into the draft. > > > > I'd suggest that Mike Ko post an initial draft of the text for the new > section 5.1.4 to the list when he resurfaces ... > > > > Thanks, > > --David > > > > From: Michael Ko [mailto:mkosjc@gmail.com] mkosjc@gmail.com]> > > Sent: Monday, May 21, 2012 10:23 AM > > To: Alexander Nezhinsky > > Cc: storm@ietf.org; Black, David; Or Gerlitz; > Mike Christie > > Subject: Re: iSER - problem with unsolicited NOP-IN right after final > Login Response > > > > Alex, > > > > The iSER Hello support has never been removed in the latest spec. Only > its use is made optional. So during login negotiation, just negotiate > iSERHelloRequired to Yes. > > > > Mike > > On Mon, May 21, 2012 at 6:15 AM, Alexander Nezhinsky < > nezhinsky@gmail.com> wrote: > > Hi > > > > I understand that it is a bad timing for sending this kind of mail, now > that iSER draft was submitted, but actually we still have a small problem. > > It is related to the final Login Response handling and the transition to > Full-Featured phase on the initiator side in Infiniband setups. > > > > When the target receives the final Login Request it send the final Login > Response and from its perspective the connection is now in Full Featured > Phase (assuming that it agreed to transition in the Login Response being > sent). > > > > This means that the target is ready to accept SCSI commands, Text > Requests etc. sent by the initiator. > > It also means that the target is eligible to send some unsolicited PDUs, > notably unsolicited NOP-INs. > > > > With IB sending NOP-IN periodically is the easiest (an almost only > feasible) way to determine closed connections reliably, because this kind > of error is delivered to user only in response to a previously initiated TX > operation. > > > > This leaves the initiator in a dubious position. It posts its RX buffers > for that connection only when the final Login Response arrives. But during > that time (after the target had sent the Last Login Response but before the > Full Featured phase related RX-buffers are posted on the initiator side) > the target may send the first NOP-IN as it considers the connection in Full > Featured phase already and NumOfUnsolicited PDUs accounting for NOP-INs has > been negotiated to a non-zero value. > > > > If the initiator works with a single RX-buffer posted during the entire > login phase (which is a logical thing to do judging by the login exchange > protocol) then an error occurs, as no buffers are posted when the NOP-IN > arrives and the connection is shut down. > > > > Posting a single extra buffer before sending the last Login Request only > alleviates the problem. Although this often solves it in practical terms > (as the target most probably sends the next NOP-IN only after some timeout > period measuring seconds or hundreds of milliseconds), it does not solves > it in terms of protocol completeness, as the target MAY theoretically send > more than one NOP-IN until FF buffers are posted. > > > > This issue was encountered recently in linux iscsi/iser initiator and > the above solution has been applied to solve it against the existing target > implementation (STGT), but the initiator remains exposed to this kind of > errors. > > > > The solution is actually quite simple (theoretically) - if we bring back > the requirement for iSER Hello exchange then the iSER assisted Full > Featured phase does not commence until HelloReply PDU arrives at the target > and the initiator has a definitive point in time when it can safely post > its RX buffers - after the final LoginResponse returns but before it sends > iSER Hello PDU. > > > > In practical terms it means that iSER Hello support requirement should > be brought back to spec, which is a hassle. > > > > Should we decide on this now? > > > > Alexander > > > > P.S. : Thanx to Mike Christie and Or Gerlitz, the maintainers of linux > iSCSI and iSER initiator for raising the issue. > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > > > > > > > > > > --20cf3005dcb21bd6de04c23cd589 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I agree with Mallikarjun.=A0 In the scenario described by Alex, the initiat= or side was not compliant with the standard.

From section 5.1.1 of t= he iSER draft on the initiator behavior during connection setup:

&qu= ot;If the outcome of the iSCSI negotiation is to enable iSER-assisted mode,= then on the initiator side, prior to sending the Login Request with the T = (Transit) bit set to 1 and the NSG (Next Stage) field set to FullFeaturePha= se, the iSCSI Layer MUST request the iSER Layer to allocate the connection = resources necessary to support RCaP by invoking the Allocate_Connection_Res= ources Operational Primitive."

It clearly states that the initiator must "allocate the connection= resources necessary to support RCaP" before sending the Login Request= to transition to FullFeaturePhase.=A0 Had the initiator been compliant wit= h this requirement, it would not run into the insufficient Rx buffer proble= m.=A0 Negotiating iSERHelloRequired to "Yes" is a way to shortcut= this requirement and buy time for the initiator if both sides support it, = but it should not be the modus operandi, and I don't think clarifying t= ext is needed to justify its usage to solve this particular problem caused = by non-compliance..

Mike
On Mon, Jun 11, 2012 at 11:56 AM, Ma= llikarjun Chadalapaka <cbm@chadalapaka.com> wrote:
Paul,

I completely agree. This promise is conceptually similar to the CmdSN windo= w promise. FWIW, I did not believe both sides were compliant in this case.<= br>
Mallikarjun


-----Original Message-----
From: Paul_Koning@Dell.com [mailto:= Paul_Koning@Dell.com]
Sent: Monday, June 11, 2012 11:41 AM
To: Mallikarjun Chadalapaka
Cc: david.black@emc.com; mkosjc@gmail.com; nezhinsky@gmail.com; ogerlitz@mellanox.com; mi= chaelc@cs.wisc.edu; storm@ietf.org
Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after fin= al Login Response

If the standard says that the expected semantics is that the buffers promis= ed are all there at end of negotiation, then yes, it's an implementatio= n issue. =A0In that case, the original premise (both ends compliant) would = not be true -- one end is in violation. =A0On the other hand, if the specif= ication of the negotiation process doesn't say when the promised resour= ces have to be available, that's a hole in the standard.

=A0 =A0 =A0 =A0paul

On Jun 11, 2012, at 2:32 PM, Mallikarjun Chadalapaka wrote:

> Sorry, if the implementation is promising x number of unsolicited buff= ers but it is has <x buffers ready at the end of negotiation, I would co= nsider the implementation to not comply with the standard anymore. Standard= on its part should clearly define the semantics of the promise - which I p= resume it does in this case.
>
> I am all for providing helpful implementation guidance in the standard= , but we have to be cautious and re-confirm the need for the guidance, when= we start providing guidance down to latency numbers (see #2) - that's = what got me concerned when I read the original note.
>
> Thanks.
>
> Mallikarjun
>
>
> -----Original Message-----
> From: Paul_Koning@Dell.com [mailto:Paul_Koning@Dell.com]
> Sent: Monday, June 11, 2012 10:59 AM
> To: david.black@emc.com
> Cc: Mallikarjun Chadalapaka; mkosj= c@gmail.com; nezhinsky@gmail.com= ; ogerlitz@mellanox.com; <= a href=3D"mailto:michaelc@cs.wisc.edu">michaelc@cs.wisc.edu; storm@ietf.org
> Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right afte= r final Login Response
>
> That sounds like the right way to look at it. =A0The whole point of pr= otocol standards is to specify what is needed to interoperate. =A0Not more,= but also not less. =A0Whenever two implementations both comply with a stan= dard yet do not interoperate, that's a bug in the standard.
>
> paul
>
> On Jun 11, 2012, at 11:34 AM, <david.black@emc.com<mailto:david.black@emc.com>>
> <david.black@emc.com<= mailto:david.black@emc.com>&g= t; wrote:
>
> Hi Mallikarjun,
>
> IMHO, the reason that this is a protocol issue is that two implementat= ions that comply with the RFC can nonetheless reliably and repeatedly fail = to interoperate because the lack of additional buffers causes the RCaP conn= ection to close before full feature phase is reached by the initiator.
>
> I believe that we have a responsibility to tell implementers what can = go wrong here and how to avoid it - the technique you describe (post all un= solicited buffers before sending final negotiation message) could be mentio= ned as part of this, and we should also describe what's possible with u= se of a single buffer, as that approach is being pursued as Alexander descr= ibes.
>
> Thanks,
> --David
>
> From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
> Sent: Friday, May 25, 2012 9:02 PM
> To: Black, David; mkosjc@gmail.com= <mailto:mkosjc@gmail.com>= ; nezhinsky@gmail.com<mailto:= nezhinsky@gmail.com>
> Cc: ogerlitz@mellanox.com= <mailto:ogerlitz@mellanox.com>; michaelc@cs.wisc.edu<= ;mailto:michaelc@cs.wisc.edu>= ;; storm@ietf.org<mailto:storm@ietf.org>
> Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right afte= r final Login Response
>
> I am curious to understand a bit more on why this is a protocol issue = per se.
>
> Seems like one way to address this problem is via an implementation ap= proach with the initiator posting in advance the negotiated number of unsol= icited PDU buffers, at the same time it makes the (final) negotiation offer= . As the in-bound unsolicited PDUs can technically arrive any time after th= e offer, due to standard network latency mechanics that Alexander summarize= d. Has that approach been considered?
>
> Mallikarjun
>
>
>
>
> From: storm-bounces@ietf.org= <mailto:storm-bounces@ietf= .org> [mailto:storm-bounce= s@ietf.org] On Behalf Of david.b= lack@emc.com<mailto:david.bla= ck@emc.com>
> Sent: Thursday, May 24, 2012 12:03 AM
> To: mkosjc@gmail.com<mailto= :mkosjc@gmail.com>; nezhinsky@gmail.com<mailto:nezhinsky@gmail.com>
> Cc: ogerlitz@mellanox.com= <mailto:ogerlitz@mellanox.com>; michaelc@cs.wisc.edu<= ;mailto:michaelc@cs.wisc.edu>= ;; storm@ietf.org<mailto:storm@ietf.org>
> Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right afte= r final Login Response
> Importance: High
>
> Mike (Ko) and Alexander,
>
> Mike is of course correct that iSER Hello usage can be forced by negot= iating iSERHelloRequired to "Yes". =A0However, existing implement= ations are likely to reply with iSERHelloRequired=3DNotUnderstood, so we do= need to specify what should be done in order to interoperate with an imple= mentation that refuses to deal with the iSER Hello exchange.
>
> I think the situation that Alexander described should be documented in= a new section 5.1.4 of the iSER draft. =A0My general rule of thumb on this= sort of surprise found by implementers in the "running code" is = that it indicates that something is missing in the spec. =A0I believe that = Alexander has described the solution - more below.
>
> The new section 5.1.4 (suggested section title: Omission of the iSER H= ello Exchange) should describe default omission of the exchange, use of iSE= RHelloRequired key to omit the iSER Hello exchange, and the consequences of= target use of unsolicited PDUs after login when the exchange is omitted, i= ncluding IB's use of NOP-IN (as a keep-alive measure, right?)
>
> The crucial requirements points that I take away from Alexander's = description are that if the iSER Hello exchange is omitted, then:
>
> 1) The target MAY send *one* unsolicited PDU immediately after sending= the Login Response.
>
> 2) The target MUST wait at least 200ms (use some other number if 200ms= isn't a good choice) or until it receives a full feature mode PDU from= the initiator before sending a second unsolicited PDU in order to ensure t= hat initiator has sufficient
> =A0 =A0 =A0time to allocate the full feature buffer resources for the = connection.
> 3) The initiator SHOULD allocate at least one additional buffer for us= e during login (so that at least two buffers are in use during login) in or= der to receive an unsolicited PDU that may follow login completion. =A0Fail= ure to allocate this second buffer may cause connection termination if no b= uffer is available when an unsolicited PDU arrives.
>
> Both Mike and I are on vacation, so it may be a few weeks until we can= agree on the new text and get a -12 version of the draft with that new tex= t submitted. =A0In the interim, I've asked our AD (Martin Stiemerling) = to hold off on further processing of the iSER draft until a -12 version wit= h this new text is submitted. =A0I'd prefer to work this text out now r= ather than deal with it as an IETF Last Call comment - as the problem turne= d up in actual implementations, I think it's worth the extra month that= it's likely to take to get correct text on how to avoid the problem in= to the draft.
>
> I'd suggest that Mike Ko post an initial draft of the text for the= new section 5.1.4 to the list when he resurfaces ...
>
> Thanks,
> --David
>
> From: Michael Ko [mailto:mkosjc@gm= ail.com]<mailto:[mailto:mkosjc@g= mail.com]>
> Sent: Monday, May 21, 2012 10:23 AM
> To: Alexander Nezhinsky
> Cc: storm@ietf.org<mailto:storm@ietf.org>; Black, David; Or Ger= litz; Mike Christie
> Subject: Re: iSER - problem with unsolicited NOP-IN right after final = Login Response
>
> Alex,
>
> The iSER Hello support has never been removed in the latest spec. =A0O= nly its use is made optional. =A0So during login negotiation, just negotiat= e iSERHelloRequired to Yes.
>
> Mike
> On Mon, May 21, 2012 at 6:15 AM, Alexander Nezhinsky <nezhinsky@gmail.com<mailto:nezhinsky@gmail.com>> wrote:
> Hi
>
> I understand that it is a bad timing for sending this kind of mail, no= w that iSER draft was submitted, but actually we still have a small problem= .
> It is related to the final Login Response handling and the transition = to Full-Featured phase on the initiator side in Infiniband setups.
>
> When the target receives the final Login Request it send the final Log= in Response and from its perspective the connection is now in Full Featured= Phase (assuming that it agreed to transition in the Login Response being s= ent).
>
> This means that the target is ready to accept SCSI commands, Text Requ= ests etc. sent by the initiator.
> It also means that the target is eligible to send some unsolicited PDU= s, notably unsolicited NOP-INs.
>
> With IB sending NOP-IN periodically is the easiest (an almost only fea= sible) way to determine closed connections reliably, because this kind of e= rror is delivered to user only in response to a previously initiated TX ope= ration.
>
> This leaves the initiator in a dubious position. It posts its RX buffe= rs for that connection only when the final Login Response arrives. But duri= ng that time (after the target had sent the Last Login Response but before = the Full Featured phase related RX-buffers are posted on the initiator side= ) the target may send the first NOP-IN as it considers the connection in Fu= ll Featured phase already and NumOfUnsolicited PDUs accounting for NOP-INs = has been negotiated to a non-zero value.
>
> If the initiator works with a single RX-buffer posted during the entir= e login phase (which is a logical thing to do judging by the login exchange= protocol) then an error occurs, as no buffers are posted when the NOP-IN a= rrives and the connection is shut down.
>
> Posting a single extra buffer before sending the last Login Request on= ly alleviates the problem. Although this often solves it in practical terms= (as the target most probably sends the next NOP-IN only after some timeout= period measuring seconds or hundreds of milliseconds), it does not solves = it in terms of protocol completeness, as the target MAY theoretically send = more than one NOP-IN until FF buffers are posted.
>
> This issue was encountered recently in linux iscsi/iser initiator and = the above solution has been applied to solve it against the existing target= implementation (STGT), but the initiator remains exposed to this kind of e= rrors.
>
> The solution is actually quite simple (theoretically) - if we bring ba= ck the requirement for iSER Hello exchange then the iSER assisted Full Feat= ured phase does not commence until HelloReply PDU arrives at the target and= the initiator has a definitive point in time when it can safely post its R= X buffers - after the final LoginResponse returns but before it sends iSER = Hello PDU.
>
> In practical terms it means that iSER Hello support requirement should= be brought back to spec, which is a hassle.
>
> Should we decide on this now?
>
> Alexander
>
> P.S. : Thanx to Mike Christie and Or Gerlitz, the maintainers of linux= iSCSI and iSER initiator for raising the issue.
>
> _______________________________________________
> storm mailing list
> storm@ietf.org<mailto:storm@ietf.org>
> https://www.ietf.org/mailman/listinfo/storm
>
>
>




--20cf3005dcb21bd6de04c23cd589-- From mkosjc@gmail.com Mon Jun 11 19:08:36 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7DA8C21F8564 for ; Mon, 11 Jun 2012 19:08:36 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.598 X-Spam-Level: X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Kzx7KYfSGErH for ; Mon, 11 Jun 2012 19:08:34 -0700 (PDT) Received: from mail-qc0-f172.google.com (mail-qc0-f172.google.com [209.85.216.172]) by ietfa.amsl.com (Postfix) with ESMTP id 20C7821F8565 for ; Mon, 11 Jun 2012 19:08:34 -0700 (PDT) Received: by qcsq13 with SMTP id q13so2813320qcs.31 for ; Mon, 11 Jun 2012 19:08:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=odK5u35zZahOEKvfDDkjp+i/qwF/Wuj8lE3ciIQLf0k=; b=zx4h7Ol9RNm/eAy69bUZQe+xjWRjucjOqAhZ2rloPTEHqESQEDVECy96OWNxpqGlPS j0xz0ukXI29uVYDSOla+Vj8J8GRnMOY7X4D+1v9J7+rtmOmhsob5ZwEFUJCp96CYiFfg IxdypK/3vbWB94m7gKU9aUqZPsyaS74w+2kAEdtsQ4n+oFs8GYTV0kxDE3/HvRMleWTM gpdySiivdXqNlEWMI+RWrCuo6uyb6y8WoUt3gaf8BgFaTDOb3uXuPOwGu7Q7ppJ7P+A8 gmacNPHMeVYwm64gwSqij9sb4hWmVcUghXaX02rS90qdFrGV9TWIQ5pAMoUI4NkDFQGU oIcA== MIME-Version: 1.0 Received: by 10.224.191.74 with SMTP id dl10mr17259025qab.65.1339466913443; Mon, 11 Jun 2012 19:08:33 -0700 (PDT) Received: by 10.229.130.233 with HTTP; Mon, 11 Jun 2012 19:08:33 -0700 (PDT) In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120707E8D5@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71205813C92@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120707E7F1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120707E8D5@MX15A.corp.emc.com> Date: Mon, 11 Jun 2012 19:08:33 -0700 Message-ID: From: Michael Ko To: david.black@emc.com Content-Type: multipart/alternative; boundary=20cf3005dcb2906e2404c23cf155 Cc: storm@ietf.org Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2012 02:08:36 -0000 --20cf3005dcb2906e2404c23cf155 Content-Type: text/plain; charset=ISO-8859-1 David, The current implemenation must be changed to correct this problem. There are two possible solutions. 1. Implement the exchange of iSERHelloRequired key and negotiate its use to Yes. This require both sides support iSERHelloRequired, leading to the need for clarifying texts (as you suggested) in the spec in order to specify what should be done otherwise. 2. Fix the non-compliance and adhere to the spec as currently defined. I agree with changing the spec to comply with the implementation when the current implementation works fine as is. But this is not the case here. Mike On Mon, Jun 11, 2012 at 6:44 PM, wrote: > > FWIW, I did not believe both sides were compliant in this case. > > There's an additional consideration here - one of the goals of the iSER > update is to match "running code", and we have already made a number of > changes that diverge from RFC 5046 for this reason, including in this > area of iSER start-up (see Appendix A of the iSER draft for details). > As a result, we are already well into non-compliance territory, as > omission of the Hello messages does not comply with RFC 5046, but those > messages are simply not implemented in practice. > > The implementation under discussion is (I believe) the predominant iSER > implementation and hence I think the new iSER RFC needs to explain what > that implementation has done (even if it may not be what should have been > done), and (more importantly) explain how to interoperate. The key > requirement is the following (IMHO, a fine example of "be conservative > in what you send"): > > > > 2) The target MUST wait at least 200ms (use some other number if 200ms > > > isn't a good choice) or until it receives a full feature mode PDU from > > > the initiator before sending a second unsolicited PDU in order to > ensure > > > that initiator has sufficient time to allocate the full feature buffer > > > resources for the connection. > > Not putting this into the iSER update requires either that Hello messages > be implemented (not done, primarily for optimization reasons), or that the > initiator effectively commit the buffer resources when starting > negotiation. > > Thanks, > --David > > > -----Original Message----- > > From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] > > Sent: Monday, June 11, 2012 2:57 PM > > To: Paul_Koning@Dell.com > > Cc: Black, David; mkosjc@gmail.com; nezhinsky@gmail.com; > > ogerlitz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org > > Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right after > final > > Login Response > > > > Paul, > > > > I completely agree. This promise is conceptually similar to the CmdSN > window > > promise. FWIW, I did not believe both sides were compliant in this case. > > > > Mallikarjun > > > > > > -----Original Message----- > > From: Paul_Koning@Dell.com [mailto:Paul_Koning@Dell.com] > > Sent: Monday, June 11, 2012 11:41 AM > > To: Mallikarjun Chadalapaka > > Cc: david.black@emc.com; mkosjc@gmail.com; nezhinsky@gmail.com; > > ogerlitz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org > > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after > final > > Login Response > > > > If the standard says that the expected semantics is that the buffers > promised > > are all there at end of negotiation, then yes, it's an implementation > issue. > > In that case, the original premise (both ends compliant) would not be > true -- > > one end is in violation. On the other hand, if the specification of the > > negotiation process doesn't say when the promised resources have to be > > available, that's a hole in the standard. > > > > paul > > > > On Jun 11, 2012, at 2:32 PM, Mallikarjun Chadalapaka wrote: > > > > > Sorry, if the implementation is promising x number of unsolicited > buffers > > but it is has consider the > > implementation to not comply with the standard anymore. Standard on its > part > > should clearly define the semantics of the promise - which I presume it > does > > in this case. > > > > > > I am all for providing helpful implementation guidance in the > standard, but > > we have to be cautious and re-confirm the need for the guidance, when we > start > > providing guidance down to latency numbers (see #2) - that's what got me > > concerned when I read the original note. > > > > > > Thanks. > > > > > > Mallikarjun > > > > > > > > > -----Original Message----- > > > From: Paul_Koning@Dell.com [mailto:Paul_Koning@Dell.com] > > > Sent: Monday, June 11, 2012 10:59 AM > > > To: david.black@emc.com > > > Cc: Mallikarjun Chadalapaka; mkosjc@gmail.com; nezhinsky@gmail.com; > > ogerlitz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org > > > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after > > final Login Response > > > > > > That sounds like the right way to look at it. The whole point of > protocol > > standards is to specify what is needed to interoperate. Not more, but > also > > not less. Whenever two implementations both comply with a standard yet > do not > > interoperate, that's a bug in the standard. > > > > > > paul > > > > > > On Jun 11, 2012, at 11:34 AM, > > > > > > > wrote: > > > > > > Hi Mallikarjun, > > > > > > IMHO, the reason that this is a protocol issue is that two > implementations > > that comply with the RFC can nonetheless reliably and repeatedly fail to > > interoperate because the lack of additional buffers causes the RCaP > connection > > to close before full feature phase is reached by the initiator. > > > > > > I believe that we have a responsibility to tell implementers what can > go > > wrong here and how to avoid it - the technique you describe (post all > > unsolicited buffers before sending final negotiation message) could be > > mentioned as part of this, and we should also describe what's possible > with > > use of a single buffer, as that approach is being pursued as Alexander > > describes. > > > > > > Thanks, > > > --David > > > > > > From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] > > > Sent: Friday, May 25, 2012 9:02 PM > > > To: Black, David; mkosjc@gmail.com; > > nezhinsky@gmail.com > > > Cc: ogerlitz@mellanox.com; > > michaelc@cs.wisc.edu; > > storm@ietf.org > > > Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right after > > final Login Response > > > > > > I am curious to understand a bit more on why this is a protocol issue > per > > se. > > > > > > Seems like one way to address this problem is via an implementation > approach > > with the initiator posting in advance the negotiated number of > unsolicited PDU > > buffers, at the same time it makes the (final) negotiation offer. As the > in- > > bound unsolicited PDUs can technically arrive any time after the offer, > due to > > standard network latency mechanics that Alexander summarized. Has that > > approach been considered? > > > > > > Mallikarjun > > > > > > > > > > > > > > > From: storm-bounces@ietf.org [mailto: > storm- > > bounces@ietf.org] On Behalf Of david.black@emc.com david.black@emc.com> > > > Sent: Thursday, May 24, 2012 12:03 AM > > > To: mkosjc@gmail.com; > > nezhinsky@gmail.com > > > Cc: ogerlitz@mellanox.com; > > michaelc@cs.wisc.edu; > > storm@ietf.org > > > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after > > final Login Response > > > Importance: High > > > > > > Mike (Ko) and Alexander, > > > > > > Mike is of course correct that iSER Hello usage can be forced by > negotiating > > iSERHelloRequired to "Yes". However, existing implementations are > likely to > > reply with iSERHelloRequired=NotUnderstood, so we do need to specify what > > should be done in order to interoperate with an implementation that > refuses to > > deal with the iSER Hello exchange. > > > > > > I think the situation that Alexander described should be documented in > a new > > section 5.1.4 of the iSER draft. My general rule of thumb on this sort > of > > surprise found by implementers in the "running code" is that it > indicates that > > something is missing in the spec. I believe that Alexander has > described the > > solution - more below. > > > > > > The new section 5.1.4 (suggested section title: Omission of the iSER > Hello > > Exchange) should describe default omission of the exchange, use of > > iSERHelloRequired key to omit the iSER Hello exchange, and the > consequences of > > target use of unsolicited PDUs after login when the exchange is omitted, > > including IB's use of NOP-IN (as a keep-alive measure, right?) > > > > > > The crucial requirements points that I take away from Alexander's > > description are that if the iSER Hello exchange is omitted, then: > > > > > > 1) The target MAY send *one* unsolicited PDU immediately after sending > the > > Login Response. > > > > > > 2) The target MUST wait at least 200ms (use some other number if 200ms > isn't > > a good choice) or until it receives a full feature mode PDU from the > initiator > > before sending a second unsolicited PDU in order to ensure that > initiator has > > sufficient > > > time to allocate the full feature buffer resources for the > connection. > > > 3) The initiator SHOULD allocate at least one additional buffer for use > > during login (so that at least two buffers are in use during login) in > order > > to receive an unsolicited PDU that may follow login completion. Failure > to > > allocate this second buffer may cause connection termination if no > buffer is > > available when an unsolicited PDU arrives. > > > > > > Both Mike and I are on vacation, so it may be a few weeks until we can > agree > > on the new text and get a -12 version of the draft with that new text > > submitted. In the interim, I've asked our AD (Martin Stiemerling) to > hold off > > on further processing of the iSER draft until a -12 version with this > new text > > is submitted. I'd prefer to work this text out now rather than deal > with it > > as an IETF Last Call comment - as the problem turned up in actual > > implementations, I think it's worth the extra month that it's likely to > take > > to get correct text on how to avoid the problem into the draft. > > > > > > I'd suggest that Mike Ko post an initial draft of the text for the new > > section 5.1.4 to the list when he resurfaces ... > > > > > > Thanks, > > > --David > > > > > > From: Michael Ko [mailto:mkosjc@gmail.com] mkosjc@gmail.com]> > > > Sent: Monday, May 21, 2012 10:23 AM > > > To: Alexander Nezhinsky > > > Cc: storm@ietf.org; Black, David; Or Gerlitz; > Mike > > Christie > > > Subject: Re: iSER - problem with unsolicited NOP-IN right after final > Login > > Response > > > > > > Alex, > > > > > > The iSER Hello support has never been removed in the latest spec. > Only its > > use is made optional. So during login negotiation, just negotiate > > iSERHelloRequired to Yes. > > > > > > Mike > > > On Mon, May 21, 2012 at 6:15 AM, Alexander Nezhinsky > > > wrote: > > > Hi > > > > > > I understand that it is a bad timing for sending this kind of mail, > now that > > iSER draft was submitted, but actually we still have a small problem. > > > It is related to the final Login Response handling and the transition > to > > Full-Featured phase on the initiator side in Infiniband setups. > > > > > > When the target receives the final Login Request it send the final > Login > > Response and from its perspective the connection is now in Full Featured > Phase > > (assuming that it agreed to transition in the Login Response being sent). > > > > > > This means that the target is ready to accept SCSI commands, Text > Requests > > etc. sent by the initiator. > > > It also means that the target is eligible to send some unsolicited > PDUs, > > notably unsolicited NOP-INs. > > > > > > With IB sending NOP-IN periodically is the easiest (an almost only > feasible) > > way to determine closed connections reliably, because this kind of error > is > > delivered to user only in response to a previously initiated TX > operation. > > > > > > This leaves the initiator in a dubious position. It posts its RX > buffers for > > that connection only when the final Login Response arrives. But during > that > > time (after the target had sent the Last Login Response but before the > Full > > Featured phase related RX-buffers are posted on the initiator side) the > target > > may send the first NOP-IN as it considers the connection in Full Featured > > phase already and NumOfUnsolicited PDUs accounting for NOP-INs has been > > negotiated to a non-zero value. > > > > > > If the initiator works with a single RX-buffer posted during the entire > > login phase (which is a logical thing to do judging by the login exchange > > protocol) then an error occurs, as no buffers are posted when the NOP-IN > > arrives and the connection is shut down. > > > > > > Posting a single extra buffer before sending the last Login Request > only > > alleviates the problem. Although this often solves it in practical terms > (as > > the target most probably sends the next NOP-IN only after some timeout > period > > measuring seconds or hundreds of milliseconds), it does not solves it in > terms > > of protocol completeness, as the target MAY theoretically send more than > one > > NOP-IN until FF buffers are posted. > > > > > > This issue was encountered recently in linux iscsi/iser initiator and > the > > above solution has been applied to solve it against the existing target > > implementation (STGT), but the initiator remains exposed to this kind of > > errors. > > > > > > The solution is actually quite simple (theoretically) - if we bring > back the > > requirement for iSER Hello exchange then the iSER assisted Full Featured > phase > > does not commence until HelloReply PDU arrives at the target and the > initiator > > has a definitive point in time when it can safely post its RX buffers - > after > > the final LoginResponse returns but before it sends iSER Hello PDU. > > > > > > In practical terms it means that iSER Hello support requirement should > be > > brought back to spec, which is a hassle. > > > > > > Should we decide on this now? > > > > > > Alexander > > > > > > P.S. : Thanx to Mike Christie and Or Gerlitz, the maintainers of linux > iSCSI > > and iSER initiator for raising the issue. > > > > > > _______________________________________________ > > > storm mailing list > > > storm@ietf.org > > > https://www.ietf.org/mailman/listinfo/storm > > > > > > > > > > > > > > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > --20cf3005dcb2906e2404c23cf155 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable David,

The current implemenation must be changed to correct this pro= blem.=A0 There are two possible solutions.

1. Implement the exchange= of iSERHelloRequired key and negotiate its use to Yes.=A0 This require bot= h sides support iSERHelloRequired, leading to the need for clarifying texts= (as you suggested) in the spec in order to specify what should be done oth= erwise.

2. Fix the non-compliance and adhere to the spec as currently defined.<= br>
I agree with changing the spec to comply with the implementation whe= n the current implementation works fine as is.=A0 But this is not the case = here.

Mike

On Mon, Jun 11, 2012 at 6:44 PM,= <david.black@emc.com> wrote:
> FWIW, I did not believe both sides were compliant in this case.

There's an additional consideration here - one of the goals of the iSER=
update is to match "running code", and we have already made a num= ber of
changes that diverge from RFC 5046 for this reason, including in this
area of iSER start-up (see Appendix A of the iSER draft for details).
As a result, we are =A0already well into non-compliance territory, as
omission of the Hello messages does not comply with RFC 5046, but those
messages are simply not implemented in practice.

The implementation under discussion is (I believe) the predominant iSER
implementation and hence I think the new iSER RFC needs to explain what
that implementation has done (even if it may not be what should have been done), and (more importantly) explain how to interoperate. =A0The key
requirement is the following (IMHO, a fine example of "be conservative=
in what you send"):

> > 2) The target MUST wait at least 200ms (use some other number if = 200ms
> > isn't a good choice) or until it receives a full feature mode= PDU from
> > the initiator before sending a second unsolicited PDU in order to= ensure
> > that initiator has sufficient time to allocate the full feature b= uffer
> > resources for the connection.

Not putting this into the iSER update requires either that Hello messages be implemented (not done, primarily for optimization reasons), or that the<= br> initiator effectively commit the buffer resources when starting negotiation= .

Thanks,
--David

> -----Original Message-----
> From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
> Sent: Monday, June 11, 2012 2:57 PM
> To: Paul_Koning@Dell.com
> Cc: Black, David; mkosjc@gmail.com= ; nezhinsky@gmail.com;
> ogerlitz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org
> Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right afte= r final
> Login Response
>
> Paul,
>
> I completely agree. This promise is conceptually similar to the CmdSN = window
> promise. FWIW, I did not believe both sides were compliant in this cas= e.
>
> Mallikarjun
>
>
> -----Original Message-----
> From: Paul_Koning@Dell.com [mailto:Paul_Koning@Dell.com]
> Sent: Monday, June 11, 2012 11:41 AM
> To: Mallikarjun Chadalapaka
> Cc: david.black@emc.com; mkosjc@gmail.com; nezhinsky@gmail.com;
> ogerlitz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org
> Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right afte= r final
> Login Response
>
> If the standard says that the expected semantics is that the buffers p= romised
> are all there at end of negotiation, then yes, it's an implementat= ion issue.
> In that case, the original premise (both ends compliant) would not be = true --
> one end is in violation. =A0On the other hand, if the specification of= the
> negotiation process doesn't say when the promised resources have t= o be
> available, that's a hole in the standard.
>
> =A0 =A0 =A0 paul
>
> On Jun 11, 2012, at 2:32 PM, Mallikarjun Chadalapaka wrote:
>
> > Sorry, if the implementation is promising x number of unsolicited= buffers
> but it is has <x buffers ready at the end of negotiation, I would c= onsider the
> implementation to not comply with the standard anymore. Standard on it= s part
> should clearly define the semantics of the promise - which I presume i= t does
> in this case.
> >
> > I am all for providing helpful implementation guidance in the sta= ndard, but
> we have to be cautious and re-confirm the need for the guidance, when = we start
> providing guidance down to latency numbers (see #2) - that's what = got me
> concerned when I read the original note.
> >
> > Thanks.
> >
> > Mallikarjun
> >
> >
> > -----Original Message-----
> > From: Paul_Koning@Dell.com [mailto:Paul_Koning@Dell.com]
> > Sent: Monday, June 11, 2012 10:59 AM
> > To: david.black@emc.com
> > Cc: Mallikarjun Chadalapaka; = mkosjc@gmail.com; nezhinsky@gmai= l.com;
> ogerlitz@mellanox.com; michaelc@cs.wisc.edu; storm@ietf.org
> > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right= after
> final Login Response
> >
> > That sounds like the right way to look at it. =A0The whole point = of protocol
> standards is to specify what is needed to interoperate. =A0Not more, b= ut also
> not less. =A0Whenever two implementations both comply with a standard = yet do not
> interoperate, that's a bug in the standard.
> >
> > paul
> >
> > On Jun 11, 2012, at 11:34 AM,
> <david.black@emc.com<= mailto:david.black@emc.com>&g= t;
> > <david.black@emc.com<mailto:david.black@emc.com&= gt;> wrote:
> >
> > Hi Mallikarjun,
> >
> > IMHO, the reason that this is a protocol issue is that two implem= entations
> that comply with the RFC can nonetheless reliably and repeatedly fail = to
> interoperate because the lack of additional buffers causes the RCaP co= nnection
> to close before full feature phase is reached by the initiator.
> >
> > I believe that we have a responsibility to tell implementers what= can go
> wrong here and how to avoid it - the technique you describe (post all<= br> > unsolicited buffers before sending final negotiation message) could be=
> mentioned as part of this, and we should also describe what's poss= ible with
> use of a single buffer, as that approach is being pursued as Alexander=
> describes.
> >
> > Thanks,
> > --David
> >
> > From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
> > Sent: Friday, May 25, 2012 9:02 PM
> > To: Black, David; mkosjc@gmai= l.com<mailto:mkosjc@gmail.com>;
> nezhinsky@gmail.com<mail= to:nezhinsky@gmail.com>
> > Cc: ogerlitz@mellanox.co= m<mailto:ogerlitz@mellanox.= com>;
> michaelc@cs.wisc.edu<ma= ilto:michaelc@cs.wisc.edu>;<= br> > storm@ietf.org<mailto:storm@ietf.org>
> > Subject: RE: [storm] iSER - problem with unsolicited NOP-IN right= after
> final Login Response
> >
> > I am curious to understand a bit more on why this is a protocol i= ssue per
> se.
> >
> > Seems like one way to address this problem is via an implementati= on approach
> with the initiator posting in advance the negotiated number of unsolic= ited PDU
> buffers, at the same time it makes the (final) negotiation offer. As t= he in-
> bound unsolicited PDUs can technically arrive any time after the offer= , due to
> standard network latency mechanics that Alexander summarized. Has that=
> approach been considered?
> >
> > Mallikarjun
> >
> >
> >
> >
> > From: storm-bounces@iet= f.org<mailto:storm-bounces= @ietf.org> [mailto:storm-
> bounces@ietf.org] On Behalf Of= david.black@emc.com<mailto:<= a href=3D"mailto:david.black@emc.com">david.black@emc.com>
> > Sent: Thursday, May 24, 2012 12:03 AM
> > To: mkosjc@gmail.com<m= ailto:mkosjc@gmail.com>;
> nezhinsky@gmail.com<mail= to:nezhinsky@gmail.com>
> > Cc: ogerlitz@mellanox.co= m<mailto:ogerlitz@mellanox.= com>;
> michaelc@cs.wisc.edu<ma= ilto:michaelc@cs.wisc.edu>;<= br> > storm@ietf.org<mailto:storm@ietf.org>
> > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right= after
> final Login Response
> > Importance: High
> >
> > Mike (Ko) and Alexander,
> >
> > Mike is of course correct that iSER Hello usage can be forced by = negotiating
> iSERHelloRequired to "Yes". =A0However, existing implementat= ions are likely to
> reply with iSERHelloRequired=3DNotUnderstood, so we do need to specify= what
> should be done in order to interoperate with an implementation that re= fuses to
> deal with the iSER Hello exchange.
> >
> > I think the situation that Alexander described should be document= ed in a new
> section 5.1.4 of the iSER draft. =A0My general rule of thumb on this s= ort of
> surprise found by implementers in the "running code" is that= it indicates that
> something is missing in the spec. =A0I believe that Alexander has desc= ribed the
> solution - more below.
> >
> > The new section 5.1.4 (suggested section title: Omission of the i= SER Hello
> Exchange) should describe default omission of the exchange, use of
> iSERHelloRequired key to omit the iSER Hello exchange, and the consequ= ences of
> target use of unsolicited PDUs after login when the exchange is omitte= d,
> including IB's use of NOP-IN (as a keep-alive measure, right?)
> >
> > The crucial requirements points that I take away from Alexander&#= 39;s
> description are that if the iSER Hello exchange is omitted, then:
> >
> > 1) The target MAY send *one* unsolicited PDU immediately after se= nding the
> Login Response.
> >
> > 2) The target MUST wait at least 200ms (use some other number if = 200ms isn't
> a good choice) or until it receives a full feature mode PDU from the i= nitiator
> before sending a second unsolicited PDU in order to ensure that initia= tor has
> sufficient
> > =A0 =A0 =A0time to allocate the full feature buffer resources for= the connection.
> > 3) The initiator SHOULD allocate at least one additional buffer f= or use
> during login (so that at least two buffers are in use during login) in= order
> to receive an unsolicited PDU that may follow login completion. =A0Fai= lure to
> allocate this second buffer may cause connection termination if no buf= fer is
> available when an unsolicited PDU arrives.
> >
> > Both Mike and I are on vacation, so it may be a few weeks until w= e can agree
> on the new text and get a -12 version of the draft with that new text<= br> > submitted. =A0In the interim, I've asked our AD (Martin Stiemerlin= g) to hold off
> on further processing of the iSER draft until a -12 version with this = new text
> is submitted. =A0I'd prefer to work this text out now rather than = deal with it
> as an IETF Last Call comment - as the problem turned up in actual
> implementations, I think it's worth the extra month that it's = likely to take
> to get correct text on how to avoid the problem into the draft.
> >
> > I'd suggest that Mike Ko post an initial draft of the text fo= r the new
> section 5.1.4 to the list when he resurfaces ...
> >
> > Thanks,
> > --David
> >
> > From: Michael Ko [mailto:mkos= jc@gmail.com]<mailto:[mailto:mko= sjc@gmail.com]>
> > Sent: Monday, May 21, 2012 10:23 AM
> > To: Alexander Nezhinsky
> > Cc: storm@ietf.org<mailt= o:storm@ietf.org>; Black, David; O= r Gerlitz; Mike
> Christie
> > Subject: Re: iSER - problem with unsolicited NOP-IN right after f= inal Login
> Response
> >
> > Alex,
> >
> > The iSER Hello support has never been removed in the latest spec.= =A0Only its
> use is made optional. =A0So during login negotiation, just negotiate > iSERHelloRequired to Yes.
> >
> > Mike
> > On Mon, May 21, 2012 at 6:15 AM, Alexander Nezhinsky
> <nezhinsky@gmail.com<= mailto:nezhinsky@gmail.com>&g= t; wrote:
> > Hi
> >
> > I understand that it is a bad timing for sending this kind of mai= l, now that
> iSER draft was submitted, but actually we still have a small problem.<= br> > > It is related to the final Login Response handling and the transi= tion to
> Full-Featured phase on the initiator side in Infiniband setups.
> >
> > When the target receives the final Login Request it send the fina= l Login
> Response and from its perspective the connection is now in Full Featur= ed Phase
> (assuming that it agreed to transition in the Login Response being sen= t).
> >
> > This means that the target is ready to accept SCSI commands, Text= Requests
> etc. sent by the initiator.
> > It also means that the target is eligible to send some unsolicite= d PDUs,
> notably unsolicited NOP-INs.
> >
> > With IB sending NOP-IN periodically is the easiest (an almost onl= y feasible)
> way to determine closed connections reliably, because this kind of err= or is
> delivered to user only in response to a previously initiated TX operat= ion.
> >
> > This leaves the initiator in a dubious position. It posts its RX = buffers for
> that connection only when the final Login Response arrives. But during= that
> time (after the target had sent the Last Login Response but before the= Full
> Featured phase related RX-buffers are posted on the initiator side) th= e target
> may send the first NOP-IN as it considers the connection in Full Featu= red
> phase already and NumOfUnsolicited PDUs accounting for NOP-INs has bee= n
> negotiated to a non-zero value.
> >
> > If the initiator works with a single RX-buffer posted during the = entire
> login phase (which is a logical thing to do judging by the login excha= nge
> protocol) then an error occurs, as no buffers are posted when the NOP-= IN
> arrives and the connection is shut down.
> >
> > Posting a single extra buffer before sending the last Login Reque= st only
> alleviates the problem. Although this often solves it in practical ter= ms (as
> the target most probably sends the next NOP-IN only after some timeout= period
> measuring seconds or hundreds of milliseconds), it does not solves it = in terms
> of protocol completeness, as the target MAY theoretically send more th= an one
> NOP-IN until FF buffers are posted.
> >
> > This issue was encountered recently in linux iscsi/iser initiator= and the
> above solution has been applied to solve it against the existing targe= t
> implementation (STGT), but the initiator remains exposed to this kind = of
> errors.
> >
> > The solution is actually quite simple (theoretically) - if we bri= ng back the
> requirement for iSER Hello exchange then the iSER assisted Full Featur= ed phase
> does not commence until HelloReply PDU arrives at the target and the i= nitiator
> has a definitive point in time when it can safely post its RX buffers = - after
> the final LoginResponse returns but before it sends iSER Hello PDU. > >
> > In practical terms it means that iSER Hello support requirement s= hould be
> brought back to spec, which is a hassle.
> >
> > Should we decide on this now?
> >
> > Alexander
> >
> > P.S. : Thanx to Mike Christie and Or Gerlitz, the maintainers of = linux iSCSI
> and iSER initiator for raising the issue.
> >
> > _______________________________________________
> > storm mailing list
> > storm@ietf.org<mailto:storm@ietf.org>
> > https://www.ietf.org/mailman/listinfo/storm
> >
> >
> >
>
>

_______________________________________________
storm mailing list
storm@ietf.org
h= ttps://www.ietf.org/mailman/listinfo/storm

--20cf3005dcb2906e2404c23cf155-- From Paul_Koning@Dell.com Tue Jun 12 06:57:02 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CAAD821F85AC for ; Tue, 12 Jun 2012 06:57:02 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -109.599 X-Spam-Level: X-Spam-Status: No, score=-109.599 tagged_above=-999 required=5 tests=[AWL=1.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oOSMqOtE9qoo for ; Tue, 12 Jun 2012 06:57:02 -0700 (PDT) Received: from ausxipps301.us.dell.com (ausxipps301.us.dell.com [143.166.148.223]) by ietfa.amsl.com (Postfix) with ESMTP id 06B3521F859A for ; Tue, 12 Jun 2012 06:56:58 -0700 (PDT) X-Loopcount0: from 10.175.216.250 From: To: Thread-Topic: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response Thread-Index: AQHNOXtOIIaCinNGykWqG66+HjtgdZbbOw+ggAecuICAEwDkAP//sNUggABa4AD//62tIIAAcLpAgAEkiAA= Date: Tue, 12 Jun 2012 13:56:56 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE71205813C92@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120707E7F1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120707E8D5@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120707E8D5@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.175.217.62] Content-Type: text/plain; charset="us-ascii" Content-ID: <5F7D721A8DEB4C40818C3CA52E1323A4@dell.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: storm@ietf.org Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2012 13:57:03 -0000 On Jun 11, 2012, at 9:44 PM, wrote: >> FWIW, I did not believe both sides were compliant in this case. >=20 > There's an additional consideration here - one of the goals of the iSER > update is to match "running code", and we have already made a number of > changes that diverge from RFC 5046 for this reason, including in this > area of iSER start-up (see Appendix A of the iSER draft for details). > As a result, we are already well into non-compliance territory, as > omission of the Hello messages does not comply with RFC 5046, but those > messages are simply not implemented in practice. >=20 > The implementation under discussion is (I believe) the predominant iSER > implementation and hence I think the new iSER RFC needs to explain what > that implementation has done (even if it may not be what should have been > done), and (more importantly) explain how to interoperate. The key > requirement is the following (IMHO, a fine example of "be conservative > in what you send"): >=20 >>> 2) The target MUST wait at least 200ms (use some other number if 200ms >>> isn't a good choice) or until it receives a full feature mode PDU from >>> the initiator before sending a second unsolicited PDU in order to ensur= e >>> that initiator has sufficient time to allocate the full feature buffer >>> resources for the connection. I don't understand how a protocol standard can have such language. "200 ms= (use some other number if...") is not a testable property. If you delete = the "(use...)" then you have standard, or if you delete "at least... or" b= ut as written, an implementer has no clue what is expected. paul From david.black@emc.com Tue Jun 12 11:01:46 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2B33021F85A4 for ; Tue, 12 Jun 2012 11:01:46 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 81YVeaiVIVBb for ; Tue, 12 Jun 2012 11:01:45 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 8BE8021F8593 for ; Tue, 12 Jun 2012 11:01:45 -0700 (PDT) Received: from hop04-l1d11-si01.isus.emc.com (HOP04-L1D11-SI01.isus.emc.com [10.254.111.54]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5CI1cMc013531 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 12 Jun 2012 14:01:39 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.222.130]) by hop04-l1d11-si01.isus.emc.com (RSA Interceptor); Tue, 12 Jun 2012 14:01:13 -0400 Received: from mxhub33.corp.emc.com (mxhub33.corp.emc.com [10.254.93.81]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5CI1Cie024045; Tue, 12 Jun 2012 14:01:12 -0400 Received: from mx15a.corp.emc.com ([169.254.1.101]) by mxhub33.corp.emc.com ([::1]) with mapi; Tue, 12 Jun 2012 14:01:12 -0400 From: To: Date: Tue, 12 Jun 2012 14:01:11 -0400 Thread-Topic: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response Thread-Index: AQHNOXtOIIaCinNGykWqG66+HjtgdZbbOw+ggAecuICAEwDkAP//sNUggABa4AD//62tIIAAcLpAgAEkiAD///AyEA== Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120707EA1A@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71205813C92@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120707E7F1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120707E8D5@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Cc: storm@ietf.org Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2012 18:01:46 -0000 Hi Paul, We're in violent agreement, and I wasn't clear in what I wrote. I'm proposing 200ms unless someone on the list has a better idea ...=20 Thanks, --David > -----Original Message----- > From: Paul_Koning@Dell.com [mailto:Paul_Koning@Dell.com] > Sent: Tuesday, June 12, 2012 9:57 AM > To: Black, David > Cc: cbm@chadalapaka.com; storm@ietf.org > Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after f= inal > Login Response >=20 >=20 > On Jun 11, 2012, at 9:44 PM, > wrote: >=20 > >> FWIW, I did not believe both sides were compliant in this case. > > > > There's an additional consideration here - one of the goals of the iSER > > update is to match "running code", and we have already made a number of > > changes that diverge from RFC 5046 for this reason, including in this > > area of iSER start-up (see Appendix A of the iSER draft for details). > > As a result, we are already well into non-compliance territory, as > > omission of the Hello messages does not comply with RFC 5046, but those > > messages are simply not implemented in practice. > > > > The implementation under discussion is (I believe) the predominant iSER > > implementation and hence I think the new iSER RFC needs to explain what > > that implementation has done (even if it may not be what should have be= en > > done), and (more importantly) explain how to interoperate. The key > > requirement is the following (IMHO, a fine example of "be conservative > > in what you send"): > > > >>> 2) The target MUST wait at least 200ms (use some other number if 200m= s > >>> isn't a good choice) or until it receives a full feature mode PDU fro= m > >>> the initiator before sending a second unsolicited PDU in order to ens= ure > >>> that initiator has sufficient time to allocate the full feature buffe= r > >>> resources for the connection. >=20 > I don't understand how a protocol standard can have such language. "200 = ms > (use some other number if...") is not a testable property. If you delete= the > "(use...)" then you have standard, or if you delete "at least... or" but= as > written, an implementer has no clue what is expected. >=20 > paul >=20 From david.black@emc.com Tue Jun 26 06:17:02 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 802EC21F8615 for ; Tue, 26 Jun 2012 06:17:02 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z39Pwiupdqqd for ; Tue, 26 Jun 2012 06:17:02 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id CFB1E21F842D for ; Tue, 26 Jun 2012 06:17:01 -0700 (PDT) Received: from hop04-l1d11-si02.isus.emc.com (HOP04-L1D11-SI02.isus.emc.com [10.254.111.55]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5QDH0nX009676 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 26 Jun 2012 09:17:00 -0400 Received: from mailhub.lss.emc.com (mailhubhoprd02.lss.emc.com [10.254.221.253]) by hop04-l1d11-si02.isus.emc.com (RSA Interceptor) for ; Tue, 26 Jun 2012 09:16:41 -0400 Received: from mxhub10.corp.emc.com (mxhub10.corp.emc.com [10.254.92.105]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5QDGesN019039 for ; Tue, 26 Jun 2012 09:16:40 -0400 Received: from mx15a.corp.emc.com ([169.254.1.189]) by mxhub10.corp.emc.com ([10.254.92.105]) with mapi; Tue, 26 Jun 2012 09:16:40 -0400 From: To: Importance: high X-Priority: 1 Date: Tue, 26 Jun 2012 09:16:38 -0400 Thread-Topic: iSER - what to do Thread-Index: Ac1TnetKxyzW/d6WTNGw6HqSI6vOSQ== Message-ID: <8D3D17ACE214DC429325B2B98F3AE71208C14966@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: [storm] iSER - what to do X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jun 2012 13:17:02 -0000 The discussion about whether the buffer behavior of the Linux iSER implementation complies with RFC 5046 is missing at least one important point - to my knowledge, there are *no* iSER implementations that comply with RFC 5046. After RFC 5046 was issued, implementers looked at its requirements and decided to do a number of things differently - the important difference for this discussion is that the iSER Hello exchange is not implemented, even though RFC 5046 requires that it be used. There are a number of other "running code" changes in the current iSER draft that reflect implementations and that are incompatible with RFC 5046, see Appendix A of the draft for details. The iSER work item was added to the storm WG's charter in order to produce an RFC specification of iSER that matches the "running code" and hence will interoperate with existing implementations. Here's the text from the WG charter: (6) iSER: Experience with Infiniband implementations suggest a few minor updates to reflect what has been done in practice. As WG chair, I'm going to insist on the following: 1) The iSER draft has to match the implementations in order to be published as an RFC. Replacing RFC 5046 with another RFC that won't interoperate with the implementations is pointless. 2) The issue encountered by the Linux iSER implementation cannot be ignored by the WG. The workaround (1 extra buffer, at most one unsolicited PDU until initiator has time to deploy resources) at least needs to be explained in the draft. 3) While requiring the iSER Hello exchange may be the proverbial "right thing to do" going forward, it is necessary to provide at least guidance on how to interoperate with current iSER implementations that don't support the Hello exchange. In contrast, as WG chair, I'm not going to insist that we adopt the workaround as part of the standard. For example, it's possible to recommend (SHOULD) use of the iSER Hello exchange (explicitly negotiated using the new key) and explain the workaround as what should be done in order to interoperate if that key results in a NotUnderstood response. For this approach to the workaround, I think a lower case "should" may be acceptable (e.g., as opposed to a "SHOULD" or "MUST", but that's subject to further discussion. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- From david.black@emc.com Tue Jun 26 10:40:16 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BA89511E8072 for ; Tue, 26 Jun 2012 10:40:16 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.572 X-Spam-Level: X-Spam-Status: No, score=-102.572 tagged_above=-999 required=5 tests=[AWL=0.027, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M-f1cGf2VCug for ; Tue, 26 Jun 2012 10:40:16 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id DF3BA21F85FD for ; Tue, 26 Jun 2012 10:40:15 -0700 (PDT) Received: from hop04-l1d11-si03.isus.emc.com (HOP04-L1D11-SI03.isus.emc.com [10.254.111.23]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5QHeFnu026877 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 26 Jun 2012 13:40:15 -0400 Received: from mailhub.lss.emc.com (mailhubhoprd03.lss.emc.com [10.254.221.145]) by hop04-l1d11-si03.isus.emc.com (RSA Interceptor) for ; Tue, 26 Jun 2012 13:40:02 -0400 Received: from mxhub34.corp.emc.com (mxhub34.corp.emc.com [10.254.93.82]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5QHe1GV008699 for ; Tue, 26 Jun 2012 13:40:02 -0400 Received: from mx15a.corp.emc.com ([169.254.1.189]) by mxhub34.corp.emc.com ([::1]) with mapi; Tue, 26 Jun 2012 13:40:01 -0400 From: To: Importance: high X-Priority: 1 Date: Tue, 26 Jun 2012 13:40:01 -0400 Thread-Topic: Draft status and possible Vancouver meeting cancellation Thread-Index: Ac1TwrZo2CnyF3rWQxqnmyA1qRlKgw== Message-ID: <8D3D17ACE214DC429325B2B98F3AE71208C14A49@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: [storm] Draft status and possible Vancouver meeting cancellation X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jun 2012 17:40:16 -0000 Here's what one of your WG chairs believes the status of the WG's drafts are. Based on this, your WG chairs are seriously considering cancelling the Vancouver meeting of the storm WG, as there doesn't seem to be anything here that has a compelling need for scarce IETF meeting time. If anyone disagrees and thinks we should meet in Vancouver, please speak up quickly with an explanation of why. -- iSCSI Area Director review comments have been received on both the consolidated and new features (-iscsi-sam) drafts. New versions of both drafts will be needed before IETF Last Call, but the revisions to the new features draft are rather minor. The revisions to the consolidated draft will be more extensive, as our new AD (Martin) did a complete text review, and found a bunch of things - most of them are areas where tighter/clearer language is needed to capture existing requirements (that implementations already meet. Expect an email to the list soon explaining the proposed changes to normative text (as opposed to editorial cleanups). One notable exception is that the recent publication of RFC 6648 requires that we take a hard look at the X- and X# extension text keys, and hence also the related Z- and Z# authentication extension text keys. There has been some behind the scenes discussion that will result in an email of proposed changes to the list - these changes *will not break* any existing implementations, and please wait for that separate email before engaging in that discussion on the list. All of these topics appear amenable to email resolution. -- iSCSI MIB One of the authors recently changed jobs, which has delayed things. The MIB Doctor Review has been received, and unlike the MIB Doctor Reviews that I'm used to, this one proposed a number of MIB functionality enhancements, all of which appear to be relatively minor. The next step is for the authors to summarize those enhancements in an email to the list, soon please (hint). =20 -- iSER This is our "problem child" draft as it was supposed to be out of the WG on its way to RFC Publication 6 months ago, and has been caught by a couple of last minute issues. I sent a separate email earlier today on what needs to be done to close the current issue that is keeping the iSER draft in the WG - please read and respond to that, but I believe that can be worked out via email. -- RDMA Extensions This draft is supposed to go to Working Group Last Call this month (June). That's been delayed by the iSER draft bouncing back into the WG after my first attempt to submit it for RFC publication. Nonetheless, your WG chairs (Tom and myself) expect (well at least hope) to post our reviews and comments on this draft to the list later this week. I don't think the possible issues with this draft justify a Vancouver meeti= ng.=20 Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- From mkosjc@gmail.com Tue Jun 26 11:45:21 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1BDDA11E80C5 for ; Tue, 26 Jun 2012 11:45:21 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.598 X-Spam-Level: X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Jysh4BE2QsfB for ; Tue, 26 Jun 2012 11:45:19 -0700 (PDT) Received: from mail-qc0-f170.google.com (mail-qc0-f170.google.com [209.85.216.170]) by ietfa.amsl.com (Postfix) with ESMTP id A1A5611E809F for ; Tue, 26 Jun 2012 11:45:13 -0700 (PDT) Received: by qcmt36 with SMTP id t36so133743qcm.15 for ; Tue, 26 Jun 2012 11:45:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=hfzSPI2e2om3xmF0Z/68ah7xiv4++5dnsENG6NAZvwg=; b=zmoiQcAB93UlW6qhfdDMJfiEbSnrwfjcJ6x76b2t59cckhTU9A33v1sTZObu4cNLva haH3hfZMVpQ3rhvs2RgguaN+6FfQoXeDUcQ7pkpVgiGfToiZHhN81uAp/pQfRl6fieuc XGXFR8MisrwTuy2qJy8zyFPwJ6hXcnVlaAoA2lSf1nEnMgnNio2SpwCTE1f7yuwuY3qz x17hH73f/KrPZmfqFllGyxqv5M6hN9CfT8N8X3ZSt5ypOTJqtypl+j4/o0xV7oed3QHQ COJ6F/qv74dT2o0JjI1JY29fjAOwtV9zAHWnlXHZd9at8PL2n7sZo5W8+KeTMELUdGBC yBVg== MIME-Version: 1.0 Received: by 10.229.135.147 with SMTP id n19mr7488332qct.11.1340736312898; Tue, 26 Jun 2012 11:45:12 -0700 (PDT) Received: by 10.229.130.233 with HTTP; Tue, 26 Jun 2012 11:45:12 -0700 (PDT) In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE71208C14966@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71208C14966@MX15A.corp.emc.com> Date: Tue, 26 Jun 2012 11:45:12 -0700 Message-ID: From: Michael Ko To: david.black@emc.com Content-Type: multipart/alternative; boundary=00248c769122aaf56d04c3647f0e Cc: storm@ietf.org Subject: Re: [storm] iSER - what to do X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jun 2012 18:45:21 -0000 --00248c769122aaf56d04c3647f0e Content-Type: text/plain; charset=ISO-8859-1 The last minute problem as related by Alex stems from the fact that the iSER initiator posts its RX buffers for that connection only after the final Login Response arrives. During that time (after the target had sent the Last Login Response but before the Full Featured phase related RX-buffers are posted on the initiator side) the target may send the first NOP-IN as it considers the connection in Full Featured phase already and MaxOutstandingUnexpectedPDUs has been negotiated to a non-zero value. The current solution is for the initiator to post an extra buffer before sending the last Login Request as the target most probably sends the next NOP-IN only after some timeout period measuring in seconds or hundreds of milliseconds. But this is not a foolproof solution as the target MAY theoretically send up to MaxOustandingUnexpectedPUDs to the initiator. The new proposed solution is to require iSER Hello exchanges in order to delay the onset of the iSER assisted Full Featured phase until the iSERHelloReply PDU has arrived at the target. This allows the initiator to delay posting its RX buffers until after receiving the final LoginResponse returns but before sending the iSER Hello PDU. To support this new proposed solution, the iSERHelloRequired key must be negotiated to Yes. This require both sides support the new iSERHelloRequired key, leading to the need for clarifying texts suggested by David in the spec in order to specify what should be done otherwise. But if we look at the current solution, the initiator is already posting an extra buffer before sending the last Login Request. Given that the existing implementation is broken and has to be fixed anyway, how difficult would it be for it to post MaxOustandingUnexpectedPDU instead of just an extra one before sending the last Login Request? This complies with the spec and is guaranteed to work. The alternative is to require the use of and implement iSER Hello exchanges. If the partner does not support the new iSERHelloRequired key, the solution fails anyway. It seems to me it makes more sense (and a cleaner soluton besides) for the initiator to allocate the iSER resources, i.e., MaxOustandingUnexpectedPDUs, before sending the last Login Request as stated in the spec than requiring both sides to support iSER Hello exchanges and negotiate the new key. Mike On Tue, Jun 26, 2012 at 6:16 AM, wrote: > > > The discussion about whether the buffer behavior of the Linux iSER > implementation complies with RFC 5046 is missing at least one > important point - to my knowledge, there are *no* iSER > implementations that comply with RFC 5046. > > After RFC 5046 was issued, implementers looked at its requirements and > decided to do a number of things differently - the important difference > for this discussion is that the iSER Hello exchange is not implemented, > even though RFC 5046 requires that it be used. There are a number of > other "running code" changes in the current iSER draft that reflect > implementations and that are incompatible with RFC 5046, see Appendix A > of the draft for details. > > The iSER work item was added to the storm WG's charter in order to produce > an RFC specification of iSER that matches the "running code" and hence will > interoperate with existing implementations. Here's the text from the > WG charter: > > (6) iSER: Experience with Infiniband implementations suggest > a few minor updates to reflect what has been done in practice. > > As WG chair, I'm going to insist on the following: > 1) The iSER draft has to match the implementations in order to be > published as an RFC. Replacing RFC 5046 with another RFC that > won't interoperate with the implementations is pointless. > 2) The issue encountered by the Linux iSER implementation cannot be > ignored by the WG. The workaround (1 extra buffer, at most > one unsolicited PDU until initiator has time to deploy > resources) at least needs to be explained in the draft. > 3) While requiring the iSER Hello exchange may be the proverbial > "right thing to do" going forward, it is necessary to provide > at least guidance on how to interoperate with current iSER > implementations that don't support the Hello exchange. > > In contrast, as WG chair, I'm not going to insist that we adopt the > workaround as part of the standard. For example, it's possible to > recommend (SHOULD) use of the iSER Hello exchange (explicitly negotiated > using the new key) and explain the workaround as what should be > done in order to interoperate if that key results in a NotUnderstood > response. For this approach to the workaround, I think a lower case > "should" may be acceptable (e.g., as opposed to a "SHOULD" or "MUST", > but that's subject to further discussion. > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > david.black@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > --00248c769122aaf56d04c3647f0e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable The last minute problem as related by Alex stems from the fact that the iSE= R initiator posts its RX buffers for that connection only after the final L= ogin Response arrives.=A0 During that time (after the target had sent the L= ast Login Response but before
the Full Featured phase related RX-buffers are posted on the initiator side= ) the target may send the first NOP-IN as it considers the connection in Fu= ll Featured phase already and MaxOutstandingUnexpectedPDUs has been negotia= ted to a non-zero value.

The current solution is for the initiator to post an extra buffer befor= e sending the last Login Request as the target most probably sends the next= NOP-IN only after some timeout period measuring in seconds or hundreds of = milliseconds.=A0 But this is not a foolproof solution as the target MAY the= oretically send up to MaxOustandingUnexpectedPUDs to the initiator.

The new proposed solution is to require iSER Hello exchanges in order t= o delay the onset of the iSER assisted Full Featured phase until the iSERHe= lloReply PDU has arrived at the target.=A0 This allows the initiator to del= ay posting its RX buffers until after receiving the final LoginResponse ret= urns but before sending the iSER Hello PDU.=A0 To support this new proposed= solution, the iSERHelloRequired key must be negotiated to Yes.=A0 This req= uire both sides support the new iSERHelloRequired key, leading to the need = for clarifying texts suggested by David in the spec in order to specify wha= t should be done otherwise.

But if we look at the current solution, the initiator is already postin= g an extra buffer before sending the last Login Request.=A0 Given that the = existing implementation is broken and has to be fixed anyway, how difficult= would it be for it to post MaxOustandingUnexpectedPDU instead of just an e= xtra one before sending the last Login Request?=A0 This complies with the s= pec and is guaranteed to work.=A0 The alternative is to require the use of = and implement iSER Hello exchanges.=A0 If the partner does not support the = new iSERHelloRequired key, the solution fails anyway.=A0 It seems to me it = makes more sense (and a cleaner soluton besides) for the initiator to alloc= ate the iSER resources, i.e., MaxOustandingUnexpectedPDUs, before sending t= he last Login Request as stated in the spec than requiring both sides to su= pport iSER Hello exchanges and negotiate the new key.

Mike

On Tue, Jun 26, 2012 at 6:16 AM,= <david.black@emc.com> wrote:
<WG chair hat on>

The discussion about whether the buffer behavior of the Linux iSER
implementation complies with RFC 5046 is missing at least one
important point - to my knowledge, there are *no* iSER
implementations that comply with RFC 5046.

After RFC 5046 was issued, implementers looked at its requirements and
decided to do a number of things differently - the important difference
for this discussion is that the iSER Hello exchange is not implemented,
even though RFC 5046 requires that it be used. =A0There are a number of
other "running code" changes in the current iSER draft that refle= ct
implementations and that are incompatible with RFC 5046, see Appendix A
of the draft for details.

The iSER work item was added to the storm WG's charter in order to prod= uce
an RFC specification of iSER that matches the "running code" and = hence will
interoperate with existing implementations. =A0Here's the text from the=
WG charter:

=A0 =A0 =A0 =A0(6) iSER: Experience with Infiniband implementations sugges= t
=A0 =A0 =A0 =A0a few minor updates to reflect what has been done in practi= ce.

As WG chair, I'm going to insist on the following:
1) The iSER draft has to match the implementations in order to be
=A0 =A0 =A0 =A0published as an RFC. =A0Replacing RFC 5046 with another RFC= that
=A0 =A0 =A0 =A0won't interoperate with the implementations is pointles= s.
2) The issue encountered by the Linux iSER implementation cannot be
=A0 =A0 =A0 =A0ignored by the WG. =A0The workaround (1 extra buffer, at mo= st
=A0 =A0 =A0 =A0one unsolicited PDU until initiator has time to deploy
=A0 =A0 =A0 =A0resources) at least needs to be explained in the draft.
3) While requiring the iSER Hello exchange may be the proverbial
=A0 =A0 =A0 =A0"right thing to do" going forward, it is necessar= y to provide
=A0 =A0 =A0 =A0at least guidance on how to interoperate with current iSER<= br> =A0 =A0 =A0 =A0implementations that don't support the Hello exchange.<= br>
In contrast, as WG chair, I'm not going to insist that we adopt the
workaround as part of the standard. =A0For example, it's possible to recommend (SHOULD) use of the iSER Hello exchange (explicitly negotiated using the new key) and explain the workaround as what should be
done in order to interoperate if that key results in a NotUnderstood
response. =A0For this approach to the workaround, I think a lower case
"should" may be acceptable (e.g., as opposed to a "SHOULD&qu= ot; or "MUST",
but that's subject to further discussion.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA=A0 01748
+1 (508)= 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-7786
david.black@emc.com=A0=A0=A0=A0= =A0=A0=A0 Mobile: +1 (978) 394-7754
----------------------------------------------------


_______________________________________________
storm mailing list
storm@ietf.org
h= ttps://www.ietf.org/mailman/listinfo/storm

--00248c769122aaf56d04c3647f0e-- From david.black@emc.com Tue Jun 26 12:02:32 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5F5B211E80CA for ; Tue, 26 Jun 2012 12:02:32 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.373 X-Spam-Level: X-Spam-Status: No, score=-102.373 tagged_above=-999 required=5 tests=[AWL=0.225, BAYES_00=-2.599, HTML_MESSAGE=0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id F3+YUKclnEiz for ; Tue, 26 Jun 2012 12:02:28 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 4FBCF21F8569 for ; Tue, 26 Jun 2012 12:02:27 -0700 (PDT) Received: from hop04-l1d11-si02.isus.emc.com (HOP04-L1D11-SI02.isus.emc.com [10.254.111.55]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5QJ2NhC031477 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 26 Jun 2012 15:02:24 -0400 Received: from mailhub.lss.emc.com (mailhubhoprd03.lss.emc.com [10.254.221.145]) by hop04-l1d11-si02.isus.emc.com (RSA Interceptor); Tue, 26 Jun 2012 15:02:11 -0400 Received: from mxhub21.corp.emc.com (mxhub21.corp.emc.com [128.222.70.133]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5QJ2AHX017420; Tue, 26 Jun 2012 15:02:10 -0400 Received: from mx15a.corp.emc.com ([169.254.1.189]) by mxhub21.corp.emc.com ([128.222.70.133]) with mapi; Tue, 26 Jun 2012 15:02:10 -0400 From: To: Date: Tue, 26 Jun 2012 15:02:08 -0400 Thread-Topic: [storm] iSER - what to do Thread-Index: Ac1Ty+GpVpipEWmqT+25wCmH5a9MOgAASPsg Message-ID: <8D3D17ACE214DC429325B2B98F3AE71208C14A8F@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71208C14966@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_8D3D17ACE214DC429325B2B98F3AE71208C14A8FMX15Acorpemccom_" MIME-Version: 1.0 X-EMM-MHVC: 1 Cc: storm@ietf.org Subject: Re: [storm] iSER - what to do X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jun 2012 19:02:32 -0000 --_000_8D3D17ACE214DC429325B2B98F3AE71208C14A8FMX15Acorpemccom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Mike, Thanks for responding - we will eventually get to a conclusion in this disc= ussion :-). I'm adding two comments to try to sharpen the discussion of the current sol= ution: > The current solution is for the initiator to post an extra buffer before > sending the last Login Request as the target most probably sends the next > NOP-IN only after some timeout period measuring in seconds or hundreds of > milliseconds. But this is not a foolproof solution as the target MAY > theoretically send up to MaxOustandingUnexpectedPUDs to the initiator. Right - my thought about how to make it fool-resistant is to require that the target MUST NOT send the next unsolicited PDU until at least 200ms after the first one or after receipt of a full-phase PDU from the initiator (that requirement could be a "SHOULD NOT" with a warning that sending sooner risks the initiator closing the RCaP connection). I pulled 200ms out of my proverbial "hat" and would like to see comments on what a good number would be for both IB and iWarp. > But if we look at the current solution, the initiator is already > posting an extra buffer before sending the last Login Request. Given > that the existing implementation is broken and has to be fixed anyway, > how difficult would it be for it to post MaxOustandingUnexpectedPDU > instead of just an extra one before sending the last Login Request? That's a question for Alexander and other implementers. My guess is that we're talking about quite a few buffers, and there may be resource manageme= nt advantages to not allocating that full set of resources until the initiator is committed to the connection. Thanks, --David From: Michael Ko [mailto:mkosjc@gmail.com] Sent: Tuesday, June 26, 2012 2:45 PM To: Black, David Cc: storm@ietf.org Subject: Re: [storm] iSER - what to do The last minute problem as related by Alex stems from the fact that the iSE= R initiator posts its RX buffers for that connection only after the final L= ogin Response arrives. During that time (after the target had sent the Las= t Login Response but before the Full Featured phase related RX-buffers are posted on the initiator side= ) the target may send the first NOP-IN as it considers the connection in Fu= ll Featured phase already and MaxOutstandingUnexpectedPDUs has been negotia= ted to a non-zero value. The current solution is for the initiator to post an extra buffer before se= nding the last Login Request as the target most probably sends the next NOP= -IN only after some timeout period measuring in seconds or hundreds of mill= iseconds. But this is not a foolproof solution as the target MAY theoretic= ally send up to MaxOustandingUnexpectedPUDs to the initiator. The new proposed solution is to require iSER Hello exchanges in order to de= lay the onset of the iSER assisted Full Featured phase until the iSERHelloR= eply PDU has arrived at the target. This allows the initiator to delay pos= ting its RX buffers until after receiving the final LoginResponse returns b= ut before sending the iSER Hello PDU. To support this new proposed solutio= n, the iSERHelloRequired key must be negotiated to Yes. This require both = sides support the new iSERHelloRequired key, leading to the need for clarif= ying texts suggested by David in the spec in order to specify what should b= e done otherwise. But if we look at the current solution, the initiator is already posting an= extra buffer before sending the last Login Request. Given that the existi= ng implementation is broken and has to be fixed anyway, how difficult would= it be for it to post MaxOustandingUnexpectedPDU instead of just an extra o= ne before sending the last Login Request? This complies with the spec and = is guaranteed to work. The alternative is to require the use of and implem= ent iSER Hello exchanges. If the partner does not support the new iSERHell= oRequired key, the solution fails anyway. It seems to me it makes more sen= se (and a cleaner soluton besides) for the initiator to allocate the iSER r= esources, i.e., MaxOustandingUnexpectedPDUs, before sending the last Login = Request as stated in the spec than requiring both sides to support iSER Hel= lo exchanges and negotiate the new key. Mike On Tue, Jun 26, 2012 at 6:16 AM, > wrote: The discussion about whether the buffer behavior of the Linux iSER implementation complies with RFC 5046 is missing at least one important point - to my knowledge, there are *no* iSER implementations that comply with RFC 5046. After RFC 5046 was issued, implementers looked at its requirements and decided to do a number of things differently - the important difference for this discussion is that the iSER Hello exchange is not implemented, even though RFC 5046 requires that it be used. There are a number of other "running code" changes in the current iSER draft that reflect implementations and that are incompatible with RFC 5046, see Appendix A of the draft for details. The iSER work item was added to the storm WG's charter in order to produce an RFC specification of iSER that matches the "running code" and hence will interoperate with existing implementations. Here's the text from the WG charter: (6) iSER: Experience with Infiniband implementations suggest a few minor updates to reflect what has been done in practice. As WG chair, I'm going to insist on the following: 1) The iSER draft has to match the implementations in order to be published as an RFC. Replacing RFC 5046 with another RFC that won't interoperate with the implementations is pointless. 2) The issue encountered by the Linux iSER implementation cannot be ignored by the WG. The workaround (1 extra buffer, at most one unsolicited PDU until initiator has time to deploy resources) at least needs to be explained in the draft. 3) While requiring the iSER Hello exchange may be the proverbial "right thing to do" going forward, it is necessary to provide at least guidance on how to interoperate with current iSER implementations that don't support the Hello exchange. In contrast, as WG chair, I'm not going to insist that we adopt the workaround as part of the standard. For example, it's possible to recommend (SHOULD) use of the iSER Hello exchange (explicitly negotiated using the new key) and explain the workaround as what should be done in order to interoperate if that key results in a NotUnderstood response. For this approach to the workaround, I think a lower case "should" may be acceptable (e.g., as opposed to a "SHOULD" or "MUST", but that's subject to further discussion. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508= ) 293-7786 david.black@emc.com Mobile: +1 (978) 394= -7754 ---------------------------------------------------- _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm --_000_8D3D17ACE214DC429325B2B98F3AE71208C14A8FMX15Acorpemccom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Mike,

 

<WG= chair hat off>

 <= /span>

Thanks for responding - we will eventually get t= o a conclusion in this discussion :-).

I’m adding two comment= s to try to sharpen the discussion of the current solution:

 

> Th= e current solution is for the initiator to post an extra buffer before=

> sending the last Login Request as th= e target most probably sends the next

> NOP-IN only after some timeout period measuring in seconds or hundred= s of

> milliseconds.  But thi= s is not a foolproof solution as the target MAY

> theoretically send up to MaxOustandingUnexpectedPUDs to the= initiator.

Right - my thou= ght about how to make it fool-resistant is to require that

the target MUST NOT send the next unsolicited PDU unt= il at least 200ms

after the first one= or after receipt of a full-phase PDU from the initiator<= /p>

(that requirement could be a “SHOULD NOT” w= ith a warning that sending

sooner ris= ks the initiator closing the RCaP connection).  I pulled 200ms

out of my proverbial “hat” and w= ould like to see comments on what a

g= ood number would be for both IB and iWarp.

 

> But if we look at t= he current solution, the initiator is already

> posting an extra buffer before sending the last Login Request= .  Given

> that the existing = implementation is broken and has to be fixed anyway,

<= p class=3DMsoNormal>> how difficult would it be for it to post MaxOustanding= UnexpectedPDU

> instead of just an= extra one before sending the last Login Request? 

 

That’= s a question for Alexander and other implementers.  My guess is that

we’re talking about quite a few = buffers, and there may be resource management

advantages to not allocating that full set of resources until the = initiator

is committed to the connect= ion.

 

Thanks,
--David

 

F= rom: Michael Ko [mailto:mkosjc@gmail.com]
Sent: Tuesday, June= 26, 2012 2:45 PM
To: Black, David
Cc: storm@ietf.orgSubject: Re: [storm] iSER - what to do

=

 

The last minute problem as related by Alex stems = from the fact that the iSER initiator posts its RX buffers for that connect= ion only after the final Login Response arrives.  During that time (af= ter the target had sent the Last Login Response but before
the Full Feat= ured phase related RX-buffers are posted on the initiator side) the target = may send the first NOP-IN as it considers the connection in Full Featured p= hase already and MaxOutstandingUnexpectedPDUs has been negotiated to a non-= zero value.

The current solution is for the initiator to post an ext= ra buffer before sending the last Login Request as the target most probably= sends the next NOP-IN only after some timeout period measuring in seconds = or hundreds of milliseconds.  But this is not a foolproof solution as = the target MAY theoretically send up to MaxOustandingUnexpectedPUDs to the = initiator.

The new proposed solution is to require iSER Hello exchan= ges in order to delay the onset of the iSER assisted Full Featured phase un= til the iSERHelloReply PDU has arrived at the target.  This allows the= initiator to delay posting its RX buffers until after receiving the final = LoginResponse returns but before sending the iSER Hello PDU.  To suppo= rt this new proposed solution, the iSERHelloRequired key must be negotiated= to Yes.  This require both sides support the new iSERHelloRequired ke= y, leading to the need for clarifying texts suggested by David in the spec = in order to specify what should be done otherwise.

But if we look at= the current solution, the initiator is already posting an extra buffer bef= ore sending the last Login Request.  Given that the existing implement= ation is broken and has to be fixed anyway, how difficult would it be for i= t to post MaxOustandingUnexpectedPDU instead of just an extra one before se= nding the last Login Request?  This complies with the spec and is guar= anteed to work.  The alternative is to require the use of and implemen= t iSER Hello exchanges.  If the partner does not support the new iSERH= elloRequired key, the solution fails anyway.  It seems to me it makes = more sense (and a cleaner soluton besides) for the initiator to allocate th= e iSER resources, i.e., MaxOustandingUnexpectedPDUs, before sending the las= t Login Request as stated in the spec than requiring both sides to support = iSER Hello exchanges and negotiate the new key.

Mike

<= div>

On Tue, Jun 26, 2012 at 6:16 AM, <david.black@emc.com> wro= te:

<WG chair hat on>

The d= iscussion about whether the buffer behavior of the Linux iSER
implementa= tion complies with RFC 5046 is missing at least one
important point - to= my knowledge, there are *no* iSER
implementations that comply with RFC = 5046.

After RFC 5046 was issued, implementers looked at its requirem= ents and
decided to do a number of things differently - the important di= fference
for this discussion is that the iSER Hello exchange is not impl= emented,
even though RFC 5046 requires that it be used.  There are = a number of
other "running code" changes in the current iSER d= raft that reflect
implementations and that are incompatible with RFC 504= 6, see Appendix A
of the draft for details.

The iSER work item wa= s added to the storm WG's charter in order to produce
an RFC specificati= on of iSER that matches the "running code" and hence will
inte= roperate with existing implementations.  Here's the text from the
W= G charter:

       (6) iSER: Experience with Infi= niband implementations suggest
       a few minor up= dates to reflect what has been done in practice.

As WG chair, I'm go= ing to insist on the following:
1) The iSER draft has to match the imple= mentations in order to be
       published as an RFC= .  Replacing RFC 5046 with another RFC that
      &n= bsp;won't interoperate with the implementations is pointless.
2) The iss= ue encountered by the Linux iSER implementation cannot be
    =    ignored by the WG.  The workaround (1 extra buffer, at mo= st
       one unsolicited PDU until initiator has ti= me to deploy
       resources) at least needs to be = explained in the draft.
3) While requiring the iSER Hello exchange may b= e the proverbial
       "right thing to do"= ; going forward, it is necessary to provide
       a= t least guidance on how to interoperate with current iSER
    =    implementations that don't support the Hello exchange.

= In contrast, as WG chair, I'm not going to insist that we adopt the
work= around as part of the standard.  For example, it's possible to
reco= mmend (SHOULD) use of the iSER Hello exchange (explicitly negotiated
usi= ng the new key) and explain the workaround as what should be
done in ord= er to interoperate if that key results in a NotUnderstood
response. &nbs= p;For this approach to the workaround, I think a lower case
"should= " may be acceptable (e.g., as opposed to a "SHOULD" or "= ;MUST",
but that's subject to further discussion.

Thanks,--David
----------------------------------------------------
David L= . Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinto= n, MA  01748
+1 (508) 2= 93-7953          &nbs= p;  FAX: +1 (508) 293-7786=
david.black@emc.com = ;       Mobile: +1 (978) 394-7754
-----------------------------------= -----------------


______________________________________________= _
storm mailing list
storm@ietf.org=
https://www.ietf.org/mailman/listinfo/storm

=

 

= --_000_8D3D17ACE214DC429325B2B98F3AE71208C14A8FMX15Acorpemccom_-- From david.black@emc.com Thu Jun 28 08:58:59 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CAF8021F85A4 for ; Thu, 28 Jun 2012 08:58:59 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.384 X-Spam-Level: X-Spam-Status: No, score=-102.384 tagged_above=-999 required=5 tests=[AWL=0.215, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IKXU7ciiZs9e for ; Thu, 28 Jun 2012 08:58:58 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id BEC2B21F85AA for ; Thu, 28 Jun 2012 08:58:58 -0700 (PDT) Received: from hop04-l1d11-si03.isus.emc.com (HOP04-L1D11-SI03.isus.emc.com [10.254.111.23]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5SFwtoA028127 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 28 Jun 2012 11:58:56 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.222.129]) by hop04-l1d11-si03.isus.emc.com (RSA Interceptor) for ; Thu, 28 Jun 2012 11:58:44 -0400 Received: from mxhub15.corp.emc.com (mxhub15.corp.emc.com [128.222.70.236]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5SFwiuZ004969 for ; Thu, 28 Jun 2012 11:58:44 -0400 Received: from mx15a.corp.emc.com ([169.254.1.189]) by mxhub15.corp.emc.com ([128.222.70.236]) with mapi; Thu, 28 Jun 2012 11:58:44 -0400 From: To: Date: Thu, 28 Jun 2012 11:58:43 -0400 Thread-Topic: iSCSI - NOP loop avoidance Thread-Index: Ac1VRuReo0UnniUrQz2rHgxcMSDpKA== Message-ID: <8D3D17ACE214DC429325B2B98F3AE71208D3A71A@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: [storm] iSCSI - NOP loop avoidance X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jun 2012 15:58:59 -0000 This is the first of a number of emails to come on relatively small technical issues that have been discovered in reviews of the consolidated iSCSI draft. Resolution of all of these issues should not require changes to implementations, but they do require the WG's attention. The specific issue for this email is NOP-loop avoidance, specifically tightening up the language in the draft that tells implementers what to do to avoid a lengthy sequence of NOP-In and NOP-Out PDUs sent in response to each other. This is not believed to be a problem in practice because NOP-* PDUs are pure overhead, and there are no requirements to respond in a fashion that would create such a lengthy sequence, so implementations are likely to not respond in order to spend those resources on something more useful. Nonetheless, the language in the draft on this subject is weak, and in particular contains no requirement (MUST/ SHOULD/MAY) to not respond in a fashion that would create a lengthy sequence. Yes, this is a corner case ;-). The Initiator Task Tag (ITT) and Target Transfer Tag (TTT) are crucial to understanding what's going on here. The NOP-In and NOP-Out PDUs (see 11.18 and 11.19) use these tags to support "ping" round trip functionality: 1) The originator of the "ping" sets its tag to a valid value, and sets the other tag to an invalid value in the first NOP-* PDU. 2) The responder recognizes the valid value of the originator's tag as a response request, and preserves those values in the response NOP-* PDU. 3) The originator is expected not to continue because the responder's tag is invalid in that second PDU. There's one additional twist. A 3-way "ping" is allowed when the Target is the originator - at step 2) above, the Initiator (but not the Target) is allowed to use valid values for both tags, in which case the target will respond with a NOP-In PDU that uses an invalid TTT value, and the "expected not to continue" in 3) applies when the initiator receives that PDU. Here are the proposed text changes: ----------------------------- 11.18. NOP-Out Existing text: Upon receipt of a NOP-In with the Target Transfer Tag set to a valid value (not the reserved 0xffffffff), the initiator MUST respond with a NOP-Out. In this case, the NOP-Out Target Transfer Tag MUST contain a copy of the NOP-In Target Transfer Tag. Add the following sentence to the end of that paragraph: The initiator SHOULD NOT send a NOP-Out in response to any other received NOP-In in order to avoid lengthy sequences of NOP-In and NOP-Out PDUs sent in response to each other. 11.19. NOP-In Existing text: When a target receives the NOP-Out with a valid Initiator Task Tag (not the reserved value 0xffffffff), it MUST respond with a NOP-In with the same Initiator Task Tag that was provided in the NOP-Out request. It MUST also duplicate up to the first MaxRecvDataSegmentLength bytes of the initiator provided Ping Data. For such a response, the Target Transfer Tag MUST be 0xffffffff. Add the following sentence to the end of that paragraph: The target SHOULD NOT send a NOP-In in response to any other received NOP-Out in order to avoid lengthy sequences of NOP-In and NOP-Out PDUs sent in response to each other. 11.19.1. Target Transfer Tag Change "is set to" to "MUST be set to" in the following two sentences: If the target is responding to a NOP-Out, this is set to the reserved value 0xffffffff. If the target is sending a NOP-In as a Ping (intending to receive a corresponding NOP-Out), this field is set to a valid value (not the reserved 0xffffffff). ----------------------------- Please comment - as noted above, this should have no impact on existing implementations, but does tighten up and explicitly state what was an implicit requirement. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- From david.black@emc.com Fri Jun 29 06:17:40 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1EF0E21F85E4 for ; Fri, 29 Jun 2012 06:17:40 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.556 X-Spam-Level: X-Spam-Status: No, score=-102.556 tagged_above=-999 required=5 tests=[AWL=0.043, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZIHGOhDF7KXR for ; Fri, 29 Jun 2012 06:17:39 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 696A421F855F for ; Fri, 29 Jun 2012 06:17:39 -0700 (PDT) Received: from hop04-l1d11-si02.isus.emc.com (HOP04-L1D11-SI02.isus.emc.com [10.254.111.55]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5TDHatE007472 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 29 Jun 2012 09:17:37 -0400 Received: from mailhub.lss.emc.com (mailhubhoprd01.lss.emc.com [10.254.221.251]) by hop04-l1d11-si02.isus.emc.com (RSA Interceptor) for ; Fri, 29 Jun 2012 09:17:18 -0400 Received: from mxhub28.corp.emc.com (mxhub28.corp.emc.com [10.254.110.184]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5TDHGiT020827 for ; Fri, 29 Jun 2012 09:17:17 -0400 Received: from mx15a.corp.emc.com ([169.254.1.189]) by mxhub28.corp.emc.com ([10.254.110.184]) with mapi; Fri, 29 Jun 2012 09:17:16 -0400 From: To: Importance: high X-Priority: 1 Date: Fri, 29 Jun 2012 09:17:16 -0400 Thread-Topic: Vancouver meeting cancelled Thread-Index: Ac1V+YDczOX/z5QsRLeEEa0vpKraDA== Message-ID: <8D3D17ACE214DC429325B2B98F3AE71208D3A882@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: [storm] Vancouver meeting cancelled X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jun 2012 13:17:40 -0000 Having seem no comment on the list since the possibility of canceling the storm WG's meeting in Vancouver was sent out on Tuesday, that meeting is now cancelled. I need to do this today, as the draft Vancouver agenda just appeared last night, and the slot juggling to deal with conflicts is starting Thanks, --David (storm WG chair) ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- From julians@infinidat.com Thu Jun 28 10:59:41 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 01FCF21F85AF for ; Thu, 28 Jun 2012 10:59:41 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.203 X-Spam-Level: X-Spam-Status: No, score=-2.203 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_QP_LONG_LINE=1.396, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UVK4G7UFUjQZ for ; Thu, 28 Jun 2012 10:59:37 -0700 (PDT) Received: from mail-wi0-f172.google.com (mail-wi0-f172.google.com [209.85.212.172]) by ietfa.amsl.com (Postfix) with ESMTP id CE0CF21F85D8 for ; Thu, 28 Jun 2012 10:59:36 -0700 (PDT) Received: by wibhm11 with SMTP id hm11so285902wib.13 for ; Thu, 28 Jun 2012 10:59:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=references:in-reply-to:mime-version:content-type:message-id :content-transfer-encoding:cc:x-mailer:from:subject:date:to :x-gm-message-state; bh=DkA+UrTlnUhMm05BcUwuik/EbNCLEKOUdj5im3G5Sr4=; b=EZAQ7SaFzqVAvjG1MwW3Fg3Q7IdhHko30mxJyRmMD9yhyIuTIbwJ6I92YqoU3WlMtG q5Vp7mGNONwrTMmcprmwsjBlpwboL+N5V+8UgZE2cNxNGLmcZPzANlY5eqoJSJU5cx6N 0OOVdnyyiMyj+BWOL1WXye3fk97onGsrnaHbdFFfiFjKDqkUVbEe3WTknoK4LmZiQGK5 WTK7+3upSrYcr9ivpm71OnrgPoijw0VWQPRHrwMIK+4+mBAX7t20c46n7+b77/p8US7i ygKghWVuiWoZMVRSDMLduYoKD7O/aycIb2m2kYwlGJFZTX1ZSHxAG72opa5QjYi4O/Dx aqMA== Received: by 10.216.143.105 with SMTP id k83mr1509741wej.99.1340906375448; Thu, 28 Jun 2012 10:59:35 -0700 (PDT) Received: from [192.168.1.4] ([109.160.140.196]) by mx.google.com with ESMTPS id cl8sm1370159wib.10.2012.06.28.10.59.32 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 28 Jun 2012 10:59:34 -0700 (PDT) References: <8D3D17ACE214DC429325B2B98F3AE71208D3A71A@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE71208D3A71A@MX15A.corp.emc.com> Mime-Version: 1.0 (1.0) Content-Type: text/plain; charset=utf-8 Message-Id: <89361195-CDC2-415B-AB70-A8DAF1637AFB@infinidat.com> Content-Transfer-Encoding: quoted-printable X-Mailer: iPad Mail (9B206) From: Julian Satran Date: Thu, 28 Jun 2012 20:59:31 +0300 To: "" X-Gm-Message-State: ALoCoQm3Y7cR1ot18ndXf+Zl0IDXIs00avhEET+gotCoyFqrke4JetGtvqTKB6rlhy4J7xM+3+7G X-Mailman-Approved-At: Fri, 29 Jun 2012 08:45:13 -0700 Cc: "" Subject: Re: [storm] iSCSI - NOP loop avoidance X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jun 2012 18:09:15 -0000 the text sounds fine although the added text to 11.18.1 is mostly decorative= . Sent from my iPad On 28 =D7=91=D7=99=D7=95=D7=A0 2012, at 18:58, wrote: > >=20 > This is the first of a number of emails to come on relatively small > technical issues that have been discovered in reviews of the consolidated > iSCSI draft. Resolution of all of these issues should not require changes= > to implementations, but they do require the WG's attention. >=20 > > >=20 > The specific issue for this email is NOP-loop avoidance, specifically > tightening up the language in the draft that tells implementers what to > do to avoid a lengthy sequence of NOP-In and NOP-Out PDUs sent in > response to each other. This is not believed to be a problem in practice > because NOP-* PDUs are pure overhead, and there are no requirements to > respond in a fashion that would create such a lengthy sequence, so > implementations are likely to not respond in order to spend those resource= s > on something more useful. Nonetheless, the language in the draft on > this subject is weak, and in particular contains no requirement (MUST/ > SHOULD/MAY) to not respond in a fashion that would create a lengthy > sequence. Yes, this is a corner case ;-). >=20 > The Initiator Task Tag (ITT) and Target Transfer Tag (TTT) are crucial > to understanding what's going on here. The NOP-In and NOP-Out PDUs > (see 11.18 and 11.19) use these tags to support "ping" round trip > functionality: >=20 > 1) The originator of the "ping" sets its tag to a valid value, > and sets the other tag to an invalid value in the first > NOP-* PDU. > 2) The responder recognizes the valid value of the originator's > tag as a response request, and preserves those values in the > response NOP-* PDU. > 3) The originator is expected not to continue because the responder's > tag is invalid in that second PDU. >=20 > There's one additional twist. A 3-way "ping" is allowed when the > Target is the originator - at step 2) above, the Initiator (but > not the Target) is allowed to use valid values for both tags, in > which case the target will respond with a NOP-In PDU that uses > an invalid TTT value, and the "expected not to continue" in 3) > applies when the initiator receives that PDU. >=20 > Here are the proposed text changes: >=20 > ----------------------------- >=20 > 11.18. NOP-Out >=20 > Existing text: >=20 > Upon receipt of a NOP-In with the Target Transfer Tag set to a > valid value (not the reserved 0xffffffff), the initiator MUST > respond with a NOP-Out. In this case, the NOP-Out Target Transfer > Tag MUST contain a copy of the NOP-In Target Transfer Tag. >=20 > Add the following sentence to the end of that paragraph: >=20 > The initiator SHOULD NOT send a NOP-Out in response > to any other received NOP-In in order to avoid lengthy > sequences of NOP-In and NOP-Out PDUs sent in response to each other. >=20 > 11.19. NOP-In >=20 > Existing text: >=20 > When a target receives the NOP-Out with a valid Initiator Task Tag > (not the reserved value 0xffffffff), it MUST respond with a NOP-In > with the same Initiator Task Tag that was provided in the NOP-Out > request. It MUST also duplicate up to the first > MaxRecvDataSegmentLength bytes of the initiator provided Ping > Data. For such a response, the Target Transfer Tag MUST be > 0xffffffff. >=20 > Add the following sentence to the end of that paragraph: >=20 > The target SHOULD NOT send a NOP-In in response > to any other received NOP-Out in order to avoid lengthy > sequences of NOP-In and NOP-Out PDUs sent in response to each other. >=20 > 11.19.1. Target Transfer Tag >=20 > Change "is set to" to "MUST be set to" in the following two sentences: >=20 > If the target is responding to a NOP-Out, this is set to the > reserved value 0xffffffff. >=20 > If the target is sending a NOP-In as a Ping (intending to receive > a corresponding NOP-Out), this field is set to a valid value (not > the reserved 0xffffffff). >=20 > ----------------------------- >=20 > Please comment - as noted above, this should have no impact on > existing implementations, but does tighten up and explicitly state > what was an implicit requirement. >=20 > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > david.black@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- >=20 >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From david.black@emc.com Fri Jun 29 11:57:25 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2C5AC21F886A for ; Fri, 29 Jun 2012 11:57:25 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.558 X-Spam-Level: X-Spam-Status: No, score=-103.558 tagged_above=-999 required=5 tests=[AWL=1.041, BAYES_00=-2.599, GB_I_LETTER=-2, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iPO7-I4QqIA5 for ; Fri, 29 Jun 2012 11:57:24 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 3976E21F8868 for ; Fri, 29 Jun 2012 11:57:23 -0700 (PDT) Received: from hop04-l1d11-si03.isus.emc.com (HOP04-L1D11-SI03.isus.emc.com [10.254.111.23]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5TIvIwJ021469 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 29 Jun 2012 14:57:22 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.222.130]) by hop04-l1d11-si03.isus.emc.com (RSA Interceptor) for ; Fri, 29 Jun 2012 14:56:56 -0400 Received: from mxhub02.corp.emc.com (mxhub02.corp.emc.com [10.254.141.104]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q5TIuup8000346 for ; Fri, 29 Jun 2012 14:56:56 -0400 Received: from mx15a.corp.emc.com ([169.254.1.189]) by mxhub02.corp.emc.com ([10.254.141.104]) with mapi; Fri, 29 Jun 2012 14:56:56 -0400 From: To: Importance: high X-Priority: 1 Date: Fri, 29 Jun 2012 14:56:54 -0400 Thread-Topic: iSCSI: Protocol extension items Thread-Index: Ac1WKPOAo6f8UxMERK2SRvioM8mwng== Message-ID: <8D3D17ACE214DC429325B2B98F3AE71208D3A987@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: [storm] iSCSI: Protocol extension items X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jun 2012 18:57:25 -0000 The recent publication of RFC 6648 requires that we make some changes to negotiation of iSCSI protocol extensions. This email explains why changes are needed, describes what appears to be necessary and summarizes the proposed detailed changes to requirements. These proposed changes are to be made in the consolidated iSCSI draft, and hence will apply to existing implementations. These proposed changes should be backwards compatible with existing implementations, although implementers should comment, especially on whether the migration from X- to V- as the prefix for newly defined private extension text keys is likely to cause problems. An example of such a problem is - receipt of a key using the V- format is likely to generate a protocol error instead of a NotUnderstood response to the V- new key. Existing X- private use text keys are *not affected* by these proposed changes, and can continue to be used. iSCSI currently allows extension text keys (for negotiation), digest (checksum) algorithms and authentication methods to be defined using X, Y and Z (respectively) as the first character of the identifier. In each case, there are two defined formats, e.g., for text keys: X-reversed.vendor.dns_name.do_something [dash-based] X<#> [hash-based] The dash-based form (X-) is for private parameters, the hash-based form (X#) is for public parameters registered with IANA. RFC 6648 has recently been published. To borrow a line from Sesame Street, this RFC was brought to you by the letter X ;-). That RFC strongly recommen= ds against use of the letter X in this fashion for non-private parameters, and similarly discourages analogous prefixes that do not contain the letter X. For iSCSI, RFC 6648 applies to the X#, Y# and Z# formats. The history to date of use of the X#, Y# and Z# formats is that there has been exactly one use, X#NodeArchitecture, which wound up being standardized as a fully standard key. That full standardization is an example of why RFC 6648 recommends not using an X-based prefix for public parameters - if the parameter is successful, it will become part of the standard. Hence the proposed approach is to retire the hash-based formats (MUST NOT use) for public extensions with the exception that the X#NodeArchitecture key needs to be preserved. It is still highly desirable to keep the dash-based formats (e.g., Y-) in order to cleanly separate the private and public namespaces for each of the= se extension items. Among the reasons for doing so is that IANA is in the pro= cess of authorizing a number of new top-level domains that will make it somewhat harder to recognize a reversed domain name. Nonetheless, in keeping with the spirit of RFC 6648, it seems prudent to encourage use of a prefix other than X- for new text keys, without doing anything to break current X- text keys. V- is suggested as the new prefix (V can be thought of a short for "vendor"). OTOH, if this will break exist= ing implementations, it's a bad idea (and we'll need to stick with X-) - see the first part of this email. =20 The following approach is therefore currently proposed: 1. Existing private X-keys may continue to remain private as before. However, new private keys SHOULD NOT use the X- prefix in keeping with [RFC6648]. Instead, new keys SHOULD start with a "V-" prefix, followed by a reversed domain name, e.g. V-com.example.do_something Note: If IANA registers something starting with V- as a new top level domain, this approach still works because the V- prefix is always added. For example, if V-org is a new top level domain, an iSCSI text key for example.v-org would have the form V-V-org.example.do_something, which is clearly identifiable as a private extension key, even under case-insensitive character string processing. 2. Each new public key in the course of standardization MUST define the acceptable responses to the key, including NotUnderstood as appropriate. There is no need for a standard name prefix for keys that allow NotUnderstood response. Note that NotUnderstood will generally have to be allowed for new public keys for backwards compatibility. 3. Thus the name prefix "X#" in new public key names does not carry any significance. New public key names MUST NOT begin with "X#" prefix to avoid confusion. 4. The existing public X# key - X#NodeArchitecture - will remain unchanged. 5. Based on our decade-long iSCSI usage experience, we know that Y# and Z# name prefixes, for digest and authentication method extensions respectively, were never used in practice. Similar to X# name prefix for keys, there is no need for this namespace separation either. So for the same reasons cited as for X# prefix, new digest and authentication method extensions MUST NOT respectively use Y# and Z# name prefixes. 6. Private digest and authentication method extensions can continue to respectively use Y- and Z- prefixes as in the current specification. 7. IANA will be requested to do the following: - Move X#NodeArchitecture to the iSCSI Login/Text Keys registry - Remove the iSCSI extended key registry - Not accept registrations of Y# extended digest algorithms or Z# extended authentication methods (and hence not create registries for these extension items). Please comment, particularly on whether moving from X- to V- for new text keys is likely to cause problems. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ----------------------------------------------------