From david.black@emc.com Mon Sep 10 06:00:51 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C750521F85E7 for ; Mon, 10 Sep 2012 06:00:51 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.001, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HK8uLgiJKci7 for ; Mon, 10 Sep 2012 06:00:51 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 39D7221F8686 for ; Mon, 10 Sep 2012 06:00:50 -0700 (PDT) Received: from hop04-l1d11-si02.isus.emc.com (HOP04-L1D11-SI02.isus.emc.com [10.254.111.55]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8AD0nWs014046 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 10 Sep 2012 09:00:49 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.222.130]) by hop04-l1d11-si02.isus.emc.com (RSA Interceptor) for ; Mon, 10 Sep 2012 09:00:44 -0400 Received: from mxhub23.corp.emc.com (mxhub23.corp.emc.com [128.222.70.135]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8AD0hVL003350 for ; Mon, 10 Sep 2012 09:00:43 -0400 Received: from mx15a.corp.emc.com ([169.254.2.46]) by mxhub23.corp.emc.com ([128.222.70.135]) with mapi; Mon, 10 Sep 2012 09:00:43 -0400 From: "Black, David" To: "storm@ietf.org" Date: Mon, 10 Sep 2012 09:00:41 -0400 Thread-Topic: No storm meeting in Atlanta in November Thread-Index: Ac2PVEhviymPDY2sSUeCUlZHnGxNrw== Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120B1D33A6@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: [storm] No storm meeting in Atlanta in November X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Sep 2012 13:00:51 -0000 The storm WG will not meet in November in Atlanta. We do have a bunch of things that need attention (by authors and/or on the list). I'll send a summary of draft status and what happens next to the list soon. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- From david.black@emc.com Mon Sep 10 11:59:12 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8969221E8063 for ; Mon, 10 Sep 2012 11:59:12 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nLf3EuyJ2elj for ; Mon, 10 Sep 2012 11:59:12 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id A421C21E808A for ; Mon, 10 Sep 2012 11:58:46 -0700 (PDT) Received: from hop04-l1d11-si03.isus.emc.com (HOP04-L1D11-SI03.isus.emc.com [10.254.111.23]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8AIwjfF001290 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 10 Sep 2012 14:58:45 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.222.129]) by hop04-l1d11-si03.isus.emc.com (RSA Interceptor) for ; Mon, 10 Sep 2012 14:58:36 -0400 Received: from mxhub03.corp.emc.com (mxhub03.corp.emc.com [10.254.141.105]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8AIwZGE006943 for ; Mon, 10 Sep 2012 14:58:36 -0400 Received: from mx15a.corp.emc.com ([169.254.2.46]) by mxhub03.corp.emc.com ([10.254.141.105]) with mapi; Mon, 10 Sep 2012 14:58:35 -0400 From: "Black, David" To: "storm@ietf.org" Date: Mon, 10 Sep 2012 14:58:33 -0400 Thread-Topic: storm WG: Draft status Thread-Index: Ac2PhkbEsYnDQ1cLRyCkwSEZToFTAw== Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120B1D34EF@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: [storm] storm WG: Draft status X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Sep 2012 18:59:13 -0000 August was a slow month - here's a quick summary of what I believe the status is of all of the storm WG drafts (please send a note to me and/or the list if something seems to be wrong): - iSCSI consolidated and new features (-sam-) drafts: IETF Last Call is complete, although there may be one more review coming on the consolidated draft. Revised versions of both drafts are needed; I'll be in touch with the authors directly about revising these drafts. - iSCSI MIB draft: The MIB Doctor review was more than expected. The currently-posted revised version of this draft contains all of the routine changes, but the MIB Doctor review also requested some functional additions that require more list discussion. Expect an initial proposal for what to do and not do on the list shortly. - iSER draft: I need to review Mike Ko's latest posted proposed text change= s; once we have agreement on that, the next version of this draft should *finally* make it out of the WG and into our AD's hands successfully. - RDMAP extensions draft: Complications with the above drafts, especially iSER (the issues that have delayed iSER have been real and important), have delayed dealing with the RDMAP extensions draft. Tom and I owe the authors reviews of the current version. A best guess for RFC Publication Request timing for this draft is now around the end of the year (sorry). Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- From david.black@emc.com Fri Sep 14 17:26:17 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5CC7921F8566 for ; Fri, 14 Sep 2012 17:26:17 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.598 X-Spam-Level: X-Spam-Status: No, score=-102.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bBqj2KwAILY1 for ; Fri, 14 Sep 2012 17:26:09 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id E1C6521F84B8 for ; Fri, 14 Sep 2012 17:26:08 -0700 (PDT) Received: from hop04-l1d11-si01.isus.emc.com (HOP04-L1D11-SI01.isus.emc.com [10.254.111.54]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8F0Q5E6029412 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 14 Sep 2012 20:26:07 -0400 Received: from mailhub.lss.emc.com (mailhubhoprd02.lss.emc.com [10.254.221.253]) by hop04-l1d11-si01.isus.emc.com (RSA Interceptor); Fri, 14 Sep 2012 20:25:55 -0400 Received: from mxhub36.corp.emc.com (mxhub36.corp.emc.com [10.254.93.84]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8F0Ps7b003545; Fri, 14 Sep 2012 20:25:54 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub36.corp.emc.com ([::1]) with mapi; Fri, 14 Sep 2012 20:25:54 -0400 From: "Black, David" To: Michael Ko Importance: high X-Priority: 1 Date: Fri, 14 Sep 2012 20:25:53 -0400 Thread-Topic: [storm] Proposed text changes for the connection allocation problem Thread-Index: Ac1+df4XYuFNnxYFSjaRgfcu8Dwa2gUX55FQ Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DCD1566@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71208F127E3@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_8D3D17ACE214DC429325B2B98F3AE7120DCD1566MX15Acorpemccom_" MIME-Version: 1.0 X-EMM-MHVC: 1 Cc: "storm@ietf.org" Subject: Re: [storm] Proposed text changes for the connection allocation problem X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Sep 2012 00:26:17 -0000 --_000_8D3D17ACE214DC429325B2B98F3AE7120DCD1566MX15Acorpemccom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Mike, I apologize for the delay in getting to this. I think your latest text is = close to what's needed, but I want to rewrite the latter part of your second paragraph for = clarity and to remove a "MUST" that appears easy to misinterpret (IMHO): OLD To prevent the potential problem caused by the target sending the number= of unexpected iSCSI control-type PDUs as determined by the MaxOustandingUne= xpectedPDUs key allowed in the FullFeaturePhase when "late" connection allocation is= practised, the iSER Hello exchanges MUST be used. This allows the initiator to all= ocate the connection resources before sending the iSER Hello Messages. The iSERHe= lloRequired key is used by the initiator to determine if it is dealing with a target= that supports the iSER Hello exchanges. NEW An initiator that employs "late" connection allocation may encounter pro= blems (e.g., RCaP connection closure) with a target that sends unexpected iSCSI PDUs = immediately upon transitioning to Full Feature Phase, as allowed by the negotiated v= alue of the the MaxOustandingUnexpectedPDUs key. The only way to prevent this situa= tion in full generality is to use iSER Hello Messages, as they enable the initia= tor to allocate its connection resources before sending its iSER Hello Message. The iSE= RHelloRequired key is used by the initiator to determine if it is dealing with a target= that supports the iSER Hello exchanges. Fortunately, known iSER target imple= mentations do not take full advantage of the number of allowed unexpected PDUs imme= diately upon transition into full feature phase, enabling an initiator workaround tha= t involves a smaller quantity of connection resources prior to full-feature phase, = as explained further below. END Would you please go ahead and submit a revised version of the draft with th= e text changes that we've agreed to. I'll want to review the changes one more time in the= context of an entire draft before (finally) sending it to our AD with a request to publis= h it as an RFC. Thank you for your patience with this. Thanks, --David From: Michael Ko [mailto:mkosjc@gmail.com] Sent: Sunday, August 19, 2012 9:49 PM To: Black, David Cc: storm@ietf.org Subject: Re: [storm] Proposed text changes for the connection allocation pr= oblem Changed "must be able to deal with" to "needs to be able to deal with" as s= uggested. For the rest of the suggestions, I have created new text that replaces the = "new" text in section 5.1.3: An initiator that conforms to [RFC5046] allocates connection resources befo= re seding the Login Request with the T (Transit) bit set to 1 and the NSG (= Next Stage) field set to FullFeaturePhase. (For brevity, this is referred = to as "early" connection allocation.) The current iSER specification relax= es this requirement to allow an initiator to allocate connection resources = after it receives the final Login Response PDU from the target. (For brevi= ty, this is referred to as "late" connection allocation.) To prevent the p= otential problem caused by the target sending the number of unexpected iSCS= I control-type PDUs as determined by the MaxOustandingUnexpectedPDUs key al= lowed in the FullFeaturePhase when "late" connection allocation is practise= d, the iSER Hello exchanges MUST be used. This allows the initiator to all= ocate the connection resources before sending the iSER Hello Messages. The= iSERHelloRequired key is used by the initiator to determine if it is deali= ng with a target that supports the iSER Hello exchanges. In the following summary where "late" connection allocation is practised, a= n initiator that follows [RFC5046] is referred to as an "old" initiator; ot= herwise it is referred to as a "new" initiator. Similarly, a target that d= oes not support the iSERHelloRequired key (and responds with "NotUnderstood= " when negotiating the iSERHelloRequired key) is referred to as an "old" ta= rget; otherwise it is referred to as a "new" target. Note that an "old" ta= rget can still support the iSER Hello exchanges but this fact is not known = by the initiator. A "new" target can also respond with "No" when negotiati= ng the iSERHelloREquired key. In this case its beahvior with respect to "l= ate" connection allocation is similar to an "old" target. A "new" initiator will work fine with a "new" target. For an "old" initiator and an "old" target, the failure by the initiator to= handle the number of unexpected iSCSI control-type PDUs that are sent by t= he target before the buffer resources are allocated at the initiator can re= sult in the failure of the iSER session caused by closure of the underlying= RCaP connection. For the "old" target, there is known implementation that= sends one unexpected iSCSI control-type PDU after sending the final Login = Response and then waits awhile before sending the next one. This tends to = alleviate somewhat the buffer allocation problem at the initiator. For a "new" initiator and an "old" target, the failure by the initiator to = handle the number of unexpected iSCSI control-type PDUs that are sent by th= e target before the buffer resources are allocated at the initiator can res= ult in the failure of the iSER session caused by closure of the underlying = RCaP connection. A "new" initiator MAY choose to terminate the connection;= otherwise it SHOULD do one of the following: 1. Allocate the connection resources before sending the final Login Request= PDU. 2. Allocate one or more buffers for receiving unexpected control-type PDUs = from the target before sending the final Login Request PDU. This reduces t= he possibility of the unexpected control-type PDUs causing the RCaP connect= ion to close before the connection resources have been allocated. For an "old" initiator and a "new" target, if the iSERHelloRequired key is = not negotiated, a "new" target MUST still respond with the iSER HelloReply = Message when it receives the iSER Hello Message. If the iSERHelloRequired = key is negotiated to "No" or "NotUnderstood", a "new" target MAY choose to = terminate the connection; otherwise it SHOULD delay sending any unexpected = control-type PDUs until one of the following events has occurred: 1. A PDU is received from the initiator after it sends the final Login Resp= onse PDU. 2. A system configurable timeout period, say one second, has expired. Mike On Mon, Aug 13, 2012 at 3:27 PM, Black, David > wrote: Mike, Thanks for posting this new text, and I apologize for the delayed response = - too much "stuff" going on in Vancouver ... I think we could also use some additional text that summarizes the new/old,= old/new and new/new implementation interactions. Some comments on your posted text= ... > If the iSCSI Layer on the initiator side allocates the connection resourc= es > to support RCaP only after it receives the final Login Response PDU from = the > target, then it may not be able to handle the number of unexpected iSCSI > control-type PDUs (as declared by the MaxOutstandingUnexpectedPDUs key fr= om > the initiator) that can be sent by the target before the buffer resources > are allocated at the initiator side. In this case the iSERHelloRequired = key > MUST be negotiated to Yes so that the initiator can allocate the connecti= on > resources before sending the iSER Hello Message. See section 5.1.3 for m= ore > details. That "MUST be negotiated to Yes" seems wrong, as that can't be done if the = target responds with NotUnderstood. Also, that text doesn't provide a sufficientl= y blunt warning, IMHO - it ought to state that the consequences of "may not be able= to handle" include failure of the iSER session caused by closure of the underl= ying RCaP connection. > As the iSERHelloRequired key is not defined in [RFC5046], an initiator/ta= rget > must be able to deal with a target/initiator that does not understand the= key > or support the iSER Hello exchanges. That's correct, although I would change: "must be able to deal with" ->"needs to be able to deal with". > For a target, if the iSERHelloRequired key is not negotiated, it MUST sti= ll > respond with the iSER HelloReply Message when it receives the iSER Hello = Message. > If the iSERHelloRequired key is negotiated to No or NotUnderstood, a targ= et MAY > choose to terminate the connection; otherwise it SHOULD delay sending any > unexpected control-type PDUs until one of the following events has occurr= ed: > > 1. A PDU is received from the initiator after it sends the final Login Re= sponse PDU. > > 2. A system configurable timeout period, say one second, has expired. That's ok, I think additional text is needed somewhere to point out that th= ere is known target code that sends one PDU and then delays - that might go in the= additional summary text that I suggest above, and also should be included as part of t= he rationale for: > For an initiator, if the iSERHelloRequired key is negotiated to No or Not= Understood, > it MAY choose to terminate the connection; otherwise it SHOULD do one of = the following: > > 1. Allocate the connection resources before sending the final Login Reque= st PDU. > > 2. Allocate one or more buffers for receiving unexpected control-type PDU= s from > the target before sending the final Login Request PDU. This reduces the = possibility > of the unexpected control-type PDUs causing the RCaP connection to close = before > the connection resources have been allocated. Thanks, --David From: storm-bounces@ietf.org [mailto:storm-b= ounces@ietf.org] On Behalf Of Michael Ko Sent: Tuesday, July 24, 2012 8:27 PM To: storm@ietf.org Subject: [storm] Proposed text changes for the connection allocation proble= m These are the changes I intend to incorporate into the existing draft to so= lve the connection allocation problem by using the iSER Hello exchanges. P= lease comment. 5.1.1 Initiator Behavior original: If the outcome of the iSCSI negotiation is to enable iSER-assisted mode, th= en on the initiator side, prior to sending the Login Request with the T (Tr= ansit) bit set to 1 and the NSG (Next Stage) field set to FullFeaturePhase,= the iSCSI Layer MUST request the iSER Layer to allocate the connection res= ources necessary to support RCaP by invoking the Allocate_Connection_Resour= ces Operational Primitive. replacement: If the outcome of the iSCSI negotiation is to enable iSER-assisted mode, th= en on the initiator side, prior to sending the Login Request with the T (Tr= ansit) bit set to 1 and the NSG (Next Stage) field set to FullFeaturePhase,= the iSCSI Layer SHOULD request the iSER Layer to allocate the connection r= esources necessary to support RCaP by invoking the Allocate_Connection_Reso= urces Operational Primitive. deleted: If the iSER Layer at the initiator is successful in allocating the connecti= on resources necessary to support RCaP, the following events MUST occur in = the specified sequence: [The numbered items will be retained but without t= he numbering.] new: If the iSCSI Layer on the initiator side allocates the connection resources= to support RCaP only after it receives the final Login Response PDU from t= he target, then it may not be able to handle the number of unexpected iSCSI= control-type PDUs (as declared by the MaxOutstandingUnexpectedPDUs key fro= m the initiator) that can be sent by the target before the buffer resources= are allocated at the initiator side. In this case the iSERHelloRequired k= ey MUST be negotiated to Yes so that the initiator can allocate the connect= ion resources before sending the iSER Hello Message. See section 5.1.3 for= more details. 5.1.3 iSER Hello Exchange new: The connection resources can be allocated at the initiator side after it ha= s received the final Login Response PDU from the target. To prevent the po= tential problem caused by the target sending the number of unexpected iSCSI= control-type PDUs as determined by the MaxOustandingUnexpectedPDUs key all= owed in the FullFeaturePhase, the iSER Hello exchanges can be used. This a= llows the initiator to allocate the connection resources before sending the= iSER Hello Message. The iSERHelloRequired key is used by the initiator to= determine if it is dealing with a target that supports the iSER Hello exch= anges. As the iSERHelloRequired key is not defined in [RFC5046], an initiator/targ= et must be able to deal with a target/initiator that does not understand th= e key or support the iSER Hello exchanges. For a target, if the iSERHelloRequired key is not negotiated, it MUST still= respond with the iSER HelloReply Message when it recieves the iSER Hello M= essage. If the iSERHelloRequired key is negotiated to No or NotUnderstood,= a target MAY choose to terminate the connection; otherwise it SHOULD delay= sending any unexpected control-type PDUs until one of the following events= has occurred: 1. A PDU is received from the initiator after it sends the final Login Resp= onse PDU. 2. A system configurable timeout period, say one second, has expired. For an initiator, if the iSERHelloRequired key is negotiated to No or NotUn= derstood, it MAY choose to terminate the connection; otherwise it SHOULD do= one of the following: 1. Allocate the connection resources before sending the final Login Request= PDU. 2. Allocate one or more buffers for receiving unexpected control-type PDUs = from the target before sending the final Login Request PDU. This reduces t= he possibility of the unexpected control-type PDUs causing the RCaP connect= ion to close before the connection resources have been allocated. 10.1.3.4 iSER Protocol Errors original: Conversely, it is a protocol error if the iSER Hello Message is sent by the= iSER Layer at the initiator when iSERHelloRequired is negotiated to "No". new: Conversely, if the iSER Hello Message is sent by the iSER Layer at the init= iator when iSERHelloRequired is negotiated to "No", the iSER Layer at the t= arget MAY treat this as a protocol error or respond with an iSER HelloReply= Message. Mike --_000_8D3D17ACE214DC429325B2B98F3AE7120DCD1566MX15Acorpemccom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Mike,

 

I apol= ogize for the delay in getting to this.  I think your latest text is c= lose to what’s

needed, but I w= ant to rewrite the latter part of your second paragraph for clarity and to<= o:p>

remove a “MUST” that appe= ars easy to misinterpret (IMHO):

=  

OLD

   To prevent the potential problem caused by the target= sending the number of

   u= nexpected iSCSI control-type PDUs as determined by the MaxOustandingUnexpec= tedPDUs

   key allowed in t= he FullFeaturePhase when "late" connection allocation is practise= d,

   the iSER Hello exchan= ges MUST be used.  This allows the initiator to allocate the

   connection resources before sendi= ng the iSER Hello Messages.  The iSERHelloRequired

   key is used by the initiator to determine i= f it is dealing with a target that

&n= bsp;  supports the iSER Hello exchanges.

NEW
   An initiator that employs “late” c= onnection allocation may encounter problems (e.g.,

   RCaP connection closure) with a target that send= s unexpected iSCSI PDUs immediately

&= nbsp;  upon transitioning to Full Feature Phase, as allowed by the neg= otiated value of the

   th= e MaxOustandingUnexpectedPDUs key.  The only way to prevent this situa= tion in

   full gener= ality is to use iSER Hello Messages, as they enable the initiator to alloca= te

   its connection resour= ces before sending its iSER Hello Message.  The iSERHelloRequired=

   key is used by the initiator= to determine if it is dealing with a target that

   supports the iSER Hello exchanges.  Fortunat= ely, known iSER target implementations

  = transition into full feature phase, enabling an initiator workaround that = involves

   a smaller quant= ity of connection resources prior to full-feature phase, as explained<= /o:p>

   further below.

END

&nbs= p;

Would you please go ahead and submit a = revised version of the draft with the text changes

that we’ve agreed to.  I’ll want to review t= he changes one more time in the context of an

entire draft before (finally) sending it to our AD with a request = to publish it as an RFC.

 <= /o:p>

Thank you for your patience with this.

 

Thanks,
--David

 

<= span style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From: = Michael Ko [mailto:mkosjc@gmail.com]
Sent: Sunday, August 19, 20= 12 9:49 PM
To: Black, David
Cc: storm@ietf.org
Su= bject: Re: [storm] Proposed text changes for the connection allocation = problem

 

Changed “= must be able to deal with” to “needs to be able to deal with= 221; as suggested.

For the rest of the suggestions, I have created n= ew text that replaces the "new" text in section 5.1.3:

An = initiator that conforms to [RFC5046] allocates connection resources before = seding the Login Request with the T (Transit) bit set to 1 and the NSG (Nex= t Stage) field set to FullFeaturePhase.  (For brevity, this is referre= d to as "early" connection allocation.)  The current iSER sp= ecification relaxes this requirement to allow an initiator to allocate conn= ection resources after it receives the final Login Response PDU from the ta= rget.  (For brevity, this is referred to as "late" connectio= n allocation.)  To prevent the potential problem caused by the target = sending the number of unexpected iSCSI control-type PDUs as determined by t= he MaxOustandingUnexpectedPDUs key allowed in the FullFeaturePhase when &qu= ot;late" connection allocation is practised, the iSER Hello exchanges = MUST be used.  This allows the initiator to allocate the connection re= sources before sending the iSER Hello Messages.  The iSERHelloRequired= key is used by the initiator to determine if it is dealing with a target t= hat supports the iSER Hello exchanges.

In the following summary wher= e "late" connection allocation is practised, an initiator that fo= llows [RFC5046] is referred to as an "old" initiator; otherwise i= t is referred to as a "new" initiator.  Similarly, a target = that does not support the iSERHelloRequired key (and responds with "No= tUnderstood" when negotiating the iSERHelloRequired key) is referred t= o as an "old" target; otherwise it is referred to as a "new&= quot; target.  Note that an "old" target can still support t= he iSER Hello exchanges but this fact is not known by the initiator.  = A "new" target can also respond with "No" when negotiat= ing the iSERHelloREquired key.  In this case its beahvior with respect= to "late" connection allocation is similar to an "old"= target.

A "new" initiator will work fine with a "new= " target.

For an "old" initiator and an "old&quo= t; target, the failure by the initiator to handle the number of unexpected = iSCSI control-type PDUs that are sent by the target before the buffer resou= rces are allocated at the initiator can result in the failure of the iSER s= ession caused by closure of the underlying RCaP connection.  For the &= quot;old" target, there is known implementation that sends one unexpec= ted iSCSI control-type PDU after sending the final Login Response and then = waits awhile before sending the next one.  This tends to alleviate som= ewhat the buffer allocation problem at the initiator.

For a "ne= w" initiator and an "old" target, the failure by the initiat= or to handle the number of unexpected iSCSI control-type PDUs that are sent= by the target before the buffer resources are allocated at the initiator c= an result in the failure of the iSER session caused by closure of the under= lying RCaP connection.  A "new" initiator MAY choose to term= inate the connection; otherwise it SHOULD do one of the following:

1= . Allocate the connection resources before sending the final Login Request = PDU.

2. Allocate one or more buffers for receiving unexpected contro= l-type PDUs from the target before sending the final Login Request PDU.&nbs= p; This reduces the possibility of the unexpected control-type PDUs causing= the RCaP connection to close before the connection resources have been all= ocated.

For an "old" initiator and a "new" targe= t, if the iSERHelloRequired key is not negotiated, a "new" target= MUST still respond with the iSER HelloReply Message when it receives the i= SER Hello Message.  If the iSERHelloRequired key is negotiated to &quo= t;No" or "NotUnderstood", a "new" target MAY choos= e to terminate the connection; otherwise it SHOULD delay sending any unexpe= cted control-type PDUs until one of the following events has occurred:
<= br>1. A PDU is received from the initiator after it sends the final Login R= esponse PDU.

2. A system configurable timeout period, say one second= , has expired.

Mike

On Mon, = Aug 13, 2012 at 3:27 PM, Black, David <david.black@emc.com> wrote:

M= ike,

 

Thanks for posting this new text, and I= apologize for the delayed response -

too much “stuffR= 21; going on in Vancouver ...

 

I think we cou= ld also use some additional text that summarizes the new/old, old/new

and new/new implementation interactions.  Some comments on your= posted text ...

 

> If the iSCSI Layer on = the initiator side allocates the connection resources

=

> to s= upport RCaP only after it receives the final Login Response PDU from the

> target, then it may not be able to handle the number of unex= pected iSCSI

> control-type PDUs (as declared by the MaxOuts= tandingUnexpectedPDUs key from

> the initiator) that can be = sent by the target before the buffer resources

= > are allocat= ed at the initiator side.  In this case the iSERHelloRequired key

> MUST be negotiated to Yes so that the initiator can allocate t= he connection

> resources before sending the iSER Hello Mess= age.  See section 5.1.3 for more

> details.=

 

That “MUST be negotiated to Yes” seems wron= g, as that can’t be done if the target

<= span style=3D'font-size:10.0pt;font-family:"Courier New"'>responds with Not= Understood.  Also, that text doesn’t provide a sufficiently blun= t

warning, IMHO - it ought to state that the consequences of &#= 8220;may not be able to

handle” include failure of the iS= ER session caused by closure of the underlying

= RCaP connection.=

 

> As the iSERHelloRequired key is not de= fined in [RFC5046], an initiator/target

> must be able to de= al with a target/initiator that does not understand the key

> or s= upport the iSER Hello exchanges.

That’s correct, although= I would change:

“must be able to deal= with” ->“needs to be able to deal with”.<= /o:p>

=  

> For a target, if the iSERHelloRequired key is not n= egotiated, it MUST still
> respond with the iSER HelloReply Message w= hen it receives the iSER Hello Message.

> If the iSERHelloRe= quired key is negotiated to No or NotUnderstood, a target MAY

&= gt; choose to terminate the connection; otherwise it SHOULD delay sending a= ny

> unexpected control-type PDUs until one of the following= events has occurred:

> 

<= span style=3D'font-size:10.0pt;font-family:"Courier New"'>> 1. A PDU is = received from the initiator after it sends the final Login Response PDU.>
> 2. A system configurable timeout period, say one second, has = expired.

 

That’s ok, I think additional= text is needed somewhere to point out that there is

<= p class=3DMsoNormal style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:= auto'>known targ= et code that sends one PDU and then delays - that might go in the additiona= l

summary text that I suggest above, and also should be include= d as part of the

rationale for:

<= span style=3D'font-size:10.0pt;font-family:"Courier New"'> 

> it MAY choose to terminate the connection= ; otherwise it SHOULD do one of the following:
>
> 1. Allocate= the connection resources before sending the final Login Request PDU.
&g= t;
> 2. Allocate one or more buffers for receiving unexpected contro= l-type PDUs from

> the target before sending the final Login= Request PDU.  This reduces the possibility
> of the unexpected = control-type PDUs causing the RCaP connection to close before

> th= e connection resources have been allocated.

<= span style=3D'font-size:10.0pt;font-family:"Courier New"'> 

Thanks,
--David

 

From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf = Of Michael Ko
Sent: Tuesday, July 24, 2012 8:27 PM
To:<= /b> storm@ietf.org<= br>Subject: [storm] Proposed text changes for the connection allocat= ion problem

 

T= hese are the changes I intend to incorporate into the existing draft to sol= ve the connection allocation problem by using the iSER Hello exchanges.&nbs= p; Please comment.

5.1.1  Initiator Behavior

original:
If the outcome of the iSCSI negotiation is to enable iSER-assisted mo= de, then on the initiator side, prior to sending the Login Request with the= T (Transit) bit set to 1 and the NSG (Next Stage) field set to FullFeature= Phase, the iSCSI Layer MUST request the iSER Layer to allocate the connecti= on resources necessary to support RCaP by invoking the Allocate_Connection_= Resources Operational Primitive.

replacement:

If the outcome = of the iSCSI negotiation is to enable iSER-assisted mode, then on the initi= ator side, prior to sending the Login Request with the T (Transit) bit set = to 1 and the NSG (Next Stage) field set to FullFeaturePhase, the iSCSI Laye= r SHOULD request the iSER Layer to allocate the connection resources necess= ary to support RCaP by invoking the Allocate_Connection_Resources Operation= al Primitive.

deleted:

If the iSER Layer at the initiator is = successful in allocating the connection resources necessary to support RCaP= , the following events MUST occur in the specified sequence:  [The num= bered items will be retained but without the numbering.]

new:
If the iSCSI Layer on the initiator side allocates the connection resource= s to support RCaP only after it receives the final Login Response PDU from = the target, then it may not be able to handle the number of unexpected iSCS= I control-type PDUs (as declared by the MaxOutstandingUnexpectedPDUs key fr= om the initiator) that can be sent by the target before the buffer resource= s are allocated at the initiator side.  In this case the iSERHelloRequ= ired key MUST be negotiated to Yes so that the initiator can allocate the c= onnection resources before sending the iSER Hello Message.  See sectio= n 5.1.3 for more details.

5.1.3  iSER Hello Exchange

new= :

The connection resources can be allocated at the initiator side af= ter it has received the final Login Response PDU from the target.  To = prevent the potential problem caused by the target sending the number of un= expected iSCSI control-type PDUs as determined by the MaxOustandingUnexpect= edPDUs key allowed in the FullFeaturePhase, the iSER Hello exchanges can be= used.  This allows the initiator to allocate the connection resources= before sending the iSER Hello Message.  The iSERHelloRequired key is = used by the initiator to determine if it is dealing with a target that supp= orts the iSER Hello exchanges.

As the iSERHelloRequired key is not d= efined in [RFC5046], an initiator/target must be able to deal with a target= /initiator that does not understand the key or support the iSER Hello excha= nges.

For a target, if the iSERHelloRequired key is not negotiated, = it MUST still respond with the iSER HelloReply Message when it recieves the= iSER Hello Message.  If the iSERHelloRequired key is negotiated to No= or NotUnderstood, a target MAY choose to terminate the connection; otherwi= se it SHOULD delay sending any unexpected control-type PDUs until one of th= e following events has occurred:

1. A PDU is received from the initi= ator after it sends the final Login Response PDU.

2. A system config= urable timeout period, say one second, has expired.

For an initiator= , if the iSERHelloRequired key is negotiated to No or NotUnderstood, it MAY= choose to terminate the connection; otherwise it SHOULD do one of the foll= owing:

1. Allocate the connection resources before sending the final= Login Request PDU.

2. Allocate one or more buffers for receiving un= expected control-type PDUs from the target before sending the final Login R= equest PDU.  This reduces the possibility of the unexpected control-ty= pe PDUs causing the RCaP connection to close before the connection resource= s have been allocated.

10.1.3.4 iSER Protocol Errors

original= :

Conversely, it is a protocol error if the iSER Hello Message is se= nt by the iSER Layer at the initiator when iSERHelloRequired is negotiated = to "No".

new:

Conversely, if the iSER Hello Message= is sent by the iSER Layer at the initiator when iSERHelloRequired is negot= iated to "No", the iSER Layer at the target MAY treat this as a p= rotocol error or respond with an iSER HelloReply Message.

Mike<= /o:p>

 

= = --_000_8D3D17ACE214DC429325B2B98F3AE7120DCD1566MX15Acorpemccom_-- From david.black@emc.com Fri Sep 14 18:06:28 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A38BE21F85AA for ; Fri, 14 Sep 2012 18:06:28 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mtKRygrYTggH for ; Fri, 14 Sep 2012 18:06:28 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id DCC3321F85A8 for ; Fri, 14 Sep 2012 18:06:27 -0700 (PDT) Received: from hop04-l1d11-si02.isus.emc.com (HOP04-L1D11-SI02.isus.emc.com [10.254.111.55]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8F16Q95001689 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 14 Sep 2012 21:06:27 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.221.145]) by hop04-l1d11-si02.isus.emc.com (RSA Interceptor) for ; Fri, 14 Sep 2012 21:06:16 -0400 Received: from mxhub25.corp.emc.com (mxhub25.corp.emc.com [10.254.110.181]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8F16FmX032116 for ; Fri, 14 Sep 2012 21:06:15 -0400 Received: from mxhub40.corp.emc.com (128.222.70.107) by mxhub25.corp.emc.com (10.254.110.181) with Microsoft SMTP Server (TLS) id 8.3.213.0; Fri, 14 Sep 2012 21:06:15 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub40.corp.emc.com ([128.222.70.107]) with mapi; Fri, 14 Sep 2012 21:06:15 -0400 From: "Black, David" To: "storm@ietf.org" Date: Fri, 14 Sep 2012 21:06:13 -0400 Thread-Topic: iSCSI MIB: Functional enhancements proposal Thread-Index: Ac2S3k0VAOHDeEHlQP66E4/tbPSxkA== Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DCD1567@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: [storm] iSCSI MIB: Functional enhancements proposal X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Sep 2012 01:06:28 -0000 The iSCSI MIB has been waiting for a WG decision about what to incorporate = from the list of functional enhancements suggested by the MIB Doctor review. With , based on consulting with the MIB draft editors, I = offer the following initial proposal for the WG's consideration ... in the hope o= f getting to a decision. --- Make the following functional enhancements: [1] Add NOP counters at iSCSI session scope for heartbeat tracking [2] Add a port number to the iscsiTgtLoginFailure and iscsiIntrLoginFailure= notifications, and to the last failure info in iscsiInitiatorAttributesEntry. [3] Add a description string to the iSCSI portal: iscsiPortalDescr OBJECT-TYPE SYNTAX SnmpAdminString MAX-ACCESS read-only STATUS current DESCRIPTION "A UTF-8 string, determined by the implementation to describe the iSCSI portal. When only a single instance is present, this object may be set to the zero-length string; with multiple iSCSI portals, it may be used in an implementation-dependent manner to describe the=20 respective portal, and could include information such as HBA model, description and version or software driver and version." [4] Support "Target Unmapped" session failure reporting in the iscsiInstSes= sionFailure notification: Editors: FailureType is an OID pointing to a stat in iscsiInstSsnErrorStats= Table, so we would need to add this: iscsiInstSsnTgtUnmappedErrors Counter32 [5] Add a port number to iscsiPortalAttributesEntry [6] Add a timeout counter to iscsiInitiatorLogoutStatsEntry to distinguish = implicit logouts caused by timeouts from implicit logouts caused by other reasons. --- Do not make any other functional enhancements, including the following [A] Summary notifications for large numbers of targets or initiators. (At = most a single summary with a count would be appropriate for each notification t= ype). [B] RowPointer or other connection to transport state as part of iscsiInstanceSsnErrorStatsEntry (can be found indirectly in current struct= ure). [C] Support for reporting more than two digest algorithms (only one is stan= dardized). [D] Additional list of authentication methods for negotiation (already avai= lable based on reporting of authorized entities). [E] Additional support for mutual authentication (check the MIBs at both en= ds instead). [F] Support for digest provisioning at the iSCSI node level (use iSCSI port= al instead). [G] Additional session reporting, including digest, error recovery level, s= ession type and additional counters for missing PDUs, data in PDUs, data out PDUs, asy= nc PDUs. [H] Report initial remote IP address and port for connections in addition t= o actual IP address and port - this would be for situations in which a redirection = has occurred. [I] Report list of discovered targets (out of scope for this MIB). -------------------- Of the above list, I'd be particularly interested in comments on [A], [G] a= nd [H] - if these were added to the MIB, would they be likely to be implemented and = used? Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- From internet-drafts@ietf.org Sun Sep 16 10:14:20 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C2BDC21F852B; Sun, 16 Sep 2012 10:14:20 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8pOe+AgOIhB3; Sun, 16 Sep 2012 10:14:19 -0700 (PDT) Received: from ietfa.amsl.com (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A186921F84EC; Sun, 16 Sep 2012 10:13:53 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: internet-drafts@ietf.org To: i-d-announce@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 4.34 Message-ID: <20120916171353.21741.88519.idtracker@ietfa.amsl.com> Date: Sun, 16 Sep 2012 10:13:53 -0700 Cc: storm@ietf.org Subject: [storm] I-D Action: draft-ietf-storm-iser-12.txt X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Sep 2012 17:14:21 -0000 A New Internet-Draft is available from the on-line Internet-Drafts director= ies. This draft is a work item of the STORage Maintenance Working Group of the = IETF. Title : iSCSI Extensions for RDMA Specification Author(s) : Michael Ko Alexander Nezhinsky Filename : draft-ietf-storm-iser-12.txt Pages : 97 Date : 2012-09-16 Abstract: iSCSI Extensions for RDMA provides the RDMA data transfer capability to iSCSI by layering iSCSI on top of an RDMA-Capable Protocol. An RDMA-Capable Protocol provides RDMA Read and Write services, which enable data to be transferred directly into SCSI I/O Buffers without intermediate data copies. This document describes the extensions to the iSCSI protocol to support RDMA services as provided by an RDMA- Capable Protocol. This document obsoletes RFC 5046. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-storm-iser There's also a htmlized version available at: http://tools.ietf.org/html/draft-ietf-storm-iser-12 A diff from the previous version is available at: http://www.ietf.org/rfcdiff?url2=3Ddraft-ietf-storm-iser-12 Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ From mkosjc@gmail.com Sun Sep 16 10:19:40 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E059721F8530 for ; Sun, 16 Sep 2012 10:19:40 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.598 X-Spam-Level: X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9OCmbpPuppGh for ; Sun, 16 Sep 2012 10:19:40 -0700 (PDT) Received: from mail-qc0-f172.google.com (mail-qc0-f172.google.com [209.85.216.172]) by ietfa.amsl.com (Postfix) with ESMTP id A4E7521F852E for ; Sun, 16 Sep 2012 10:19:39 -0700 (PDT) Received: by qcac10 with SMTP id c10so4631686qca.31 for ; Sun, 16 Sep 2012 10:19:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=bQ4PcmyogK5naPheh75sL+yQ1aEZ61C/ZhhHATU459A=; b=DfDUDqw1IEsGl69CxhBIS5CNuRlKg5rPSgNIHboPpLdWnIy5HnXWDDTrO2bgp8584t /K5WjXJg3/XhiBDain/OjefhJhQPzp1udf7Z/vgNB2SkEFMsbqHp1FrF8CKKfIaukkD9 KtYccl7etwjOlSwgfwMpzlMmnH/zY31z42lzLqFCOXALXVQZc9puVwafg4Gg0e6fazDA AoImCAUHLjnIukvTFVXHhoqq474yGjyii92zP/j2ftoUs6WOTWMirJK3D0+vCWHqLupX dWWbjucmDQMTw7vUgkNqHR9gYw1LNmYAjJWbN6S1E7RCX4ZN/CXkW0D4rZvt8/WfB9eY 8JAA== MIME-Version: 1.0 Received: by 10.224.203.193 with SMTP id fj1mr22219569qab.13.1347815979193; Sun, 16 Sep 2012 10:19:39 -0700 (PDT) Received: by 10.229.231.205 with HTTP; Sun, 16 Sep 2012 10:19:39 -0700 (PDT) In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DCD1566@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71208F127E3@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DCD1566@MX15A.corp.emc.com> Date: Sun, 16 Sep 2012 10:19:39 -0700 Message-ID: From: Michael Ko To: "Black, David" Content-Type: multipart/alternative; boundary=20cf30051232a994f404c9d4dc2c Cc: "storm@ietf.org" Subject: Re: [storm] Proposed text changes for the connection allocation problem X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Sep 2012 17:19:41 -0000 --20cf30051232a994f404c9d4dc2c Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable David, I have submiited a new version of the iSER draft incorporating all the latest changes so far, including your recommendations below. I have also reword item #7 in the summary of changes to reflect the new use of the iSERHelloRequired key. Mike On Fri, Sep 14, 2012 at 5:25 PM, Black, David wrote: > Mike,**** > > ** ** > > I apologize for the delay in getting to this. I think your latest text i= s > close to what=92s**** > > needed, but I want to rewrite the latter part of your second paragraph fo= r > clarity and to**** > > remove a =93MUST=94 that appears easy to misinterpret (IMHO):**** > > ** ** > > OLD**** > > To prevent the potential problem caused by the target sending the > number of**** > > unexpected iSCSI control-type PDUs as determined by the > MaxOustandingUnexpectedPDUs**** > > key allowed in the FullFeaturePhase when "late" connection allocation > is practised,**** > > the iSER Hello exchanges MUST be used. This allows the initiator to > allocate the**** > > connection resources before sending the iSER Hello Messages. The > iSERHelloRequired**** > > key is used by the initiator to determine if it is dealing with a > target that**** > > supports the iSER Hello exchanges.**** > > NEW > An initiator that employs =93late=94 connection allocation may encount= er > problems (e.g.,**** > > RCaP connection closure) with a target that sends unexpected iSCSI PDU= s > immediately**** > > upon transitioning to Full Feature Phase, as allowed by the negotiated > value of the**** > > the MaxOustandingUnexpectedPDUs key. The only way to prevent this > situation in **** > > full generality is to use iSER Hello Messages, as they enable the > initiator to allocate**** > > its connection resources before sending its iSER Hello Message. The > iSERHelloRequired**** > > key is used by the initiator to determine if it is dealing with a > target that**** > > supports the iSER Hello exchanges. Fortunately, known iSER target > implementations**** > > do not take full advantage of the number of allowed unexpected PDUs > immediately upon**** > > transition into full feature phase, enabling an initiator workaround > that involves**** > > a smaller quantity of connection resources prior to full-feature phase= , > as explained**** > > further below.**** > > END**** > > ** ** > > Would you please go ahead and submit a revised version of the draft with > the text changes**** > > that we=92ve agreed to. I=92ll want to review the changes one more time = in > the context of an**** > > entire draft before (finally) sending it to our AD with a request to > publish it as an RFC.**** > > ** ** > > Thank you for your patience with this.**** > > ** ** > > Thanks, > --David**** > --20cf30051232a994f404c9d4dc2c Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable David,

I have submiited a new version of the iSER draft incorporatin= g all the latest changes so far, including your recommendations below.=A0 I= have also reword item #7 in the summary of changes to reflect the new use = of the iSERHelloRequired key.

Mike

On Fri, Sep 14, 2012 at 5:25 PM,= Black, David <david.black@emc.com> wrote:

Mi= ke,

=A0

I apologize for the delay in getting to this.=A0 I think y= our latest text is close to what=92s

needed= , but I want to rewrite the latter part of your second paragraph for clarit= y and to

remove a =93MUST=94 that = appears easy to misinterpret (IMHO):

=A0

OLD=

=A0=A0 To prevent the potential problem caused by the targ= et sending the number of

=A0=A0 un= expected iSCSI control-type PDUs as determined by the MaxOustandingUnexpect= edPDUs

=A0=A0 key allowed in the FullFeaturePhase when "late= " connection allocation is practised,

=A0=A0= the iSER Hello exchanges MUST be used.=A0 This allows the initiator to all= ocate the

=A0=A0 connection resour= ces before sending the iSER Hello Messages.=A0 The iSERHelloRequired=

=A0=A0 key is used by the initiator to determine if it is = dealing with a target that

<= span style=3D"font-size:10.0pt;font-family:"Courier New"">=A0=A0 = supports the iSER Hello exchanges.

NEW
=A0=A0 An initiator that employs =93late=94 connect= ion allocation may encounter problems (e.g.,

=A0=A0= RCaP connection closure) with a target that sends unexpected iSCSI PDUs im= mediately

=A0=A0 upon transitionin= g to Full Feature Phase, as allowed by the negotiated value of the

=A0=A0 the MaxOustandingUnexpectedPDUs key.=A0 The only wa= y to prevent this situation in

= =A0=A0=A0full generality is to use iSER Hello Messages, as they enable the = initiator to allocate

=A0=A0 its connection resources before sending its iSER He= llo Message.=A0 The iSERHelloRequired

=A0=A0= key is used by the initiator to determine if it is dealing with a target t= hat

=A0=A0 supports the iSER Hello= exchanges.=A0 Fortunately, known iSER target implementations=

=A0=A0 do not take full advantage of the number of allowed= unexpected PDUs immediately upon

=A0=A0= transition into full feature phase, enabling an initiator workaround that = involves

=A0=A0 a smaller quantity= of connection resources prior to full-feature phase, as explained

=A0=A0 further below.

END

=A0

Would you ple= ase go ahead and submit a revised version of the draft with the text change= s

that we=92ve agreed to.=A0 I=92ll want to review the chang= es one more time in the context of an

entire= draft before (finally) sending it to our AD with a request to publish it a= s an RFC.

=A0=

Thank you for your patience with this.

=A0

Thanks,
--David

--20cf30051232a994f404c9d4dc2c-- From david.black@emc.com Mon Sep 17 08:30:23 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BAEBA21F867F for ; Mon, 17 Sep 2012 08:30:23 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.598 X-Spam-Level: X-Spam-Status: No, score=-102.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id In4Rj01GsZsO for ; Mon, 17 Sep 2012 08:30:19 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 01E5021F8610 for ; Mon, 17 Sep 2012 08:30:18 -0700 (PDT) Received: from hop04-l1d11-si02.isus.emc.com (HOP04-L1D11-SI02.isus.emc.com [10.254.111.55]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8HFUDNh017462 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 17 Sep 2012 11:30:16 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.221.145]) by hop04-l1d11-si02.isus.emc.com (RSA Interceptor); Mon, 17 Sep 2012 11:29:51 -0400 Received: from mxhub04.corp.emc.com (mxhub04.corp.emc.com [10.254.141.106]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8HFTptq025381; Mon, 17 Sep 2012 11:29:51 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub04.corp.emc.com ([10.254.141.106]) with mapi; Mon, 17 Sep 2012 11:29:50 -0400 From: "Black, David" To: Michael Ko Date: Mon, 17 Sep 2012 11:29:49 -0400 Thread-Topic: iSER draft progress! Thread-Index: Ac2UL4Vi7wcV05IqSwSkpSuWNXVIoAAuIQ0A Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DCD16C7@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71208F127E3@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DCD1566@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/mixed; boundary="_004_8D3D17ACE214DC429325B2B98F3AE7120DCD16C7MX15Acorpemccom_" MIME-Version: 1.0 X-EMM-MHVC: 1 Cc: "storm@ietf.org" Subject: [storm] iSER draft progress! X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Sep 2012 15:30:23 -0000 --_004_8D3D17ACE214DC429325B2B98F3AE7120DCD16C7MX15Acorpemccom_ Content-Type: multipart/alternative; boundary="_000_8D3D17ACE214DC429325B2B98F3AE7120DCD16C7MX15Acorpemccom_" --_000_8D3D17ACE214DC429325B2B98F3AE7120DCD16C7MX15Acorpemccom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Mike, That new -12 version of the iSER draft looks fine, so I've sent it onward t= o our AD (Martin) for further review, along with a minor change to the PROT= O writeup (new writeup is attached). I do hope that this third attempt to = complete this stage of storm WG technical work on iSER turns out to be the = charm :). Thank you for the quick turn-around on the new draft version. Many thanks to everyone who's contributed to working through the iSER issue= s since the start of the year. Although the delays to the iSER draft have = not been ideal, they've occurred for good reasons, as we do need to make su= re that the technical content of the draft is actually correct and matches = the implementations. Also - RDMAP draft authors - this is your heads-up that I will get to takin= g a look at the RDMAP draft in the next week or so ... Thanks, --David (storm WG co-chair) From: Michael Ko [mailto:mkosjc@gmail.com] Sent: Sunday, September 16, 2012 1:20 PM To: Black, David Cc: storm@ietf.org Subject: Re: [storm] Proposed text changes for the connection allocation pr= oblem David, I have submiited a new version of the iSER draft incorporating all the late= st changes so far, including your recommendations below. I have also rewor= d item #7 in the summary of changes to reflect the new use of the iSERHello= Required key. Mike On Fri, Sep 14, 2012 at 5:25 PM, Black, David > wrote: Mike, I apologize for the delay in getting to this. I think your latest text is = close to what's needed, but I want to rewrite the latter part of your second paragraph for = clarity and to remove a "MUST" that appears easy to misinterpret (IMHO): OLD To prevent the potential problem caused by the target sending the number= of unexpected iSCSI control-type PDUs as determined by the MaxOustandingUne= xpectedPDUs key allowed in the FullFeaturePhase when "late" connection allocation is= practised, the iSER Hello exchanges MUST be used. This allows the initiator to all= ocate the connection resources before sending the iSER Hello Messages. The iSERHe= lloRequired key is used by the initiator to determine if it is dealing with a target= that supports the iSER Hello exchanges. NEW An initiator that employs "late" connection allocation may encounter pro= blems (e.g., RCaP connection closure) with a target that sends unexpected iSCSI PDUs = immediately upon transitioning to Full Feature Phase, as allowed by the negotiated v= alue of the the MaxOustandingUnexpectedPDUs key. The only way to prevent this situa= tion in full generality is to use iSER Hello Messages, as they enable the initia= tor to allocate its connection resources before sending its iSER Hello Message. The iSE= RHelloRequired key is used by the initiator to determine if it is dealing with a target= that supports the iSER Hello exchanges. Fortunately, known iSER target imple= mentations do not take full advantage of the number of allowed unexpected PDUs imme= diately upon transition into full feature phase, enabling an initiator workaround tha= t involves a smaller quantity of connection resources prior to full-feature phase, = as explained further below. END Would you please go ahead and submit a revised version of the draft with th= e text changes that we've agreed to. I'll want to review the changes one more time in the= context of an entire draft before (finally) sending it to our AD with a request to publis= h it as an RFC. Thank you for your patience with this. Thanks, --David --_000_8D3D17ACE214DC429325B2B98F3AE7120DCD16C7MX15Acorpemccom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Mike,

 

That n= ew -12 version of the iSER draft looks fine, so I’ve sent it onward t= o our AD (Martin) for further review, along with a minor change to the PROT= O writeup (new writeup is attached).  I do hope that this third attemp= t to complete this stage of storm WG technical work on iSER turns out to be= the charm J.  Thank you for the quick turn-around on the new draft = version.

 

=

Many thanks to everyone who’s contributed to working= through the iSER issues since the start of the year.  Although the de= lays to the iSER draft have not been ideal, they’ve occurred for good= reasons, as we do need to make sure that the technical content of the draf= t is actually correct and matches the implementations.

 

Also - RDMAP= draft authors - this is your heads-up that I will get to taking a look at = the RDMAP draft in the next week or so ...

 

Thanks,
--David = (storm WG co-chair)

&nb= sp;

From: Michael K= o [mailto:mkosjc@gmail.com]
Sent: Sunday, September 16, 2012 1:2= 0 PM
To: Black, David
Cc: storm@ietf.org
Subject:= Re: [storm] Proposed text changes for the connection allocation proble= m

 

David,

I have = submiited a new version of the iSER draft incorporating all the latest chan= ges so far, including your recommendations below.  I have also reword = item #7 in the summary of changes to reflect the new use of the iSERHelloRe= quired key.

Mike

On Fri, Sep= 14, 2012 at 5:25 PM, Black, David <david.black@emc.com> wrote:

=

Mike= ,

 

I apologize for the delay in getting to th= is.  I think your latest text is close to what’s

nee= ded, but I want to rewrite the latter part of your second paragraph for cla= rity and to

remove a “MUST” that appears easy to mi= sinterpret (IMHO):

 

OLD

=

 &nb= sp; To prevent the potential problem caused by the target sending the numbe= r of

   unexpected iSCSI control-type PDUs as determi= ned by the MaxOustandingUnexpectedPDUs

   key allowed= in the FullFeaturePhase when "late" connection allocation is pra= ctised,

   the iSER Hello exchanges MUST be used.&nbs= p; This allows the initiator to allocate the

<= span style=3D'font-size:10.0pt;font-family:"Courier New"'>   conn= ection resources before sending the iSER Hello Messages.  The iSERHell= oRequired

   key is used by the initiator to determin= e if it is dealing with a target that

   supports the= iSER Hello exchanges.

NEW
   An initiator that em= ploys “late” connection allocation may encounter problems (e.g.= ,

   RCaP connection closure) with a target that send= s unexpected iSCSI PDUs immediately

   upon transitio= ning to Full Feature Phase, as allowed by the negotiated value of the

   the MaxOustandingUnexpectedPDUs key.  The only way= to prevent this situation in

   full generalit= y is to use iSER Hello Messages, as they enable the initiator to allocate

   its connection resources before sending its iSER He= llo Message.  The iSERHelloRequired

   key is us= ed by the initiator to determine if it is dealing with a target that=

   supports the iSER Hello exchanges.  Fortunately, kn= own iSER target implementations

   do not take full a= dvantage of the number of allowed unexpected PDUs immediately upon

   transition into full feature phase, enabling an initiator = workaround that involves

   a smaller quantity of con= nection resources prior to full-feature phase, as explained

&nb= sp;  further below.

END

 

Wou= ld you please go ahead and submit a revised version of the draft with the t= ext changes

that we’ve agreed to.  I’ll want t= o review the changes one more time in the context of an

entire = draft before (finally) sending it to our AD with a request to publish it as= an RFC.

 

Thank you for your patience with th= is.

 

Thanks,
--David

= --_000_8D3D17ACE214DC429325B2B98F3AE7120DCD16C7MX15Acorpemccom_-- --_004_8D3D17ACE214DC429325B2B98F3AE7120DCD16C7MX15Acorpemccom_ Content-Type: text/plain; name="iSER Proto Writeup v2.txt" Content-Description: iSER Proto Writeup v2.txt Content-Disposition: attachment; filename="iSER Proto Writeup v2.txt"; size=8379; creation-date="Mon, 17 Sep 2012 14:58:49 GMT"; modification-date="Mon, 17 Sep 2012 15:13:01 GMT" Content-Transfer-Encoding: base64 DQpQUk9UTyB3cml0ZXVwOg0KDQogICAgICAgICAgICAgICAgICBpU0NTSSBFeHRlbnNpb25zIGZv ciBSRE1BIFNwZWNpZmljYXRpb24NCiAgICAgICAgICAgICAgICAgICAgICAgZHJhZnQtaWV0Zi1z dG9ybS1pc2VyLTEyLnR4dA0KDQpQUk9UTyBzaGVwaGVyZDogRGF2aWQgTC4gQmxhY2sgKFNUT1JN IFdHIENvLUNoYWlyKQ0KLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQoNCigxKSBXaGF0IHR5cGUgb2YgUkZDIGlz IGJlaW5nIHJlcXVlc3RlZCAoQkNQLCBQcm9wb3NlZCBTdGFuZGFyZCwNCkludGVybmV0IFN0YW5k YXJkLCBJbmZvcm1hdGlvbmFsLCBFeHBlcmltZW50YWwsIG9yIEhpc3RvcmljKT8gIFdoeQ0KaXMg dGhpcyB0aGUgcHJvcGVyIHR5cGUgb2YgUkZDPyAgSXMgdGhpcyB0eXBlIG9mIFJGQyBpbmRpY2F0 ZWQgaW4gdGhlDQp0aXRsZSBwYWdlIGhlYWRlcj8NCg0KICAgUHJvcG9zZWQgU3RhbmRhcmQgUkZD IGlzIHJlcXVlc3RlZCBiZWNhdXNlIHRoaXMgZHJhZnQgdXBkYXRlcyBhbmQNCiAgIHJlcGxhY2Vz IFJGQyA1MDQ2LCBhIFByb3Bvc2VkIFN0YW5kYXJkIFJGQy4gIFByb3Bvc2VkIFN0YW5kYXJkIGlz DQogICBpbmRpY2F0ZWQgYXMgdGhlIGludGVuZGVkIHN0YXR1cyBpcyBpbiB0aGUgdGl0bGUgcGFn ZSBoZWFkZXIuDQoNCigyKSBUaGUgSUVTRyBhcHByb3ZhbCBhbm5vdW5jZW1lbnQgaW5jbHVkZXMg YSBEb2N1bWVudCBBbm5vdW5jZW1lbnQNCldyaXRlLVVwLiBQbGVhc2UgcHJvdmlkZSBzdWNoIGEg RG9jdW1lbnQgQW5ub3VuY2VtZW50IFdyaXRlLVVwLiBSZWNlbnQNCmV4YW1wbGVzIGNhbiBiZSBm b3VuZCBpbiB0aGUgIkFjdGlvbiIgYW5ub3VuY2VtZW50cyBmb3IgYXBwcm92ZWQNCmRvY3VtZW50 cy4gVGhlIGFwcHJvdmFsIGFubm91bmNlbWVudCBjb250YWlucyB0aGUgZm9sbG93aW5nIHNlY3Rp b25zOg0KDQpUZWNobmljYWwgU3VtbWFyeQ0KDQogICBpU0NTSSBFeHRlbnNpb25zIGZvciBSRE1B IHByb3ZpZGVzIHRoZSBSRE1BIGRhdGEgdHJhbnNmZXIgY2FwYWJpbGl0eQ0KICAgdG8gaVNDU0kg YnkgbGF5ZXJpbmcgaVNDU0kgb24gdG9wIG9mIGFuIFJETUEtQ2FwYWJsZSBQcm90b2NvbC4gIEFu DQogICBSRE1BLUNhcGFibGUgUHJvdG9jb2wgcHJvdmlkZXMgUkRNQSBSZWFkIGFuZCBXcml0ZSBz ZXJ2aWNlcywgd2hpY2gNCiAgIGVuYWJsZSBkYXRhIHRvIGJlIHRyYW5zZmVycmVkIGRpcmVjdGx5 IGludG8gU0NTSSBJL08gQnVmZmVycyB3aXRob3V0DQogICBpbnRlcm1lZGlhdGUgZGF0YSBjb3Bp ZXMuICBUaGlzIGRvY3VtZW50IGRlc2NyaWJlcyB0aGUgZXh0ZW5zaW9ucyB0bw0KICAgdGhlIGlT Q1NJIHByb3RvY29sIHRvIHN1cHBvcnQgUkRNQSBzZXJ2aWNlcyBhcyBwcm92aWRlZCBieSBhbiBS RE1BLQ0KICAgQ2FwYWJsZSBQcm90b2NvbC4NCg0KV29ya2luZyBHcm91cCBTdW1tYXJ5DQoNCiAg IFRoaXMgZG9jdW1lbnQgaXMgYSBtaW5vciB1cGRhdGUgdG8gUkZDIDUwNDYsIHByaW1hcmlseSB0 byByZWZsZWN0DQogICB3aGF0IGhhcyBhY3R1YWxseSBiZWVuIGRvbmUgaW4gaW1wbGVtZW50YXRp b25zLiAgV0cgTGFzdCBDYWxsIHR1cm5lZA0KICAgdXAgc2V2ZXJhbCBpc3N1ZXMgYXJvdW5kIHJl bGF4aW5nIFJGQyA1MDQ2J3MgcmVxdWlyZW1lbnRzIGJhc2VkIG9uIHdoYXQNCiAgIGltcGxlbWVu dGF0aW9ucyBoYXZlIGRvbmUuICBUaGVzZSBpc3N1ZXMgaW52b2x2ZWQgdXNlIG9mIFNlbmQgbWVz c2FnZQ0KICAgdHlwZXMgdGhhdCBoYXZlIHNpZGUgZWZmZWN0cyAoaW1wbGVtZW50YXRpb25zIG9m dGVuIGRvIG5vdCB1c2UgdGhlc2UpDQogICBhbmQgdGhlIGNvbnNlcXVlY2VzIG9mIGRlbGF5ZWQg cmVzb3VyY2UgYWxsb2NhdGlvbiAoaW1wbGVtZW50YXRpb25zDQogICBoYXZlIHJ1biBpbnRvIGEg cmFjZSBjb25kaXRpb24gdGhhdCBjYW4gdGVybWluYXRlIGFuIGlTRVIgY29ubmVjdGlvbg0KICAg aWYgbWVhc3VyZXMgdG8gYXZvaWQgaXQgYXJlIG5vdCB0YWtlbikuICBBbGwgb2YgdGhlc2UgaXNz dWVzIGhhdmUNCiAgIGJlZW4gcmVzb2x2ZWQgaW4gdGhlIGN1cnJlbnQgdmVyc2lvbiBvZiB0aGlz IGRvY3VtZW50LCBhbHRob3VnaA0KICAgdGhlIHJhY2UgY29uZGl0aW9uIGF2b2lkYW5jZSBpcyBu b3QgcGVyZmVjdCBkdWUgdG8gdGhlIG5lZWQgdG8NCiAgIGNvcGUgd2lodCBjdXJyZW50ICJydW5u aW5nIGNvZGUiIGluIGltcGxlbWVudGF0aW9ucy4NCg0KRG9jdW1lbnQgUXVhbGl0eQ0KDQogICBU aGVyZSBhcmUgbXVsdGlwbGUgaW1wbGVtZW50YXRpb25zIG9mIHRoZSBpU0VSIHByb3RvY29sOw0K ICAgdGhlIHByaW1hcnkgcHVycG9zZSBvZiB0aGlzIGRvY3VtZW50IGlzIHRvIHJlZmxlY3QgaW1w bGVtZW50YXRpb24NCiAgIGV4cGVyaWVuY2Ugc28gdGhhdCB0aGUgaVNFUiBwcm90b2NvbCBzcGVj aWZpY2F0aW9uIG1hdGNoZXMgdGhlDQogICAicnVubmluZyBjb2RlIi4gIEhlbWFsIFNoYWgncyBy ZXZpZXcgb2YgdGhlIGRvY3VtZW50IHJlc3VsdGVkIGluDQogICBzb21lIGltcG9ydGFudCBjaGFu Z2VzIGluIHRoZSB0ZXh0IGRlc2NyaWJpbmcgdXNlIG9mIHZlcnNpb25zIG9mDQogICB0aGUgU2Vu ZCBtZXNzYWdlLiAgQWxleGFuZGVyIE5lc2hpbnNreSByZXBvcnRlZCB0aGUgcmVzb3VyY2UNCiAg IGFsbG9jYXRpb24gcHJvYmxlbSBzZWVuIGluIGltcGxtZW5ldGF0aW9ucyBhbmQgcHJvdmlkZWQg dmFsdWFibGUNCiAgIGhlbHAgaW4gd29ya2luZyBvdXQgYW4gYXBwcm9hY2ggdGhhdCBlbmNvbXBh c3NlcyBib2h0IHRoZSAicmlnaHQNCiAgIHRoaW5nIiB0byBkbyBnb2luZyBmb3J3YXJkIGFuZCBu ZWNlc3NhcnkgbWVhc3VyZXMgdG8gY29wZSB3aXRoDQogICBjdXJyZW50ICJydW5uaW5nIGNvZGUi IGluIGltcGxlbWVudGF0aW9ucy4NCg0KUGVyc29ubmVsDQoNCiAgIERvY3VtZW50IFNoZXBoZXJk OiBEYXZpZCBCbGFjayAoc3Rvcm0gV0cgY28tY2hhaXIpDQogICBSZXNwb25zaWJsZSBBcmVhIERp cmVjdG9yOiBNYXJ0aW4gU3RpZW1lcmxpbmcgKFRyYW5zcG9ydCkNCg0KKDMpIEJyaWVmbHkgZGVz Y3JpYmUgdGhlIHJldmlldyBvZiB0aGlzIGRvY3VtZW50IHRoYXQgd2FzIHBlcmZvcm1lZCBieQ0K dGhlIERvY3VtZW50IFNoZXBoZXJkLiAgSWYgdGhpcyB2ZXJzaW9uIG9mIHRoZSBkb2N1bWVudCBp cyBub3QgcmVhZHkNCmZvciBwdWJsaWNhdGlvbiwgcGxlYXNlIGV4cGxhaW4gd2h5IHRoZSBkb2N1 bWVudCBpcyBiZWluZyBmb3J3YXJkZWQgdG8NCnRoZSBJRVNHLg0KDQogIFRoZSBEb2N1bWVudCBT aGVwaGVyZCByZWFkIHRoZSBkb2N1bWVudCBpbiBpdHMgZW50aXJldHkgZm9yIFdHIExhc3QNCiAg Q2FsbCBhbmQgY29tcGFyZWQgaXQgdG8gUkZDIDUwNDYgYXQgdGhhdCB0aW1tZS4gIFRoZSBEb2N1 bWVudCBTaGVwaGVyZA0KICBoYXMgcmV2aWV3ZWQgYWxsIG9mIHRoZSBzdWJzZXF1ZW50IGNoYW5n ZXMuICBOdW1lcm91cyBjaGFuZ2VzIGhhdmUNCiAgYmVlbiBtYWRlIHRvIHRoaXMgZG9jdW1lbnQg aW4gIHJlc3BvbnNlIHRvIHRoZSBEb2N1bWVudCBTaGVwaGVyZCdzDQogIGNvbW1lbnRzIGF0IFdH IExhc3QgQ2FsbCBhbmQgc3Vic2VxdWVudGx5LiAgVGhlIERvY3VtZW50IFNoZXBoZXJkDQogIGJl bGlldmVzIHRoYXQgdGhpcyBkb2N1bWVudCBpcyBub3cgcmVhZHkgZm9yIFJGQyBwdWJsaWNhdGlv bi4NCg0KKDQpIERvZXMgdGhlIGRvY3VtZW50IFNoZXBoZXJkIGhhdmUgYW55IGNvbmNlcm5zIGFi b3V0IHRoZSBkZXB0aCBvcg0KYnJlYWR0aCBvZiB0aGUgcmV2aWV3cyB0aGF0IGhhdmUgYmVlbiBw ZXJmb3JtZWQ/ICANCg0KICAgTm8uDQoNCig1KSBEbyBwb3J0aW9ucyBvZiB0aGUgZG9jdW1lbnQg bmVlZCByZXZpZXcgZnJvbSBhIHBhcnRpY3VsYXIgb3IgZnJvbQ0KYnJvYWRlciBwZXJzcGVjdGl2 ZSwgZS5nLiwgc2VjdXJpdHksIG9wZXJhdGlvbmFsIGNvbXBsZXhpdHksIEFBQSwgRE5TLA0KREhD UCwgWE1MLCBvciBpbnRlcm5hdGlvbmFsaXphdGlvbj8gDQoNCiAgIE5vLiANCg0KKDYpIERlc2Ny aWJlIGFueSBzcGVjaWZpYyBjb25jZXJucyBvciBpc3N1ZXMgdGhhdCB0aGUgRG9jdW1lbnQgU2hl cGhlcmQNCmhhcyB3aXRoIHRoaXMgZG9jdW1lbnQgdGhhdCB0aGUgUmVzcG9uc2libGUgQXJlYSBE aXJlY3RvciBhbmQvb3IgdGhlDQpJRVNHIHNob3VsZCBiZSBhd2FyZSBvZj8gRm9yIGV4YW1wbGUs IHBlcmhhcHMgaGUgb3Igc2hlIGlzIHVuY29tZm9ydGFibGUNCndpdGggY2VydGFpbiBwYXJ0cyBv ZiB0aGUgZG9jdW1lbnQsIG9yIGhhcyBjb25jZXJucyB3aGV0aGVyIHRoZXJlIHJlYWxseQ0KaXMg YSBuZWVkIGZvciBpdC4gSW4gYW55IGV2ZW50LCBpZiB0aGUgV0cgaGFzIGRpc2N1c3NlZCB0aG9z ZSBpc3N1ZXMgYW5kDQpoYXMgaW5kaWNhdGVkIHRoYXQgaXQgc3RpbGwgd2lzaGVzIHRvIGFkdmFu Y2UgdGhlIGRvY3VtZW50LCBkZXRhaWwgdGhvc2UNCmNvbmNlcm5zIGhlcmUuDQoNCiAgIE5vbmUu DQoNCig3KSBIYXMgZWFjaCBhdXRob3IgY29uZmlybWVkIHRoYXQgYW55IGFuZCBhbGwgYXBwcm9w cmlhdGUgSVBSDQpkaXNjbG9zdXJlcyByZXF1aXJlZCBmb3IgZnVsbCBjb25mb3JtYW5jZSB3aXRo IHRoZSBwcm92aXNpb25zIG9mIEJDUCA3OA0KYW5kIEJDUCA3OSBoYXZlIGFscmVhZHkgYmVlbiBm aWxlZC4gSWYgbm90LCBleHBsYWluIHdoeS4NCg0KICAgWWVzLg0KDQooOCkgSGFzIGFuIElQUiBk aXNjbG9zdXJlIGJlZW4gZmlsZWQgdGhhdCByZWZlcmVuY2VzIHRoaXMgZG9jdW1lbnQ/DQoNCiAg IE5vLg0KDQooOSkgSG93IHNvbGlkIGlzIHRoZSBXRyBjb25zZW5zdXMgYmVoaW5kIHRoaXMgZG9j dW1lbnQ/IERvZXMgaXQgDQpyZXByZXNlbnQgdGhlIHN0cm9uZyBjb25jdXJyZW5jZSBvZiBhIGZl dyBpbmRpdmlkdWFscywgd2l0aCBvdGhlcnMNCmJlaW5nIHNpbGVudCwgb3IgZG9lcyB0aGUgV0cg YXMgYSB3aG9sZSB1bmRlcnN0YW5kIGFuZCBhZ3JlZSB3aXRoIGl0PyAgIA0KDQogICBUaGUgc3Rv cm0gKFNUT1JhZ2UgTWFpbnRlbmFuY2UpIFdHIGlzIGEgbWFpbnRlbmFuY2UgV0cgdGhhdCB3b3Jr cyBvbg0KICAgYSBudW1iZXIgb2Ygc3RvcmFnZSB0ZWNobm9sb2dpZXMsIGFuZCBoZW5jZSBub3Qg ZXZlcnkgcGFydGljaXBhbnQgaXMNCiAgIGludGVyZXN0ZWQgaW4gZXZlcnkgdGVjaG5vbG9neS4g IFRoZSBtZW1iZXJzIG9mIHRoZSBXRyB3aG8gYXJlDQogICBpbnRlcmVzdGVkIGluIGlTRVIgdW5k ZXJzdGFuZCBhbmQgYWdyZWUgd2l0aCB0aGlzIGRvY3VtZW50Lg0KDQooMTApIEhhcyBhbnlvbmUg dGhyZWF0ZW5lZCBhbiBhcHBlYWwgb3Igb3RoZXJ3aXNlIGluZGljYXRlZCBleHRyZW1lIA0KZGlz Y29udGVudD8NCg0KICAgTm8uDQoNCigxMSkgSWRlbnRpZnkgYW55IElEIG5pdHMgdGhlIERvY3Vt ZW50IFNoZXBoZXJkIGhhcyBmb3VuZCBpbiB0aGlzDQpkb2N1bWVudC4gKFNlZSBodHRwOi8vd3d3 LmlldGYub3JnL3Rvb2xzL2lkbml0cy8gYW5kIHRoZSBJbnRlcm5ldC1EcmFmdHMNCkNoZWNrbGlz dCkuIEJvaWxlcnBsYXRlIGNoZWNrcyBhcmUgbm90IGVub3VnaDsgdGhpcyBjaGVjayBuZWVkcyB0 byBiZQ0KdGhvcm91Z2guDQoNCiAgIE5vIG5pdHMgLSBpZG5pdHMgMi4xMi4xMyBpc3N1ZXMgdGhy ZWUgd2FybmluZ3MgdGhhdCBkbyBub3QgcmVmbGVjdA0KICAgYWN0dWFsIHByb2JsZW1zIHdpdGgg dGhlIGRyYWZ0Lg0KDQooMTIpIERlc2NyaWJlIGhvdyB0aGUgZG9jdW1lbnQgbWVldHMgYW55IHJl cXVpcmVkIGZvcm1hbCByZXZpZXcNCmNyaXRlcmlhLCBzdWNoIGFzIHRoZSBNSUIgRG9jdG9yLCBt ZWRpYSB0eXBlLCBhbmQgVVJJIHR5cGUgcmV2aWV3cy4NCg0KICAgTm90IGFwcGxpY2FibGUuDQoN CigxMykgSGF2ZSBhbGwgcmVmZXJlbmNlcyB3aXRoaW4gdGhpcyBkb2N1bWVudCBiZWVuIGlkZW50 aWZpZWQgYXMNCmVpdGhlciBub3JtYXRpdmUgb3IgaW5mb3JtYXRpdmU/DQoNCiAgIFllcy4NCg0K KDE0KSBBcmUgdGhlcmUgbm9ybWF0aXZlIHJlZmVyZW5jZXMgdG8gZG9jdW1lbnRzIHRoYXQgYXJl IG5vdCByZWFkeSBmb3INCmFkdmFuY2VtZW50IG9yIGFyZSBvdGhlcndpc2UgaW4gYW4gdW5jbGVh ciBzdGF0ZT8NCg0KICAgTm8uDQoNCigxNSkgQXJlIHRoZXJlIGRvd253YXJkIG5vcm1hdGl2ZSBy ZWZlcmVuY2VzIHJlZmVyZW5jZXMgKHNlZSBSRkMgMzk2Nyk/DQoNCiAgIE5vLg0KDQooMTYpIFdp bGwgcHVibGljYXRpb24gb2YgdGhpcyBkb2N1bWVudCBjaGFuZ2UgdGhlIHN0YXR1cyBvZiBhbnkN CmV4aXN0aW5nIFJGQ3M/IEFyZSB0aG9zZSBSRkNzIGxpc3RlZCBvbiB0aGUgdGl0bGUgcGFnZSBo ZWFkZXIsIGxpc3RlZA0KaW4gdGhlIGFic3RyYWN0LCBhbmQgZGlzY3Vzc2VkIGluIHRoZSBpbnRy b2R1Y3Rpb24/IElmIHRoZSBSRkNzIGFyZSBub3QNCmxpc3RlZCBpbiB0aGUgQWJzdHJhY3QgYW5k IEludHJvZHVjdGlvbiwgZXhwbGFpbiB3aHksIGFuZCBwb2ludCB0byB0aGUNCnBhcnQgb2YgdGhl IGRvY3VtZW50IHdoZXJlIHRoZSByZWxhdGlvbnNoaXAgb2YgdGhpcyBkb2N1bWVudCB0byB0aGUN Cm90aGVyIFJGQ3MgaXMgZGlzY3Vzc2VkLiBJZiB0aGlzIGluZm9ybWF0aW9uIGlzIG5vdCBpbiB0 aGUgZG9jdW1lbnQsDQpleHBsYWluIHdoeSB0aGUgV0cgY29uc2lkZXJzIGl0IHVubmVjZXNzYXJ5 Lg0KDQogICBUaGlzIGRvY3VtZW50IG9ic29sZXRlcyBSRkMgNTA0NjsgdGhhdCBpcyBsaXN0ZWQg aW4gdGhlIHRpdGxlIHBhZ2UgaGVhZGVyDQogICBhbmQgc3RhdGVkIGluIHRoZSBBYnN0cmFjdC4g IFRoZSBJbnRyb2R1Y3Rpb24gc2VjdGlvbiBzdGFydHMgb24gcC4xNSwgYW5kDQogICBpdCBkb2Vz IG5vdCBzZWVtIG5lY2Vzc2FyeSB0byByZXBlYXQgdGhlIG9ic29sZXNjZW5jZSBzdGF0ZW1lbnQg dGhhdCBmYXINCiAgIGludG8gdGhlIGRvY3VtZW50Lg0KDQooMTcpIERlc2NyaWJlIHRoZSBEb2N1 bWVudCBTaGVwaGVyZCdzIHJldmlldyBvZiB0aGUgSUFOQSBjb25zaWRlcmF0aW9ucw0Kc2VjdGlv biwgZXNwZWNpYWxseSB3aXRoIHJlZ2FyZCB0byBpdHMgY29uc2lzdGVuY3kgd2l0aCB0aGUgYm9k eSBvZiB0aGUNCmRvY3VtZW50LiBDb25maXJtIHRoYXQgYWxsIHByb3RvY29sIGV4dGVuc2lvbnMg dGhhdCB0aGUgZG9jdW1lbnQgbWFrZXMNCmFyZSBhc3NvY2lhdGVkIHdpdGggdGhlIGFwcHJvcHJp YXRlIHJlc2VydmF0aW9ucyBpbiBJQU5BIHJlZ2lzdHJpZXMuDQpDb25maXJtIHRoYXQgYW55IHJl ZmVyZW5jZWQgSUFOQSByZWdpc3RyaWVzIGhhdmUgYmVlbiBjbGVhcmx5IGlkZW50aWZpZWQuIA0K DQogICBUaGUgSUFOQSBjb25zaWRlcmF0aW9ucyBzZWN0aW9uIGFkZHMgdGhyZWUga2V5cyB0byB0 aGUgaVNDU0kNCiAgIExvZ2luL1RleHQgS2V5cyIgcmVnaXN0cnkgb2YgImlTQ1NJIFBhcmFtZXRl cnMiLCBhbmQgcmVxdWVzdHMgcmVmZXJlbmNlDQogICB1cGRhdGVzIHRvIG90aGVyIGlTRVIga2V5 cyB0aGF0IHdpbGwgYmUgY29uZmlybWVkIGJ5IElBTkEuICBUaGUgZG9jdW1lbnQNCiAgIHNoZXBo ZXJkIGhhcyBjaGVja2VkIHRoZSBJQU5BIENvbnNpZGVyYXRpb25zIHNlY3Rpb24gYW5kIGJlbGll dmVzIGl0DQogICB0byBiZSBjb3JyZWN0IGFuZCBzdWZmaWNpZW50Lg0KDQooMTctY29udGludWVk KSBDb25maXJtIHRoYXQgbmV3bHkgY3JlYXRlZCBJQU5BIHJlZ2lzdHJpZXMgaW5jbHVkZSBhDQpk ZXRhaWxlZCBzcGVjaWZpY2F0aW9uIG9mIHRoZSBpbml0aWFsIGNvbnRlbnRzIGZvciB0aGUgcmVn aXN0cnksIHRoYXQNCmFsbG9jYXRpb25zIHByb2NlZHVyZXMgZm9yIGZ1dHVyZSByZWdpc3RyYXRp b25zIGFyZSBkZWZpbmVkLCBhbmQgYQ0KcmVhc29uYWJsZSBuYW1lIGZvciB0aGUgbmV3IHJlZ2lz dHJ5IGhhcyBiZWVuIHN1Z2dlc3RlZCAoc2VlIFJGQyA1MjI2KS4NCg0KICAgTm90IGFwcGxpY2Fi bGUuDQoNCigxOCkgTGlzdCBhbnkgbmV3IElBTkEgcmVnaXN0cmllcyB0aGF0IHJlcXVpcmUgRXhw ZXJ0IFJldmlldyBmb3IgZnV0dXJlDQphbGxvY2F0aW9ucy4gUHJvdmlkZSBhbnkgcHVibGljIGd1 aWRhbmNlIHRoYXQgdGhlIElFU0cgd291bGQgZmluZA0KdXNlZnVsIGluIHNlbGVjdGluZyB0aGUg SUFOQSBFeHBlcnRzIGZvciB0aGVzZSBuZXcgcmVnaXN0cmllcy4NCg0KICAgTm90IGFwcGxpY2Fi bGUuDQoNCigxOSkgRGVzY3JpYmUgcmV2aWV3cyBhbmQgYXV0b21hdGVkIGNoZWNrcyBwZXJmb3Jt ZWQgYnkgdGhlIERvY3VtZW50DQpTaGVwaGVyZCB0byB2YWxpZGF0ZSBzZWN0aW9ucyBvZiB0aGUg ZG9jdW1lbnQgd3JpdHRlbiBpbiBhIGZvcm1hbA0KbGFuZ3VhZ2UsIHN1Y2ggYXMgWE1MIGNvZGUs IEJORiBydWxlcywgTUlCIGRlZmluaXRpb25zLCBldGMuDQoNCiAgIE5vdCBhcHBsaWNhYmxlLg0K --_004_8D3D17ACE214DC429325B2B98F3AE7120DCD16C7MX15Acorpemccom_-- From david.black@emc.com Fri Sep 21 12:55:20 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AEB4111E8097 for ; Fri, 21 Sep 2012 12:55:20 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id o+DJgj5R9lIP for ; Fri, 21 Sep 2012 12:55:20 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id F023A11E808D for ; Fri, 21 Sep 2012 12:55:19 -0700 (PDT) Received: from hop04-l1d11-si03.isus.emc.com (HOP04-L1D11-SI03.isus.emc.com [10.254.111.23]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8LJtF3d013229 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 21 Sep 2012 15:55:17 -0400 Received: from mailhub.lss.emc.com (mailhubhoprd04.lss.emc.com [10.254.222.226]) by hop04-l1d11-si03.isus.emc.com (RSA Interceptor) for ; Fri, 21 Sep 2012 15:55:02 -0400 Received: from mxhub32.corp.emc.com (mxhub32.corp.emc.com [128.222.70.172]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8LJt1su016722 for ; Fri, 21 Sep 2012 15:55:01 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub32.corp.emc.com ([128.222.70.172]) with mapi; Fri, 21 Sep 2012 15:55:00 -0400 From: "Black, David" To: "Black, David" , "storm@ietf.org" Importance: high X-Priority: 1 Date: Fri, 21 Sep 2012 15:55:00 -0400 Thread-Topic: iSCSI MIB: Functional enhancements proposal Thread-Index: Ac2S3k0VAOHDeEHlQP66E4/tbPSxkAFVDrNQ Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE78FD8@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE7120DCD1567@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DCD1567@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: Re: [storm] iSCSI MIB: Functional enhancements proposal X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Sep 2012 19:55:20 -0000 In order to move forward, I suggest that the authors make the functional changes [1] - [6], not make changes [A] - [F] and [I}, and use their best judgment on what (if anything) to do about [G] and [H] > [G] Additional session reporting, including digest, error recovery level,= session type > and additional counters for missing PDUs, data in PDUs, data out PDUs, a= sync PDUs. > [H] Report initial remote IP address and port for connections in addition= to actual > IP address and port - this would be for situations in which a redirectio= n has > occurred. Please submit a revised version of the MIB draft soon. Thanks, --David (storm WG co-chair). > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of > Black, David > Sent: Friday, September 14, 2012 9:06 PM > To: storm@ietf.org > Subject: [storm] iSCSI MIB: Functional enhancements proposal >=20 > The iSCSI MIB has been waiting for a WG decision about what to incorporat= e from > the list of functional enhancements suggested by the MIB Doctor review. >=20 > With , based on consulting with the MIB draft editors, = I offer > the following initial proposal for the WG's consideration ... in the hope= of > getting to a decision. >=20 > --- Make the following functional enhancements: >=20 > [1] Add NOP counters at iSCSI session scope for heartbeat tracking > [2] Add a port number to the iscsiTgtLoginFailure and iscsiIntrLoginFailu= re notifications, > and to the last failure info in iscsiInitiatorAttributesEntry. > [3] Add a description string to the iSCSI portal: >=20 > iscsiPortalDescr OBJECT-TYPE > SYNTAX SnmpAdminString > MAX-ACCESS read-only > STATUS current > DESCRIPTION > "A UTF-8 string, determined by the implementation to > describe the iSCSI portal. When only a single instance > is present, this object may be set to the zero-length > string; with multiple iSCSI portals, it may be used in > an implementation-dependent manner to describe the > respective portal, and could include information such as > HBA model, description and version or software driver and > version." >=20 > [4] Support "Target Unmapped" session failure reporting in the > iscsiInstSessionFailure notification: >=20 > Editors: FailureType is an OID pointing to a stat in > iscsiInstSsnErrorStatsTable, so we would need to add this: >=20 > iscsiInstSsnTgtUnmappedErrors Counter32 >=20 > [5] Add a port number to iscsiPortalAttributesEntry >=20 > [6] Add a timeout counter to iscsiInitiatorLogoutStatsEntry to distinguis= h implicit > logouts caused by timeouts from implicit logouts caused by other reasons= . >=20 > --- Do not make any other functional enhancements, including the followin= g >=20 > [A] Summary notifications for large numbers of targets or initiators. (A= t most > a single summary with a count would be appropriate for each notification= type). > [B] RowPointer or other connection to transport state as part of > iscsiInstanceSsnErrorStatsEntry (can be found indirectly in current stru= cture). > [C] Support for reporting more than two digest algorithms (only one is st= andardized). > [D] Additional list of authentication methods for negotiation (already av= ailable based > on reporting of authorized entities). > [E] Additional support for mutual authentication (check the MIBs at both = ends instead). > [F] Support for digest provisioning at the iSCSI node level (use iSCSI po= rtal instead). > [G] Additional session reporting, including digest, error recovery level,= session type > and additional counters for missing PDUs, data in PDUs, data out PDUs, a= sync PDUs. > [H] Report initial remote IP address and port for connections in addition= to actual > IP address and port - this would be for situations in which a redirectio= n has > occurred. > [I] Report list of discovered targets (out of scope for this MIB). >=20 > -------------------- >=20 > Of the above list, I'd be particularly interested in comments on [A], [G]= and [H] - > if these were added to the MIB, would they be likely to be implemented an= d > used? >=20 > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer > EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 > +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-7= 786 > david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 > ---------------------------------------------------- >=20 >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From cbm@chadalapaka.com Sat Sep 22 22:33:58 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8891221F8523 for ; Sat, 22 Sep 2012 22:33:58 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6pF2SzDFArug for ; Sat, 22 Sep 2012 22:33:58 -0700 (PDT) Received: from ch1outboundpool.messaging.microsoft.com (ch1ehsobe001.messaging.microsoft.com [216.32.181.181]) by ietfa.amsl.com (Postfix) with ESMTP id DAEA121F851B for ; Sat, 22 Sep 2012 22:33:57 -0700 (PDT) Received: from mail80-ch1-R.bigfish.com (10.43.68.229) by CH1EHSOBE012.bigfish.com (10.43.70.62) with Microsoft SMTP Server id 14.1.225.23; Sun, 23 Sep 2012 05:33:56 +0000 Received: from mail80-ch1 (localhost [127.0.0.1]) by mail80-ch1-R.bigfish.com (Postfix) with ESMTP id 52C0780149 for ; Sun, 23 Sep 2012 05:33:56 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.240.117; KIP:(null); UIP:(null); IPV:NLI; H:BL2PRD0610HT002.namprd06.prod.outlook.com; RD:none; EFVD:NLI X-SpamScore: -26 X-BigFish: PS-26(zz9371I542M4015Id6f1izz1202h1d1ah1d2ahzz1033IL8275bh8275dhz2fh2a8h668h839hd25hf0ah107ah1288h12a5h12a9h12bdh137ah1155h) Received-SPF: pass (mail80-ch1: domain of chadalapaka.com designates 157.56.240.117 as permitted sender) client-ip=157.56.240.117; envelope-from=cbm@chadalapaka.com; helo=BL2PRD0610HT002.namprd06.prod.outlook.com ; .outlook.com ; Received: from mail80-ch1 (localhost.localdomain [127.0.0.1]) by mail80-ch1 (MessageSwitch) id 1348378434531735_15242; Sun, 23 Sep 2012 05:33:54 +0000 (UTC) Received: from CH1EHSMHS020.bigfish.com (snatpool1.int.messaging.microsoft.com [10.43.68.240]) by mail80-ch1.bigfish.com (Postfix) with ESMTP id 738C3120044; Sun, 23 Sep 2012 05:33:54 +0000 (UTC) Received: from BL2PRD0610HT002.namprd06.prod.outlook.com (157.56.240.117) by CH1EHSMHS020.bigfish.com (10.43.70.20) with Microsoft SMTP Server (TLS) id 14.1.225.23; Sun, 23 Sep 2012 05:33:54 +0000 Received: from BL2PRD0610MB361.namprd06.prod.outlook.com ([169.254.12.31]) by BL2PRD0610HT002.namprd06.prod.outlook.com ([10.255.101.37]) with mapi id 14.16.0190.008; Sun, 23 Sep 2012 05:33:53 +0000 From: Mallikarjun Chadalapaka To: "david.black@emc.com" , "storm@ietf.org" Thread-Topic: iSCSI drafts - T10 and T11 references Thread-Index: Ac1rNRGDFlJk/LPPRq2a5Pa224AxSwuFP5jw Date: Sun, 23 Sep 2012 05:33:52 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [71.197.226.181] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: chadalapaka.com Subject: Re: [storm] iSCSI drafts - T10 and T11 references X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Sep 2012 05:33:58 -0000 Hi David, I have noticed this comment as I am working through all Last Call comments. The references to SAM-3 and SAM-4 are keyed to this paragraph in Section 13= .9: "[SAM2] and [SAM3] specifications note in their informative text that TPGT = value should be non-zero, note that it is incorrect. A zero value is allow= ed as a legal value for TPGT. This discrepancy currently stands corrected i= n [SAM4]." Additionally, SAM4 reference is also used in citing the new ASC/ASCQ codes = that it defines, in Section 11.14.5 of the draft. So, I believe references to all three SAM specs would continue to be approp= riate. The current draft references FC-FS in discussing the NAA format in a few pl= aces, so the current reference at the end is intentionally the same. Is the= older reference a problem? If so, I suppose we can replace all references = to FC-FS-4, but that seems to be still in early stages(0.1 spec posted on A= pril 2012), so perhaps we should use FC-FS3? Thanks. Mallikarjun -----Original Message----- From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of d= avid.black@emc.com Sent: Thursday, July 26, 2012 6:47 AM To: storm@ietf.org Subject: [storm] iSCSI drafts - T10 and T11 references In making up the lists of T10 and T11 documents that are currently referenc= ed, some of those references don't look right to me, for example: - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. - T11: The FC-FS reference is out of date. As part of dealing with the comments from IETF Last Call, we'll need to go = over all of the T10 and T11 references (and should check all the references= ) to make sure that they're up to date and necessary. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From Frederick.Knight@netapp.com Sun Sep 23 03:38:23 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id F132321F847B for ; Sun, 23 Sep 2012 03:38:22 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5+4ZACooDT4F for ; Sun, 23 Sep 2012 03:38:22 -0700 (PDT) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id F007721F8476 for ; Sun, 23 Sep 2012 03:38:21 -0700 (PDT) X-IronPort-AV: E=Sophos;i="4.80,469,1344236400"; d="scan'208";a="693010849" Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx2-out.netapp.com with ESMTP; 23 Sep 2012 03:38:21 -0700 Received: from vmwexceht01-prd.hq.netapp.com (vmwexceht01-prd.hq.netapp.com [10.106.76.239]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8NAcKFU016822; Sun, 23 Sep 2012 03:38:20 -0700 (PDT) Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.102]) by vmwexceht01-prd.hq.netapp.com ([10.106.76.239]) with mapi id 14.02.0309.002; Sun, 23 Sep 2012 03:38:20 -0700 From: "Knight, Frederick" To: Mallikarjun Chadalapaka , "david.black@emc.com" , "storm@ietf.org" Thread-Topic: iSCSI drafts - T10 and T11 references Thread-Index: Ac1rNRGDFlJk/LPPRq2a5Pa224AxSwuFP5jwAAlZtBA= Date: Sun, 23 Sep 2012 10:38:18 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.104.60.116] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [storm] iSCSI drafts - T10 and T11 references X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Sep 2012 10:38:23 -0000 Interesting, I just came across a question in a related area (only for SAM-= 2). RFC3720 was written in between SAM-2 and SAM-3; some concepts were drawn fr= om SAM-2, and some from SAM-3. We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus loss= concept does not exist in SAM-2. SAM-2 does include references to RESERVE= and RELEASE commands. SAM-3 includes I_T Nexus loss, but it has no refere= nce to RESERVE state. There are allusions in RFC3720 about the intersectio= n of RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit. RF= C3720 references other documents (SAM2, SAM3, SBC, SPC3), none of which add= ress the situation. Consider text such as the following: 4.4.3.1. I_T Nexus State Certain nexus relationships contain an explicit state (e.g., initiator-specific mode pages) that may need to be preserved by the device server [SAM2] in a logical unit through changes or failures in the iSCSI layer (e.g., session failures). In order for that state to be restored, the iSCSI initiator should reestablish its session (re-login) to the same Target Portal Group using the previous ISID. That is, it should perform session recovery as described in Chapter 6. This is because the SCSI initiator port identifier and the SCSI target port identifier (or relative target port) form the datum that the SCSI logical unit device server uses to identify the I_T nexus. Is RESERVE state - explicit state? When an I_T nexus is reestablished, ini= tiator-specific mode pages are preserved. That implies that an I_T nexus l= oss should be treated in the way that SAM-2 describes via CLEAR TASK SET (w= here mode pages and RESERVE state are preserved). If a session reestablish= ment caused saved mode pages to be restored rather than the established ini= tiator-specific mode pages to be restored, then it would be like a LU reset= (where the RESERVE state is lost). Is RESERVE state explicit state that i= s to be preserved? Appendix E covers clearing effects during various events, but it never ment= ions RESERVE state. Section E.2 contains the following text: The only iSCSI protocol action that can effect clearing actions on SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 Loss of Nexus notification). [SPC3] describes the clearing effects of this notification on a variety of SCSI attributes. In addition, SCSI standards documents (such as [SAM2] and [SBC]) define additional clearing actions that may take place for several SCSI objects on SCSI events such as LU resets and power-on resets. I could not find any text in SPC3 that describes these clearing effects (I = don't know if this SPC3 reference is incorrect, but at a minimum, it is ine= ffective). The SAM2 text for LU resets and power-on resets is easily found= , and clearly says that any RESERVE is released in those cases. As an asid= e, the reference to section 6.3.5.1 is a circular reference). The text in = E.2 goes on: Since iSCSI defines a target cold reset as a protocol-equivalent to a target power-cycle, the iSCSI target cold reset must also be considered as the power-on reset event in interpreting the actions defined in the SCSI standards. When the iSCSI session is reconstructed (between the same SCSI ports with the same nexus identifier) reestablishing the same I_T nexus, all SCSI objects that are defined to not clear on the "I_T nexus loss" notification event, such as persistent reservations, are automatically associated to this new session. So, it is clear what happens to PERSISTENT RESERVE state, but not so clear = about RESERVE state. It also states that all SCSI objects that are defined= to not clear on the I_T nexus loss are retained. But again, the only list= I can find of what is cleared is here in this Appendix, since SAM-2, SAM-3= , SAM-4 are all silent on the interaction between RESERVE and I_T Nexus los= s. T10 has never had to deal with this because they removed RESERVE before the= y added I_T Nexus loss. However, iSCSI did not do that; we discuss BOTH, b= ut we fail to describe how they interact. iSCSI also has the feature of re= establishment of a session that does not exist in SAM-2 (SAM-2 was still pa= rallel SCSI focused, where connections didn't exist). So again, if session= reestablishment is examined in the light of parallel SCSI, the intent woul= d be to retain the RESERVE state. Here is another section on the topic: 6.3.5.1. Loss of Nexus Notification The iSCSI layer provides the SCSI layer with the "I_T nexus loss" notification when any one of the following events happens: Successful completion of session reinstatement. Session closure event. Session timeout event. Certain SCSI object clearing actions may result due to the notification in the SCSI end nodes, as documented in Appendix F. - Clearing Effects of Various Events on Targets. FWIW - the reference above is incorrect - there is no Appendix F in the con= solidated draft - this should be Appendix E (and also, BTW - this is a circ= ular reference). So anyway, it seems like the intent was that session reinstatement would pr= eserve the RESERVE state across the I_T Nexus loss event. =20 So aside from the incorrect reference (and circular nature of it), any thou= ghts on whether we should get explicit about this situation? Should we rem= ain silent; should we tweak Appendix E along these lines (or other lines th= at others might suggest): 4.4.3.1. I_T Nexus State Certain nexus relationships contain an explicit state (e.g., initiator-specific mode pages, and RESERVE state) that may need to be pre= served by the device server [SAM2] in a logical unit through changes or failures in the iSCSI layer (e.g., session failures).... Fred Knight -----Original Message----- From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of M= allikarjun Chadalapaka Sent: Sunday, September 23, 2012 1:34 AM To: david.black@emc.com; storm@ietf.org Subject: Re: [storm] iSCSI drafts - T10 and T11 references Hi David, I have noticed this comment as I am working through all Last Call comments. The references to SAM-3 and SAM-4 are keyed to this paragraph in Section 13= .9: "[SAM2] and [SAM3] specifications note in their informative text that TPGT = value should be non-zero, note that it is incorrect. A zero value is allow= ed as a legal value for TPGT. This discrepancy currently stands corrected i= n [SAM4]." Additionally, SAM4 reference is also used in citing the new ASC/ASCQ codes = that it defines, in Section 11.14.5 of the draft. So, I believe references to all three SAM specs would continue to be approp= riate. The current draft references FC-FS in discussing the NAA format in a few pl= aces, so the current reference at the end is intentionally the same. Is the= older reference a problem? If so, I suppose we can replace all references = to FC-FS-4, but that seems to be still in early stages(0.1 spec posted on A= pril 2012), so perhaps we should use FC-FS3? Thanks. Mallikarjun -----Original Message----- From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of d= avid.black@emc.com Sent: Thursday, July 26, 2012 6:47 AM To: storm@ietf.org Subject: [storm] iSCSI drafts - T10 and T11 references In making up the lists of T10 and T11 documents that are currently referenc= ed, some of those references don't look right to me, for example: - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. - T11: The FC-FS reference is out of date. As part of dealing with the comments from IETF Last Call, we'll need to go = over all of the T10 and T11 references (and should check all the references= ) to make sure that they're up to date and necessary. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From cbm@chadalapaka.com Sun Sep 23 15:34:49 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E21E121F84FD for ; Sun, 23 Sep 2012 15:34:49 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M2whCnZOjWsY for ; Sun, 23 Sep 2012 15:34:48 -0700 (PDT) Received: from am1outboundpool.messaging.microsoft.com (am1ehsobe004.messaging.microsoft.com [213.199.154.207]) by ietfa.amsl.com (Postfix) with ESMTP id 4172A21F84F5 for ; Sun, 23 Sep 2012 15:34:47 -0700 (PDT) Received: from mail7-am1-R.bigfish.com (10.3.201.248) by AM1EHSOBE006.bigfish.com (10.3.204.26) with Microsoft SMTP Server id 14.1.225.23; Sun, 23 Sep 2012 22:34:46 +0000 Received: from mail7-am1 (localhost [127.0.0.1]) by mail7-am1-R.bigfish.com (Postfix) with ESMTP id D1C332000F0; Sun, 23 Sep 2012 22:34:46 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.240.117; KIP:(null); UIP:(null); IPV:NLI; H:BL2PRD0610HT003.namprd06.prod.outlook.com; RD:none; EFVD:NLI X-SpamScore: -28 X-BigFish: PS-28(zz9371I146fI148cI542M4015Id6f1izz1202h1d1ah1d2ahzz1033IL8275bh8275dhz2fh2a8h668h839hd25hf0ah107ah1288h12a5h12a9h12bdh137ah1155h) Received-SPF: pass (mail7-am1: domain of chadalapaka.com designates 157.56.240.117 as permitted sender) client-ip=157.56.240.117; envelope-from=cbm@chadalapaka.com; helo=BL2PRD0610HT003.namprd06.prod.outlook.com ; .outlook.com ; Received: from mail7-am1 (localhost.localdomain [127.0.0.1]) by mail7-am1 (MessageSwitch) id 1348439684495534_324; Sun, 23 Sep 2012 22:34:44 +0000 (UTC) Received: from AM1EHSMHS005.bigfish.com (unknown [10.3.201.249]) by mail7-am1.bigfish.com (Postfix) with ESMTP id 753E2E0071; Sun, 23 Sep 2012 22:34:44 +0000 (UTC) Received: from BL2PRD0610HT003.namprd06.prod.outlook.com (157.56.240.117) by AM1EHSMHS005.bigfish.com (10.3.207.105) with Microsoft SMTP Server (TLS) id 14.1.225.23; Sun, 23 Sep 2012 22:34:39 +0000 Received: from BL2PRD0610MB361.namprd06.prod.outlook.com ([169.254.12.31]) by BL2PRD0610HT003.namprd06.prod.outlook.com ([10.255.101.38]) with mapi id 14.16.0190.008; Sun, 23 Sep 2012 22:34:37 +0000 From: Mallikarjun Chadalapaka To: "Knight, Frederick" , "david.black@emc.com" , "storm@ietf.org" Thread-Topic: iSCSI drafts - T10 and T11 references Thread-Index: Ac1rNRGDFlJk/LPPRq2a5Pa224AxSwuFP5jwAAlZtBAAF2slIA== Date: Sun, 23 Sep 2012 22:34:36 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [71.197.226.181] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: chadalapaka.com Subject: Re: [storm] iSCSI drafts - T10 and T11 references X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Sep 2012 22:34:50 -0000 Hi Fred, SPC-3 discusses clearing effects of I-T nexus loss in several places. Section 5.8.2.9 for ALUA Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for background self-test Section 5.6.1 on persistent reservations Section 5.13 on timestamps ...... Thanks for pointing out the broken cross-reference - Appendix F instead of = E. I have just fixed it - and reviewed all other Appendix references elsewh= ere in the draft. However, I do not see a circular reference problem per se= - Appendix E is correctly referencing Section 6.3.5 for definition of term= s, e.g. session continuation. And Section 6.3.5.1 is correctly deferring to= Appendix E when discussing clearing effects. Let me know if I'm overlookin= g something here. Now, the reserve/release management state, :) : I agree the formal I_T nexu= s loss event didn't exist in SAM-2, however the notion of a new I_T nexus s= tate did co-exist with a session-oriented I_T nexus well before iSCSI - FCP= process login (PRLO) comes to mind here. So I'd be curious about tradition= al FCP handling of reserve/release management state. And I'm also checking = internally here at MS. From what I know now though, it does *appear* that r= eserve/release management state should not be cleared with I_T nexus loss. Having said all that, note that reserve/release management state is squarel= y in the SCSI territory. iSCSI spec has so far by design deferred to SCSI s= tandards in specifying clearing effects on SCSI objects - as Appendix E.2 m= akes it clear. What we do specify is the clearing effects of SCSI actions, = e.g. LU reset, on iSCSI objects. Thanks. Mallikarjun -----Original Message----- From: Knight, Frederick [mailto:Frederick.Knight@netapp.com]=20 Sent: Sunday, September 23, 2012 3:38 AM To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org Subject: RE: iSCSI drafts - T10 and T11 references Interesting, I just came across a question in a related area (only for SAM-= 2). RFC3720 was written in between SAM-2 and SAM-3; some concepts were drawn fr= om SAM-2, and some from SAM-3. We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus loss= concept does not exist in SAM-2. SAM-2 does include references to RESERVE= and RELEASE commands. SAM-3 includes I_T Nexus loss, but it has no refere= nce to RESERVE state. There are allusions in RFC3720 about the intersectio= n of RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit. RF= C3720 references other documents (SAM2, SAM3, SBC, SPC3), none of which add= ress the situation. Consider text such as the following: 4.4.3.1. I_T Nexus State Certain nexus relationships contain an explicit state (e.g., initiator-specific mode pages) that may need to be preserved by the device server [SAM2] in a logical unit through changes or failures in the iSCSI layer (e.g., session failures). In order for that state to be restored, the iSCSI initiator should reestablish its session (re-login) to the same Target Portal Group using the previous ISID. That is, it should perform session recovery as described in Chapter 6. This is because the SCSI initiator port identifier and the SCSI target port identifier (or relative target port) form the datum that the SCSI logical unit device server uses to identify the I_T nexus. Is RESERVE state - explicit state? When an I_T nexus is reestablished, ini= tiator-specific mode pages are preserved. That implies that an I_T nexus l= oss should be treated in the way that SAM-2 describes via CLEAR TASK SET (w= here mode pages and RESERVE state are preserved). If a session reestablish= ment caused saved mode pages to be restored rather than the established ini= tiator-specific mode pages to be restored, then it would be like a LU reset= (where the RESERVE state is lost). Is RESERVE state explicit state that i= s to be preserved? Appendix E covers clearing effects during various events, but it never ment= ions RESERVE state. Section E.2 contains the following text: The only iSCSI protocol action that can effect clearing actions on SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 Loss of Nexus notification). [SPC3] describes the clearing effects of this notification on a variety of SCSI attributes. In addition, SCSI standards documents (such as [SAM2] and [SBC]) define additional clearing actions that may take place for several SCSI objects on SCSI events such as LU resets and power-on resets. I could not find any text in SPC3 that describes these clearing effects (I = don't know if this SPC3 reference is incorrect, but at a minimum, it is ine= ffective). The SAM2 text for LU resets and power-on resets is easily found= , and clearly says that any RESERVE is released in those cases. As an asid= e, the reference to section 6.3.5.1 is a circular reference). The text in = E.2 goes on: Since iSCSI defines a target cold reset as a protocol-equivalent to a target power-cycle, the iSCSI target cold reset must also be considered as the power-on reset event in interpreting the actions defined in the SCSI standards. When the iSCSI session is reconstructed (between the same SCSI ports with the same nexus identifier) reestablishing the same I_T nexus, all SCSI objects that are defined to not clear on the "I_T nexus loss" notification event, such as persistent reservations, are automatically associated to this new session. So, it is clear what happens to PERSISTENT RESERVE state, but not so clear = about RESERVE state. It also states that all SCSI objects that are defined= to not clear on the I_T nexus loss are retained. But again, the only list= I can find of what is cleared is here in this Appendix, since SAM-2, SAM-3= , SAM-4 are all silent on the interaction between RESERVE and I_T Nexus los= s. T10 has never had to deal with this because they removed RESERVE before the= y added I_T Nexus loss. However, iSCSI did not do that; we discuss BOTH, b= ut we fail to describe how they interact. iSCSI also has the feature of re= establishment of a session that does not exist in SAM-2 (SAM-2 was still pa= rallel SCSI focused, where connections didn't exist). So again, if session= reestablishment is examined in the light of parallel SCSI, the intent woul= d be to retain the RESERVE state. Here is another section on the topic: 6.3.5.1. Loss of Nexus Notification The iSCSI layer provides the SCSI layer with the "I_T nexus loss" notification when any one of the following events happens: Successful completion of session reinstatement. Session closure event. Session timeout event. Certain SCSI object clearing actions may result due to the notification in the SCSI end nodes, as documented in Appendix F. - Clearing Effects of Various Events on Targets. FWIW - the reference above is incorrect - there is no Appendix F in the con= solidated draft - this should be Appendix E (and also, BTW - this is a circ= ular reference). So anyway, it seems like the intent was that session reinstatement would pr= eserve the RESERVE state across the I_T Nexus loss event. =20 So aside from the incorrect reference (and circular nature of it), any thou= ghts on whether we should get explicit about this situation? Should we rem= ain silent; should we tweak Appendix E along these lines (or other lines th= at others might suggest): 4.4.3.1. I_T Nexus State Certain nexus relationships contain an explicit state (e.g., initiator-specific mode pages, and RESERVE state) that may need to be pre= served by the device server [SAM2] in a logical unit through changes or failures in the iSCSI layer (e.g., session failures).... Fred Knight -----Original Message----- From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of M= allikarjun Chadalapaka Sent: Sunday, September 23, 2012 1:34 AM To: david.black@emc.com; storm@ietf.org Subject: Re: [storm] iSCSI drafts - T10 and T11 references Hi David, I have noticed this comment as I am working through all Last Call comments. The references to SAM-3 and SAM-4 are keyed to this paragraph in Section 13= .9: "[SAM2] and [SAM3] specifications note in their informative text that TPGT = value should be non-zero, note that it is incorrect. A zero value is allow= ed as a legal value for TPGT. This discrepancy currently stands corrected i= n [SAM4]." Additionally, SAM4 reference is also used in citing the new ASC/ASCQ codes = that it defines, in Section 11.14.5 of the draft. So, I believe references to all three SAM specs would continue to be approp= riate. The current draft references FC-FS in discussing the NAA format in a few pl= aces, so the current reference at the end is intentionally the same. Is the= older reference a problem? If so, I suppose we can replace all references = to FC-FS-4, but that seems to be still in early stages(0.1 spec posted on A= pril 2012), so perhaps we should use FC-FS3? Thanks. Mallikarjun -----Original Message----- From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of d= avid.black@emc.com Sent: Thursday, July 26, 2012 6:47 AM To: storm@ietf.org Subject: [storm] iSCSI drafts - T10 and T11 references In making up the lists of T10 and T11 documents that are currently referenc= ed, some of those references don't look right to me, for example: - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. - T11: The FC-FS reference is out of date. As part of dealing with the comments from IETF Last Call, we'll need to go = over all of the T10 and T11 references (and should check all the references= ) to make sure that they're up to date and necessary. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From Frederick.Knight@netapp.com Mon Sep 24 05:57:53 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3518821F8685 for ; Mon, 24 Sep 2012 05:57:53 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nZAbbJxN7jkz for ; Mon, 24 Sep 2012 05:57:52 -0700 (PDT) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id 0E50E21F8682 for ; Mon, 24 Sep 2012 05:57:52 -0700 (PDT) X-IronPort-AV: E=Sophos;i="4.80,475,1344236400"; d="scan'208";a="693368624" Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx2-out.netapp.com with ESMTP; 24 Sep 2012 05:57:51 -0700 Received: from vmwexceht05-prd.hq.netapp.com (vmwexceht05-prd.hq.netapp.com [10.106.77.35]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8OCvoDW015381; Mon, 24 Sep 2012 05:57:51 -0700 (PDT) Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.102]) by vmwexceht05-prd.hq.netapp.com ([10.106.77.35]) with mapi id 14.02.0309.002; Mon, 24 Sep 2012 05:57:50 -0700 From: "Knight, Frederick" To: Mallikarjun Chadalapaka , "david.black@emc.com" , "storm@ietf.org" Thread-Topic: iSCSI drafts - T10 and T11 references Thread-Index: Ac1rNRGDFlJk/LPPRq2a5Pa224AxSwuFP5jwAAlZtBAAF2slIAAgt6qw Date: Mon, 24 Sep 2012 12:57:49 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.106.53.53] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [storm] iSCSI drafts - T10 and T11 references X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Sep 2012 12:57:53 -0000 FCP is also not explicit (it defers to the SCSI specs - which do not say). = The closest statements I can find still requires extrapolation: Here are some key points (from SAM 6.3.4 I_T nexus loss): 1) I_T nexus loss clears all ACA conditions; 2) established a UA - one of the following is suggested: "If the logical unit retains state information for the I_T nexus that is lo= st, its response to the subsequent I_T nexus re-establishment for the logical unit should include establishing a unit at= tention with an additional sense code set to I_T NEXUS LOSS OCCURRED. If the logical unit does not retain state information for the I_T nexus tha= t is lost, it shall consider the subsequent I_T nexus re-establishment, if any, as the formation of a new I_T nexus for= which there is no past history (e.g., establish a unit attention with an additional sense code set to POWER ON OC= CURRED)." But, there is no definition of what it means to "retain state". The conclu= sion I could jump to based on #1 above, is that RESERVE state is also clear= ed (this action lines up more with LU RESET than CLEAR TASK SET); and based= on the POWER ON CONDITION UA of 29/01 (or 29/00 - both are power on condit= ions) I could also assume that the RESERVE state is cleared; but based on t= he UA of 29/07 (retain state), I could jump to the conclusion that RESERVE = state is retained. But these are all conclusions I've jump to with very limited basis in the t= ext. Based on what RFC3720 says about explicit state (e.g., mode pages), I would= assume that hosts would believe that every command they explicitly send (l= ike MODE SELECT or RESERVE) would have created explicit state and therefore= the host would assume RESERVE state is retained. It doesn't make sense to= have mode page values retained, but have RESERVE state lost. Certain nexus relationships contain an explicit state (e.g., initiator-specific mode pages) that may need to be preserved by the device server [SAM2] in a logical unit through changes or failures in the iSCSI layer (e.g., session failures). It will be interesting to hear what a real host assumes about this situatio= n. Fred -----Original Message----- From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]=20 Sent: Sunday, September 23, 2012 6:35 PM To: Knight, Frederick; david.black@emc.com; storm@ietf.org Subject: RE: iSCSI drafts - T10 and T11 references Hi Fred, SPC-3 discusses clearing effects of I-T nexus loss in several places. Section 5.8.2.9 for ALUA Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for background sel= f-test Section 5.6.1 on persistent reservations Section 5.13 on timestamps = ...... Thanks for pointing out the broken cross-reference - Appendix F instead of = E. I have just fixed it - and reviewed all other Appendix references elsewh= ere in the draft. However, I do not see a circular reference problem per se= - Appendix E is correctly referencing Section 6.3.5 for definition of term= s, e.g. session continuation. And Section 6.3.5.1 is correctly deferring to= Appendix E when discussing clearing effects. Let me know if I'm overlookin= g something here. Now, the reserve/release management state, :) : I agree the formal I_T nex= us loss event didn't exist in SAM-2, however the notion of a new I_T nexus = state did co-exist with a session-oriented I_T nexus well before iSCSI - FC= P process login (PRLO) comes to mind here. So I'd be curious about traditio= nal FCP handling of reserve/release management state. And I'm also checking= internally here at MS. From what I know now though, it does *appear* that = reserve/release management state should not be cleared with I_T nexus loss. Having said all that, note that reserve/release management state is squarel= y in the SCSI territory. iSCSI spec has so far by design deferred to SCSI s= tandards in specifying clearing effects on SCSI objects - as Appendix E.2 m= akes it clear. What we do specify is the clearing effects of SCSI actions, = e.g. LU reset, on iSCSI objects. Thanks. Mallikarjun -----Original Message----- From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] Sent: Sunday, September 23, 2012 3:38 AM To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org Subject: RE: iSCSI drafts - T10 and T11 references Interesting, I just came across a question in a related area (only for SAM-= 2). RFC3720 was written in between SAM-2 and SAM-3; some concepts were drawn fr= om SAM-2, and some from SAM-3. We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus loss= concept does not exist in SAM-2. SAM-2 does include references to RESERVE= and RELEASE commands. SAM-3 includes I_T Nexus loss, but it has no refere= nce to RESERVE state. There are allusions in RFC3720 about the intersectio= n of RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit. RF= C3720 references other documents (SAM2, SAM3, SBC, SPC3), none of which add= ress the situation. Consider text such as the following: 4.4.3.1. I_T Nexus State Certain nexus relationships contain an explicit state (e.g., initiator-specific mode pages) that may need to be preserved by the device server [SAM2] in a logical unit through changes or failures in the iSCSI layer (e.g., session failures). In order for that state to be restored, the iSCSI initiator should reestablish its session (re-login) to the same Target Portal Group using the previous ISID. That is, it should perform session recovery as described in Chapter 6. This is because the SCSI initiator port identifier and the SCSI target port identifier (or relative target port) form the datum that the SCSI logical unit device server uses to identify the I_T nexus. Is RESERVE state - explicit state? When an I_T nexus is reestablished, ini= tiator-specific mode pages are preserved. That implies that an I_T nexus l= oss should be treated in the way that SAM-2 describes via CLEAR TASK SET (w= here mode pages and RESERVE state are preserved). If a session reestablish= ment caused saved mode pages to be restored rather than the established ini= tiator-specific mode pages to be restored, then it would be like a LU reset= (where the RESERVE state is lost). Is RESERVE state explicit state that i= s to be preserved? Appendix E covers clearing effects during various events, but it never ment= ions RESERVE state. Section E.2 contains the following text: The only iSCSI protocol action that can effect clearing actions on SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 Loss of Nexus notification). [SPC3] describes the clearing effects of this notification on a variety of SCSI attributes. In addition, SCSI standards documents (such as [SAM2] and [SBC]) define additional clearing actions that may take place for several SCSI objects on SCSI events such as LU resets and power-on resets. I could not find any text in SPC3 that describes these clearing effects (I = don't know if this SPC3 reference is incorrect, but at a minimum, it is ine= ffective). The SAM2 text for LU resets and power-on resets is easily found= , and clearly says that any RESERVE is released in those cases. As an asid= e, the reference to section 6.3.5.1 is a circular reference). The text in = E.2 goes on: Since iSCSI defines a target cold reset as a protocol-equivalent to a target power-cycle, the iSCSI target cold reset must also be considered as the power-on reset event in interpreting the actions defined in the SCSI standards. When the iSCSI session is reconstructed (between the same SCSI ports with the same nexus identifier) reestablishing the same I_T nexus, all SCSI objects that are defined to not clear on the "I_T nexus loss" notification event, such as persistent reservations, are automatically associated to this new session. So, it is clear what happens to PERSISTENT RESERVE state, but not so clear = about RESERVE state. It also states that all SCSI objects that are defined= to not clear on the I_T nexus loss are retained. But again, the only list= I can find of what is cleared is here in this Appendix, since SAM-2, SAM-3= , SAM-4 are all silent on the interaction between RESERVE and I_T Nexus los= s. T10 has never had to deal with this because they removed RESERVE before the= y added I_T Nexus loss. However, iSCSI did not do that; we discuss BOTH, b= ut we fail to describe how they interact. iSCSI also has the feature of re= establishment of a session that does not exist in SAM-2 (SAM-2 was still pa= rallel SCSI focused, where connections didn't exist). So again, if session= reestablishment is examined in the light of parallel SCSI, the intent woul= d be to retain the RESERVE state. Here is another section on the topic: 6.3.5.1. Loss of Nexus Notification The iSCSI layer provides the SCSI layer with the "I_T nexus loss" notification when any one of the following events happens: Successful completion of session reinstatement. Session closure event. Session timeout event. Certain SCSI object clearing actions may result due to the notification in the SCSI end nodes, as documented in Appendix F. - Clearing Effects of Various Events on Targets. FWIW - the reference above is incorrect - there is no Appendix F in the con= solidated draft - this should be Appendix E (and also, BTW - this is a circ= ular reference). So anyway, it seems like the intent was that session reinstatement would pr= eserve the RESERVE state across the I_T Nexus loss event. =20 So aside from the incorrect reference (and circular nature of it), any thou= ghts on whether we should get explicit about this situation? Should we rem= ain silent; should we tweak Appendix E along these lines (or other lines th= at others might suggest): 4.4.3.1. I_T Nexus State Certain nexus relationships contain an explicit state (e.g., initiator-specific mode pages, and RESERVE state) that may need to be pre= served by the device server [SAM2] in a logical unit through changes or failures in the iSCSI layer (e.g., session failures).... Fred Knight -----Original Message----- From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of M= allikarjun Chadalapaka Sent: Sunday, September 23, 2012 1:34 AM To: david.black@emc.com; storm@ietf.org Subject: Re: [storm] iSCSI drafts - T10 and T11 references Hi David, I have noticed this comment as I am working through all Last Call comments. The references to SAM-3 and SAM-4 are keyed to this paragraph in Section 13= .9: "[SAM2] and [SAM3] specifications note in their informative text that TPGT = value should be non-zero, note that it is incorrect. A zero value is allow= ed as a legal value for TPGT. This discrepancy currently stands corrected i= n [SAM4]." Additionally, SAM4 reference is also used in citing the new ASC/ASCQ codes = that it defines, in Section 11.14.5 of the draft. So, I believe references to all three SAM specs would continue to be approp= riate. The current draft references FC-FS in discussing the NAA format in a few pl= aces, so the current reference at the end is intentionally the same. Is the= older reference a problem? If so, I suppose we can replace all references = to FC-FS-4, but that seems to be still in early stages(0.1 spec posted on A= pril 2012), so perhaps we should use FC-FS3? Thanks. Mallikarjun -----Original Message----- From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of d= avid.black@emc.com Sent: Thursday, July 26, 2012 6:47 AM To: storm@ietf.org Subject: [storm] iSCSI drafts - T10 and T11 references In making up the lists of T10 and T11 documents that are currently referenc= ed, some of those references don't look right to me, for example: - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. - T11: The FC-FS reference is out of date. As part of dealing with the comments from IETF Last Call, we'll need to go = over all of the T10 and T11 references (and should check all the references= ) to make sure that they're up to date and necessary. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From david.black@emc.com Mon Sep 24 08:37:06 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7737121F87D2 for ; Mon, 24 Sep 2012 08:37:06 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mnXzRamarsAC for ; Mon, 24 Sep 2012 08:37:05 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 1636021F87C1 for ; Mon, 24 Sep 2012 08:37:04 -0700 (PDT) Received: from hop04-l1d11-si04.isus.emc.com (HOP04-L1D11-SI04.isus.emc.com [10.254.111.24]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8OFax6F006317 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 24 Sep 2012 11:37:02 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.221.251]) by hop04-l1d11-si04.isus.emc.com (RSA Interceptor); Mon, 24 Sep 2012 11:36:45 -0400 Received: from mxhub01.corp.emc.com (mxhub01.corp.emc.com [10.254.141.103]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8OFagMH016732; Mon, 24 Sep 2012 11:36:44 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub01.corp.emc.com ([10.254.141.103]) with mapi; Mon, 24 Sep 2012 11:36:42 -0400 From: "Black, David" To: "Knight, Frederick" , Mallikarjun Chadalapaka , "storm@ietf.org" Date: Mon, 24 Sep 2012 11:36:41 -0400 Thread-Topic: iSCSI drafts - T10 and T11 references Thread-Index: Ac1rNRGDFlJk/LPPRq2a5Pa224AxSwuFP5jwAAlZtBAAPXLRYA== Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE79192@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: Re: [storm] iSCSI drafts - T10 and T11 references X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Sep 2012 15:37:06 -0000 Fred, Thank you for the discussion, particularly around the dependence on SAM-3. In light of that, and a few other things (e.g., a strong preference that the iSCSI consolidated draft not reference work in progress), my recommendations on the T10 and T11 references are: --- T10 (SCSI) --- 1) [SBC2] and [SPC3] are the current published standards; those should be t= he correct references for the iSCSI consolidated draft. [SPC3] is clearly normative, but I think [SBC2] is (and ought to be) informative - [SBC2] is only cited once in what looks like an informative fashion in Appendix E.2: In addition, SCSI standards documents (such as [SAM2] and [SBC2]) define additional clearing actions that may take place for several SCSI objects on SCSI events such as LU resets and power-on resets. Moreover, there is no requirement to implement the SBC command set for all iSCSI implementations, whereas the SPC command set is a requirement for obvious reasons (P =3D Primary). 2) [SAM2] and [SAM3] ought to be normative references, in that we're drawin= g some concepts from [SAM-3]. OTOH, [SAM-4] ought to be an informative reference, as the TPGT text that is referred to in SAM-4: > "[SAM2] and [SAM3] specifications note in their informative text that TPG= T > value should be non-zero, note that it is incorrect. A zero value is all= owed > as a legal value for TPGT. This discrepancy currently stands corrected in > [SAM4]." is an informative table footnote in an annex. Further, > Additionally, SAM4 reference is also used in citing the new ASC/ASCQ code= s > that it defines, in Section 11.14.5 of the draft. That new ASC/Q code's name does not occur in the body of SAM-4, so I would= =20 remove that second citation of SAM-4. 3) The [SAS] reference needs to be expanded ;-). The current informative [SAS] reference needs to be replaced by references to SAS-2.1, SPL and SPL Amendment 1, see: http://www.t10.org/drafts.htm#SCSI3_SAS 4) OTOH, I recommend removing the [SRP] reference and related material, 'namely the definition of "SCSI RDMA Protocol" and the alternative expansio= n of the SRP acronym. The SRP-2 standards project to update SRP was abandone= d by T10 in 2005 due to lack of interest (see T10/05-097r0), leaving us with a vintage-2002 SRP document. --- T11 (Fibre Channel) --- The FC-FS reference ought to be replaced by a reference to the current published FS-FS-3 standard. Thanks, --David > -----Original Message----- > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > Sent: Sunday, September 23, 2012 6:38 AM > To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references >=20 > Interesting, I just came across a question in a related area (only for SA= M-2). >=20 > RFC3720 was written in between SAM-2 and SAM-3; some concepts were drawn = from > SAM-2, and some from SAM-3. >=20 > We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus lo= ss > concept does not exist in SAM-2. SAM-2 does include references to RESERV= E and > RELEASE commands. SAM-3 includes I_T Nexus loss, but it has no reference= to > RESERVE state. There are allusions in RFC3720 about the intersection of > RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit. RFC37= 20 > references other documents (SAM2, SAM3, SBC, SPC3), none of which address= the > situation. >=20 > Consider text such as the following: > 4.4.3.1. I_T Nexus State >=20 > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages) that may need to be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures). In order for > that state to be restored, the iSCSI initiator should reestablish > its session (re-login) to the same Target Portal Group using the > previous ISID. That is, it should perform session recovery as > described in Chapter 6. This is because the SCSI initiator port > identifier and the SCSI target port identifier (or relative target > port) form the datum that the SCSI logical unit device server uses > to identify the I_T nexus. >=20 > Is RESERVE state - explicit state? When an I_T nexus is reestablished, > initiator-specific mode pages are preserved. That implies that an I_T ne= xus > loss should be treated in the way that SAM-2 describes via CLEAR TASK SET > (where mode pages and RESERVE state are preserved). If a session > reestablishment caused saved mode pages to be restored rather than the > established initiator-specific mode pages to be restored, then it would b= e > like a LU reset (where the RESERVE state is lost). Is RESERVE state expl= icit > state that is to be preserved? >=20 > Appendix E covers clearing effects during various events, but it never > mentions RESERVE state. >=20 > Section E.2 contains the following text: >=20 > The only iSCSI protocol action that can effect clearing actions on > SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 > Loss of Nexus notification). [SPC3] describes the clearing effects > of this notification on a variety of SCSI attributes. In addition, > SCSI standards documents (such as [SAM2] and [SBC]) define > additional clearing actions that may take place for several SCSI > objects on SCSI events such as LU resets and power-on resets. >=20 > I could not find any text in SPC3 that describes these clearing effects (= I > don't know if this SPC3 reference is incorrect, but at a minimum, it is > ineffective). The SAM2 text for LU resets and power-on resets is easily > found, and clearly says that any RESERVE is released in those cases. As = an > aside, the reference to section 6.3.5.1 is a circular reference). The te= xt in > E.2 goes on: >=20 > Since iSCSI defines a target cold reset as a protocol-equivalent > to a target power-cycle, the iSCSI target cold reset must also be > considered as the power-on reset event in interpreting the actions > defined in the SCSI standards. >=20 > When the iSCSI session is reconstructed (between the same SCSI > ports with the same nexus identifier) reestablishing the same I_T > nexus, all SCSI objects that are defined to not clear on the "I_T > nexus loss" notification event, such as persistent reservations, > are automatically associated to this new session. >=20 > So, it is clear what happens to PERSISTENT RESERVE state, but not so clea= r > about RESERVE state. It also states that all SCSI objects that are defin= ed to > not clear on the I_T nexus loss are retained. But again, the only list I= can > find of what is cleared is here in this Appendix, since SAM-2, SAM-3, SAM= -4 > are all silent on the interaction between RESERVE and I_T Nexus loss. >=20 > T10 has never had to deal with this because they removed RESERVE before t= hey > added I_T Nexus loss. However, iSCSI did not do that; we discuss BOTH, b= ut we > fail to describe how they interact. iSCSI also has the feature of > reestablishment of a session that does not exist in SAM-2 (SAM-2 was stil= l > parallel SCSI focused, where connections didn't exist). So again, if ses= sion > reestablishment is examined in the light of parallel SCSI, the intent wou= ld be > to retain the RESERVE state. Here is another section on the topic: >=20 > 6.3.5.1. Loss of Nexus Notification >=20 > The iSCSI layer provides the SCSI layer with the "I_T nexus loss" > notification when any one of the following events happens: >=20 > Successful completion of session reinstatement. > Session closure event. > Session timeout event. >=20 > Certain SCSI object clearing actions may result due to the > notification in the SCSI end nodes, as documented in Appendix F. - > Clearing Effects of Various Events on Targets. >=20 > FWIW - the reference above is incorrect - there is no Appendix F in the > consolidated draft - this should be Appendix E (and also, BTW - this is a > circular reference). >=20 > So anyway, it seems like the intent was that session reinstatement would > preserve the RESERVE state across the I_T Nexus loss event. >=20 > So aside from the incorrect reference (and circular nature of it), any > thoughts on whether we should get explicit about this situation? Should = we > remain silent; should we tweak Appendix E along these lines (or other lin= es > that others might suggest): >=20 > 4.4.3.1. I_T Nexus State >=20 > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages, and RESERVE state) that may need to be > preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures).... >=20 > Fred Knight >=20 >=20 >=20 > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of > Mallikarjun Chadalapaka > Sent: Sunday, September 23, 2012 1:34 AM > To: david.black@emc.com; storm@ietf.org > Subject: Re: [storm] iSCSI drafts - T10 and T11 references >=20 > Hi David, >=20 > I have noticed this comment as I am working through all Last Call comment= s. >=20 > The references to SAM-3 and SAM-4 are keyed to this paragraph in Section = 13.9: >=20 > "[SAM2] and [SAM3] specifications note in their informative text that TPG= T > value should be non-zero, note that it is incorrect. A zero value is all= owed > as a legal value for TPGT. This discrepancy currently stands corrected in > [SAM4]." >=20 > Additionally, SAM4 reference is also used in citing the new ASC/ASCQ code= s > that it defines, in Section 11.14.5 of the draft. >=20 > So, I believe references to all three SAM specs would continue to be > appropriate. >=20 > The current draft references FC-FS in discussing the NAA format in a few > places, so the current reference at the end is intentionally the same. Is= the > older reference a problem? If so, I suppose we can replace all references= to > FC-FS-4, but that seems to be still in early stages(0.1 spec posted on Ap= ril > 2012), so perhaps we should use FC-FS3? >=20 > Thanks. >=20 > Mallikarjun >=20 >=20 >=20 >=20 > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Thursday, July 26, 2012 6:47 AM > To: storm@ietf.org > Subject: [storm] iSCSI drafts - T10 and T11 references >=20 > In making up the lists of T10 and T11 documents that are currently refere= nced, > some of those references don't look right to me, for example: >=20 > - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. > - T11: The FC-FS reference is out of date. >=20 > As part of dealing with the comments from IETF Last Call, we'll need to g= o > over all of the T10 and T11 references (and should check all the referenc= es) > to make sure that they're up to date and necessary. >=20 > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer > EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 > +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-7= 786 > david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 > ---------------------------------------------------- >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm >=20 >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From david.black@emc.com Mon Sep 24 09:42:46 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D9B2121F881C for ; Mon, 24 Sep 2012 09:42:46 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CigaHLpLZ45m for ; Mon, 24 Sep 2012 09:42:45 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 5493721F8813 for ; Mon, 24 Sep 2012 09:42:45 -0700 (PDT) Received: from hop04-l1d11-si01.isus.emc.com (HOP04-L1D11-SI01.isus.emc.com [10.254.111.54]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8OGgcBx023846 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 24 Sep 2012 12:42:42 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.221.253]) by hop04-l1d11-si01.isus.emc.com (RSA Interceptor); Mon, 24 Sep 2012 12:42:15 -0400 Received: from mxhub10.corp.emc.com (mxhub10.corp.emc.com [10.254.92.105]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8OGg9IM014209; Mon, 24 Sep 2012 12:42:09 -0400 Received: from mxhub38.corp.emc.com (128.222.70.105) by mxhub10.corp.emc.com (10.254.92.105) with Microsoft SMTP Server (TLS) id 8.3.213.0; Mon, 24 Sep 2012 12:42:09 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub38.corp.emc.com ([128.222.70.105]) with mapi; Mon, 24 Sep 2012 12:42:09 -0400 From: "Black, David" To: "Knight, Frederick" , Mallikarjun Chadalapaka , "storm@ietf.org" Date: Mon, 24 Sep 2012 12:42:07 -0400 Thread-Topic: RESERVE state - clearing effects Thread-Index: Ac1rNRGDFlJk/LPPRq2a5Pa224AxSwuFP5jwAAlZtBAAF2slIAAgt6qwAAaehPA= Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: [storm] RESERVE state - clearing effects X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Sep 2012 16:42:47 -0000 Fred, This is interesting ... > Here are some key points (from SAM 6.3.4 I_T nexus loss): > > 1) I_T nexus loss clears all ACA conditions; > 2) established a UA - one of the following is suggested: > > "If the logical unit retains state information for the I_T nexus that > is lost, its response to the subsequent I_T nexus > re-establishment for the logical unit should include establishing a > unit attention with an additional sense code set > to I_T NEXUS LOSS OCCURRED. > > If the logical unit does not retain state information for the I_T nexus > that is lost, it shall consider the subsequent > I_T nexus re-establishment, if any, as the formation of a new I_T nexus > for which there is no past history (e.g., > establish a unit attention with an additional sense code set to POWER ON > OCCURRED)." That "state information" has to identify the endpoints of the I_T Nexus - for iSCSI that means at least the ISID and TPGT. If that's gone, I think the RESERVE state is gone as well, in contrast to PERSISTENT RESERVE state. On RESERVE state, SPC-2 has this to say in 5.5.2 "The Reserve/Release management method": The reserve/release management method commands, RESERVE(6), RESERVE(10), RELEASE(6), and RELEASE(10) are used among multiple initiators that do not require operations to be protected across initiator failures (and subsequent hard resets). I think it's fair to regard I_T Nexus loss as an initiator failure. I also don't see any good way to do a 3rd-party release on a RESERVE reservation that persists after I_T Nexus loss. I also believe that initiator software that uses Reserve/Release does not expect reservations to persist across I_T Nexus loss. As to what needs to be stated in the iSCSI consolidated draft - the SPC reference has been updated to SPC-3 in which the Reserve/Release management method is obsolete, so there isn't any absolute requirement to cover that. OTOH, it might be reasonable to add a sentence to say that Reserve/Release state is implicit (e.g., as there's no way for an initiator to figure out which I_T Nexus holds a reservation), and that treating Reserve/Release state as indefinitely persistent after I_T Nexus Loss is a bad idea ("should not", but lower case) as it may require the initiator to reset the target device in order to clear out that state (issuing RELEASE commands isn't enough). This sort of text would require a reference to SPC-2. Thanks, --David > -----Original Message----- > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > Sent: Monday, September 24, 2012 8:58 AM > To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references > > FCP is also not explicit (it defers to the SCSI specs - which do not say)= . > The closest statements I can find still requires extrapolation: > > Here are some key points (from SAM 6.3.4 I_T nexus loss): > > 1) I_T nexus loss clears all ACA conditions; > 2) established a UA - one of the following is suggested: > > "If the logical unit retains state information for the I_T nexus that is = lost, > its response to the subsequent I_T nexus > re-establishment for the logical unit should include establishing a unit > attention with an additional sense code set > to I_T NEXUS LOSS OCCURRED. > > If the logical unit does not retain state information for the I_T nexus t= hat > is lost, it shall consider the subsequent > I_T nexus re-establishment, if any, as the formation of a new I_T nexus f= or > which there is no past history (e.g., > establish a unit attention with an additional sense code set to POWER ON > OCCURRED)." > > But, there is no definition of what it means to "retain state". The > conclusion I could jump to based on #1 above, is that RESERVE state is al= so > cleared (this action lines up more with LU RESET than CLEAR TASK SET); an= d > based on the POWER ON CONDITION UA of 29/01 (or 29/00 - both are power on > conditions) I could also assume that the RESERVE state is cleared; but ba= sed > on the UA of 29/07 (retain state), I could jump to the conclusion that RE= SERVE > state is retained. > > But these are all conclusions I've jump to with very limited basis in the > text. > > Based on what RFC3720 says about explicit state (e.g., mode pages), I wou= ld > assume that hosts would believe that every command they explicitly send (= like > MODE SELECT or RESERVE) would have created explicit state and therefore t= he > host would assume RESERVE state is retained. It doesn't make sense to ha= ve > mode page values retained, but have RESERVE state lost. > > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages) that may need to be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures). > > It will be interesting to hear what a real host assumes about this situat= ion. > > Fred > > -----Original Message----- > From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] > Sent: Sunday, September 23, 2012 6:35 PM > To: Knight, Frederick; david.black@emc.com; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references > > Hi Fred, > > SPC-3 discusses clearing effects of I-T nexus loss in several places. > > Section 5.8.2.9 for ALUA > Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for background s= elf- > test Section 5.6.1 on persistent reservations Section 5.13 on timestamps > ...... > > Thanks for pointing out the broken cross-reference - Appendix F instead o= f E. > I have just fixed it - and reviewed all other Appendix references elsewhe= re in > the draft. However, I do not see a circular reference problem per se - > Appendix E is correctly referencing Section 6.3.5 for definition of terms= , > e.g. session continuation. And Section 6.3.5.1 is correctly deferring to > Appendix E when discussing clearing effects. Let me know if I'm overlooki= ng > something here. > > Now, the reserve/release management state, :) : I agree the formal I_T n= exus > loss event didn't exist in SAM-2, however the notion of a new I_T nexus s= tate > did co-exist with a session-oriented I_T nexus well before iSCSI - FCP pr= ocess > login (PRLO) comes to mind here. So I'd be curious about traditional FCP > handling of reserve/release management state. And I'm also checking inter= nally > here at MS. From what I know now though, it does *appear* that reserve/re= lease > management state should not be cleared with I_T nexus loss. > > Having said all that, note that reserve/release management state is squar= ely > in the SCSI territory. iSCSI spec has so far by design deferred to SCSI > standards in specifying clearing effects on SCSI objects - as Appendix E.= 2 > makes it clear. What we do specify is the clearing effects of SCSI action= s, > e.g. LU reset, on iSCSI objects. > > Thanks. > > Mallikarjun > > > > > > -----Original Message----- > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > Sent: Sunday, September 23, 2012 3:38 AM > To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references > > Interesting, I just came across a question in a related area (only for SA= M-2). > > RFC3720 was written in between SAM-2 and SAM-3; some concepts were drawn = from > SAM-2, and some from SAM-3. > > We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus lo= ss > concept does not exist in SAM-2. SAM-2 does include references to RESERV= E and > RELEASE commands. SAM-3 includes I_T Nexus loss, but it has no reference= to > RESERVE state. There are allusions in RFC3720 about the intersection of > RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit. RFC37= 20 > references other documents (SAM2, SAM3, SBC, SPC3), none of which address= the > situation. > > Consider text such as the following: > 4.4.3.1. I_T Nexus State > > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages) that may need to be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures). In order for > that state to be restored, the iSCSI initiator should reestablish > its session (re-login) to the same Target Portal Group using the > previous ISID. That is, it should perform session recovery as > described in Chapter 6. This is because the SCSI initiator port > identifier and the SCSI target port identifier (or relative target > port) form the datum that the SCSI logical unit device server uses > to identify the I_T nexus. > > Is RESERVE state - explicit state? When an I_T nexus is reestablished, > initiator-specific mode pages are preserved. That implies that an I_T ne= xus > loss should be treated in the way that SAM-2 describes via CLEAR TASK SET > (where mode pages and RESERVE state are preserved). If a session > reestablishment caused saved mode pages to be restored rather than the > established initiator-specific mode pages to be restored, then it would b= e > like a LU reset (where the RESERVE state is lost). Is RESERVE state expl= icit > state that is to be preserved? > > Appendix E covers clearing effects during various events, but it never > mentions RESERVE state. > > Section E.2 contains the following text: > > The only iSCSI protocol action that can effect clearing actions on > SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 > Loss of Nexus notification). [SPC3] describes the clearing effects > of this notification on a variety of SCSI attributes. In addition, > SCSI standards documents (such as [SAM2] and [SBC]) define > additional clearing actions that may take place for several SCSI > objects on SCSI events such as LU resets and power-on resets. > > I could not find any text in SPC3 that describes these clearing effects (= I > don't know if this SPC3 reference is incorrect, but at a minimum, it is > ineffective). The SAM2 text for LU resets and power-on resets is easily > found, and clearly says that any RESERVE is released in those cases. As = an > aside, the reference to section 6.3.5.1 is a circular reference). The te= xt in > E.2 goes on: > > Since iSCSI defines a target cold reset as a protocol-equivalent > to a target power-cycle, the iSCSI target cold reset must also be > considered as the power-on reset event in interpreting the actions > defined in the SCSI standards. > > When the iSCSI session is reconstructed (between the same SCSI > ports with the same nexus identifier) reestablishing the same I_T > nexus, all SCSI objects that are defined to not clear on the "I_T > nexus loss" notification event, such as persistent reservations, > are automatically associated to this new session. > > So, it is clear what happens to PERSISTENT RESERVE state, but not so clea= r > about RESERVE state. It also states that all SCSI objects that are defin= ed to > not clear on the I_T nexus loss are retained. But again, the only list I= can > find of what is cleared is here in this Appendix, since SAM-2, SAM-3, SAM= -4 > are all silent on the interaction between RESERVE and I_T Nexus loss. > > T10 has never had to deal with this because they removed RESERVE before t= hey > added I_T Nexus loss. However, iSCSI did not do that; we discuss BOTH, b= ut we > fail to describe how they interact. iSCSI also has the feature of > reestablishment of a session that does not exist in SAM-2 (SAM-2 was stil= l > parallel SCSI focused, where connections didn't exist). So again, if ses= sion > reestablishment is examined in the light of parallel SCSI, the intent wou= ld be > to retain the RESERVE state. Here is another section on the topic: > > 6.3.5.1. Loss of Nexus Notification > > The iSCSI layer provides the SCSI layer with the "I_T nexus loss" > notification when any one of the following events happens: > > Successful completion of session reinstatement. > Session closure event. > Session timeout event. > > Certain SCSI object clearing actions may result due to the > notification in the SCSI end nodes, as documented in Appendix F. - > Clearing Effects of Various Events on Targets. > > FWIW - the reference above is incorrect - there is no Appendix F in the > consolidated draft - this should be Appendix E (and also, BTW - this is a > circular reference). > > So anyway, it seems like the intent was that session reinstatement would > preserve the RESERVE state across the I_T Nexus loss event. > > So aside from the incorrect reference (and circular nature of it), any > thoughts on whether we should get explicit about this situation? Should = we > remain silent; should we tweak Appendix E along these lines (or other lin= es > that others might suggest): > > 4.4.3.1. I_T Nexus State > > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages, and RESERVE state) that may need to be > preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures).... > > Fred Knight > > > > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of > Mallikarjun Chadalapaka > Sent: Sunday, September 23, 2012 1:34 AM > To: david.black@emc.com; storm@ietf.org > Subject: Re: [storm] iSCSI drafts - T10 and T11 references > > Hi David, > > I have noticed this comment as I am working through all Last Call comment= s. > > The references to SAM-3 and SAM-4 are keyed to this paragraph in Section = 13.9: > > "[SAM2] and [SAM3] specifications note in their informative text that TPG= T > value should be non-zero, note that it is incorrect. A zero value is all= owed > as a legal value for TPGT. This discrepancy currently stands corrected in > [SAM4]." > > Additionally, SAM4 reference is also used in citing the new ASC/ASCQ code= s > that it defines, in Section 11.14.5 of the draft. > > So, I believe references to all three SAM specs would continue to be > appropriate. > > The current draft references FC-FS in discussing the NAA format in a few > places, so the current reference at the end is intentionally the same. Is= the > older reference a problem? If so, I suppose we can replace all references= to > FC-FS-4, but that seems to be still in early stages(0.1 spec posted on Ap= ril > 2012), so perhaps we should use FC-FS3? > > Thanks. > > Mallikarjun > > > > > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Thursday, July 26, 2012 6:47 AM > To: storm@ietf.org > Subject: [storm] iSCSI drafts - T10 and T11 references > > In making up the lists of T10 and T11 documents that are currently refere= nced, > some of those references don't look right to me, for example: > > - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. > - T11: The FC-FS reference is out of date. > > As part of dealing with the comments from IETF Last Call, we'll need to g= o > over all of the T10 and T11 references (and should check all the referenc= es) > to make sure that they're up to date and necessary. > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > david.black@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > From roweber@ieee.org Mon Sep 24 10:29:03 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8994C21F8821 for ; Mon, 24 Sep 2012 10:29:03 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PbpJRcKsnae5 for ; Mon, 24 Sep 2012 10:29:02 -0700 (PDT) Received: from vms173015pub.verizon.net (vms173015pub.verizon.net [206.46.173.15]) by ietfa.amsl.com (Postfix) with ESMTP id 20ADF21F8790 for ; Mon, 24 Sep 2012 10:29:01 -0700 (PDT) Received: from [192.168.1.3] ([unknown] [96.226.101.225]) by vms173015.mailsrvcs.net (Sun Java(tm) System Messaging Server 7u2-7.02 32bit (built Apr 16 2009)) with ESMTPA id <0MAV007VJ77HEP00@vms173015.mailsrvcs.net> for storm@ietf.org; Mon, 24 Sep 2012 12:28:30 -0500 (CDT) Message-id: <50609840.7060909@ieee.org> Date: Mon, 24 Sep 2012 12:28:32 -0500 From: Ralph Weber User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120907 Thunderbird/15.0.1 MIME-version: 1.0 To: storm@ietf.org References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> In-reply-to: <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7bit Subject: Re: [storm] RESERVE state - clearing effects X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Sep 2012 17:29:03 -0000 Because there is no practical way to say anything about this subject in the modern SCSI standards (i.e., RESERVE/RELEASE has been obsolete for years), I support adding a few well-chosen words to the consolidated iSCSI draft/RFC. Thanks, Ralph Weber On 9/24/2012 11:42 AM, Black, David wrote: > Fred, > > This is interesting ... > >> Here are some key points (from SAM 6.3.4 I_T nexus loss): >> >> 1) I_T nexus loss clears all ACA conditions; >> 2) established a UA - one of the following is suggested: >> >> "If the logical unit retains state information for the I_T nexus that >> is lost, its response to the subsequent I_T nexus >> re-establishment for the logical unit should include establishing a >> unit attention with an additional sense code set >> to I_T NEXUS LOSS OCCURRED. >> >> If the logical unit does not retain state information for the I_T nexus >> that is lost, it shall consider the subsequent >> I_T nexus re-establishment, if any, as the formation of a new I_T nexus >> for which there is no past history (e.g., >> establish a unit attention with an additional sense code set to POWER ON >> OCCURRED)." > That "state information" has to identify the endpoints of the I_T Nexus > - for iSCSI that means at least the ISID and TPGT. If that's gone, I > think the RESERVE state is gone as well, in contrast to PERSISTENT > RESERVE state. On RESERVE state, SPC-2 has this to say in 5.5.2 "The > Reserve/Release management method": > > The reserve/release management method commands, RESERVE(6), > RESERVE(10), RELEASE(6), and RELEASE(10) are used among multiple > initiators that do not require operations to be protected across > initiator failures (and subsequent hard resets). > > I think it's fair to regard I_T Nexus loss as an initiator failure. I > also don't see any good way to do a 3rd-party release on a RESERVE > reservation that persists after I_T Nexus loss. I also believe that > initiator software that uses Reserve/Release does not expect > reservations to persist across I_T Nexus loss. > > As to what needs to be stated in the iSCSI consolidated draft - the SPC > reference has been updated to SPC-3 in which the Reserve/Release > management method is obsolete, so there isn't any absolute requirement > to cover that. OTOH, it might be reasonable to add a sentence to say > that Reserve/Release state is implicit (e.g., as there's no way for an > initiator to figure out which I_T Nexus holds a reservation), and that > treating Reserve/Release state as indefinitely persistent after I_T > Nexus Loss is a bad idea ("should not", but lower case) as it may > require the initiator to reset the target device in order to clear out > that state (issuing RELEASE commands isn't enough). This sort of text > would require a reference to SPC-2. > > Thanks, > --David > >> -----Original Message----- >> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] >> Sent: Monday, September 24, 2012 8:58 AM >> To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org >> Subject: RE: iSCSI drafts - T10 and T11 references >> >> FCP is also not explicit (it defers to the SCSI specs - which do not say). >> The closest statements I can find still requires extrapolation: >> >> Here are some key points (from SAM 6.3.4 I_T nexus loss): >> >> 1) I_T nexus loss clears all ACA conditions; >> 2) established a UA - one of the following is suggested: >> >> "If the logical unit retains state information for the I_T nexus that is lost, >> its response to the subsequent I_T nexus >> re-establishment for the logical unit should include establishing a unit >> attention with an additional sense code set >> to I_T NEXUS LOSS OCCURRED. >> >> If the logical unit does not retain state information for the I_T nexus that >> is lost, it shall consider the subsequent >> I_T nexus re-establishment, if any, as the formation of a new I_T nexus for >> which there is no past history (e.g., >> establish a unit attention with an additional sense code set to POWER ON >> OCCURRED)." >> >> But, there is no definition of what it means to "retain state". The >> conclusion I could jump to based on #1 above, is that RESERVE state is also >> cleared (this action lines up more with LU RESET than CLEAR TASK SET); and >> based on the POWER ON CONDITION UA of 29/01 (or 29/00 - both are power on >> conditions) I could also assume that the RESERVE state is cleared; but based >> on the UA of 29/07 (retain state), I could jump to the conclusion that RESERVE >> state is retained. >> >> But these are all conclusions I've jump to with very limited basis in the >> text. >> >> Based on what RFC3720 says about explicit state (e.g., mode pages), I would >> assume that hosts would believe that every command they explicitly send (like >> MODE SELECT or RESERVE) would have created explicit state and therefore the >> host would assume RESERVE state is retained. It doesn't make sense to have >> mode page values retained, but have RESERVE state lost. >> >> Certain nexus relationships contain an explicit state (e.g., >> initiator-specific mode pages) that may need to be preserved by >> the device server [SAM2] in a logical unit through changes or >> failures in the iSCSI layer (e.g., session failures). >> >> It will be interesting to hear what a real host assumes about this situation. >> >> Fred >> >> -----Original Message----- >> From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] >> Sent: Sunday, September 23, 2012 6:35 PM >> To: Knight, Frederick; david.black@emc.com; storm@ietf.org >> Subject: RE: iSCSI drafts - T10 and T11 references >> >> Hi Fred, >> >> SPC-3 discusses clearing effects of I-T nexus loss in several places. >> >> Section 5.8.2.9 for ALUA >> Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for background self- >> test Section 5.6.1 on persistent reservations Section 5.13 on timestamps >> ...... >> >> Thanks for pointing out the broken cross-reference - Appendix F instead of E. >> I have just fixed it - and reviewed all other Appendix references elsewhere in >> the draft. However, I do not see a circular reference problem per se - >> Appendix E is correctly referencing Section 6.3.5 for definition of terms, >> e.g. session continuation. And Section 6.3.5.1 is correctly deferring to >> Appendix E when discussing clearing effects. Let me know if I'm overlooking >> something here. >> >> Now, the reserve/release management state, :) : I agree the formal I_T nexus >> loss event didn't exist in SAM-2, however the notion of a new I_T nexus state >> did co-exist with a session-oriented I_T nexus well before iSCSI - FCP process >> login (PRLO) comes to mind here. So I'd be curious about traditional FCP >> handling of reserve/release management state. And I'm also checking internally >> here at MS. From what I know now though, it does *appear* that reserve/release >> management state should not be cleared with I_T nexus loss. >> >> Having said all that, note that reserve/release management state is squarely >> in the SCSI territory. iSCSI spec has so far by design deferred to SCSI >> standards in specifying clearing effects on SCSI objects - as Appendix E.2 >> makes it clear. What we do specify is the clearing effects of SCSI actions, >> e.g. LU reset, on iSCSI objects. >> >> Thanks. >> >> Mallikarjun >> >> >> >> >> >> -----Original Message----- >> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] >> Sent: Sunday, September 23, 2012 3:38 AM >> To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org >> Subject: RE: iSCSI drafts - T10 and T11 references >> >> Interesting, I just came across a question in a related area (only for SAM-2). >> >> RFC3720 was written in between SAM-2 and SAM-3; some concepts were drawn from >> SAM-2, and some from SAM-3. >> >> We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus loss >> concept does not exist in SAM-2. SAM-2 does include references to RESERVE and >> RELEASE commands. SAM-3 includes I_T Nexus loss, but it has no reference to >> RESERVE state. There are allusions in RFC3720 about the intersection of >> RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit. RFC3720 >> references other documents (SAM2, SAM3, SBC, SPC3), none of which address the >> situation. >> >> Consider text such as the following: >> 4.4.3.1. I_T Nexus State >> >> Certain nexus relationships contain an explicit state (e.g., >> initiator-specific mode pages) that may need to be preserved by >> the device server [SAM2] in a logical unit through changes or >> failures in the iSCSI layer (e.g., session failures). In order for >> that state to be restored, the iSCSI initiator should reestablish >> its session (re-login) to the same Target Portal Group using the >> previous ISID. That is, it should perform session recovery as >> described in Chapter 6. This is because the SCSI initiator port >> identifier and the SCSI target port identifier (or relative target >> port) form the datum that the SCSI logical unit device server uses >> to identify the I_T nexus. >> >> Is RESERVE state - explicit state? When an I_T nexus is reestablished, >> initiator-specific mode pages are preserved. That implies that an I_T nexus >> loss should be treated in the way that SAM-2 describes via CLEAR TASK SET >> (where mode pages and RESERVE state are preserved). If a session >> reestablishment caused saved mode pages to be restored rather than the >> established initiator-specific mode pages to be restored, then it would be >> like a LU reset (where the RESERVE state is lost). Is RESERVE state explicit >> state that is to be preserved? >> >> Appendix E covers clearing effects during various events, but it never >> mentions RESERVE state. >> >> Section E.2 contains the following text: >> >> The only iSCSI protocol action that can effect clearing actions on >> SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 >> Loss of Nexus notification). [SPC3] describes the clearing effects >> of this notification on a variety of SCSI attributes. In addition, >> SCSI standards documents (such as [SAM2] and [SBC]) define >> additional clearing actions that may take place for several SCSI >> objects on SCSI events such as LU resets and power-on resets. >> >> I could not find any text in SPC3 that describes these clearing effects (I >> don't know if this SPC3 reference is incorrect, but at a minimum, it is >> ineffective). The SAM2 text for LU resets and power-on resets is easily >> found, and clearly says that any RESERVE is released in those cases. As an >> aside, the reference to section 6.3.5.1 is a circular reference). The text in >> E.2 goes on: >> >> Since iSCSI defines a target cold reset as a protocol-equivalent >> to a target power-cycle, the iSCSI target cold reset must also be >> considered as the power-on reset event in interpreting the actions >> defined in the SCSI standards. >> >> When the iSCSI session is reconstructed (between the same SCSI >> ports with the same nexus identifier) reestablishing the same I_T >> nexus, all SCSI objects that are defined to not clear on the "I_T >> nexus loss" notification event, such as persistent reservations, >> are automatically associated to this new session. >> >> So, it is clear what happens to PERSISTENT RESERVE state, but not so clear >> about RESERVE state. It also states that all SCSI objects that are defined to >> not clear on the I_T nexus loss are retained. But again, the only list I can >> find of what is cleared is here in this Appendix, since SAM-2, SAM-3, SAM-4 >> are all silent on the interaction between RESERVE and I_T Nexus loss. >> >> T10 has never had to deal with this because they removed RESERVE before they >> added I_T Nexus loss. However, iSCSI did not do that; we discuss BOTH, but we >> fail to describe how they interact. iSCSI also has the feature of >> reestablishment of a session that does not exist in SAM-2 (SAM-2 was still >> parallel SCSI focused, where connections didn't exist). So again, if session >> reestablishment is examined in the light of parallel SCSI, the intent would be >> to retain the RESERVE state. Here is another section on the topic: >> >> 6.3.5.1. Loss of Nexus Notification >> >> The iSCSI layer provides the SCSI layer with the "I_T nexus loss" >> notification when any one of the following events happens: >> >> Successful completion of session reinstatement. >> Session closure event. >> Session timeout event. >> >> Certain SCSI object clearing actions may result due to the >> notification in the SCSI end nodes, as documented in Appendix F. - >> Clearing Effects of Various Events on Targets. >> >> FWIW - the reference above is incorrect - there is no Appendix F in the >> consolidated draft - this should be Appendix E (and also, BTW - this is a >> circular reference). >> >> So anyway, it seems like the intent was that session reinstatement would >> preserve the RESERVE state across the I_T Nexus loss event. >> >> So aside from the incorrect reference (and circular nature of it), any >> thoughts on whether we should get explicit about this situation? Should we >> remain silent; should we tweak Appendix E along these lines (or other lines >> that others might suggest): >> >> 4.4.3.1. I_T Nexus State >> >> Certain nexus relationships contain an explicit state (e.g., >> initiator-specific mode pages, and RESERVE state) that may need to be >> preserved by >> the device server [SAM2] in a logical unit through changes or >> failures in the iSCSI layer (e.g., session failures).... >> >> Fred Knight >> >> >> >> -----Original Message----- >> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of >> Mallikarjun Chadalapaka >> Sent: Sunday, September 23, 2012 1:34 AM >> To: david.black@emc.com; storm@ietf.org >> Subject: Re: [storm] iSCSI drafts - T10 and T11 references >> >> Hi David, >> >> I have noticed this comment as I am working through all Last Call comments. >> >> The references to SAM-3 and SAM-4 are keyed to this paragraph in Section 13.9: >> >> "[SAM2] and [SAM3] specifications note in their informative text that TPGT >> value should be non-zero, note that it is incorrect. A zero value is allowed >> as a legal value for TPGT. This discrepancy currently stands corrected in >> [SAM4]." >> >> Additionally, SAM4 reference is also used in citing the new ASC/ASCQ codes >> that it defines, in Section 11.14.5 of the draft. >> >> So, I believe references to all three SAM specs would continue to be >> appropriate. >> >> The current draft references FC-FS in discussing the NAA format in a few >> places, so the current reference at the end is intentionally the same. Is the >> older reference a problem? If so, I suppose we can replace all references to >> FC-FS-4, but that seems to be still in early stages(0.1 spec posted on April >> 2012), so perhaps we should use FC-FS3? >> >> Thanks. >> >> Mallikarjun >> >> >> >> >> -----Original Message----- >> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of >> david.black@emc.com >> Sent: Thursday, July 26, 2012 6:47 AM >> To: storm@ietf.org >> Subject: [storm] iSCSI drafts - T10 and T11 references >> >> In making up the lists of T10 and T11 documents that are currently referenced, >> some of those references don't look right to me, for example: >> >> - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. >> - T11: The FC-FS reference is out of date. >> >> As part of dealing with the comments from IETF Last Call, we'll need to go >> over all of the T10 and T11 references (and should check all the references) >> to make sure that they're up to date and necessary. >> >> Thanks, >> --David >> ---------------------------------------------------- >> David L. Black, Distinguished Engineer >> EMC Corporation, 176 South St., Hopkinton, MA 01748 >> +1 (508) 293-7953 FAX: +1 (508) 293-7786 >> david.black@emc.com Mobile: +1 (978) 394-7754 >> ---------------------------------------------------- >> >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm >> >> >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm >> >> > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > From rjcplym@gmail.com Mon Sep 24 10:56:37 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DA99E21E8055 for ; Mon, 24 Sep 2012 10:56:37 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.977 X-Spam-Level: X-Spam-Status: No, score=-2.977 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YUHWn5SOAqIo for ; Mon, 24 Sep 2012 10:56:35 -0700 (PDT) Received: from mail-vb0-f44.google.com (mail-vb0-f44.google.com [209.85.212.44]) by ietfa.amsl.com (Postfix) with ESMTP id 9179121E8054 for ; Mon, 24 Sep 2012 10:56:35 -0700 (PDT) Received: by vbbfc26 with SMTP id fc26so7037951vbb.31 for ; Mon, 24 Sep 2012 10:56:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=lQM90hHaBUSnn3UVKB+wyN51MgzoS18ZLUNhI+2miqQ=; b=n5h3XNBxuHaUcZcr83jNV5W3N+MwDquXG973XZ0J24qus3s5RWRnLAsTo4s7j28tjO swa7rp1Rlyqd9IHWZOe6/MOt6K7uK1ZDJHBbySh7Pbtx/L14Assdhn0R6B8WppNhZQxa M7N7bwzYQqyRoP1xly1MUZS583UjvgIkPKxj06OTYa5elJ4VcNiAn7wYyA2Q8wTg+sWE Yanq+ScAYYLHOsdq+6EmNZ5jpi2DZI9hxylDFgD8xHnrTotD4BBkQeXNDIn6COEi79/h uG9ZKAgPalEOdRiS8dd8BF1aR/L9OkAYatvVq0lwZHJ9t1BY/AYykdW+jwTetYR0Qs4x GcKg== MIME-Version: 1.0 Received: by 10.58.35.166 with SMTP id i6mr8135338vej.10.1348509394977; Mon, 24 Sep 2012 10:56:34 -0700 (PDT) Sender: rjcplym@gmail.com Received: by 10.58.91.134 with HTTP; Mon, 24 Sep 2012 10:56:34 -0700 (PDT) In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> Date: Mon, 24 Sep 2012 13:56:34 -0400 X-Google-Sender-Auth: FwmpkZvsX4E6n9bhgRADIQ7uxOw Message-ID: From: Roger Cummings To: "Black, David" , "Knight, Frederick" Content-Type: text/plain; charset=ISO-8859-1 Cc: "storm@ietf.org" Subject: Re: [storm] RESERVE state - clearing effects X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Sep 2012 17:56:38 -0000 David, Fred, I'm not actually sure if this is relevant to the point under discussion, but thought I'd raise it just in case. While the Reserve/Release method is obsolete in SCSI-3 as you say, the are actually some words (penned by yours truly) in SPC-4 section 5.13 called "Exceptions to SPC-2 RESERVE and RELEASE behavior" which handles the case when Reserve/Release and persistent reservations are mixed. This happens quite a bit in the field, and as an app vendor we were getting tired of targets doing brain-dead things in that circumstance like rejecting one command or other with a hitherto-unseen ASC/ASCQ, and so we drafted the text that essentially makes Reserve or Release a nop (and thus no reservation state is created) when those commands are received from an initiator that has access under a existing persistent reservation. I wonder if it's worth a note in the iSCSI consolidated draft that says that reservation state isn't created or cleared when an enabling persistent reservation is in place? I think us app vendors would thank you for it! FWIW, Roger Cummings On Mon, Sep 24, 2012 at 12:42 PM, Black, David wrote: > Fred, > > This is interesting ... > >> Here are some key points (from SAM 6.3.4 I_T nexus loss): >> >> 1) I_T nexus loss clears all ACA conditions; >> 2) established a UA - one of the following is suggested: >> >> "If the logical unit retains state information for the I_T nexus that >> is lost, its response to the subsequent I_T nexus >> re-establishment for the logical unit should include establishing a >> unit attention with an additional sense code set >> to I_T NEXUS LOSS OCCURRED. >> >> If the logical unit does not retain state information for the I_T nexus >> that is lost, it shall consider the subsequent >> I_T nexus re-establishment, if any, as the formation of a new I_T nexus >> for which there is no past history (e.g., >> establish a unit attention with an additional sense code set to POWER ON >> OCCURRED)." > > That "state information" has to identify the endpoints of the I_T Nexus > - for iSCSI that means at least the ISID and TPGT. If that's gone, I > think the RESERVE state is gone as well, in contrast to PERSISTENT > RESERVE state. On RESERVE state, SPC-2 has this to say in 5.5.2 "The > Reserve/Release management method": > > The reserve/release management method commands, RESERVE(6), > RESERVE(10), RELEASE(6), and RELEASE(10) are used among multiple > initiators that do not require operations to be protected across > initiator failures (and subsequent hard resets). > > I think it's fair to regard I_T Nexus loss as an initiator failure. I > also don't see any good way to do a 3rd-party release on a RESERVE > reservation that persists after I_T Nexus loss. I also believe that > initiator software that uses Reserve/Release does not expect > reservations to persist across I_T Nexus loss. > > As to what needs to be stated in the iSCSI consolidated draft - the SPC > reference has been updated to SPC-3 in which the Reserve/Release > management method is obsolete, so there isn't any absolute requirement > to cover that. OTOH, it might be reasonable to add a sentence to say > that Reserve/Release state is implicit (e.g., as there's no way for an > initiator to figure out which I_T Nexus holds a reservation), and that > treating Reserve/Release state as indefinitely persistent after I_T > Nexus Loss is a bad idea ("should not", but lower case) as it may > require the initiator to reset the target device in order to clear out > that state (issuing RELEASE commands isn't enough). This sort of text > would require a reference to SPC-2. > > Thanks, > --David > >> -----Original Message----- >> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] >> Sent: Monday, September 24, 2012 8:58 AM >> To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org >> Subject: RE: iSCSI drafts - T10 and T11 references >> >> FCP is also not explicit (it defers to the SCSI specs - which do not say). >> The closest statements I can find still requires extrapolation: >> >> Here are some key points (from SAM 6.3.4 I_T nexus loss): >> >> 1) I_T nexus loss clears all ACA conditions; >> 2) established a UA - one of the following is suggested: >> >> "If the logical unit retains state information for the I_T nexus that is lost, >> its response to the subsequent I_T nexus >> re-establishment for the logical unit should include establishing a unit >> attention with an additional sense code set >> to I_T NEXUS LOSS OCCURRED. >> >> If the logical unit does not retain state information for the I_T nexus that >> is lost, it shall consider the subsequent >> I_T nexus re-establishment, if any, as the formation of a new I_T nexus for >> which there is no past history (e.g., >> establish a unit attention with an additional sense code set to POWER ON >> OCCURRED)." >> >> But, there is no definition of what it means to "retain state". The >> conclusion I could jump to based on #1 above, is that RESERVE state is also >> cleared (this action lines up more with LU RESET than CLEAR TASK SET); and >> based on the POWER ON CONDITION UA of 29/01 (or 29/00 - both are power on >> conditions) I could also assume that the RESERVE state is cleared; but based >> on the UA of 29/07 (retain state), I could jump to the conclusion that RESERVE >> state is retained. >> >> But these are all conclusions I've jump to with very limited basis in the >> text. >> >> Based on what RFC3720 says about explicit state (e.g., mode pages), I would >> assume that hosts would believe that every command they explicitly send (like >> MODE SELECT or RESERVE) would have created explicit state and therefore the >> host would assume RESERVE state is retained. It doesn't make sense to have >> mode page values retained, but have RESERVE state lost. >> >> Certain nexus relationships contain an explicit state (e.g., >> initiator-specific mode pages) that may need to be preserved by >> the device server [SAM2] in a logical unit through changes or >> failures in the iSCSI layer (e.g., session failures). >> >> It will be interesting to hear what a real host assumes about this situation. >> >> Fred >> >> -----Original Message----- >> From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] >> Sent: Sunday, September 23, 2012 6:35 PM >> To: Knight, Frederick; david.black@emc.com; storm@ietf.org >> Subject: RE: iSCSI drafts - T10 and T11 references >> >> Hi Fred, >> >> SPC-3 discusses clearing effects of I-T nexus loss in several places. >> >> Section 5.8.2.9 for ALUA >> Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for background self- >> test Section 5.6.1 on persistent reservations Section 5.13 on timestamps >> ...... >> >> Thanks for pointing out the broken cross-reference - Appendix F instead of E. >> I have just fixed it - and reviewed all other Appendix references elsewhere in >> the draft. However, I do not see a circular reference problem per se - >> Appendix E is correctly referencing Section 6.3.5 for definition of terms, >> e.g. session continuation. And Section 6.3.5.1 is correctly deferring to >> Appendix E when discussing clearing effects. Let me know if I'm overlooking >> something here. >> >> Now, the reserve/release management state, :) : I agree the formal I_T nexus >> loss event didn't exist in SAM-2, however the notion of a new I_T nexus state >> did co-exist with a session-oriented I_T nexus well before iSCSI - FCP process >> login (PRLO) comes to mind here. So I'd be curious about traditional FCP >> handling of reserve/release management state. And I'm also checking internally >> here at MS. From what I know now though, it does *appear* that reserve/release >> management state should not be cleared with I_T nexus loss. >> >> Having said all that, note that reserve/release management state is squarely >> in the SCSI territory. iSCSI spec has so far by design deferred to SCSI >> standards in specifying clearing effects on SCSI objects - as Appendix E.2 >> makes it clear. What we do specify is the clearing effects of SCSI actions, >> e.g. LU reset, on iSCSI objects. >> >> Thanks. >> >> Mallikarjun >> >> >> >> >> >> -----Original Message----- >> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] >> Sent: Sunday, September 23, 2012 3:38 AM >> To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org >> Subject: RE: iSCSI drafts - T10 and T11 references >> >> Interesting, I just came across a question in a related area (only for SAM-2). >> >> RFC3720 was written in between SAM-2 and SAM-3; some concepts were drawn from >> SAM-2, and some from SAM-3. >> >> We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus loss >> concept does not exist in SAM-2. SAM-2 does include references to RESERVE and >> RELEASE commands. SAM-3 includes I_T Nexus loss, but it has no reference to >> RESERVE state. There are allusions in RFC3720 about the intersection of >> RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit. RFC3720 >> references other documents (SAM2, SAM3, SBC, SPC3), none of which address the >> situation. >> >> Consider text such as the following: >> 4.4.3.1. I_T Nexus State >> >> Certain nexus relationships contain an explicit state (e.g., >> initiator-specific mode pages) that may need to be preserved by >> the device server [SAM2] in a logical unit through changes or >> failures in the iSCSI layer (e.g., session failures). In order for >> that state to be restored, the iSCSI initiator should reestablish >> its session (re-login) to the same Target Portal Group using the >> previous ISID. That is, it should perform session recovery as >> described in Chapter 6. This is because the SCSI initiator port >> identifier and the SCSI target port identifier (or relative target >> port) form the datum that the SCSI logical unit device server uses >> to identify the I_T nexus. >> >> Is RESERVE state - explicit state? When an I_T nexus is reestablished, >> initiator-specific mode pages are preserved. That implies that an I_T nexus >> loss should be treated in the way that SAM-2 describes via CLEAR TASK SET >> (where mode pages and RESERVE state are preserved). If a session >> reestablishment caused saved mode pages to be restored rather than the >> established initiator-specific mode pages to be restored, then it would be >> like a LU reset (where the RESERVE state is lost). Is RESERVE state explicit >> state that is to be preserved? >> >> Appendix E covers clearing effects during various events, but it never >> mentions RESERVE state. >> >> Section E.2 contains the following text: >> >> The only iSCSI protocol action that can effect clearing actions on >> SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 >> Loss of Nexus notification). [SPC3] describes the clearing effects >> of this notification on a variety of SCSI attributes. In addition, >> SCSI standards documents (such as [SAM2] and [SBC]) define >> additional clearing actions that may take place for several SCSI >> objects on SCSI events such as LU resets and power-on resets. >> >> I could not find any text in SPC3 that describes these clearing effects (I >> don't know if this SPC3 reference is incorrect, but at a minimum, it is >> ineffective). The SAM2 text for LU resets and power-on resets is easily >> found, and clearly says that any RESERVE is released in those cases. As an >> aside, the reference to section 6.3.5.1 is a circular reference). The text in >> E.2 goes on: >> >> Since iSCSI defines a target cold reset as a protocol-equivalent >> to a target power-cycle, the iSCSI target cold reset must also be >> considered as the power-on reset event in interpreting the actions >> defined in the SCSI standards. >> >> When the iSCSI session is reconstructed (between the same SCSI >> ports with the same nexus identifier) reestablishing the same I_T >> nexus, all SCSI objects that are defined to not clear on the "I_T >> nexus loss" notification event, such as persistent reservations, >> are automatically associated to this new session. >> >> So, it is clear what happens to PERSISTENT RESERVE state, but not so clear >> about RESERVE state. It also states that all SCSI objects that are defined to >> not clear on the I_T nexus loss are retained. But again, the only list I can >> find of what is cleared is here in this Appendix, since SAM-2, SAM-3, SAM-4 >> are all silent on the interaction between RESERVE and I_T Nexus loss. >> >> T10 has never had to deal with this because they removed RESERVE before they >> added I_T Nexus loss. However, iSCSI did not do that; we discuss BOTH, but we >> fail to describe how they interact. iSCSI also has the feature of >> reestablishment of a session that does not exist in SAM-2 (SAM-2 was still >> parallel SCSI focused, where connections didn't exist). So again, if session >> reestablishment is examined in the light of parallel SCSI, the intent would be >> to retain the RESERVE state. Here is another section on the topic: >> >> 6.3.5.1. Loss of Nexus Notification >> >> The iSCSI layer provides the SCSI layer with the "I_T nexus loss" >> notification when any one of the following events happens: >> >> Successful completion of session reinstatement. >> Session closure event. >> Session timeout event. >> >> Certain SCSI object clearing actions may result due to the >> notification in the SCSI end nodes, as documented in Appendix F. - >> Clearing Effects of Various Events on Targets. >> >> FWIW - the reference above is incorrect - there is no Appendix F in the >> consolidated draft - this should be Appendix E (and also, BTW - this is a >> circular reference). >> >> So anyway, it seems like the intent was that session reinstatement would >> preserve the RESERVE state across the I_T Nexus loss event. >> >> So aside from the incorrect reference (and circular nature of it), any >> thoughts on whether we should get explicit about this situation? Should we >> remain silent; should we tweak Appendix E along these lines (or other lines >> that others might suggest): >> >> 4.4.3.1. I_T Nexus State >> >> Certain nexus relationships contain an explicit state (e.g., >> initiator-specific mode pages, and RESERVE state) that may need to be >> preserved by >> the device server [SAM2] in a logical unit through changes or >> failures in the iSCSI layer (e.g., session failures).... >> >> Fred Knight >> >> >> >> -----Original Message----- >> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of >> Mallikarjun Chadalapaka >> Sent: Sunday, September 23, 2012 1:34 AM >> To: david.black@emc.com; storm@ietf.org >> Subject: Re: [storm] iSCSI drafts - T10 and T11 references >> >> Hi David, >> >> I have noticed this comment as I am working through all Last Call comments. >> >> The references to SAM-3 and SAM-4 are keyed to this paragraph in Section 13.9: >> >> "[SAM2] and [SAM3] specifications note in their informative text that TPGT >> value should be non-zero, note that it is incorrect. A zero value is allowed >> as a legal value for TPGT. This discrepancy currently stands corrected in >> [SAM4]." >> >> Additionally, SAM4 reference is also used in citing the new ASC/ASCQ codes >> that it defines, in Section 11.14.5 of the draft. >> >> So, I believe references to all three SAM specs would continue to be >> appropriate. >> >> The current draft references FC-FS in discussing the NAA format in a few >> places, so the current reference at the end is intentionally the same. Is the >> older reference a problem? If so, I suppose we can replace all references to >> FC-FS-4, but that seems to be still in early stages(0.1 spec posted on April >> 2012), so perhaps we should use FC-FS3? >> >> Thanks. >> >> Mallikarjun >> >> >> >> >> -----Original Message----- >> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of >> david.black@emc.com >> Sent: Thursday, July 26, 2012 6:47 AM >> To: storm@ietf.org >> Subject: [storm] iSCSI drafts - T10 and T11 references >> >> In making up the lists of T10 and T11 documents that are currently referenced, >> some of those references don't look right to me, for example: >> >> - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. >> - T11: The FC-FS reference is out of date. >> >> As part of dealing with the comments from IETF Last Call, we'll need to go >> over all of the T10 and T11 references (and should check all the references) >> to make sure that they're up to date and necessary. >> >> Thanks, >> --David >> ---------------------------------------------------- >> David L. Black, Distinguished Engineer >> EMC Corporation, 176 South St., Hopkinton, MA 01748 >> +1 (508) 293-7953 FAX: +1 (508) 293-7786 >> david.black@emc.com Mobile: +1 (978) 394-7754 >> ---------------------------------------------------- >> >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm >> >> >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm >> >> > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From david.black@emc.com Mon Sep 24 11:26:54 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DCEBB21F8847 for ; Mon, 24 Sep 2012 11:26:54 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HXUTyCcjKO-6 for ; Mon, 24 Sep 2012 11:26:53 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 2123121F8845 for ; Mon, 24 Sep 2012 11:26:52 -0700 (PDT) Received: from hop04-l1d11-si01.isus.emc.com (HOP04-L1D11-SI01.isus.emc.com [10.254.111.54]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8OIQmab029096 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 24 Sep 2012 14:26:49 -0400 Received: from mailhub.lss.emc.com (mailhubhoprd01.lss.emc.com [10.254.221.251]) by hop04-l1d11-si01.isus.emc.com (RSA Interceptor); Mon, 24 Sep 2012 14:26:16 -0400 Received: from mxhub19.corp.emc.com (mxhub19.corp.emc.com [10.254.93.48]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8OIQALp009497; Mon, 24 Sep 2012 14:26:15 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub19.corp.emc.com ([10.254.93.48]) with mapi; Mon, 24 Sep 2012 14:26:13 -0400 From: "Black, David" To: Roger Cummings Date: Mon, 24 Sep 2012 14:26:12 -0400 Thread-Topic: [storm] RESERVE state - clearing effects Thread-Index: Ac2afhsV3tYy0S6iSzeSzuSbNwwMZwAAugdQ Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE79238@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Cc: "storm@ietf.org" Subject: Re: [storm] RESERVE state - clearing effects X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Sep 2012 18:26:55 -0000 Roger, IMHO, that makes sense to add, but at a somewhat more abstract level with an important initial caveat: - The Reserve/Release method is obsolete in the SCSI standards; Persistent Reservations should be used instead (lower case "should"). [SPC3] - When Persistent Reservations are used, Reserve/Release may not behave as expected - see the discussion of "Exceptions to SPC-2 RESERVE an= d RELEASE behavior" in [SPC4] (informative reference, subclause numbe= r deliberately omitted as it may change). I'd prefer to leave the explanation of when RESERVE and RELEASE are no-ops to SPC-4 ... where it belongs. Thanks, --David > -----Original Message----- > From: rjcplym@gmail.com [mailto:rjcplym@gmail.com] On Behalf Of Roger Cum= mings > Sent: Monday, September 24, 2012 1:57 PM > To: Black, David; Knight, Frederick > Cc: Mallikarjun Chadalapaka; storm@ietf.org > Subject: Re: [storm] RESERVE state - clearing effects > > David, Fred, > > I'm not actually sure if this is relevant to the point under > discussion, but thought I'd raise it just in case. > > While the Reserve/Release method is obsolete in SCSI-3 as you say, the > are actually some words (penned by yours truly) in SPC-4 section 5.13 > called "Exceptions to SPC-2 RESERVE and RELEASE behavior" which > handles the case when Reserve/Release and persistent reservations are > mixed. This happens quite a bit in the field, and as an app vendor we > were getting tired of targets doing brain-dead things in that > circumstance like rejecting one command or other with a > hitherto-unseen ASC/ASCQ, and so we drafted the text that essentially > makes Reserve or Release a nop (and thus no reservation state is > created) when those commands are received from an initiator that has > access under a existing persistent reservation. > > I wonder if it's worth a note in the iSCSI consolidated draft that > says that reservation state isn't created or cleared when an enabling > persistent reservation is in place? I think us app vendors would thank > you for it! > > FWIW, > > > > > Roger Cummings > > On Mon, Sep 24, 2012 at 12:42 PM, Black, David wrot= e: > > Fred, > > > > This is interesting ... > > > >> Here are some key points (from SAM 6.3.4 I_T nexus loss): > >> > >> 1) I_T nexus loss clears all ACA conditions; > >> 2) established a UA - one of the following is suggested: > >> > >> "If the logical unit retains state information for the I_T nexus that > >> is lost, its response to the subsequent I_T nexus > >> re-establishment for the logical unit should include establishing a > >> unit attention with an additional sense code set > >> to I_T NEXUS LOSS OCCURRED. > >> > >> If the logical unit does not retain state information for the I_T nexu= s > >> that is lost, it shall consider the subsequent > >> I_T nexus re-establishment, if any, as the formation of a new I_T nexu= s > >> for which there is no past history (e.g., > >> establish a unit attention with an additional sense code set to POWER = ON > >> OCCURRED)." > > > > That "state information" has to identify the endpoints of the I_T Nexus > > - for iSCSI that means at least the ISID and TPGT. If that's gone, I > > think the RESERVE state is gone as well, in contrast to PERSISTENT > > RESERVE state. On RESERVE state, SPC-2 has this to say in 5.5.2 "The > > Reserve/Release management method": > > > > The reserve/release management method commands, RESERVE(6), > > RESERVE(10), RELEASE(6), and RELEASE(10) are used among multipl= e > > initiators that do not require operations to be protected acros= s > > initiator failures (and subsequent hard resets). > > > > I think it's fair to regard I_T Nexus loss as an initiator failure. I > > also don't see any good way to do a 3rd-party release on a RESERVE > > reservation that persists after I_T Nexus loss. I also believe that > > initiator software that uses Reserve/Release does not expect > > reservations to persist across I_T Nexus loss. > > > > As to what needs to be stated in the iSCSI consolidated draft - the SPC > > reference has been updated to SPC-3 in which the Reserve/Release > > management method is obsolete, so there isn't any absolute requirement > > to cover that. OTOH, it might be reasonable to add a sentence to say > > that Reserve/Release state is implicit (e.g., as there's no way for an > > initiator to figure out which I_T Nexus holds a reservation), and that > > treating Reserve/Release state as indefinitely persistent after I_T > > Nexus Loss is a bad idea ("should not", but lower case) as it may > > require the initiator to reset the target device in order to clear out > > that state (issuing RELEASE commands isn't enough). This sort of text > > would require a reference to SPC-2. > > > > Thanks, > > --David > > > >> -----Original Message----- > >> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > >> Sent: Monday, September 24, 2012 8:58 AM > >> To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org > >> Subject: RE: iSCSI drafts - T10 and T11 references > >> > >> FCP is also not explicit (it defers to the SCSI specs - which do not s= ay). > >> The closest statements I can find still requires extrapolation: > >> > >> Here are some key points (from SAM 6.3.4 I_T nexus loss): > >> > >> 1) I_T nexus loss clears all ACA conditions; > >> 2) established a UA - one of the following is suggested: > >> > >> "If the logical unit retains state information for the I_T nexus that = is > lost, > >> its response to the subsequent I_T nexus > >> re-establishment for the logical unit should include establishing a un= it > >> attention with an additional sense code set > >> to I_T NEXUS LOSS OCCURRED. > >> > >> If the logical unit does not retain state information for the I_T nexu= s > that > >> is lost, it shall consider the subsequent > >> I_T nexus re-establishment, if any, as the formation of a new I_T nexu= s for > >> which there is no past history (e.g., > >> establish a unit attention with an additional sense code set to POWER = ON > >> OCCURRED)." > >> > >> But, there is no definition of what it means to "retain state". The > >> conclusion I could jump to based on #1 above, is that RESERVE state is= also > >> cleared (this action lines up more with LU RESET than CLEAR TASK SET);= and > >> based on the POWER ON CONDITION UA of 29/01 (or 29/00 - both are power= on > >> conditions) I could also assume that the RESERVE state is cleared; but > based > >> on the UA of 29/07 (retain state), I could jump to the conclusion that > RESERVE > >> state is retained. > >> > >> But these are all conclusions I've jump to with very limited basis in = the > >> text. > >> > >> Based on what RFC3720 says about explicit state (e.g., mode pages), I = would > >> assume that hosts would believe that every command they explicitly sen= d > (like > >> MODE SELECT or RESERVE) would have created explicit state and therefor= e the > >> host would assume RESERVE state is retained. It doesn't make sense to= have > >> mode page values retained, but have RESERVE state lost. > >> > >> Certain nexus relationships contain an explicit state (e.g., > >> initiator-specific mode pages) that may need to be preserved by > >> the device server [SAM2] in a logical unit through changes or > >> failures in the iSCSI layer (e.g., session failures). > >> > >> It will be interesting to hear what a real host assumes about this > situation. > >> > >> Fred > >> > >> -----Original Message----- > >> From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] > >> Sent: Sunday, September 23, 2012 6:35 PM > >> To: Knight, Frederick; david.black@emc.com; storm@ietf.org > >> Subject: RE: iSCSI drafts - T10 and T11 references > >> > >> Hi Fred, > >> > >> SPC-3 discusses clearing effects of I-T nexus loss in several places. > >> > >> Section 5.8.2.9 for ALUA > >> Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for backgroun= d > self- > >> test Section 5.6.1 on persistent reservations Section 5.13 on timestam= ps > >> ...... > >> > >> Thanks for pointing out the broken cross-reference - Appendix F instea= d of > E. > >> I have just fixed it - and reviewed all other Appendix references else= where > in > >> the draft. However, I do not see a circular reference problem per se - > >> Appendix E is correctly referencing Section 6.3.5 for definition of te= rms, > >> e.g. session continuation. And Section 6.3.5.1 is correctly deferring = to > >> Appendix E when discussing clearing effects. Let me know if I'm overlo= oking > >> something here. > >> > >> Now, the reserve/release management state, :) : I agree the formal I_= T > nexus > >> loss event didn't exist in SAM-2, however the notion of a new I_T nexu= s > state > >> did co-exist with a session-oriented I_T nexus well before iSCSI - FCP > process > >> login (PRLO) comes to mind here. So I'd be curious about traditional F= CP > >> handling of reserve/release management state. And I'm also checking > internally > >> here at MS. From what I know now though, it does *appear* that > reserve/release > >> management state should not be cleared with I_T nexus loss. > >> > >> Having said all that, note that reserve/release management state is > squarely > >> in the SCSI territory. iSCSI spec has so far by design deferred to SCS= I > >> standards in specifying clearing effects on SCSI objects - as Appendix= E.2 > >> makes it clear. What we do specify is the clearing effects of SCSI act= ions, > >> e.g. LU reset, on iSCSI objects. > >> > >> Thanks. > >> > >> Mallikarjun > >> > >> > >> > >> > >> > >> -----Original Message----- > >> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > >> Sent: Sunday, September 23, 2012 3:38 AM > >> To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org > >> Subject: RE: iSCSI drafts - T10 and T11 references > >> > >> Interesting, I just came across a question in a related area (only for= SAM- > 2). > >> > >> RFC3720 was written in between SAM-2 and SAM-3; some concepts were dra= wn > from > >> SAM-2, and some from SAM-3. > >> > >> We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus= loss > >> concept does not exist in SAM-2. SAM-2 does include references to RES= ERVE > and > >> RELEASE commands. SAM-3 includes I_T Nexus loss, but it has no refere= nce > to > >> RESERVE state. There are allusions in RFC3720 about the intersection = of > >> RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit. RF= C3720 > >> references other documents (SAM2, SAM3, SBC, SPC3), none of which addr= ess > the > >> situation. > >> > >> Consider text such as the following: > >> 4.4.3.1. I_T Nexus State > >> > >> Certain nexus relationships contain an explicit state (e.g., > >> initiator-specific mode pages) that may need to be preserved by > >> the device server [SAM2] in a logical unit through changes or > >> failures in the iSCSI layer (e.g., session failures). In order for > >> that state to be restored, the iSCSI initiator should reestablish > >> its session (re-login) to the same Target Portal Group using the > >> previous ISID. That is, it should perform session recovery as > >> described in Chapter 6. This is because the SCSI initiator port > >> identifier and the SCSI target port identifier (or relative target > >> port) form the datum that the SCSI logical unit device server uses > >> to identify the I_T nexus. > >> > >> Is RESERVE state - explicit state? When an I_T nexus is reestablished= , > >> initiator-specific mode pages are preserved. That implies that an I_T > nexus > >> loss should be treated in the way that SAM-2 describes via CLEAR TASK = SET > >> (where mode pages and RESERVE state are preserved). If a session > >> reestablishment caused saved mode pages to be restored rather than the > >> established initiator-specific mode pages to be restored, then it woul= d be > >> like a LU reset (where the RESERVE state is lost). Is RESERVE state > explicit > >> state that is to be preserved? > >> > >> Appendix E covers clearing effects during various events, but it never > >> mentions RESERVE state. > >> > >> Section E.2 contains the following text: > >> > >> The only iSCSI protocol action that can effect clearing actions on > >> SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 > >> Loss of Nexus notification). [SPC3] describes the clearing effects > >> of this notification on a variety of SCSI attributes. In addition, > >> SCSI standards documents (such as [SAM2] and [SBC]) define > >> additional clearing actions that may take place for several SCSI > >> objects on SCSI events such as LU resets and power-on resets. > >> > >> I could not find any text in SPC3 that describes these clearing effect= s (I > >> don't know if this SPC3 reference is incorrect, but at a minimum, it i= s > >> ineffective). The SAM2 text for LU resets and power-on resets is easi= ly > >> found, and clearly says that any RESERVE is released in those cases. = As an > >> aside, the reference to section 6.3.5.1 is a circular reference). The= text > in > >> E.2 goes on: > >> > >> Since iSCSI defines a target cold reset as a protocol-equivalent > >> to a target power-cycle, the iSCSI target cold reset must also be > >> considered as the power-on reset event in interpreting the actions > >> defined in the SCSI standards. > >> > >> When the iSCSI session is reconstructed (between the same SCSI > >> ports with the same nexus identifier) reestablishing the same I_T > >> nexus, all SCSI objects that are defined to not clear on the "I_T > >> nexus loss" notification event, such as persistent reservations, > >> are automatically associated to this new session. > >> > >> So, it is clear what happens to PERSISTENT RESERVE state, but not so c= lear > >> about RESERVE state. It also states that all SCSI objects that are de= fined > to > >> not clear on the I_T nexus loss are retained. But again, the only lis= t I > can > >> find of what is cleared is here in this Appendix, since SAM-2, SAM-3, = SAM-4 > >> are all silent on the interaction between RESERVE and I_T Nexus loss. > >> > >> T10 has never had to deal with this because they removed RESERVE befor= e > they > >> added I_T Nexus loss. However, iSCSI did not do that; we discuss BOTH= , but > we > >> fail to describe how they interact. iSCSI also has the feature of > >> reestablishment of a session that does not exist in SAM-2 (SAM-2 was s= till > >> parallel SCSI focused, where connections didn't exist). So again, if > session > >> reestablishment is examined in the light of parallel SCSI, the intent = would > be > >> to retain the RESERVE state. Here is another section on the topic: > >> > >> 6.3.5.1. Loss of Nexus Notification > >> > >> The iSCSI layer provides the SCSI layer with the "I_T nexus loss" > >> notification when any one of the following events happens: > >> > >> Successful completion of session reinstatement. > >> Session closure event. > >> Session timeout event. > >> > >> Certain SCSI object clearing actions may result due to the > >> notification in the SCSI end nodes, as documented in Appendix F. - > >> Clearing Effects of Various Events on Targets. > >> > >> FWIW - the reference above is incorrect - there is no Appendix F in th= e > >> consolidated draft - this should be Appendix E (and also, BTW - this i= s a > >> circular reference). > >> > >> So anyway, it seems like the intent was that session reinstatement wou= ld > >> preserve the RESERVE state across the I_T Nexus loss event. > >> > >> So aside from the incorrect reference (and circular nature of it), any > >> thoughts on whether we should get explicit about this situation? Shou= ld we > >> remain silent; should we tweak Appendix E along these lines (or other = lines > >> that others might suggest): > >> > >> 4.4.3.1. I_T Nexus State > >> > >> Certain nexus relationships contain an explicit state (e.g., > >> initiator-specific mode pages, and RESERVE state) that may need to b= e > >> preserved by > >> the device server [SAM2] in a logical unit through changes or > >> failures in the iSCSI layer (e.g., session failures).... > >> > >> Fred Knight > >> > >> > >> > >> -----Original Message----- > >> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf= Of > >> Mallikarjun Chadalapaka > >> Sent: Sunday, September 23, 2012 1:34 AM > >> To: david.black@emc.com; storm@ietf.org > >> Subject: Re: [storm] iSCSI drafts - T10 and T11 references > >> > >> Hi David, > >> > >> I have noticed this comment as I am working through all Last Call comm= ents. > >> > >> The references to SAM-3 and SAM-4 are keyed to this paragraph in Secti= on > 13.9: > >> > >> "[SAM2] and [SAM3] specifications note in their informative text that = TPGT > >> value should be non-zero, note that it is incorrect. A zero value is > allowed > >> as a legal value for TPGT. This discrepancy currently stands corrected= in > >> [SAM4]." > >> > >> Additionally, SAM4 reference is also used in citing the new ASC/ASCQ c= odes > >> that it defines, in Section 11.14.5 of the draft. > >> > >> So, I believe references to all three SAM specs would continue to be > >> appropriate. > >> > >> The current draft references FC-FS in discussing the NAA format in a f= ew > >> places, so the current reference at the end is intentionally the same.= Is > the > >> older reference a problem? If so, I suppose we can replace all referen= ces > to > >> FC-FS-4, but that seems to be still in early stages(0.1 spec posted on > April > >> 2012), so perhaps we should use FC-FS3? > >> > >> Thanks. > >> > >> Mallikarjun > >> > >> > >> > >> > >> -----Original Message----- > >> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf= Of > >> david.black@emc.com > >> Sent: Thursday, July 26, 2012 6:47 AM > >> To: storm@ietf.org > >> Subject: [storm] iSCSI drafts - T10 and T11 references > >> > >> In making up the lists of T10 and T11 documents that are currently > referenced, > >> some of those references don't look right to me, for example: > >> > >> - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. > >> - T11: The FC-FS reference is out of date. > >> > >> As part of dealing with the comments from IETF Last Call, we'll need t= o go > >> over all of the T10 and T11 references (and should check all the > references) > >> to make sure that they're up to date and necessary. > >> > >> Thanks, > >> --David > >> ---------------------------------------------------- > >> David L. Black, Distinguished Engineer > >> EMC Corporation, 176 South St., Hopkinton, MA 01748 > >> +1 (508) 293-7953 FAX: +1 (508) 293-7786 > >> david.black@emc.com Mobile: +1 (978) 394-7754 > >> ---------------------------------------------------- > >> > >> _______________________________________________ > >> storm mailing list > >> storm@ietf.org > >> https://www.ietf.org/mailman/listinfo/storm > >> > >> > >> _______________________________________________ > >> storm mailing list > >> storm@ietf.org > >> https://www.ietf.org/mailman/listinfo/storm > >> > >> > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm From Frederick.Knight@netapp.com Mon Sep 24 11:30:59 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8AA1421F885A for ; Mon, 24 Sep 2012 11:30:59 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CH9i67rb+lDh for ; Mon, 24 Sep 2012 11:30:58 -0700 (PDT) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id 0720F21F8858 for ; Mon, 24 Sep 2012 11:30:58 -0700 (PDT) X-IronPort-AV: E=Sophos;i="4.80,477,1344236400"; d="scan'208";a="693498537" Received: from smtp2.corp.netapp.com ([10.57.159.114]) by mx2-out.netapp.com with ESMTP; 24 Sep 2012 11:30:57 -0700 Received: from vmwexceht01-prd.hq.netapp.com (vmwexceht01-prd.hq.netapp.com [10.106.76.239]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8OIUvRw012476; Mon, 24 Sep 2012 11:30:57 -0700 (PDT) Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.102]) by vmwexceht01-prd.hq.netapp.com ([10.106.76.239]) with mapi id 14.02.0309.002; Mon, 24 Sep 2012 11:30:57 -0700 From: "Knight, Frederick" To: Roger Cummings , "Black, David" Thread-Topic: [storm] RESERVE state - clearing effects Thread-Index: AQHNmnOgfkCvZihy0EOAZUHR/pZPlpeaPFAA//+NJyA= Date: Mon, 24 Sep 2012 18:30:56 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.106.53.53] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "storm@ietf.org" Subject: Re: [storm] RESERVE state - clearing effects X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Sep 2012 18:30:59 -0000 I would say this is already covered in SPC (which we reference), and we don= 't need to repeat it. SCSI stuff in SCSI documents, and transport stuff in= transport documents. Maybe saying that "the interaction between RESERVE state and PERSISTENT RES= ERVE state is defined in SPC-x" would be OK. But, since you're an application, what can you share about your recovery ac= tions for 29/07? Does an I_T Nexus loss cause you to fully reestablish sta= te, or do you assume some of that state was retained? Do you treat 29/07 a= ny different from 29/00? Fred -----Original Message----- From: rjcplym@gmail.com [mailto:rjcplym@gmail.com] On Behalf Of Roger Cummi= ngs Sent: Monday, September 24, 2012 1:57 PM To: Black, David; Knight, Frederick Cc: Mallikarjun Chadalapaka; storm@ietf.org Subject: Re: [storm] RESERVE state - clearing effects David, Fred, I'm not actually sure if this is relevant to the point under discussion, bu= t thought I'd raise it just in case. While the Reserve/Release method is obsolete in SCSI-3 as you say, the are = actually some words (penned by yours truly) in SPC-4 section 5.13 called "E= xceptions to SPC-2 RESERVE and RELEASE behavior" which handles the case whe= n Reserve/Release and persistent reservations are mixed. This happens quite= a bit in the field, and as an app vendor we were getting tired of targets = doing brain-dead things in that circumstance like rejecting one command or = other with a hitherto-unseen ASC/ASCQ, and so we drafted the text that esse= ntially makes Reserve or Release a nop (and thus no reservation state is created) when those commands are received from an initiator that has access= under a existing persistent reservation. I wonder if it's worth a note in the iSCSI consolidated draft that says tha= t reservation state isn't created or cleared when an enabling persistent re= servation is in place? I think us app vendors would thank you for it! FWIW, Roger Cummings On Mon, Sep 24, 2012 at 12:42 PM, Black, David wrote: > Fred, > > This is interesting ... > >> Here are some key points (from SAM 6.3.4 I_T nexus loss): >> >> 1) I_T nexus loss clears all ACA conditions; >> 2) established a UA - one of the following is suggested: >> >> "If the logical unit retains state information for the I_T nexus that=20 >> is lost, its response to the subsequent I_T nexus re-establishment=20 >> for the logical unit should include establishing a unit attention=20 >> with an additional sense code set to I_T NEXUS LOSS OCCURRED. >> >> If the logical unit does not retain state information for the I_T=20 >> nexus that is lost, it shall consider the subsequent I_T nexus=20 >> re-establishment, if any, as the formation of a new I_T nexus for=20 >> which there is no past history (e.g., establish a unit attention with=20 >> an additional sense code set to POWER ON OCCURRED)." > > That "state information" has to identify the endpoints of the I_T=20 > Nexus > - for iSCSI that means at least the ISID and TPGT. If that's gone, I=20 > think the RESERVE state is gone as well, in contrast to PERSISTENT=20 > RESERVE state. On RESERVE state, SPC-2 has this to say in 5.5.2 "The=20 > Reserve/Release management method": > > The reserve/release management method commands, RESERVE(6), > RESERVE(10), RELEASE(6), and RELEASE(10) are used among multiple > initiators that do not require operations to be protected across > initiator failures (and subsequent hard resets). > > I think it's fair to regard I_T Nexus loss as an initiator failure. I=20 > also don't see any good way to do a 3rd-party release on a RESERVE=20 > reservation that persists after I_T Nexus loss. I also believe that=20 > initiator software that uses Reserve/Release does not expect=20 > reservations to persist across I_T Nexus loss. > > As to what needs to be stated in the iSCSI consolidated draft - the=20 > SPC reference has been updated to SPC-3 in which the Reserve/Release=20 > management method is obsolete, so there isn't any absolute requirement=20 > to cover that. OTOH, it might be reasonable to add a sentence to say=20 > that Reserve/Release state is implicit (e.g., as there's no way for an=20 > initiator to figure out which I_T Nexus holds a reservation), and that=20 > treating Reserve/Release state as indefinitely persistent after I_T=20 > Nexus Loss is a bad idea ("should not", but lower case) as it may=20 > require the initiator to reset the target device in order to clear out=20 > that state (issuing RELEASE commands isn't enough). This sort of text=20 > would require a reference to SPC-2. > > Thanks, > --David > >> -----Original Message----- >> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] >> Sent: Monday, September 24, 2012 8:58 AM >> To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org >> Subject: RE: iSCSI drafts - T10 and T11 references >> >> FCP is also not explicit (it defers to the SCSI specs - which do not say= ). >> The closest statements I can find still requires extrapolation: >> >> Here are some key points (from SAM 6.3.4 I_T nexus loss): >> >> 1) I_T nexus loss clears all ACA conditions; >> 2) established a UA - one of the following is suggested: >> >> "If the logical unit retains state information for the I_T nexus that=20 >> is lost, its response to the subsequent I_T nexus re-establishment=20 >> for the logical unit should include establishing a unit attention=20 >> with an additional sense code set to I_T NEXUS LOSS OCCURRED. >> >> If the logical unit does not retain state information for the I_T=20 >> nexus that is lost, it shall consider the subsequent I_T nexus=20 >> re-establishment, if any, as the formation of a new I_T nexus for=20 >> which there is no past history (e.g., establish a unit attention with=20 >> an additional sense code set to POWER ON OCCURRED)." >> >> But, there is no definition of what it means to "retain state". The=20 >> conclusion I could jump to based on #1 above, is that RESERVE state=20 >> is also cleared (this action lines up more with LU RESET than CLEAR=20 >> TASK SET); and based on the POWER ON CONDITION UA of 29/01 (or 29/00=20 >> - both are power on >> conditions) I could also assume that the RESERVE state is cleared;=20 >> but based on the UA of 29/07 (retain state), I could jump to the=20 >> conclusion that RESERVE state is retained. >> >> But these are all conclusions I've jump to with very limited basis in=20 >> the text. >> >> Based on what RFC3720 says about explicit state (e.g., mode pages), I=20 >> would assume that hosts would believe that every command they=20 >> explicitly send (like MODE SELECT or RESERVE) would have created=20 >> explicit state and therefore the host would assume RESERVE state is=20 >> retained. It doesn't make sense to have mode page values retained, but = have RESERVE state lost. >> >> Certain nexus relationships contain an explicit state (e.g., >> initiator-specific mode pages) that may need to be preserved by >> the device server [SAM2] in a logical unit through changes or >> failures in the iSCSI layer (e.g., session failures). >> >> It will be interesting to hear what a real host assumes about this situa= tion. >> >> Fred >> >> -----Original Message----- >> From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] >> Sent: Sunday, September 23, 2012 6:35 PM >> To: Knight, Frederick; david.black@emc.com; storm@ietf.org >> Subject: RE: iSCSI drafts - T10 and T11 references >> >> Hi Fred, >> >> SPC-3 discusses clearing effects of I-T nexus loss in several places. >> >> Section 5.8.2.9 for ALUA >> Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for=20 >> background self- test Section 5.6.1 on persistent reservations=20 >> Section 5.13 on timestamps ...... >> >> Thanks for pointing out the broken cross-reference - Appendix F instead = of E. >> I have just fixed it - and reviewed all other Appendix references=20 >> elsewhere in the draft. However, I do not see a circular reference=20 >> problem per se - Appendix E is correctly referencing Section 6.3.5=20 >> for definition of terms, e.g. session continuation. And Section=20 >> 6.3.5.1 is correctly deferring to Appendix E when discussing clearing=20 >> effects. Let me know if I'm overlooking something here. >> >> Now, the reserve/release management state, :) : I agree the formal=20 >> I_T nexus loss event didn't exist in SAM-2, however the notion of a=20 >> new I_T nexus state did co-exist with a session-oriented I_T nexus=20 >> well before iSCSI - FCP process login (PRLO) comes to mind here. So=20 >> I'd be curious about traditional FCP handling of reserve/release=20 >> management state. And I'm also checking internally here at MS. From=20 >> what I know now though, it does *appear* that reserve/release management= state should not be cleared with I_T nexus loss. >> >> Having said all that, note that reserve/release management state is=20 >> squarely in the SCSI territory. iSCSI spec has so far by design=20 >> deferred to SCSI standards in specifying clearing effects on SCSI=20 >> objects - as Appendix E.2 makes it clear. What we do specify is the=20 >> clearing effects of SCSI actions, e.g. LU reset, on iSCSI objects. >> >> Thanks. >> >> Mallikarjun >> >> >> >> >> >> -----Original Message----- >> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] >> Sent: Sunday, September 23, 2012 3:38 AM >> To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org >> Subject: RE: iSCSI drafts - T10 and T11 references >> >> Interesting, I just came across a question in a related area (only for S= AM-2). >> >> RFC3720 was written in between SAM-2 and SAM-3; some concepts were=20 >> drawn from SAM-2, and some from SAM-3. >> >> We have references to I_T NEXUS LOSS events in RFC3720. The I_T=20 >> Nexus loss concept does not exist in SAM-2. SAM-2 does include=20 >> references to RESERVE and RELEASE commands. SAM-3 includes I_T Nexus=20 >> loss, but it has no reference to RESERVE state. There are allusions=20 >> in RFC3720 about the intersection of RESERVE/RELEASE and I_T NEXUS=20 >> LOSS, but there is nothing explicit. RFC3720 references other=20 >> documents (SAM2, SAM3, SBC, SPC3), none of which address the situation. >> >> Consider text such as the following: >> 4.4.3.1. I_T Nexus State >> >> Certain nexus relationships contain an explicit state (e.g., >> initiator-specific mode pages) that may need to be preserved by >> the device server [SAM2] in a logical unit through changes or >> failures in the iSCSI layer (e.g., session failures). In order for >> that state to be restored, the iSCSI initiator should reestablish >> its session (re-login) to the same Target Portal Group using the >> previous ISID. That is, it should perform session recovery as >> described in Chapter 6. This is because the SCSI initiator port >> identifier and the SCSI target port identifier (or relative target >> port) form the datum that the SCSI logical unit device server uses >> to identify the I_T nexus. >> >> Is RESERVE state - explicit state? When an I_T nexus is=20 >> reestablished, initiator-specific mode pages are preserved. That=20 >> implies that an I_T nexus loss should be treated in the way that=20 >> SAM-2 describes via CLEAR TASK SET (where mode pages and RESERVE=20 >> state are preserved). If a session reestablishment caused saved mode=20 >> pages to be restored rather than the established initiator-specific=20 >> mode pages to be restored, then it would be like a LU reset (where=20 >> the RESERVE state is lost). Is RESERVE state explicit state that is to = be preserved? >> >> Appendix E covers clearing effects during various events, but it=20 >> never mentions RESERVE state. >> >> Section E.2 contains the following text: >> >> The only iSCSI protocol action that can effect clearing actions on >> SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 >> Loss of Nexus notification). [SPC3] describes the clearing effects >> of this notification on a variety of SCSI attributes. In addition, >> SCSI standards documents (such as [SAM2] and [SBC]) define >> additional clearing actions that may take place for several SCSI >> objects on SCSI events such as LU resets and power-on resets. >> >> I could not find any text in SPC3 that describes these clearing=20 >> effects (I don't know if this SPC3 reference is incorrect, but at a=20 >> minimum, it is ineffective). The SAM2 text for LU resets and=20 >> power-on resets is easily found, and clearly says that any RESERVE is=20 >> released in those cases. As an aside, the reference to section=20 >> 6.3.5.1 is a circular reference). The text in >> E.2 goes on: >> >> Since iSCSI defines a target cold reset as a protocol-equivalent >> to a target power-cycle, the iSCSI target cold reset must also be >> considered as the power-on reset event in interpreting the actions >> defined in the SCSI standards. >> >> When the iSCSI session is reconstructed (between the same SCSI >> ports with the same nexus identifier) reestablishing the same I_T >> nexus, all SCSI objects that are defined to not clear on the "I_T >> nexus loss" notification event, such as persistent reservations, >> are automatically associated to this new session. >> >> So, it is clear what happens to PERSISTENT RESERVE state, but not so=20 >> clear about RESERVE state. It also states that all SCSI objects that=20 >> are defined to not clear on the I_T nexus loss are retained. But=20 >> again, the only list I can find of what is cleared is here in this=20 >> Appendix, since SAM-2, SAM-3, SAM-4 are all silent on the interaction be= tween RESERVE and I_T Nexus loss. >> >> T10 has never had to deal with this because they removed RESERVE=20 >> before they added I_T Nexus loss. However, iSCSI did not do that; we=20 >> discuss BOTH, but we fail to describe how they interact. iSCSI also=20 >> has the feature of reestablishment of a session that does not exist=20 >> in SAM-2 (SAM-2 was still parallel SCSI focused, where connections=20 >> didn't exist). So again, if session reestablishment is examined in=20 >> the light of parallel SCSI, the intent would be to retain the RESERVE st= ate. Here is another section on the topic: >> >> 6.3.5.1. Loss of Nexus Notification >> >> The iSCSI layer provides the SCSI layer with the "I_T nexus loss" >> notification when any one of the following events happens: >> >> Successful completion of session reinstatement. >> Session closure event. >> Session timeout event. >> >> Certain SCSI object clearing actions may result due to the >> notification in the SCSI end nodes, as documented in Appendix F. - >> Clearing Effects of Various Events on Targets. >> >> FWIW - the reference above is incorrect - there is no Appendix F in=20 >> the consolidated draft - this should be Appendix E (and also, BTW -=20 >> this is a circular reference). >> >> So anyway, it seems like the intent was that session reinstatement=20 >> would preserve the RESERVE state across the I_T Nexus loss event. >> >> So aside from the incorrect reference (and circular nature of it),=20 >> any thoughts on whether we should get explicit about this situation? =20 >> Should we remain silent; should we tweak Appendix E along these lines=20 >> (or other lines that others might suggest): >> >> 4.4.3.1. I_T Nexus State >> >> Certain nexus relationships contain an explicit state (e.g., >> initiator-specific mode pages, and RESERVE state) that may need to=20 >> be preserved by >> the device server [SAM2] in a logical unit through changes or >> failures in the iSCSI layer (e.g., session failures).... >> >> Fred Knight >> >> >> >> -----Original Message----- >> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On=20 >> Behalf Of Mallikarjun Chadalapaka >> Sent: Sunday, September 23, 2012 1:34 AM >> To: david.black@emc.com; storm@ietf.org >> Subject: Re: [storm] iSCSI drafts - T10 and T11 references >> >> Hi David, >> >> I have noticed this comment as I am working through all Last Call commen= ts. >> >> The references to SAM-3 and SAM-4 are keyed to this paragraph in Section= 13.9: >> >> "[SAM2] and [SAM3] specifications note in their informative text that=20 >> TPGT value should be non-zero, note that it is incorrect. A zero=20 >> value is allowed as a legal value for TPGT. This discrepancy=20 >> currently stands corrected in [SAM4]." >> >> Additionally, SAM4 reference is also used in citing the new ASC/ASCQ=20 >> codes that it defines, in Section 11.14.5 of the draft. >> >> So, I believe references to all three SAM specs would continue to be=20 >> appropriate. >> >> The current draft references FC-FS in discussing the NAA format in a=20 >> few places, so the current reference at the end is intentionally the=20 >> same. Is the older reference a problem? If so, I suppose we can=20 >> replace all references to FC-FS-4, but that seems to be still in=20 >> early stages(0.1 spec posted on April 2012), so perhaps we should use FC= -FS3? >> >> Thanks. >> >> Mallikarjun >> >> >> >> >> -----Original Message----- >> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On=20 >> Behalf Of david.black@emc.com >> Sent: Thursday, July 26, 2012 6:47 AM >> To: storm@ietf.org >> Subject: [storm] iSCSI drafts - T10 and T11 references >> >> In making up the lists of T10 and T11 documents that are currently=20 >> referenced, some of those references don't look right to me, for example= : >> >> - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. >> - T11: The FC-FS reference is out of date. >> >> As part of dealing with the comments from IETF Last Call, we'll need=20 >> to go over all of the T10 and T11 references (and should check all=20 >> the references) to make sure that they're up to date and necessary. >> >> Thanks, >> --David >> ---------------------------------------------------- >> David L. Black, Distinguished Engineer EMC Corporation, 176 South=20 >> St., Hopkinton, MA 01748 >> +1 (508) 293-7953 FAX: +1 (508) 293-7786 >> david.black@emc.com Mobile: +1 (978) 394-7754 >> ---------------------------------------------------- >> >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm >> >> >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm >> >> > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From Frederick.Knight@netapp.com Mon Sep 24 11:56:41 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 646F721F8898 for ; Mon, 24 Sep 2012 11:56:41 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SCJZaunvExao for ; Mon, 24 Sep 2012 11:56:40 -0700 (PDT) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id DDF7721F887A for ; Mon, 24 Sep 2012 11:56:39 -0700 (PDT) X-IronPort-AV: E=Sophos;i="4.80,477,1344236400"; d="scan'208";a="693507987" Received: from smtp2.corp.netapp.com ([10.57.159.114]) by mx2-out.netapp.com with ESMTP; 24 Sep 2012 11:56:39 -0700 Received: from vmwexceht01-prd.hq.netapp.com (vmwexceht01-prd.hq.netapp.com [10.106.76.239]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8OIudco017128; Mon, 24 Sep 2012 11:56:39 -0700 (PDT) Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.102]) by vmwexceht01-prd.hq.netapp.com ([10.106.76.239]) with mapi id 14.02.0309.002; Mon, 24 Sep 2012 11:56:39 -0700 From: "Knight, Frederick" To: "Black, David" , Mallikarjun Chadalapaka , "storm@ietf.org" Thread-Topic: RESERVE state - clearing effects Thread-Index: AQHNmnOgfkCvZihy0EOAZUHR/pZPlpeZtzaw Date: Mon, 24 Sep 2012 18:56:37 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.106.53.53] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [storm] RESERVE state - clearing effects X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Sep 2012 18:56:41 -0000 I agree that I_T Nexus loss might be an initiator failure - it might just a= s easily be a wire/Ethernet switch failure (between the initiator and targe= t), and the initiator is perfectly fine - which, is the whole point of logg= ing back in using the same ISID; and, the point of the failure case the spe= cific paragraph in RFC3720 references - "a failure in the iSCSI layer". RFC3720 classifies MODE SELECT commands (that set port specific mode parame= ters) as explicit. There is also no way for a third party to impact those = port specific mode parameters. RFC3270 is specific in this case, that this= state IS retained across the I_T Nexus loss (and SAM-2 is specific that CL= EAR TASK SET also retains that state, and that LU RESET eliminates that sta= te). I assume a command from a host is something the host explicitly asked= for, and not something that implicitly happens. Should an initiator that does a relogin using the same ISID, just be able t= o continue where it left off, or must it assume state is lost and reestabli= sh state before it can resume? If it is to assume state is lost, then why = does RFC3710 require that I_T specific mode parameters be retained? If the= host is to assume state is lost, then what is the difference between I_T N= exus loss and Power On? Is the host to assume only some state is retained?= How does the host know which state is retained and which is lost? > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages) that may need to be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures). SAM-2 was written in the context of parallel SCSI, so I would think that SA= M-2 (while silent on the subject), would have expected a parallel SCSI RESE= RVE command to retain state - if we could have ever figured out what an I_T= Nexus loss might have look like in the parallel SCSI context. I suppose that's enough questions for now. Fred -----Original Message----- From: Black, David [mailto:david.black@emc.com]=20 Sent: Monday, September 24, 2012 12:42 PM To: Knight, Frederick; Mallikarjun Chadalapaka; storm@ietf.org Subject: RESERVE state - clearing effects Fred, This is interesting ... > Here are some key points (from SAM 6.3.4 I_T nexus loss): > > 1) I_T nexus loss clears all ACA conditions; > 2) established a UA - one of the following is suggested: > > "If the logical unit retains state information for the I_T nexus that=20 > is lost, its response to the subsequent I_T nexus re-establishment for=20 > the logical unit should include establishing a unit attention with an=20 > additional sense code set to I_T NEXUS LOSS OCCURRED. > > If the logical unit does not retain state information for the I_T=20 > nexus that is lost, it shall consider the subsequent I_T nexus=20 > re-establishment, if any, as the formation of a new I_T nexus for=20 > which there is no past history (e.g., establish a unit attention with=20 > an additional sense code set to POWER ON OCCURRED)." That "state information" has to identify the endpoints of the I_T Nexus - for iSCSI that means at least the ISID and TPGT. If that's gone, I think= the RESERVE state is gone as well, in contrast to PERSISTENT RESERVE state= . On RESERVE state, SPC-2 has this to say in 5.5.2 "The Reserve/Release ma= nagement method": The reserve/release management method commands, RESERVE(6), RESERVE(10), RELEASE(6), and RELEASE(10) are used among multiple initiators that do not require operations to be protected across initiator failures (and subsequent hard resets). I think it's fair to regard I_T Nexus loss as an initiator failure. I also= don't see any good way to do a 3rd-party release on a RESERVE reservation = that persists after I_T Nexus loss. I also believe that initiator software= that uses Reserve/Release does not expect reservations to persist across I= _T Nexus loss. As to what needs to be stated in the iSCSI consolidated draft - the SPC ref= erence has been updated to SPC-3 in which the Reserve/Release management me= thod is obsolete, so there isn't any absolute requirement to cover that. O= TOH, it might be reasonable to add a sentence to say that Reserve/Release s= tate is implicit (e.g., as there's no way for an initiator to figure out wh= ich I_T Nexus holds a reservation), and that treating Reserve/Release state= as indefinitely persistent after I_T Nexus Loss is a bad idea ("should not= ", but lower case) as it may require the initiator to reset the target devi= ce in order to clear out that state (issuing RELEASE commands isn't enough)= . This sort of text would require a reference to SPC-2. Thanks, --David > -----Original Message----- > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > Sent: Monday, September 24, 2012 8:58 AM > To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references > > FCP is also not explicit (it defers to the SCSI specs - which do not say)= . > The closest statements I can find still requires extrapolation: > > Here are some key points (from SAM 6.3.4 I_T nexus loss): > > 1) I_T nexus loss clears all ACA conditions; > 2) established a UA - one of the following is suggested: > > "If the logical unit retains state information for the I_T nexus that=20 > is lost, its response to the subsequent I_T nexus re-establishment for=20 > the logical unit should include establishing a unit attention with an=20 > additional sense code set to I_T NEXUS LOSS OCCURRED. > > If the logical unit does not retain state information for the I_T=20 > nexus that is lost, it shall consider the subsequent I_T nexus=20 > re-establishment, if any, as the formation of a new I_T nexus for=20 > which there is no past history (e.g., establish a unit attention with=20 > an additional sense code set to POWER ON OCCURRED)." > > But, there is no definition of what it means to "retain state". The=20 > conclusion I could jump to based on #1 above, is that RESERVE state is=20 > also cleared (this action lines up more with LU RESET than CLEAR TASK=20 > SET); and based on the POWER ON CONDITION UA of 29/01 (or 29/00 - both=20 > are power on > conditions) I could also assume that the RESERVE state is cleared; but=20 > based on the UA of 29/07 (retain state), I could jump to the=20 > conclusion that RESERVE state is retained. > > But these are all conclusions I've jump to with very limited basis in=20 > the text. > > Based on what RFC3720 says about explicit state (e.g., mode pages), I=20 > would assume that hosts would believe that every command they=20 > explicitly send (like MODE SELECT or RESERVE) would have created=20 > explicit state and therefore the host would assume RESERVE state is=20 > retained. It doesn't make sense to have mode page values retained, but h= ave RESERVE state lost. > > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages) that may need to be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures). > > It will be interesting to hear what a real host assumes about this situat= ion. > > Fred > > -----Original Message----- > From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] > Sent: Sunday, September 23, 2012 6:35 PM > To: Knight, Frederick; david.black@emc.com; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references > > Hi Fred, > > SPC-3 discusses clearing effects of I-T nexus loss in several places. > > Section 5.8.2.9 for ALUA > Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for=20 > background self- test Section 5.6.1 on persistent reservations Section=20 > 5.13 on timestamps ...... > > Thanks for pointing out the broken cross-reference - Appendix F instead o= f E. > I have just fixed it - and reviewed all other Appendix references=20 > elsewhere in the draft. However, I do not see a circular reference=20 > problem per se - Appendix E is correctly referencing Section 6.3.5 for=20 > definition of terms, e.g. session continuation. And Section 6.3.5.1 is=20 > correctly deferring to Appendix E when discussing clearing effects.=20 > Let me know if I'm overlooking something here. > > Now, the reserve/release management state, :) : I agree the formal=20 > I_T nexus loss event didn't exist in SAM-2, however the notion of a=20 > new I_T nexus state did co-exist with a session-oriented I_T nexus=20 > well before iSCSI - FCP process login (PRLO) comes to mind here. So=20 > I'd be curious about traditional FCP handling of reserve/release=20 > management state. And I'm also checking internally here at MS. From=20 > what I know now though, it does *appear* that reserve/release management = state should not be cleared with I_T nexus loss. > > Having said all that, note that reserve/release management state is=20 > squarely in the SCSI territory. iSCSI spec has so far by design=20 > deferred to SCSI standards in specifying clearing effects on SCSI=20 > objects - as Appendix E.2 makes it clear. What we do specify is the=20 > clearing effects of SCSI actions, e.g. LU reset, on iSCSI objects. > > Thanks. > > Mallikarjun > > > > > > -----Original Message----- > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > Sent: Sunday, September 23, 2012 3:38 AM > To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references > > Interesting, I just came across a question in a related area (only for SA= M-2). > > RFC3720 was written in between SAM-2 and SAM-3; some concepts were=20 > drawn from SAM-2, and some from SAM-3. > > We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus=20 > loss concept does not exist in SAM-2. SAM-2 does include references=20 > to RESERVE and RELEASE commands. SAM-3 includes I_T Nexus loss, but=20 > it has no reference to RESERVE state. There are allusions in RFC3720=20 > about the intersection of RESERVE/RELEASE and I_T NEXUS LOSS, but=20 > there is nothing explicit. RFC3720 references other documents (SAM2,=20 > SAM3, SBC, SPC3), none of which address the situation. > > Consider text such as the following: > 4.4.3.1. I_T Nexus State > > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages) that may need to be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures). In order for > that state to be restored, the iSCSI initiator should reestablish > its session (re-login) to the same Target Portal Group using the > previous ISID. That is, it should perform session recovery as > described in Chapter 6. This is because the SCSI initiator port > identifier and the SCSI target port identifier (or relative target > port) form the datum that the SCSI logical unit device server uses > to identify the I_T nexus. > > Is RESERVE state - explicit state? When an I_T nexus is=20 > reestablished, initiator-specific mode pages are preserved. That=20 > implies that an I_T nexus loss should be treated in the way that SAM-2=20 > describes via CLEAR TASK SET (where mode pages and RESERVE state are=20 > preserved). If a session reestablishment caused saved mode pages to=20 > be restored rather than the established initiator-specific mode pages=20 > to be restored, then it would be like a LU reset (where the RESERVE=20 > state is lost). Is RESERVE state explicit state that is to be preserved? > > Appendix E covers clearing effects during various events, but it never=20 > mentions RESERVE state. > > Section E.2 contains the following text: > > The only iSCSI protocol action that can effect clearing actions on > SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 > Loss of Nexus notification). [SPC3] describes the clearing effects > of this notification on a variety of SCSI attributes. In addition, > SCSI standards documents (such as [SAM2] and [SBC]) define > additional clearing actions that may take place for several SCSI > objects on SCSI events such as LU resets and power-on resets. > > I could not find any text in SPC3 that describes these clearing=20 > effects (I don't know if this SPC3 reference is incorrect, but at a=20 > minimum, it is ineffective). The SAM2 text for LU resets and power-on=20 > resets is easily found, and clearly says that any RESERVE is released=20 > in those cases. As an aside, the reference to section 6.3.5.1 is a=20 > circular reference). The text in > E.2 goes on: > > Since iSCSI defines a target cold reset as a protocol-equivalent > to a target power-cycle, the iSCSI target cold reset must also be > considered as the power-on reset event in interpreting the actions > defined in the SCSI standards. > > When the iSCSI session is reconstructed (between the same SCSI > ports with the same nexus identifier) reestablishing the same I_T > nexus, all SCSI objects that are defined to not clear on the "I_T > nexus loss" notification event, such as persistent reservations, > are automatically associated to this new session. > > So, it is clear what happens to PERSISTENT RESERVE state, but not so=20 > clear about RESERVE state. It also states that all SCSI objects that=20 > are defined to not clear on the I_T nexus loss are retained. But=20 > again, the only list I can find of what is cleared is here in this=20 > Appendix, since SAM-2, SAM-3, SAM-4 are all silent on the interaction bet= ween RESERVE and I_T Nexus loss. > > T10 has never had to deal with this because they removed RESERVE=20 > before they added I_T Nexus loss. However, iSCSI did not do that; we=20 > discuss BOTH, but we fail to describe how they interact. iSCSI also=20 > has the feature of reestablishment of a session that does not exist in=20 > SAM-2 (SAM-2 was still parallel SCSI focused, where connections didn't=20 > exist). So again, if session reestablishment is examined in the light=20 > of parallel SCSI, the intent would be to retain the RESERVE state. Here = is another section on the topic: > > 6.3.5.1. Loss of Nexus Notification > > The iSCSI layer provides the SCSI layer with the "I_T nexus loss" > notification when any one of the following events happens: > > Successful completion of session reinstatement. > Session closure event. > Session timeout event. > > Certain SCSI object clearing actions may result due to the > notification in the SCSI end nodes, as documented in Appendix F. - > Clearing Effects of Various Events on Targets. > > FWIW - the reference above is incorrect - there is no Appendix F in=20 > the consolidated draft - this should be Appendix E (and also, BTW -=20 > this is a circular reference). > > So anyway, it seems like the intent was that session reinstatement=20 > would preserve the RESERVE state across the I_T Nexus loss event. > > So aside from the incorrect reference (and circular nature of it), any=20 > thoughts on whether we should get explicit about this situation? =20 > Should we remain silent; should we tweak Appendix E along these lines=20 > (or other lines that others might suggest): > > 4.4.3.1. I_T Nexus State > > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages, and RESERVE state) that may need to=20 > be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures).... > > Fred Knight > > > > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf=20 > Of Mallikarjun Chadalapaka > Sent: Sunday, September 23, 2012 1:34 AM > To: david.black@emc.com; storm@ietf.org > Subject: Re: [storm] iSCSI drafts - T10 and T11 references > > Hi David, > > I have noticed this comment as I am working through all Last Call comment= s. > > The references to SAM-3 and SAM-4 are keyed to this paragraph in Section = 13.9: > > "[SAM2] and [SAM3] specifications note in their informative text that=20 > TPGT value should be non-zero, note that it is incorrect. A zero=20 > value is allowed as a legal value for TPGT. This discrepancy currently=20 > stands corrected in [SAM4]." > > Additionally, SAM4 reference is also used in citing the new ASC/ASCQ=20 > codes that it defines, in Section 11.14.5 of the draft. > > So, I believe references to all three SAM specs would continue to be=20 > appropriate. > > The current draft references FC-FS in discussing the NAA format in a=20 > few places, so the current reference at the end is intentionally the=20 > same. Is the older reference a problem? If so, I suppose we can=20 > replace all references to FC-FS-4, but that seems to be still in early=20 > stages(0.1 spec posted on April 2012), so perhaps we should use FC-FS3? > > Thanks. > > Mallikarjun > > > > > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf=20 > Of david.black@emc.com > Sent: Thursday, July 26, 2012 6:47 AM > To: storm@ietf.org > Subject: [storm] iSCSI drafts - T10 and T11 references > > In making up the lists of T10 and T11 documents that are currently=20 > referenced, some of those references don't look right to me, for example: > > - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. > - T11: The FC-FS reference is out of date. > > As part of dealing with the comments from IETF Last Call, we'll need=20 > to go over all of the T10 and T11 references (and should check all the=20 > references) to make sure that they're up to date and necessary. > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer EMC Corporation, 176 South St.,=20 > Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > david.black@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > From david.black@emc.com Mon Sep 24 14:57:51 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5B3681F041D for ; Mon, 24 Sep 2012 14:57:51 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3k9OJzhzoe+d for ; Mon, 24 Sep 2012 14:57:49 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 176E921F88A3 for ; Mon, 24 Sep 2012 14:57:48 -0700 (PDT) Received: from hop04-l1d11-si02.isus.emc.com (HOP04-L1D11-SI02.isus.emc.com [10.254.111.55]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8OLvkVl031407 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 24 Sep 2012 17:57:46 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.222.130]) by hop04-l1d11-si02.isus.emc.com (RSA Interceptor); Mon, 24 Sep 2012 17:57:24 -0400 Received: from mxhub02.corp.emc.com (mxhub02.corp.emc.com [10.254.141.104]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8OLvMaR003785; Mon, 24 Sep 2012 17:57:23 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub02.corp.emc.com ([10.254.141.104]) with mapi; Mon, 24 Sep 2012 17:57:23 -0400 From: "Black, David" To: "Knight, Frederick" , Mallikarjun Chadalapaka , "storm@ietf.org" Date: Mon, 24 Sep 2012 17:57:21 -0400 Thread-Topic: RESERVE state - clearing effects Thread-Index: AQHNmnOgfkCvZihy0EOAZUHR/pZPlpeZtzawgABN03A= Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE792E8@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: Re: [storm] RESERVE state - clearing effects X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Sep 2012 21:57:51 -0000 Fred, > I agree that I_T Nexus loss might be an initiator failure - it might just= as > easily be a wire/Ethernet switch failure (between the initiator and targe= t), > and the initiator is perfectly fine - which, is the whole point of loggin= g > back in using the same ISID; and, the point of the failure case the speci= fic > paragraph in RFC3720 references - "a failure in the iSCSI layer". So far so good - the approach I'm favoring allows for both scenarios. IMHO= , it'd be ok for the target to retain RESERVE state for a while, but retainin= g it indefinitely in a situation where the initiator cannot log back in with the same ISID requires a "hard reset" to recover the target (as stated in S= PC-2). [... snip ...] > Should an initiator that does a relogin using the same ISID, just be able= to > continue where it left off, or must it assume state is lost and reestabli= sh > state before it can resume? Aha, I think I see where we're talking past each other ;-). It looks like you're concerned with the initiator successfully logging back in and being able to pick up where it left off, whereas I'm concerned with what happens when the initiator can't or doesn't log back in and specifically the side effects on other I_T Nexuses in that scenario. For an initiator that successfully logs back in with the same ISID, the question of whether it can continue is answered in the text that you quoted from SAM on which UA condition is established for such a re-login. In that situation, I'm fine with linking survival of the RESERVE state with surviva= l of I_T Nexus-specific mode state (and both would be indicated by which UA occurs). My concern is what happens when the I_T Nexus is not re-established - if th= e RESERVATION CONFLICT side effects on other I_T Nexuses persist indefinitely= , the only means of recovery is a "hard reset" via some other I_T Nexus, and I really don't want to add a new requirement for that. Thanks, --David > -----Original Message----- > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > Sent: Monday, September 24, 2012 2:57 PM > To: Black, David; Mallikarjun Chadalapaka; storm@ietf.org > Subject: RE: RESERVE state - clearing effects > > I agree that I_T Nexus loss might be an initiator failure - it might just= as > easily be a wire/Ethernet switch failure (between the initiator and targe= t), > and the initiator is perfectly fine - which, is the whole point of loggin= g > back in using the same ISID; and, the point of the failure case the speci= fic > paragraph in RFC3720 references - "a failure in the iSCSI layer". > > RFC3720 classifies MODE SELECT commands (that set port specific mode > parameters) as explicit. There is also no way for a third party to impac= t > those port specific mode parameters. RFC3270 is specific in this case, t= hat > this state IS retained across the I_T Nexus loss (and SAM-2 is specific t= hat > CLEAR TASK SET also retains that state, and that LU RESET eliminates that > state). I assume a command from a host is something the host explicitly = asked > for, and not something that implicitly happens. > > Should an initiator that does a relogin using the same ISID, just be able= to > continue where it left off, or must it assume state is lost and reestabli= sh > state before it can resume? If it is to assume state is lost, then why d= oes > RFC3710 require that I_T specific mode parameters be retained? If the ho= st is > to assume state is lost, then what is the difference between I_T Nexus lo= ss > and Power On? Is the host to assume only some state is retained? How do= es > the host know which state is retained and which is lost? > > > Certain nexus relationships contain an explicit state (e.g., > > initiator-specific mode pages) that may need to be preserved by > > the device server [SAM2] in a logical unit through changes or > > failures in the iSCSI layer (e.g., session failures). > > SAM-2 was written in the context of parallel SCSI, so I would think that = SAM-2 > (while silent on the subject), would have expected a parallel SCSI RESERV= E > command to retain state - if we could have ever figured out what an I_T N= exus > loss might have look like in the parallel SCSI context. > > I suppose that's enough questions for now. > > Fred > > > -----Original Message----- > From: Black, David [mailto:david.black@emc.com] > Sent: Monday, September 24, 2012 12:42 PM > To: Knight, Frederick; Mallikarjun Chadalapaka; storm@ietf.org > Subject: RESERVE state - clearing effects > > Fred, > > This is interesting ... > > > Here are some key points (from SAM 6.3.4 I_T nexus loss): > > > > 1) I_T nexus loss clears all ACA conditions; > > 2) established a UA - one of the following is suggested: > > > > "If the logical unit retains state information for the I_T nexus that > > is lost, its response to the subsequent I_T nexus re-establishment for > > the logical unit should include establishing a unit attention with an > > additional sense code set to I_T NEXUS LOSS OCCURRED. > > > > If the logical unit does not retain state information for the I_T > > nexus that is lost, it shall consider the subsequent I_T nexus > > re-establishment, if any, as the formation of a new I_T nexus for > > which there is no past history (e.g., establish a unit attention with > > an additional sense code set to POWER ON OCCURRED)." > > That "state information" has to identify the endpoints of the I_T Nexus > - for iSCSI that means at least the ISID and TPGT. If that's gone, I thi= nk > the RESERVE state is gone as well, in contrast to PERSISTENT RESERVE stat= e. > On RESERVE state, SPC-2 has this to say in 5.5.2 "The Reserve/Release > management method": > > The reserve/release management method commands, RESERVE(6), > RESERVE(10), RELEASE(6), and RELEASE(10) are used among multiple > initiators that do not require operations to be protected across > initiator failures (and subsequent hard resets). > > I think it's fair to regard I_T Nexus loss as an initiator failure. I al= so > don't see any good way to do a 3rd-party release on a RESERVE reservation= that > persists after I_T Nexus loss. I also believe that initiator software th= at > uses Reserve/Release does not expect reservations to persist across I_T N= exus > loss. > > As to what needs to be stated in the iSCSI consolidated draft - the SPC > reference has been updated to SPC-3 in which the Reserve/Release manageme= nt > method is obsolete, so there isn't any absolute requirement to cover that= . > OTOH, it might be reasonable to add a sentence to say that Reserve/Releas= e > state is implicit (e.g., as there's no way for an initiator to figure out > which I_T Nexus holds a reservation), and that treating Reserve/Release s= tate > as indefinitely persistent after I_T Nexus Loss is a bad idea ("should no= t", > but lower case) as it may require the initiator to reset the target devic= e in > order to clear out that state (issuing RELEASE commands isn't enough). T= his > sort of text would require a reference to SPC-2. > > Thanks, > --David > > > -----Original Message----- > > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > > Sent: Monday, September 24, 2012 8:58 AM > > To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org > > Subject: RE: iSCSI drafts - T10 and T11 references > > > > FCP is also not explicit (it defers to the SCSI specs - which do not sa= y). > > The closest statements I can find still requires extrapolation: > > > > Here are some key points (from SAM 6.3.4 I_T nexus loss): > > > > 1) I_T nexus loss clears all ACA conditions; > > 2) established a UA - one of the following is suggested: > > > > "If the logical unit retains state information for the I_T nexus that > > is lost, its response to the subsequent I_T nexus re-establishment for > > the logical unit should include establishing a unit attention with an > > additional sense code set to I_T NEXUS LOSS OCCURRED. > > > > If the logical unit does not retain state information for the I_T > > nexus that is lost, it shall consider the subsequent I_T nexus > > re-establishment, if any, as the formation of a new I_T nexus for > > which there is no past history (e.g., establish a unit attention with > > an additional sense code set to POWER ON OCCURRED)." > > > > But, there is no definition of what it means to "retain state". The > > conclusion I could jump to based on #1 above, is that RESERVE state is > > also cleared (this action lines up more with LU RESET than CLEAR TASK > > SET); and based on the POWER ON CONDITION UA of 29/01 (or 29/00 - both > > are power on > > conditions) I could also assume that the RESERVE state is cleared; but > > based on the UA of 29/07 (retain state), I could jump to the > > conclusion that RESERVE state is retained. > > > > But these are all conclusions I've jump to with very limited basis in > > the text. > > > > Based on what RFC3720 says about explicit state (e.g., mode pages), I > > would assume that hosts would believe that every command they > > explicitly send (like MODE SELECT or RESERVE) would have created > > explicit state and therefore the host would assume RESERVE state is > > retained. It doesn't make sense to have mode page values retained, but= have > RESERVE state lost. > > > > Certain nexus relationships contain an explicit state (e.g., > > initiator-specific mode pages) that may need to be preserved by > > the device server [SAM2] in a logical unit through changes or > > failures in the iSCSI layer (e.g., session failures). > > > > It will be interesting to hear what a real host assumes about this > situation. > > > > Fred > > > > -----Original Message----- > > From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] > > Sent: Sunday, September 23, 2012 6:35 PM > > To: Knight, Frederick; david.black@emc.com; storm@ietf.org > > Subject: RE: iSCSI drafts - T10 and T11 references > > > > Hi Fred, > > > > SPC-3 discusses clearing effects of I-T nexus loss in several places. > > > > Section 5.8.2.9 for ALUA > > Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for > > background self- test Section 5.6.1 on persistent reservations Section > > 5.13 on timestamps ...... > > > > Thanks for pointing out the broken cross-reference - Appendix F instead= of > E. > > I have just fixed it - and reviewed all other Appendix references > > elsewhere in the draft. However, I do not see a circular reference > > problem per se - Appendix E is correctly referencing Section 6.3.5 for > > definition of terms, e.g. session continuation. And Section 6.3.5.1 is > > correctly deferring to Appendix E when discussing clearing effects. > > Let me know if I'm overlooking something here. > > > > Now, the reserve/release management state, :) : I agree the formal > > I_T nexus loss event didn't exist in SAM-2, however the notion of a > > new I_T nexus state did co-exist with a session-oriented I_T nexus > > well before iSCSI - FCP process login (PRLO) comes to mind here. So > > I'd be curious about traditional FCP handling of reserve/release > > management state. And I'm also checking internally here at MS. From > > what I know now though, it does *appear* that reserve/release managemen= t > state should not be cleared with I_T nexus loss. > > > > Having said all that, note that reserve/release management state is > > squarely in the SCSI territory. iSCSI spec has so far by design > > deferred to SCSI standards in specifying clearing effects on SCSI > > objects - as Appendix E.2 makes it clear. What we do specify is the > > clearing effects of SCSI actions, e.g. LU reset, on iSCSI objects. > > > > Thanks. > > > > Mallikarjun > > > > > > > > > > > > -----Original Message----- > > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > > Sent: Sunday, September 23, 2012 3:38 AM > > To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org > > Subject: RE: iSCSI drafts - T10 and T11 references > > > > Interesting, I just came across a question in a related area (only for = SAM- > 2). > > > > RFC3720 was written in between SAM-2 and SAM-3; some concepts were > > drawn from SAM-2, and some from SAM-3. > > > > We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus > > loss concept does not exist in SAM-2. SAM-2 does include references > > to RESERVE and RELEASE commands. SAM-3 includes I_T Nexus loss, but > > it has no reference to RESERVE state. There are allusions in RFC3720 > > about the intersection of RESERVE/RELEASE and I_T NEXUS LOSS, but > > there is nothing explicit. RFC3720 references other documents (SAM2, > > SAM3, SBC, SPC3), none of which address the situation. > > > > Consider text such as the following: > > 4.4.3.1. I_T Nexus State > > > > Certain nexus relationships contain an explicit state (e.g., > > initiator-specific mode pages) that may need to be preserved by > > the device server [SAM2] in a logical unit through changes or > > failures in the iSCSI layer (e.g., session failures). In order for > > that state to be restored, the iSCSI initiator should reestablish > > its session (re-login) to the same Target Portal Group using the > > previous ISID. That is, it should perform session recovery as > > described in Chapter 6. This is because the SCSI initiator port > > identifier and the SCSI target port identifier (or relative target > > port) form the datum that the SCSI logical unit device server uses > > to identify the I_T nexus. > > > > Is RESERVE state - explicit state? When an I_T nexus is > > reestablished, initiator-specific mode pages are preserved. That > > implies that an I_T nexus loss should be treated in the way that SAM-2 > > describes via CLEAR TASK SET (where mode pages and RESERVE state are > > preserved). If a session reestablishment caused saved mode pages to > > be restored rather than the established initiator-specific mode pages > > to be restored, then it would be like a LU reset (where the RESERVE > > state is lost). Is RESERVE state explicit state that is to be preserve= d? > > > > Appendix E covers clearing effects during various events, but it never > > mentions RESERVE state. > > > > Section E.2 contains the following text: > > > > The only iSCSI protocol action that can effect clearing actions on > > SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 > > Loss of Nexus notification). [SPC3] describes the clearing effects > > of this notification on a variety of SCSI attributes. In addition, > > SCSI standards documents (such as [SAM2] and [SBC]) define > > additional clearing actions that may take place for several SCSI > > objects on SCSI events such as LU resets and power-on resets. > > > > I could not find any text in SPC3 that describes these clearing > > effects (I don't know if this SPC3 reference is incorrect, but at a > > minimum, it is ineffective). The SAM2 text for LU resets and power-on > > resets is easily found, and clearly says that any RESERVE is released > > in those cases. As an aside, the reference to section 6.3.5.1 is a > > circular reference). The text in > > E.2 goes on: > > > > Since iSCSI defines a target cold reset as a protocol-equivalent > > to a target power-cycle, the iSCSI target cold reset must also be > > considered as the power-on reset event in interpreting the actions > > defined in the SCSI standards. > > > > When the iSCSI session is reconstructed (between the same SCSI > > ports with the same nexus identifier) reestablishing the same I_T > > nexus, all SCSI objects that are defined to not clear on the "I_T > > nexus loss" notification event, such as persistent reservations, > > are automatically associated to this new session. > > > > So, it is clear what happens to PERSISTENT RESERVE state, but not so > > clear about RESERVE state. It also states that all SCSI objects that > > are defined to not clear on the I_T nexus loss are retained. But > > again, the only list I can find of what is cleared is here in this > > Appendix, since SAM-2, SAM-3, SAM-4 are all silent on the interaction > between RESERVE and I_T Nexus loss. > > > > T10 has never had to deal with this because they removed RESERVE > > before they added I_T Nexus loss. However, iSCSI did not do that; we > > discuss BOTH, but we fail to describe how they interact. iSCSI also > > has the feature of reestablishment of a session that does not exist in > > SAM-2 (SAM-2 was still parallel SCSI focused, where connections didn't > > exist). So again, if session reestablishment is examined in the light > > of parallel SCSI, the intent would be to retain the RESERVE state. Her= e is > another section on the topic: > > > > 6.3.5.1. Loss of Nexus Notification > > > > The iSCSI layer provides the SCSI layer with the "I_T nexus loss" > > notification when any one of the following events happens: > > > > Successful completion of session reinstatement. > > Session closure event. > > Session timeout event. > > > > Certain SCSI object clearing actions may result due to the > > notification in the SCSI end nodes, as documented in Appendix F. - > > Clearing Effects of Various Events on Targets. > > > > FWIW - the reference above is incorrect - there is no Appendix F in > > the consolidated draft - this should be Appendix E (and also, BTW - > > this is a circular reference). > > > > So anyway, it seems like the intent was that session reinstatement > > would preserve the RESERVE state across the I_T Nexus loss event. > > > > So aside from the incorrect reference (and circular nature of it), any > > thoughts on whether we should get explicit about this situation? > > Should we remain silent; should we tweak Appendix E along these lines > > (or other lines that others might suggest): > > > > 4.4.3.1. I_T Nexus State > > > > Certain nexus relationships contain an explicit state (e.g., > > initiator-specific mode pages, and RESERVE state) that may need to > > be preserved by > > the device server [SAM2] in a logical unit through changes or > > failures in the iSCSI layer (e.g., session failures).... > > > > Fred Knight > > > > > > > > -----Original Message----- > > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf > > Of Mallikarjun Chadalapaka > > Sent: Sunday, September 23, 2012 1:34 AM > > To: david.black@emc.com; storm@ietf.org > > Subject: Re: [storm] iSCSI drafts - T10 and T11 references > > > > Hi David, > > > > I have noticed this comment as I am working through all Last Call comme= nts. > > > > The references to SAM-3 and SAM-4 are keyed to this paragraph in Sectio= n > 13.9: > > > > "[SAM2] and [SAM3] specifications note in their informative text that > > TPGT value should be non-zero, note that it is incorrect. A zero > > value is allowed as a legal value for TPGT. This discrepancy currently > > stands corrected in [SAM4]." > > > > Additionally, SAM4 reference is also used in citing the new ASC/ASCQ > > codes that it defines, in Section 11.14.5 of the draft. > > > > So, I believe references to all three SAM specs would continue to be > > appropriate. > > > > The current draft references FC-FS in discussing the NAA format in a > > few places, so the current reference at the end is intentionally the > > same. Is the older reference a problem? If so, I suppose we can > > replace all references to FC-FS-4, but that seems to be still in early > > stages(0.1 spec posted on April 2012), so perhaps we should use FC-FS3? > > > > Thanks. > > > > Mallikarjun > > > > > > > > > > -----Original Message----- > > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf > > Of david.black@emc.com > > Sent: Thursday, July 26, 2012 6:47 AM > > To: storm@ietf.org > > Subject: [storm] iSCSI drafts - T10 and T11 references > > > > In making up the lists of T10 and T11 documents that are currently > > referenced, some of those references don't look right to me, for exampl= e: > > > > - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. > > - T11: The FC-FS reference is out of date. > > > > As part of dealing with the comments from IETF Last Call, we'll need > > to go over all of the T10 and T11 references (and should check all the > > references) to make sure that they're up to date and necessary. > > > > Thanks, > > --David > > ---------------------------------------------------- > > David L. Black, Distinguished Engineer EMC Corporation, 176 South St., > > Hopkinton, MA 01748 > > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > > david.black@emc.com Mobile: +1 (978) 394-7754 > > ---------------------------------------------------- > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > > > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > > > > > From julian@satran.net Tue Sep 25 00:41:59 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5CB4321F8804 for ; Tue, 25 Sep 2012 00:41:59 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.203 X-Spam-Level: X-Spam-Status: No, score=-5.203 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_QP_LONG_LINE=1.396, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nsFrG+2RA4j4 for ; Tue, 25 Sep 2012 00:41:58 -0700 (PDT) Received: from eu1sys200aog119.obsmtp.com (eu1sys200aog119.obsmtp.com [207.126.144.147]) by ietfa.amsl.com (Postfix) with SMTP id 2936F21F84AF for ; Tue, 25 Sep 2012 00:41:56 -0700 (PDT) Received: from mail-ee0-f44.google.com ([74.125.83.44]) (using TLSv1) by eu1sys200aob119.postini.com ([207.126.147.11]) with SMTP ID DSNKUGFgQAKYnY7BtQxWixtNuGdgBxRRfor+@postini.com; Tue, 25 Sep 2012 07:41:57 UTC Received: by eekd4 with SMTP id d4so740793eek.31 for ; Tue, 25 Sep 2012 00:41:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=content-type:content-transfer-encoding:references:in-reply-to :mime-version:message-id:cc:from:subject:date:to:x-mailer :x-gm-message-state; bh=ka7yQ+OBuVWRELV2c/MQoJ5WuZU+BcdTPtNKIQbvKJg=; b=R6N7QUQLPgnDdg+p4C3nI0P1ElwyQJXBHzOfN3Jkkzzo7X+NBRgzUrbavgwzXMrMlD AO3l+DswBLd4UvEW1T8yMg9waugb/ydvoPQ1lKu8xWRSEc/MhKKHyvved5tp2xW2tGHZ v0E4zPZ3OhlH0XZe9PBi346DaARMq/waZ1hRhvF5e0uFvsaNQo8TY9hHAiQwYETQJLko Xln/yo8Ep2SsCsOdxScLjdqBFOxIpjgEtUr1ESEDvGO4eygib8hcQEC71j5c14icgDGo QFGKvSc1DfpYUoNCY7dBDYBhpm+BdInx9B7P3vb1AEqCVz7Kp4676qSkg48TdJNSLu/O ClFg== Received: by 10.14.179.136 with SMTP id h8mr17419707eem.6.1348558912359; Tue, 25 Sep 2012 00:41:52 -0700 (PDT) Received: from [10.59.1.47] (host152-152-static.48-79-b.business.telecomitalia.it. [79.48.152.152]) by mx.google.com with ESMTPS id u47sm32066801eep.1.2012.09.25.00.41.48 (version=SSLv3 cipher=OTHER); Tue, 25 Sep 2012 00:41:51 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> Mime-Version: 1.0 (1.0) Message-Id: From: Julian Satran Date: Tue, 25 Sep 2012 06:55:03 +0200 To: "Black, David" X-Mailer: iPad Mail (10A403) X-Gm-Message-State: ALoCoQnW19q9ZIsQVIv2dxDcO+jD8S9v4u6nBipvvztV0gayqAAtL+QlnaZ+LDJdyppc26qjZUqf Cc: "storm@ietf.org" Subject: Re: [storm] RESERVE state - clearing effects X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 07:41:59 -0000 Did you consideir clusters or migrating vms? With those usually one expects t= argets to retain some state accross IT nexus loss. To avoid third party rese= rvation cleanup SCSI (t10) may want to introduce a timeout mechanism associa= ted with nexus loss (state is preserved for a while after nexus loss and the= timeout values can be set). julo Sent from my iPad On 24 =D7=91=D7=A1=D7=A4=D7=98 2012, at 18:42, "Black, David" wrote: > Fred, >=20 > This is interesting ... >=20 >> Here are some key points (from SAM 6.3.4 I_T nexus loss): >>=20 >> 1) I_T nexus loss clears all ACA conditions; >> 2) established a UA - one of the following is suggested: >>=20 >> "If the logical unit retains state information for the I_T nexus that >> is lost, its response to the subsequent I_T nexus >> re-establishment for the logical unit should include establishing a >> unit attention with an additional sense code set >> to I_T NEXUS LOSS OCCURRED. >>=20 >> If the logical unit does not retain state information for the I_T nexus >> that is lost, it shall consider the subsequent >> I_T nexus re-establishment, if any, as the formation of a new I_T nexus >> for which there is no past history (e.g., >> establish a unit attention with an additional sense code set to POWER ON >> OCCURRED)." >=20 > That "state information" has to identify the endpoints of the I_T Nexus > - for iSCSI that means at least the ISID and TPGT. If that's gone, I > think the RESERVE state is gone as well, in contrast to PERSISTENT > RESERVE state. On RESERVE state, SPC-2 has this to say in 5.5.2 "The > Reserve/Release management method": >=20 > The reserve/release management method commands, RESERVE(6), > RESERVE(10), RELEASE(6), and RELEASE(10) are used among multiple > initiators that do not require operations to be protected across > initiator failures (and subsequent hard resets). >=20 > I think it's fair to regard I_T Nexus loss as an initiator failure. I > also don't see any good way to do a 3rd-party release on a RESERVE > reservation that persists after I_T Nexus loss. I also believe that > initiator software that uses Reserve/Release does not expect > reservations to persist across I_T Nexus loss. >=20 > As to what needs to be stated in the iSCSI consolidated draft - the SPC > reference has been updated to SPC-3 in which the Reserve/Release > management method is obsolete, so there isn't any absolute requirement > to cover that. OTOH, it might be reasonable to add a sentence to say > that Reserve/Release state is implicit (e.g., as there's no way for an > initiator to figure out which I_T Nexus holds a reservation), and that > treating Reserve/Release state as indefinitely persistent after I_T > Nexus Loss is a bad idea ("should not", but lower case) as it may > require the initiator to reset the target device in order to clear out > that state (issuing RELEASE commands isn't enough). This sort of text > would require a reference to SPC-2. >=20 > Thanks, > --David >=20 >> -----Original Message----- >> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] >> Sent: Monday, September 24, 2012 8:58 AM >> To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org >> Subject: RE: iSCSI drafts - T10 and T11 references >>=20 >> FCP is also not explicit (it defers to the SCSI specs - which do not say)= . >> The closest statements I can find still requires extrapolation: >>=20 >> Here are some key points (from SAM 6.3.4 I_T nexus loss): >>=20 >> 1) I_T nexus loss clears all ACA conditions; >> 2) established a UA - one of the following is suggested: >>=20 >> "If the logical unit retains state information for the I_T nexus that is l= ost, >> its response to the subsequent I_T nexus >> re-establishment for the logical unit should include establishing a unit >> attention with an additional sense code set >> to I_T NEXUS LOSS OCCURRED. >>=20 >> If the logical unit does not retain state information for the I_T nexus t= hat >> is lost, it shall consider the subsequent >> I_T nexus re-establishment, if any, as the formation of a new I_T nexus f= or >> which there is no past history (e.g., >> establish a unit attention with an additional sense code set to POWER ON >> OCCURRED)." >>=20 >> But, there is no definition of what it means to "retain state". The >> conclusion I could jump to based on #1 above, is that RESERVE state is al= so >> cleared (this action lines up more with LU RESET than CLEAR TASK SET); an= d >> based on the POWER ON CONDITION UA of 29/01 (or 29/00 - both are power on= >> conditions) I could also assume that the RESERVE state is cleared; but ba= sed >> on the UA of 29/07 (retain state), I could jump to the conclusion that RE= SERVE >> state is retained. >>=20 >> But these are all conclusions I've jump to with very limited basis in the= >> text. >>=20 >> Based on what RFC3720 says about explicit state (e.g., mode pages), I wou= ld >> assume that hosts would believe that every command they explicitly send (= like >> MODE SELECT or RESERVE) would have created explicit state and therefore t= he >> host would assume RESERVE state is retained. It doesn't make sense to ha= ve >> mode page values retained, but have RESERVE state lost. >>=20 >> Certain nexus relationships contain an explicit state (e.g., >> initiator-specific mode pages) that may need to be preserved by >> the device server [SAM2] in a logical unit through changes or >> failures in the iSCSI layer (e.g., session failures). >>=20 >> It will be interesting to hear what a real host assumes about this situat= ion. >>=20 >> Fred >>=20 >> -----Original Message----- >> From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] >> Sent: Sunday, September 23, 2012 6:35 PM >> To: Knight, Frederick; david.black@emc.com; storm@ietf.org >> Subject: RE: iSCSI drafts - T10 and T11 references >>=20 >> Hi Fred, >>=20 >> SPC-3 discusses clearing effects of I-T nexus loss in several places. >>=20 >> Section 5.8.2.9 for ALUA >> Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for background s= elf- >> test Section 5.6.1 on persistent reservations Section 5.13 on timestamps >> ...... >>=20 >> Thanks for pointing out the broken cross-reference - Appendix F instead o= f E. >> I have just fixed it - and reviewed all other Appendix references elsewhe= re in >> the draft. However, I do not see a circular reference problem per se - >> Appendix E is correctly referencing Section 6.3.5 for definition of terms= , >> e.g. session continuation. And Section 6.3.5.1 is correctly deferring to >> Appendix E when discussing clearing effects. Let me know if I'm overlooki= ng >> something here. >>=20 >> Now, the reserve/release management state, :) : I agree the formal I_T n= exus >> loss event didn't exist in SAM-2, however the notion of a new I_T nexus s= tate >> did co-exist with a session-oriented I_T nexus well before iSCSI - FCP pr= ocess >> login (PRLO) comes to mind here. So I'd be curious about traditional FCP >> handling of reserve/release management state. And I'm also checking inter= nally >> here at MS. =46rom what I know now though, it does *appear* that reserve/= release >> management state should not be cleared with I_T nexus loss. >>=20 >> Having said all that, note that reserve/release management state is squar= ely >> in the SCSI territory. iSCSI spec has so far by design deferred to SCSI >> standards in specifying clearing effects on SCSI objects - as Appendix E.= 2 >> makes it clear. What we do specify is the clearing effects of SCSI action= s, >> e.g. LU reset, on iSCSI objects. >>=20 >> Thanks. >>=20 >> Mallikarjun >>=20 >>=20 >>=20 >>=20 >>=20 >> -----Original Message----- >> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] >> Sent: Sunday, September 23, 2012 3:38 AM >> To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org >> Subject: RE: iSCSI drafts - T10 and T11 references >>=20 >> Interesting, I just came across a question in a related area (only for SA= M-2). >>=20 >> RFC3720 was written in between SAM-2 and SAM-3; some concepts were drawn f= rom >> SAM-2, and some from SAM-3. >>=20 >> We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus lo= ss >> concept does not exist in SAM-2. SAM-2 does include references to RESERV= E and >> RELEASE commands. SAM-3 includes I_T Nexus loss, but it has no reference= to >> RESERVE state. There are allusions in RFC3720 about the intersection of >> RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit. RFC37= 20 >> references other documents (SAM2, SAM3, SBC, SPC3), none of which address= the >> situation. >>=20 >> Consider text such as the following: >> 4.4.3.1. I_T Nexus State >>=20 >> Certain nexus relationships contain an explicit state (e.g., >> initiator-specific mode pages) that may need to be preserved by >> the device server [SAM2] in a logical unit through changes or >> failures in the iSCSI layer (e.g., session failures). In order for >> that state to be restored, the iSCSI initiator should reestablish >> its session (re-login) to the same Target Portal Group using the >> previous ISID. That is, it should perform session recovery as >> described in Chapter 6. This is because the SCSI initiator port >> identifier and the SCSI target port identifier (or relative target >> port) form the datum that the SCSI logical unit device server uses >> to identify the I_T nexus. >>=20 >> Is RESERVE state - explicit state? When an I_T nexus is reestablished, >> initiator-specific mode pages are preserved. That implies that an I_T ne= xus >> loss should be treated in the way that SAM-2 describes via CLEAR TASK SET= >> (where mode pages and RESERVE state are preserved). If a session >> reestablishment caused saved mode pages to be restored rather than the >> established initiator-specific mode pages to be restored, then it would b= e >> like a LU reset (where the RESERVE state is lost). Is RESERVE state expl= icit >> state that is to be preserved? >>=20 >> Appendix E covers clearing effects during various events, but it never >> mentions RESERVE state. >>=20 >> Section E.2 contains the following text: >>=20 >> The only iSCSI protocol action that can effect clearing actions on >> SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 >> Loss of Nexus notification). [SPC3] describes the clearing effects >> of this notification on a variety of SCSI attributes. In addition, >> SCSI standards documents (such as [SAM2] and [SBC]) define >> additional clearing actions that may take place for several SCSI >> objects on SCSI events such as LU resets and power-on resets. >>=20 >> I could not find any text in SPC3 that describes these clearing effects (= I >> don't know if this SPC3 reference is incorrect, but at a minimum, it is >> ineffective). The SAM2 text for LU resets and power-on resets is easily >> found, and clearly says that any RESERVE is released in those cases. As a= n >> aside, the reference to section 6.3.5.1 is a circular reference). The te= xt in >> E.2 goes on: >>=20 >> Since iSCSI defines a target cold reset as a protocol-equivalent >> to a target power-cycle, the iSCSI target cold reset must also be >> considered as the power-on reset event in interpreting the actions >> defined in the SCSI standards. >>=20 >> When the iSCSI session is reconstructed (between the same SCSI >> ports with the same nexus identifier) reestablishing the same I_T >> nexus, all SCSI objects that are defined to not clear on the "I_T >> nexus loss" notification event, such as persistent reservations, >> are automatically associated to this new session. >>=20 >> So, it is clear what happens to PERSISTENT RESERVE state, but not so clea= r >> about RESERVE state. It also states that all SCSI objects that are defin= ed to >> not clear on the I_T nexus loss are retained. But again, the only list I= can >> find of what is cleared is here in this Appendix, since SAM-2, SAM-3, SAM= -4 >> are all silent on the interaction between RESERVE and I_T Nexus loss. >>=20 >> T10 has never had to deal with this because they removed RESERVE before t= hey >> added I_T Nexus loss. However, iSCSI did not do that; we discuss BOTH, b= ut we >> fail to describe how they interact. iSCSI also has the feature of >> reestablishment of a session that does not exist in SAM-2 (SAM-2 was stil= l >> parallel SCSI focused, where connections didn't exist). So again, if ses= sion >> reestablishment is examined in the light of parallel SCSI, the intent wou= ld be >> to retain the RESERVE state. Here is another section on the topic: >>=20 >> 6.3.5.1. Loss of Nexus Notification >>=20 >> The iSCSI layer provides the SCSI layer with the "I_T nexus loss" >> notification when any one of the following events happens: >>=20 >> Successful completion of session reinstatement. >> Session closure event. >> Session timeout event. >>=20 >> Certain SCSI object clearing actions may result due to the >> notification in the SCSI end nodes, as documented in Appendix F. - >> Clearing Effects of Various Events on Targets. >>=20 >> FWIW - the reference above is incorrect - there is no Appendix F in the >> consolidated draft - this should be Appendix E (and also, BTW - this is a= >> circular reference). >>=20 >> So anyway, it seems like the intent was that session reinstatement would >> preserve the RESERVE state across the I_T Nexus loss event. >>=20 >> So aside from the incorrect reference (and circular nature of it), any >> thoughts on whether we should get explicit about this situation? Should w= e >> remain silent; should we tweak Appendix E along these lines (or other lin= es >> that others might suggest): >>=20 >> 4.4.3.1. I_T Nexus State >>=20 >> Certain nexus relationships contain an explicit state (e.g., >> initiator-specific mode pages, and RESERVE state) that may need to be >> preserved by >> the device server [SAM2] in a logical unit through changes or >> failures in the iSCSI layer (e.g., session failures).... >>=20 >> Fred Knight >>=20 >>=20 >>=20 >> -----Original Message----- >> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of= >> Mallikarjun Chadalapaka >> Sent: Sunday, September 23, 2012 1:34 AM >> To: david.black@emc.com; storm@ietf.org >> Subject: Re: [storm] iSCSI drafts - T10 and T11 references >>=20 >> Hi David, >>=20 >> I have noticed this comment as I am working through all Last Call comment= s. >>=20 >> The references to SAM-3 and SAM-4 are keyed to this paragraph in Section 1= 3.9: >>=20 >> "[SAM2] and [SAM3] specifications note in their informative text that TPG= T >> value should be non-zero, note that it is incorrect. A zero value is all= owed >> as a legal value for TPGT. This discrepancy currently stands corrected in= >> [SAM4]." >>=20 >> Additionally, SAM4 reference is also used in citing the new ASC/ASCQ code= s >> that it defines, in Section 11.14.5 of the draft. >>=20 >> So, I believe references to all three SAM specs would continue to be >> appropriate. >>=20 >> The current draft references FC-FS in discussing the NAA format in a few >> places, so the current reference at the end is intentionally the same. Is= the >> older reference a problem? If so, I suppose we can replace all references= to >> FC-FS-4, but that seems to be still in early stages(0.1 spec posted on Ap= ril >> 2012), so perhaps we should use FC-FS3? >>=20 >> Thanks. >>=20 >> Mallikarjun >>=20 >>=20 >>=20 >>=20 >> -----Original Message----- >> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of= >> david.black@emc.com >> Sent: Thursday, July 26, 2012 6:47 AM >> To: storm@ietf.org >> Subject: [storm] iSCSI drafts - T10 and T11 references >>=20 >> In making up the lists of T10 and T11 documents that are currently refere= nced, >> some of those references don't look right to me, for example: >>=20 >> - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. >> - T11: The FC-FS reference is out of date. >>=20 >> As part of dealing with the comments from IETF Last Call, we'll need to g= o >> over all of the T10 and T11 references (and should check all the referenc= es) >> to make sure that they're up to date and necessary. >>=20 >> Thanks, >> --David >> ---------------------------------------------------- >> David L. Black, Distinguished Engineer >> EMC Corporation, 176 South St., Hopkinton, MA 01748 >> +1 (508) 293-7953 FAX: +1 (508) 293-7786 >> david.black@emc.com Mobile: +1 (978) 394-7754 >> ---------------------------------------------------- >>=20 >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm >>=20 >>=20 >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm >>=20 >>=20 >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From roweber@ieee.org Tue Sep 25 05:32:51 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 986FC21F88CE for ; Tue, 25 Sep 2012 05:32:51 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CFWfckhXfQq7 for ; Tue, 25 Sep 2012 05:32:49 -0700 (PDT) Received: from vms173013pub.verizon.net (vms173013pub.verizon.net [206.46.173.13]) by ietfa.amsl.com (Postfix) with ESMTP id 6384F21F886F for ; Tue, 25 Sep 2012 05:32:48 -0700 (PDT) MIME-version: 1.0 Content-transfer-encoding: 8BIT Content-type: text/plain; charset=UTF-8; format=flowed Received: from [192.168.1.3] ([unknown] [96.226.101.225]) by vms173013.mailsrvcs.net (Sun Java(tm) System Messaging Server 7u2-7.02 32bit (built Apr 16 2009)) with ESMTPA id <0MAW00JDRO607P90@vms173013.mailsrvcs.net> for storm@ietf.org; Tue, 25 Sep 2012 07:32:25 -0500 (CDT) Message-id: <5061A45E.7010607@ieee.org> Date: Tue, 25 Sep 2012 07:32:30 -0500 From: Ralph Weber User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120907 Thunderbird/15.0.1 To: storm@ietf.org References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> In-reply-to: Subject: Re: [storm] RESERVE state - clearing effects X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 12:32:51 -0000 Persistent Reservations were invented because Reserve/Release failed to provide the quality of reservations services demanded by clusters. Clusters that depend on Reserve/Release reservations get exactly what they deserve. As for Open VMS, my recollection is that it had only one minor dependence on reservations. The most I remember needing from reservations was a way to ensure that disks stopped processing I/Os from an expelled cluster member. As long as all commands are aborted at the same time a reservation is released, the VMS I remember will not suffer any adverse effects. In any case, VMS has had two decades to migrate to Persistent Reservations for whatever they needed or developed a dependence upon. Even for OS engineers, that should be long enough for them to have matched their software to the available services. All the best, .Ralph P.S. David and Fred, I am not venturing into the topics you are discussing because I see nothing upon which I can provide knowledgeable additional comment. Whatever agreement you all reach will be okay ... as far as I can tell. On 9/24/2012 11:55 PM, Julian Satran wrote: > Did you consideir clusters or migrating vms? With those usually one expects targets to retain some state accross IT nexus loss. To avoid third party reservation cleanup SCSI (t10) may want to introduce a timeout mechanism associated with nexus loss (state is preserved for a while after nexus loss and the timeout values can be set). > > julo > > Sent from my iPad > > On 24 בספט 2012, at 18:42, "Black, David" wrote: > >> Fred, >> >> This is interesting ... >> >>> Here are some key points (from SAM 6.3.4 I_T nexus loss): >>> >>> 1) I_T nexus loss clears all ACA conditions; >>> 2) established a UA - one of the following is suggested: >>> >>> "If the logical unit retains state information for the I_T nexus that >>> is lost, its response to the subsequent I_T nexus >>> re-establishment for the logical unit should include establishing a >>> unit attention with an additional sense code set >>> to I_T NEXUS LOSS OCCURRED. >>> >>> If the logical unit does not retain state information for the I_T nexus >>> that is lost, it shall consider the subsequent >>> I_T nexus re-establishment, if any, as the formation of a new I_T nexus >>> for which there is no past history (e.g., >>> establish a unit attention with an additional sense code set to POWER ON >>> OCCURRED)." >> That "state information" has to identify the endpoints of the I_T Nexus >> - for iSCSI that means at least the ISID and TPGT. If that's gone, I >> think the RESERVE state is gone as well, in contrast to PERSISTENT >> RESERVE state. On RESERVE state, SPC-2 has this to say in 5.5.2 "The >> Reserve/Release management method": >> >> The reserve/release management method commands, RESERVE(6), >> RESERVE(10), RELEASE(6), and RELEASE(10) are used among multiple >> initiators that do not require operations to be protected across >> initiator failures (and subsequent hard resets). >> >> I think it's fair to regard I_T Nexus loss as an initiator failure. I >> also don't see any good way to do a 3rd-party release on a RESERVE >> reservation that persists after I_T Nexus loss. I also believe that >> initiator software that uses Reserve/Release does not expect >> reservations to persist across I_T Nexus loss. >> >> As to what needs to be stated in the iSCSI consolidated draft - the SPC >> reference has been updated to SPC-3 in which the Reserve/Release >> management method is obsolete, so there isn't any absolute requirement >> to cover that. OTOH, it might be reasonable to add a sentence to say >> that Reserve/Release state is implicit (e.g., as there's no way for an >> initiator to figure out which I_T Nexus holds a reservation), and that >> treating Reserve/Release state as indefinitely persistent after I_T >> Nexus Loss is a bad idea ("should not", but lower case) as it may >> require the initiator to reset the target device in order to clear out >> that state (issuing RELEASE commands isn't enough). This sort of text >> would require a reference to SPC-2. >> >> Thanks, >> --David >> >>> -----Original Message----- >>> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] >>> Sent: Monday, September 24, 2012 8:58 AM >>> To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org >>> Subject: RE: iSCSI drafts - T10 and T11 references >>> >>> FCP is also not explicit (it defers to the SCSI specs - which do not say). >>> The closest statements I can find still requires extrapolation: >>> >>> Here are some key points (from SAM 6.3.4 I_T nexus loss): >>> >>> 1) I_T nexus loss clears all ACA conditions; >>> 2) established a UA - one of the following is suggested: >>> >>> "If the logical unit retains state information for the I_T nexus that is lost, >>> its response to the subsequent I_T nexus >>> re-establishment for the logical unit should include establishing a unit >>> attention with an additional sense code set >>> to I_T NEXUS LOSS OCCURRED. >>> >>> If the logical unit does not retain state information for the I_T nexus that >>> is lost, it shall consider the subsequent >>> I_T nexus re-establishment, if any, as the formation of a new I_T nexus for >>> which there is no past history (e.g., >>> establish a unit attention with an additional sense code set to POWER ON >>> OCCURRED)." >>> >>> But, there is no definition of what it means to "retain state". The >>> conclusion I could jump to based on #1 above, is that RESERVE state is also >>> cleared (this action lines up more with LU RESET than CLEAR TASK SET); and >>> based on the POWER ON CONDITION UA of 29/01 (or 29/00 - both are power on >>> conditions) I could also assume that the RESERVE state is cleared; but based >>> on the UA of 29/07 (retain state), I could jump to the conclusion that RESERVE >>> state is retained. >>> >>> But these are all conclusions I've jump to with very limited basis in the >>> text. >>> >>> Based on what RFC3720 says about explicit state (e.g., mode pages), I would >>> assume that hosts would believe that every command they explicitly send (like >>> MODE SELECT or RESERVE) would have created explicit state and therefore the >>> host would assume RESERVE state is retained. It doesn't make sense to have >>> mode page values retained, but have RESERVE state lost. >>> >>> Certain nexus relationships contain an explicit state (e.g., >>> initiator-specific mode pages) that may need to be preserved by >>> the device server [SAM2] in a logical unit through changes or >>> failures in the iSCSI layer (e.g., session failures). >>> >>> It will be interesting to hear what a real host assumes about this situation. >>> >>> Fred >>> >>> -----Original Message----- >>> From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] >>> Sent: Sunday, September 23, 2012 6:35 PM >>> To: Knight, Frederick; david.black@emc.com; storm@ietf.org >>> Subject: RE: iSCSI drafts - T10 and T11 references >>> >>> Hi Fred, >>> >>> SPC-3 discusses clearing effects of I-T nexus loss in several places. >>> >>> Section 5.8.2.9 for ALUA >>> Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for background self- >>> test Section 5.6.1 on persistent reservations Section 5.13 on timestamps >>> ...... >>> >>> Thanks for pointing out the broken cross-reference - Appendix F instead of E. >>> I have just fixed it - and reviewed all other Appendix references elsewhere in >>> the draft. However, I do not see a circular reference problem per se - >>> Appendix E is correctly referencing Section 6.3.5 for definition of terms, >>> e.g. session continuation. And Section 6.3.5.1 is correctly deferring to >>> Appendix E when discussing clearing effects. Let me know if I'm overlooking >>> something here. >>> >>> Now, the reserve/release management state, :) : I agree the formal I_T nexus >>> loss event didn't exist in SAM-2, however the notion of a new I_T nexus state >>> did co-exist with a session-oriented I_T nexus well before iSCSI - FCP process >>> login (PRLO) comes to mind here. So I'd be curious about traditional FCP >>> handling of reserve/release management state. And I'm also checking internally >>> here at MS. From what I know now though, it does *appear* that reserve/release >>> management state should not be cleared with I_T nexus loss. >>> >>> Having said all that, note that reserve/release management state is squarely >>> in the SCSI territory. iSCSI spec has so far by design deferred to SCSI >>> standards in specifying clearing effects on SCSI objects - as Appendix E.2 >>> makes it clear. What we do specify is the clearing effects of SCSI actions, >>> e.g. LU reset, on iSCSI objects. >>> >>> Thanks. >>> >>> Mallikarjun >>> >>> >>> >>> >>> >>> -----Original Message----- >>> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] >>> Sent: Sunday, September 23, 2012 3:38 AM >>> To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org >>> Subject: RE: iSCSI drafts - T10 and T11 references >>> >>> Interesting, I just came across a question in a related area (only for SAM-2). >>> >>> RFC3720 was written in between SAM-2 and SAM-3; some concepts were drawn from >>> SAM-2, and some from SAM-3. >>> >>> We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus loss >>> concept does not exist in SAM-2. SAM-2 does include references to RESERVE and >>> RELEASE commands. SAM-3 includes I_T Nexus loss, but it has no reference to >>> RESERVE state. There are allusions in RFC3720 about the intersection of >>> RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit. RFC3720 >>> references other documents (SAM2, SAM3, SBC, SPC3), none of which address the >>> situation. >>> >>> Consider text such as the following: >>> 4.4.3.1. I_T Nexus State >>> >>> Certain nexus relationships contain an explicit state (e.g., >>> initiator-specific mode pages) that may need to be preserved by >>> the device server [SAM2] in a logical unit through changes or >>> failures in the iSCSI layer (e.g., session failures). In order for >>> that state to be restored, the iSCSI initiator should reestablish >>> its session (re-login) to the same Target Portal Group using the >>> previous ISID. That is, it should perform session recovery as >>> described in Chapter 6. This is because the SCSI initiator port >>> identifier and the SCSI target port identifier (or relative target >>> port) form the datum that the SCSI logical unit device server uses >>> to identify the I_T nexus. >>> >>> Is RESERVE state - explicit state? When an I_T nexus is reestablished, >>> initiator-specific mode pages are preserved. That implies that an I_T nexus >>> loss should be treated in the way that SAM-2 describes via CLEAR TASK SET >>> (where mode pages and RESERVE state are preserved). If a session >>> reestablishment caused saved mode pages to be restored rather than the >>> established initiator-specific mode pages to be restored, then it would be >>> like a LU reset (where the RESERVE state is lost). Is RESERVE state explicit >>> state that is to be preserved? >>> >>> Appendix E covers clearing effects during various events, but it never >>> mentions RESERVE state. >>> >>> Section E.2 contains the following text: >>> >>> The only iSCSI protocol action that can effect clearing actions on >>> SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 >>> Loss of Nexus notification). [SPC3] describes the clearing effects >>> of this notification on a variety of SCSI attributes. In addition, >>> SCSI standards documents (such as [SAM2] and [SBC]) define >>> additional clearing actions that may take place for several SCSI >>> objects on SCSI events such as LU resets and power-on resets. >>> >>> I could not find any text in SPC3 that describes these clearing effects (I >>> don't know if this SPC3 reference is incorrect, but at a minimum, it is >>> ineffective). The SAM2 text for LU resets and power-on resets is easily >>> found, and clearly says that any RESERVE is released in those cases. As an >>> aside, the reference to section 6.3.5.1 is a circular reference). The text in >>> E.2 goes on: >>> >>> Since iSCSI defines a target cold reset as a protocol-equivalent >>> to a target power-cycle, the iSCSI target cold reset must also be >>> considered as the power-on reset event in interpreting the actions >>> defined in the SCSI standards. >>> >>> When the iSCSI session is reconstructed (between the same SCSI >>> ports with the same nexus identifier) reestablishing the same I_T >>> nexus, all SCSI objects that are defined to not clear on the "I_T >>> nexus loss" notification event, such as persistent reservations, >>> are automatically associated to this new session. >>> >>> So, it is clear what happens to PERSISTENT RESERVE state, but not so clear >>> about RESERVE state. It also states that all SCSI objects that are defined to >>> not clear on the I_T nexus loss are retained. But again, the only list I can >>> find of what is cleared is here in this Appendix, since SAM-2, SAM-3, SAM-4 >>> are all silent on the interaction between RESERVE and I_T Nexus loss. >>> >>> T10 has never had to deal with this because they removed RESERVE before they >>> added I_T Nexus loss. However, iSCSI did not do that; we discuss BOTH, but we >>> fail to describe how they interact. iSCSI also has the feature of >>> reestablishment of a session that does not exist in SAM-2 (SAM-2 was still >>> parallel SCSI focused, where connections didn't exist). So again, if session >>> reestablishment is examined in the light of parallel SCSI, the intent would be >>> to retain the RESERVE state. Here is another section on the topic: >>> >>> 6.3.5.1. Loss of Nexus Notification >>> >>> The iSCSI layer provides the SCSI layer with the "I_T nexus loss" >>> notification when any one of the following events happens: >>> >>> Successful completion of session reinstatement. >>> Session closure event. >>> Session timeout event. >>> >>> Certain SCSI object clearing actions may result due to the >>> notification in the SCSI end nodes, as documented in Appendix F. - >>> Clearing Effects of Various Events on Targets. >>> >>> FWIW - the reference above is incorrect - there is no Appendix F in the >>> consolidated draft - this should be Appendix E (and also, BTW - this is a >>> circular reference). >>> >>> So anyway, it seems like the intent was that session reinstatement would >>> preserve the RESERVE state across the I_T Nexus loss event. >>> >>> So aside from the incorrect reference (and circular nature of it), any >>> thoughts on whether we should get explicit about this situation? Should we >>> remain silent; should we tweak Appendix E along these lines (or other lines >>> that others might suggest): >>> >>> 4.4.3.1. I_T Nexus State >>> >>> Certain nexus relationships contain an explicit state (e.g., >>> initiator-specific mode pages, and RESERVE state) that may need to be >>> preserved by >>> the device server [SAM2] in a logical unit through changes or >>> failures in the iSCSI layer (e.g., session failures).... >>> >>> Fred Knight >>> >>> >>> >>> -----Original Message----- >>> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of >>> Mallikarjun Chadalapaka >>> Sent: Sunday, September 23, 2012 1:34 AM >>> To: david.black@emc.com; storm@ietf.org >>> Subject: Re: [storm] iSCSI drafts - T10 and T11 references >>> >>> Hi David, >>> >>> I have noticed this comment as I am working through all Last Call comments. >>> >>> The references to SAM-3 and SAM-4 are keyed to this paragraph in Section 13.9: >>> >>> "[SAM2] and [SAM3] specifications note in their informative text that TPGT >>> value should be non-zero, note that it is incorrect. A zero value is allowed >>> as a legal value for TPGT. This discrepancy currently stands corrected in >>> [SAM4]." >>> >>> Additionally, SAM4 reference is also used in citing the new ASC/ASCQ codes >>> that it defines, in Section 11.14.5 of the draft. >>> >>> So, I believe references to all three SAM specs would continue to be >>> appropriate. >>> >>> The current draft references FC-FS in discussing the NAA format in a few >>> places, so the current reference at the end is intentionally the same. Is the >>> older reference a problem? If so, I suppose we can replace all references to >>> FC-FS-4, but that seems to be still in early stages(0.1 spec posted on April >>> 2012), so perhaps we should use FC-FS3? >>> >>> Thanks. >>> >>> Mallikarjun >>> >>> >>> >>> >>> -----Original Message----- >>> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of >>> david.black@emc.com >>> Sent: Thursday, July 26, 2012 6:47 AM >>> To: storm@ietf.org >>> Subject: [storm] iSCSI drafts - T10 and T11 references >>> >>> In making up the lists of T10 and T11 documents that are currently referenced, >>> some of those references don't look right to me, for example: >>> >>> - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. >>> - T11: The FC-FS reference is out of date. >>> >>> As part of dealing with the comments from IETF Last Call, we'll need to go >>> over all of the T10 and T11 references (and should check all the references) >>> to make sure that they're up to date and necessary. >>> >>> Thanks, >>> --David >>> ---------------------------------------------------- >>> David L. Black, Distinguished Engineer >>> EMC Corporation, 176 South St., Hopkinton, MA 01748 >>> +1 (508) 293-7953 FAX: +1 (508) 293-7786 >>> david.black@emc.com Mobile: +1 (978) 394-7754 >>> ---------------------------------------------------- >>> >>> _______________________________________________ >>> storm mailing list >>> storm@ietf.org >>> https://www.ietf.org/mailman/listinfo/storm >>> >>> >>> _______________________________________________ >>> storm mailing list >>> storm@ietf.org >>> https://www.ietf.org/mailman/listinfo/storm >>> >>> >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From julian@satran.net Tue Sep 25 05:51:01 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3BBAA21F8668 for ; Tue, 25 Sep 2012 05:51:01 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.203 X-Spam-Level: X-Spam-Status: No, score=-5.203 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_QP_LONG_LINE=1.396, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eIctfmmQ1WGf for ; Tue, 25 Sep 2012 05:50:59 -0700 (PDT) Received: from eu1sys200aog106.obsmtp.com (eu1sys200aog106.obsmtp.com [207.126.144.121]) by ietfa.amsl.com (Postfix) with SMTP id 3B18821F84D3 for ; Tue, 25 Sep 2012 05:50:58 -0700 (PDT) Received: from mail-ee0-f44.google.com ([74.125.83.44]) (using TLSv1) by eu1sys200aob106.postini.com ([207.126.147.11]) with SMTP ID DSNKUGGorVZ0xnHUUaOj7qItgB730ZtSqe+u@postini.com; Tue, 25 Sep 2012 12:50:58 UTC Received: by eekd4 with SMTP id d4so1018045eek.31 for ; Tue, 25 Sep 2012 05:50:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=references:in-reply-to:mime-version:content-transfer-encoding :content-type:message-id:cc:x-mailer:from:subject:date:to :x-gm-message-state; bh=pANUMtNuiKWi+O++qMWe5HnXj+kUH2BBlbKQdlBZrOo=; b=C1Kb+h1757HgderhAngrhomX+KfhRQV8QMLwZpWKyM8Im2HUfofk7A68O9eO//UzA0 IjLmRsO1nyTZaj6EBi5BfSiqZZTfFe63xgZwlnP2wjK8rFJEmsTtfUljasP+QgsjT4Os XIDW38l6xCyRUhzx864vfY/LcGdEjLpOWRka0/scQkb4HriksTjJ+EAXe8tjm7CIVmB7 wusfCrKcVd9bDqZRD8dw9zP/VERMT2S8x6BxWqlloohQcbj5xk1l6rZHoRV4qx5AskP+ dMWZ+h3qddz4hkSHKX42VirARtGbdB+8NZSrOKzZkZGK046zx0m+4sa77R9EaLYtgRQc sLbg== Received: by 10.14.224.135 with SMTP id x7mr9071759eep.34.1348577453794; Tue, 25 Sep 2012 05:50:53 -0700 (PDT) Received: from [10.59.1.47] (host152-152-static.48-79-b.business.telecomitalia.it. [79.48.152.152]) by mx.google.com with ESMTPS id z25sm260188bkz.15.2012.09.25.05.50.50 (version=SSLv3 cipher=OTHER); Tue, 25 Sep 2012 05:50:52 -0700 (PDT) References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> <5061A45E.7010607@ieee.org> In-Reply-To: <5061A45E.7010607@ieee.org> Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Message-Id: X-Mailer: iPad Mail (10A403) From: Julian Satran Date: Tue, 25 Sep 2012 14:50:58 +0200 To: Ralph Weber X-Gm-Message-State: ALoCoQnaT1cmNqPCETZ/C3YnDN06HJu9UQODV0uTMAtXccKfYISLOQ1digM7nHJYWWRQNKlrDxht Cc: "storm@ietf.org" Subject: Re: [storm] RESERVE state - clearing effects X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 12:51:01 -0000 Ralph, I understand. I missed the fact that the discussion was exclusively abour re= serve/release. And i was referring to VMs (even I am too young to remeber VM= S - that was during the trojan war wasn't it?). Julo Sent from my iPad On 25 =D7=91=D7=A1=D7=A4=D7=98 2012, at 14:32, Ralph Weber wrote: > Persistent Reservations were invented because Reserve/Release failed > to provide the quality of reservations services demanded by clusters. > Clusters that depend on Reserve/Release reservations get exactly > what they deserve. >=20 > As for Open VMS, my recollection is that it had only one minor > dependence on reservations. The most I remember needing from > reservations was a way to ensure that disks stopped processing > I/Os from an expelled cluster member. As long as all commands are > aborted at the same time a reservation is released, the VMS I > remember will not suffer any adverse effects. >=20 > In any case, VMS has had two decades to migrate to Persistent > Reservations for whatever they needed or developed a dependence > upon. Even for OS engineers, that should be long enough for > them to have matched their software to the available services. >=20 > All the best, >=20 > .Ralph >=20 > P.S. David and Fred, I am not venturing into the topics you are > discussing because I see nothing upon which I can provide knowledgeable > additional comment. Whatever agreement you all reach will be okay ... > as far as I can tell. >=20 > On 9/24/2012 11:55 PM, Julian Satran wrote: >> Did you consideir clusters or migrating vms? With those usually one expec= ts targets to retain some state accross IT nexus loss. To avoid third party r= eservation cleanup SCSI (t10) may want to introduce a timeout mechanism asso= ciated with nexus loss (state is preserved for a while after nexus loss and t= he timeout values can be set). >>=20 >> julo >>=20 >> Sent from my iPad >>=20 >> On 24 =D7=91=D7=A1=D7=A4=D7=98 2012, at 18:42, "Black, David" wrote: >>=20 >>> Fred, >>>=20 >>> This is interesting ... >>>=20 >>>> Here are some key points (from SAM 6.3.4 I_T nexus loss): >>>>=20 >>>> 1) I_T nexus loss clears all ACA conditions; >>>> 2) established a UA - one of the following is suggested: >>>>=20 >>>> "If the logical unit retains state information for the I_T nexus that >>>> is lost, its response to the subsequent I_T nexus >>>> re-establishment for the logical unit should include establishing a >>>> unit attention with an additional sense code set >>>> to I_T NEXUS LOSS OCCURRED. >>>>=20 >>>> If the logical unit does not retain state information for the I_T nexus= >>>> that is lost, it shall consider the subsequent >>>> I_T nexus re-establishment, if any, as the formation of a new I_T nexus= >>>> for which there is no past history (e.g., >>>> establish a unit attention with an additional sense code set to POWER O= N >>>> OCCURRED)." >>> That "state information" has to identify the endpoints of the I_T Nexus >>> - for iSCSI that means at least the ISID and TPGT. If that's gone, I >>> think the RESERVE state is gone as well, in contrast to PERSISTENT >>> RESERVE state. On RESERVE state, SPC-2 has this to say in 5.5.2 "The >>> Reserve/Release management method": >>>=20 >>> The reserve/release management method commands, RESERVE(6), >>> RESERVE(10), RELEASE(6), and RELEASE(10) are used among multiple >>> initiators that do not require operations to be protected across >>> initiator failures (and subsequent hard resets). >>>=20 >>> I think it's fair to regard I_T Nexus loss as an initiator failure. I >>> also don't see any good way to do a 3rd-party release on a RESERVE >>> reservation that persists after I_T Nexus loss. I also believe that >>> initiator software that uses Reserve/Release does not expect >>> reservations to persist across I_T Nexus loss. >>>=20 >>> As to what needs to be stated in the iSCSI consolidated draft - the SPC >>> reference has been updated to SPC-3 in which the Reserve/Release >>> management method is obsolete, so there isn't any absolute requirement >>> to cover that. OTOH, it might be reasonable to add a sentence to say >>> that Reserve/Release state is implicit (e.g., as there's no way for an >>> initiator to figure out which I_T Nexus holds a reservation), and that >>> treating Reserve/Release state as indefinitely persistent after I_T >>> Nexus Loss is a bad idea ("should not", but lower case) as it may >>> require the initiator to reset the target device in order to clear out >>> that state (issuing RELEASE commands isn't enough). This sort of text >>> would require a reference to SPC-2. >>>=20 >>> Thanks, >>> --David >>>=20 >>>> -----Original Message----- >>>> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] >>>> Sent: Monday, September 24, 2012 8:58 AM >>>> To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org >>>> Subject: RE: iSCSI drafts - T10 and T11 references >>>>=20 >>>> FCP is also not explicit (it defers to the SCSI specs - which do not sa= y). >>>> The closest statements I can find still requires extrapolation: >>>>=20 >>>> Here are some key points (from SAM 6.3.4 I_T nexus loss): >>>>=20 >>>> 1) I_T nexus loss clears all ACA conditions; >>>> 2) established a UA - one of the following is suggested: >>>>=20 >>>> "If the logical unit retains state information for the I_T nexus that i= s lost, >>>> its response to the subsequent I_T nexus >>>> re-establishment for the logical unit should include establishing a uni= t >>>> attention with an additional sense code set >>>> to I_T NEXUS LOSS OCCURRED. >>>>=20 >>>> If the logical unit does not retain state information for the I_T nexus= that >>>> is lost, it shall consider the subsequent >>>> I_T nexus re-establishment, if any, as the formation of a new I_T nexus= for >>>> which there is no past history (e.g., >>>> establish a unit attention with an additional sense code set to POWER O= N >>>> OCCURRED)." >>>>=20 >>>> But, there is no definition of what it means to "retain state". The >>>> conclusion I could jump to based on #1 above, is that RESERVE state is a= lso >>>> cleared (this action lines up more with LU RESET than CLEAR TASK SET); a= nd >>>> based on the POWER ON CONDITION UA of 29/01 (or 29/00 - both are power o= n >>>> conditions) I could also assume that the RESERVE state is cleared; but b= ased >>>> on the UA of 29/07 (retain state), I could jump to the conclusion that R= ESERVE >>>> state is retained. >>>>=20 >>>> But these are all conclusions I've jump to with very limited basis in t= he >>>> text. >>>>=20 >>>> Based on what RFC3720 says about explicit state (e.g., mode pages), I w= ould >>>> assume that hosts would believe that every command they explicitly send= (like >>>> MODE SELECT or RESERVE) would have created explicit state and therefore= the >>>> host would assume RESERVE state is retained. It doesn't make sense to h= ave >>>> mode page values retained, but have RESERVE state lost. >>>>=20 >>>> Certain nexus relationships contain an explicit state (e.g., >>>> initiator-specific mode pages) that may need to be preserved by >>>> the device server [SAM2] in a logical unit through changes or >>>> failures in the iSCSI layer (e.g., session failures). >>>>=20 >>>> It will be interesting to hear what a real host assumes about this situ= ation. >>>>=20 >>>> Fred >>>>=20 >>>> -----Original Message----- >>>> From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] >>>> Sent: Sunday, September 23, 2012 6:35 PM >>>> To: Knight, Frederick; david.black@emc.com; storm@ietf.org >>>> Subject: RE: iSCSI drafts - T10 and T11 references >>>>=20 >>>> Hi Fred, >>>>=20 >>>> SPC-3 discusses clearing effects of I-T nexus loss in several places. >>>>=20 >>>> Section 5.8.2.9 for ALUA >>>> Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for background= self- >>>> test Section 5.6.1 on persistent reservations Section 5.13 on timestamp= s >>>> ...... >>>>=20 >>>> Thanks for pointing out the broken cross-reference - Appendix F instead= of E. >>>> I have just fixed it - and reviewed all other Appendix references elsew= here in >>>> the draft. However, I do not see a circular reference problem per se - >>>> Appendix E is correctly referencing Section 6.3.5 for definition of ter= ms, >>>> e.g. session continuation. And Section 6.3.5.1 is correctly deferring t= o >>>> Appendix E when discussing clearing effects. Let me know if I'm overloo= king >>>> something here. >>>>=20 >>>> Now, the reserve/release management state, :) : I agree the formal I_T= nexus >>>> loss event didn't exist in SAM-2, however the notion of a new I_T nexus= state >>>> did co-exist with a session-oriented I_T nexus well before iSCSI - FCP p= rocess >>>> login (PRLO) comes to mind here. So I'd be curious about traditional FC= P >>>> handling of reserve/release management state. And I'm also checking int= ernally >>>> here at MS. =46rom what I know now though, it does *appear* that reserv= e/release >>>> management state should not be cleared with I_T nexus loss. >>>>=20 >>>> Having said all that, note that reserve/release management state is squ= arely >>>> in the SCSI territory. iSCSI spec has so far by design deferred to SCSI= >>>> standards in specifying clearing effects on SCSI objects - as Appendix E= .2 >>>> makes it clear. What we do specify is the clearing effects of SCSI acti= ons, >>>> e.g. LU reset, on iSCSI objects. >>>>=20 >>>> Thanks. >>>>=20 >>>> Mallikarjun >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>> -----Original Message----- >>>> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] >>>> Sent: Sunday, September 23, 2012 3:38 AM >>>> To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org >>>> Subject: RE: iSCSI drafts - T10 and T11 references >>>>=20 >>>> Interesting, I just came across a question in a related area (only for S= AM-2). >>>>=20 >>>> RFC3720 was written in between SAM-2 and SAM-3; some concepts were draw= n from >>>> SAM-2, and some from SAM-3. >>>>=20 >>>> We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus l= oss >>>> concept does not exist in SAM-2. SAM-2 does include references to RESE= RVE and >>>> RELEASE commands. SAM-3 includes I_T Nexus loss, but it has no referen= ce to >>>> RESERVE state. There are allusions in RFC3720 about the intersection o= f >>>> RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit. RFC= 3720 >>>> references other documents (SAM2, SAM3, SBC, SPC3), none of which addre= ss the >>>> situation. >>>>=20 >>>> Consider text such as the following: >>>> 4.4.3.1. I_T Nexus State >>>>=20 >>>> Certain nexus relationships contain an explicit state (e.g., >>>> initiator-specific mode pages) that may need to be preserved by >>>> the device server [SAM2] in a logical unit through changes or >>>> failures in the iSCSI layer (e.g., session failures). In order for >>>> that state to be restored, the iSCSI initiator should reestablish >>>> its session (re-login) to the same Target Portal Group using the >>>> previous ISID. That is, it should perform session recovery as >>>> described in Chapter 6. This is because the SCSI initiator port >>>> identifier and the SCSI target port identifier (or relative target >>>> port) form the datum that the SCSI logical unit device server uses >>>> to identify the I_T nexus. >>>>=20 >>>> Is RESERVE state - explicit state? When an I_T nexus is reestablished,= >>>> initiator-specific mode pages are preserved. That implies that an I_T n= exus >>>> loss should be treated in the way that SAM-2 describes via CLEAR TASK S= ET >>>> (where mode pages and RESERVE state are preserved). If a session >>>> reestablishment caused saved mode pages to be restored rather than the >>>> established initiator-specific mode pages to be restored, then it would= be >>>> like a LU reset (where the RESERVE state is lost). Is RESERVE state ex= plicit >>>> state that is to be preserved? >>>>=20 >>>> Appendix E covers clearing effects during various events, but it never >>>> mentions RESERVE state. >>>>=20 >>>> Section E.2 contains the following text: >>>>=20 >>>> The only iSCSI protocol action that can effect clearing actions on >>>> SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 >>>> Loss of Nexus notification). [SPC3] describes the clearing effects >>>> of this notification on a variety of SCSI attributes. In addition, >>>> SCSI standards documents (such as [SAM2] and [SBC]) define >>>> additional clearing actions that may take place for several SCSI >>>> objects on SCSI events such as LU resets and power-on resets. >>>>=20 >>>> I could not find any text in SPC3 that describes these clearing effects= (I >>>> don't know if this SPC3 reference is incorrect, but at a minimum, it is= >>>> ineffective). The SAM2 text for LU resets and power-on resets is easil= y >>>> found, and clearly says that any RESERVE is released in those cases. A= s an >>>> aside, the reference to section 6.3.5.1 is a circular reference). The t= ext in >>>> E.2 goes on: >>>>=20 >>>> Since iSCSI defines a target cold reset as a protocol-equivalent >>>> to a target power-cycle, the iSCSI target cold reset must also be >>>> considered as the power-on reset event in interpreting the actions >>>> defined in the SCSI standards. >>>>=20 >>>> When the iSCSI session is reconstructed (between the same SCSI >>>> ports with the same nexus identifier) reestablishing the same I_T >>>> nexus, all SCSI objects that are defined to not clear on the "I_T >>>> nexus loss" notification event, such as persistent reservations, >>>> are automatically associated to this new session. >>>>=20 >>>> So, it is clear what happens to PERSISTENT RESERVE state, but not so cl= ear >>>> about RESERVE state. It also states that all SCSI objects that are def= ined to >>>> not clear on the I_T nexus loss are retained. But again, the only list= I can >>>> find of what is cleared is here in this Appendix, since SAM-2, SAM-3, S= AM-4 >>>> are all silent on the interaction between RESERVE and I_T Nexus loss. >>>>=20 >>>> T10 has never had to deal with this because they removed RESERVE before= they >>>> added I_T Nexus loss. However, iSCSI did not do that; we discuss BOTH,= but we >>>> fail to describe how they interact. iSCSI also has the feature of >>>> reestablishment of a session that does not exist in SAM-2 (SAM-2 was st= ill >>>> parallel SCSI focused, where connections didn't exist). So again, if s= ession >>>> reestablishment is examined in the light of parallel SCSI, the intent w= ould be >>>> to retain the RESERVE state. Here is another section on the topic: >>>>=20 >>>> 6.3.5.1. Loss of Nexus Notification >>>>=20 >>>> The iSCSI layer provides the SCSI layer with the "I_T nexus loss" >>>> notification when any one of the following events happens: >>>>=20 >>>> Successful completion of session reinstatement. >>>> Session closure event. >>>> Session timeout event. >>>>=20 >>>> Certain SCSI object clearing actions may result due to the >>>> notification in the SCSI end nodes, as documented in Appendix F. - >>>> Clearing Effects of Various Events on Targets. >>>>=20 >>>> FWIW - the reference above is incorrect - there is no Appendix F in the= >>>> consolidated draft - this should be Appendix E (and also, BTW - this is= a >>>> circular reference). >>>>=20 >>>> So anyway, it seems like the intent was that session reinstatement woul= d >>>> preserve the RESERVE state across the I_T Nexus loss event. >>>>=20 >>>> So aside from the incorrect reference (and circular nature of it), any >>>> thoughts on whether we should get explicit about this situation? Shoul= d we >>>> remain silent; should we tweak Appendix E along these lines (or other l= ines >>>> that others might suggest): >>>>=20 >>>> 4.4.3.1. I_T Nexus State >>>>=20 >>>> Certain nexus relationships contain an explicit state (e.g., >>>> initiator-specific mode pages, and RESERVE state) that may need to be >>>> preserved by >>>> the device server [SAM2] in a logical unit through changes or >>>> failures in the iSCSI layer (e.g., session failures).... >>>>=20 >>>> Fred Knight >>>>=20 >>>>=20 >>>>=20 >>>> -----Original Message----- >>>> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf O= f >>>> Mallikarjun Chadalapaka >>>> Sent: Sunday, September 23, 2012 1:34 AM >>>> To: david.black@emc.com; storm@ietf.org >>>> Subject: Re: [storm] iSCSI drafts - T10 and T11 references >>>>=20 >>>> Hi David, >>>>=20 >>>> I have noticed this comment as I am working through all Last Call comme= nts. >>>>=20 >>>> The references to SAM-3 and SAM-4 are keyed to this paragraph in Sectio= n 13.9: >>>>=20 >>>> "[SAM2] and [SAM3] specifications note in their informative text that T= PGT >>>> value should be non-zero, note that it is incorrect. A zero value is a= llowed >>>> as a legal value for TPGT. This discrepancy currently stands corrected i= n >>>> [SAM4]." >>>>=20 >>>> Additionally, SAM4 reference is also used in citing the new ASC/ASCQ co= des >>>> that it defines, in Section 11.14.5 of the draft. >>>>=20 >>>> So, I believe references to all three SAM specs would continue to be >>>> appropriate. >>>>=20 >>>> The current draft references FC-FS in discussing the NAA format in a fe= w >>>> places, so the current reference at the end is intentionally the same. I= s the >>>> older reference a problem? If so, I suppose we can replace all referenc= es to >>>> FC-FS-4, but that seems to be still in early stages(0.1 spec posted on A= pril >>>> 2012), so perhaps we should use FC-FS3? >>>>=20 >>>> Thanks. >>>>=20 >>>> Mallikarjun >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>> -----Original Message----- >>>> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf O= f >>>> david.black@emc.com >>>> Sent: Thursday, July 26, 2012 6:47 AM >>>> To: storm@ietf.org >>>> Subject: [storm] iSCSI drafts - T10 and T11 references >>>>=20 >>>> In making up the lists of T10 and T11 documents that are currently refe= renced, >>>> some of those references don't look right to me, for example: >>>>=20 >>>> - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. >>>> - T11: The FC-FS reference is out of date. >>>>=20 >>>> As part of dealing with the comments from IETF Last Call, we'll need to= go >>>> over all of the T10 and T11 references (and should check all the refere= nces) >>>> to make sure that they're up to date and necessary. >>>>=20 >>>> Thanks, >>>> --David >>>> ---------------------------------------------------- >>>> David L. Black, Distinguished Engineer >>>> EMC Corporation, 176 South St., Hopkinton, MA 01748 >>>> +1 (508) 293-7953 FAX: +1 (508) 293-7786 >>>> david.black@emc.com Mobile: +1 (978) 394-7754 >>>> ---------------------------------------------------- >>>>=20 >>>> _______________________________________________ >>>> storm mailing list >>>> storm@ietf.org >>>> https://www.ietf.org/mailman/listinfo/storm >>>>=20 >>>>=20 >>>> _______________________________________________ >>>> storm mailing list >>>> storm@ietf.org >>>> https://www.ietf.org/mailman/listinfo/storm >>>>=20 >>>>=20 >>> _______________________________________________ >>> storm mailing list >>> storm@ietf.org >>> https://www.ietf.org/mailman/listinfo/storm >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From david.black@emc.com Tue Sep 25 06:15:43 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B6B3421F86D4 for ; Tue, 25 Sep 2012 06:15:43 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8OqkhKnuySJa for ; Tue, 25 Sep 2012 06:15:42 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id C025921F86A1 for ; Tue, 25 Sep 2012 06:15:41 -0700 (PDT) Received: from hop04-l1d11-si03.isus.emc.com (HOP04-L1D11-SI03.isus.emc.com [10.254.111.23]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PDFd4P002158 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 25 Sep 2012 09:15:39 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.221.251]) by hop04-l1d11-si03.isus.emc.com (RSA Interceptor); Tue, 25 Sep 2012 09:15:14 -0400 Received: from mxhub28.corp.emc.com (mxhub28.corp.emc.com [10.254.110.184]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PDFC3v002653; Tue, 25 Sep 2012 09:15:12 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub28.corp.emc.com ([10.254.110.184]) with mapi; Tue, 25 Sep 2012 09:15:11 -0400 From: "Black, David" To: Julian Satran , Ralph Weber Date: Tue, 25 Sep 2012 09:15:10 -0400 Thread-Topic: [storm] RESERVE state - clearing effects Thread-Index: Ac2bHLIa6cZXAjslRSajPnZ7FlExPQAALzjQ Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE793CA@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> <5061A45E.7010607@ieee.org> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-EMM-MHVC: 1 Cc: "storm@ietf.org" Subject: Re: [storm] RESERVE state - clearing effects X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 13:15:43 -0000 VG8gcHV0IHRoZSBmaW5hbCAibmFpbCBpbiB0aGUgY29mZmluIiBvZiBjbHVzdGVycyBhbmQgcHJl c2VydmF0aW9uDQpvZiByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zLCBJIGtub3cgb2Ygb25l IHJlYXNvbmFibHkgd2lkZWx5DQpkZXBsb3llZCBjbHVzdGVyIHRoYXQgZGlkIHVzZSByZXNlcnZl L3JlbGVhc2UgcmVzZXJ2YXRpb25zIC4uLg0KLi4uIGFuZCB0aGF0IGNsdXN0ZXIgdXNlZCBwZXJp b2RpYyBsb3NzIG9mIGEgcmVzZXJ2YXRpb24gKHZpYQ0KVEFSR0VUIFJFU0VUKSBhcyBhIGhlYXJ0 YmVhdCBtZWNoYW5pc20gLi4uIG1ha2luZyB0aGlzIGRpc2N1c3Npb24NCmFib3V0IHJlc2VydmF0 aW9uIHByZXNlcnZhdGlvbiBiYXNpY2FsbHkgaXJyZWxldmFudCAuLi4gYXQgbGVhc3QgdG8NCnRo YXQgY2x1c3Rlci4NCg0KVGhhdCBzb3J0IG9mIChhYil1c2Ugb2YgVEFSR0VUIFJFU0VUIChvciB0 aGUgY3VycmVudCBMT0dJQ0FMIFVOSVQNClJFU0VUKSBpcyBhbW9uZyB0aGUgcmVhc29ucyB3aHkg cGVyc2lzdGVudCByZXNlcnZhdGlvbnMgYXJlIHRoZQ0Kc3Ryb25nbHkgcHJlZmVycmVkIGFwcHJv YWNoLiAgVk0gbWlncmF0aW9uIHZhcmllcyBieSBoeXBlcnZpc29yIC0NCnBlcnNpc3RlbnQgcmVz ZXJ2YXRpb25zIHdvdWxkIGJlIHVzZWQgaWYgYSByZXNlcnZhdGlvbiBpcyBuZWVkZWQNCm9uIHRo ZSBWTSdzIHN0b3JhZ2UgYnV0IG5vdCBhbGwgaHlwZXJ2aXNvcnMgYXJlIGltcGxlbWVudGVkIHRo YXQNCndheS4gIEkga25vdyBvZiBhbiBpbXBsZW1lbnRhdGlvbiB0aGF0IHVzZXMgcmVzZXJ2ZS9y ZWxlYXNlIHRvDQpjcmVhdGUgc2hvcnQgY3JpdGljYWwgc2VjdGlvbnMgKGZvciB3aGljaCBDT01Q QVJFIEFORCBXUklURSBpcw0KdGhlIHByZWZlcnJlZCByZXBsYWNlbWVudCksIGJ1dCBpdCdzIHNh ZmUgdG8gYWJvcnQgdGhlIGNyaXRpY2FsDQpzZWN0aW9ucyBpbiBwcm9ncmVzcyBvbiBJX1QgTmV4 dXMgbG9zcy4NCg0KSSB3aWxsIHN0YXJ0IGEgbmV3IHRocmVhZCB3aXRoIHNvbWUgcHJvcG9zZWQg dGV4dC4gIEEgY3J1Y2lhbA0KY29uc2lkZXJhdGlvbiB3aWxsIGJlIHRoYXQgd2UgY2Fubm90IGlu dHJvZHVjZSBuZXcgcmVxdWlyZW1lbnRzDQppbnRvIHRoZSBjb25zb2xpZGF0ZWQgaVNDU0kgZHJh ZnQgdGhhdCBleGlzdGluZyBpbXBsZW1lbnRhdGlvbnMNCm1pZ2h0IG5vdCBzYXRpc2Z5Lg0KDQpU aGFua3MsDQotLURhdmlkDQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTog c3Rvcm0tYm91bmNlc0BpZXRmLm9yZyBbbWFpbHRvOnN0b3JtLWJvdW5jZXNAaWV0Zi5vcmddIE9u IEJlaGFsZiBPZg0KPiBKdWxpYW4gU2F0cmFuDQo+IFNlbnQ6IFR1ZXNkYXksIFNlcHRlbWJlciAy NSwgMjAxMiA4OjUxIEFNDQo+IFRvOiBSYWxwaCBXZWJlcg0KPiBDYzogc3Rvcm1AaWV0Zi5vcmcN Cj4gU3ViamVjdDogUmU6IFtzdG9ybV0gUkVTRVJWRSBzdGF0ZSAtIGNsZWFyaW5nIGVmZmVjdHMN Cj4NCj4gUmFscGgsDQo+DQo+IEkgdW5kZXJzdGFuZC4gSSBtaXNzZWQgdGhlIGZhY3QgdGhhdCB0 aGUgZGlzY3Vzc2lvbiB3YXMgZXhjbHVzaXZlbHkgYWJvdXINCj4gcmVzZXJ2ZS9yZWxlYXNlLiBB bmQgaSB3YXMgcmVmZXJyaW5nIHRvIFZNcyAoZXZlbiBJIGFtIHRvbyB5b3VuZyB0byByZW1lYmVy DQo+IFZNUyAtIHRoYXQgd2FzIGR1cmluZyB0aGUgdHJvamFuIHdhciB3YXNuJ3QgaXQ/KS4NCj4N Cj4gSnVsbw0KPg0KPiBTZW50IGZyb20gbXkgaVBhZA0KPg0KPiBPbiAyNSDXkdeh16TXmCAyMDEy LCBhdCAxNDozMiwgUmFscGggV2ViZXIgPHJvd2ViZXJAaWVlZS5vcmc+IHdyb3RlOg0KPg0KPiA+ IFBlcnNpc3RlbnQgUmVzZXJ2YXRpb25zIHdlcmUgaW52ZW50ZWQgYmVjYXVzZSBSZXNlcnZlL1Jl bGVhc2UgZmFpbGVkDQo+ID4gdG8gcHJvdmlkZSB0aGUgcXVhbGl0eSBvZiByZXNlcnZhdGlvbnMg c2VydmljZXMgZGVtYW5kZWQgYnkgY2x1c3RlcnMuDQo+ID4gQ2x1c3RlcnMgdGhhdCBkZXBlbmQg b24gUmVzZXJ2ZS9SZWxlYXNlIHJlc2VydmF0aW9ucyBnZXQgZXhhY3RseQ0KPiA+IHdoYXQgdGhl eSBkZXNlcnZlLg0KPiA+DQo+ID4gQXMgZm9yIE9wZW4gVk1TLCBteSByZWNvbGxlY3Rpb24gaXMg dGhhdCBpdCBoYWQgb25seSBvbmUgbWlub3INCj4gPiBkZXBlbmRlbmNlIG9uIHJlc2VydmF0aW9u cy4gVGhlIG1vc3QgSSByZW1lbWJlciBuZWVkaW5nIGZyb20NCj4gPiByZXNlcnZhdGlvbnMgd2Fz IGEgd2F5IHRvIGVuc3VyZSB0aGF0IGRpc2tzIHN0b3BwZWQgcHJvY2Vzc2luZw0KPiA+IEkvT3Mg ZnJvbSBhbiBleHBlbGxlZCBjbHVzdGVyIG1lbWJlci4gQXMgbG9uZyBhcyBhbGwgY29tbWFuZHMg YXJlDQo+ID4gYWJvcnRlZCBhdCB0aGUgc2FtZSB0aW1lIGEgcmVzZXJ2YXRpb24gaXMgcmVsZWFz ZWQsIHRoZSBWTVMgSQ0KPiA+IHJlbWVtYmVyIHdpbGwgbm90IHN1ZmZlciBhbnkgYWR2ZXJzZSBl ZmZlY3RzLg0KPiA+DQo+ID4gSW4gYW55IGNhc2UsIFZNUyBoYXMgaGFkIHR3byBkZWNhZGVzIHRv IG1pZ3JhdGUgdG8gUGVyc2lzdGVudA0KPiA+IFJlc2VydmF0aW9ucyBmb3Igd2hhdGV2ZXIgdGhl eSBuZWVkZWQgb3IgZGV2ZWxvcGVkIGEgZGVwZW5kZW5jZQ0KPiA+IHVwb24uIEV2ZW4gZm9yIE9T IGVuZ2luZWVycywgdGhhdCBzaG91bGQgYmUgbG9uZyBlbm91Z2ggZm9yDQo+ID4gdGhlbSB0byBo YXZlIG1hdGNoZWQgdGhlaXIgc29mdHdhcmUgdG8gdGhlIGF2YWlsYWJsZSBzZXJ2aWNlcy4NCj4g Pg0KPiA+IEFsbCB0aGUgYmVzdCwNCj4gPg0KPiA+IC5SYWxwaA0KPiA+DQo+ID4gUC5TLiBEYXZp ZCBhbmQgRnJlZCwgSSBhbSBub3QgdmVudHVyaW5nIGludG8gdGhlIHRvcGljcyB5b3UgYXJlDQo+ ID4gZGlzY3Vzc2luZyBiZWNhdXNlIEkgc2VlIG5vdGhpbmcgdXBvbiB3aGljaCBJIGNhbiBwcm92 aWRlIGtub3dsZWRnZWFibGUNCj4gPiBhZGRpdGlvbmFsIGNvbW1lbnQuIFdoYXRldmVyIGFncmVl bWVudCB5b3UgYWxsIHJlYWNoIHdpbGwgYmUgb2theSAuLi4NCj4gPiBhcyBmYXIgYXMgSSBjYW4g dGVsbC4NCj4gPg0KPiA+IE9uIDkvMjQvMjAxMiAxMTo1NSBQTSwgSnVsaWFuIFNhdHJhbiB3cm90 ZToNCj4gPj4gRGlkIHlvdSBjb25zaWRlaXIgY2x1c3RlcnMgb3IgbWlncmF0aW5nIHZtcz8gV2l0 aCB0aG9zZSB1c3VhbGx5IG9uZSBleHBlY3RzDQo+IHRhcmdldHMgdG8gcmV0YWluIHNvbWUgc3Rh dGUgYWNjcm9zcyBJVCBuZXh1cyBsb3NzLiBUbyBhdm9pZCB0aGlyZCBwYXJ0eQ0KPiByZXNlcnZh dGlvbiBjbGVhbnVwIFNDU0kgKHQxMCkgbWF5IHdhbnQgdG8gaW50cm9kdWNlIGEgdGltZW91dCBt ZWNoYW5pc20NCj4gYXNzb2NpYXRlZCB3aXRoIG5leHVzIGxvc3MgKHN0YXRlIGlzIHByZXNlcnZl ZCBmb3IgYSB3aGlsZSBhZnRlciBuZXh1cyBsb3NzDQo+IGFuZCB0aGUgdGltZW91dCB2YWx1ZXMg Y2FuIGJlIHNldCkuDQo+ID4+DQo+ID4+IGp1bG8NCj4gPj4NCj4gPj4gU2VudCBmcm9tIG15IGlQ YWQNCj4gPj4NCj4gPj4gT24gMjQg15HXodek15ggMjAxMiwgYXQgMTg6NDIsICJCbGFjaywgRGF2 aWQiIDxkYXZpZC5ibGFja0BlbWMuY29tPiB3cm90ZToNCj4gPj4NCj4gPj4+IEZyZWQsDQo+ID4+ Pg0KPiA+Pj4gVGhpcyBpcyBpbnRlcmVzdGluZyAuLi4NCj4gPj4+DQo+ID4+Pj4gSGVyZSBhcmUg c29tZSBrZXkgcG9pbnRzIChmcm9tIFNBTSA2LjMuNCBJX1QgbmV4dXMgbG9zcyk6DQo+ID4+Pj4N Cj4gPj4+PiAxKSBJX1QgbmV4dXMgbG9zcyBjbGVhcnMgYWxsIEFDQSBjb25kaXRpb25zOw0KPiA+ Pj4+IDIpIGVzdGFibGlzaGVkIGEgVUEgLSBvbmUgb2YgdGhlIGZvbGxvd2luZyBpcyBzdWdnZXN0 ZWQ6DQo+ID4+Pj4NCj4gPj4+PiAiSWYgdGhlIGxvZ2ljYWwgdW5pdCByZXRhaW5zIHN0YXRlIGlu Zm9ybWF0aW9uIGZvciB0aGUgSV9UIG5leHVzIHRoYXQNCj4gPj4+PiBpcyBsb3N0LCBpdHMgcmVz cG9uc2UgdG8gdGhlIHN1YnNlcXVlbnQgSV9UIG5leHVzDQo+ID4+Pj4gcmUtZXN0YWJsaXNobWVu dCBmb3IgdGhlIGxvZ2ljYWwgdW5pdCBzaG91bGQgaW5jbHVkZSBlc3RhYmxpc2hpbmcgYQ0KPiA+ Pj4+IHVuaXQgYXR0ZW50aW9uIHdpdGggYW4gYWRkaXRpb25hbCBzZW5zZSBjb2RlIHNldA0KPiA+ Pj4+IHRvIElfVCBORVhVUyBMT1NTIE9DQ1VSUkVELg0KPiA+Pj4+DQo+ID4+Pj4gSWYgdGhlIGxv Z2ljYWwgdW5pdCBkb2VzIG5vdCByZXRhaW4gc3RhdGUgaW5mb3JtYXRpb24gZm9yIHRoZSBJX1Qg bmV4dXMNCj4gPj4+PiB0aGF0IGlzIGxvc3QsIGl0IHNoYWxsIGNvbnNpZGVyIHRoZSBzdWJzZXF1 ZW50DQo+ID4+Pj4gSV9UIG5leHVzIHJlLWVzdGFibGlzaG1lbnQsIGlmIGFueSwgYXMgdGhlIGZv cm1hdGlvbiBvZiBhIG5ldyBJX1QgbmV4dXMNCj4gPj4+PiBmb3Igd2hpY2ggdGhlcmUgaXMgbm8g cGFzdCBoaXN0b3J5IChlLmcuLA0KPiA+Pj4+IGVzdGFibGlzaCBhIHVuaXQgYXR0ZW50aW9uIHdp dGggYW4gYWRkaXRpb25hbCBzZW5zZSBjb2RlIHNldCB0byBQT1dFUiBPTg0KPiA+Pj4+IE9DQ1VS UkVEKS4iDQo+ID4+PiBUaGF0ICJzdGF0ZSBpbmZvcm1hdGlvbiIgaGFzIHRvIGlkZW50aWZ5IHRo ZSBlbmRwb2ludHMgb2YgdGhlIElfVCBOZXh1cw0KPiA+Pj4gLSBmb3IgaVNDU0kgdGhhdCBtZWFu cyBhdCBsZWFzdCB0aGUgSVNJRCBhbmQgVFBHVC4gIElmIHRoYXQncyBnb25lLCBJDQo+ID4+PiB0 aGluayB0aGUgUkVTRVJWRSBzdGF0ZSBpcyBnb25lIGFzIHdlbGwsIGluIGNvbnRyYXN0IHRvIFBF UlNJU1RFTlQNCj4gPj4+IFJFU0VSVkUgc3RhdGUuICBPbiBSRVNFUlZFIHN0YXRlLCBTUEMtMiBo YXMgdGhpcyB0byBzYXkgaW4gNS41LjIgIlRoZQ0KPiA+Pj4gUmVzZXJ2ZS9SZWxlYXNlIG1hbmFn ZW1lbnQgbWV0aG9kIjoNCj4gPj4+DQo+ID4+PiAgICAgICAgVGhlIHJlc2VydmUvcmVsZWFzZSBt YW5hZ2VtZW50IG1ldGhvZCBjb21tYW5kcywgUkVTRVJWRSg2KSwNCj4gPj4+ICAgICAgICBSRVNF UlZFKDEwKSwgUkVMRUFTRSg2KSwgYW5kIFJFTEVBU0UoMTApIGFyZSB1c2VkIGFtb25nIG11bHRp cGxlDQo+ID4+PiAgICAgICAgaW5pdGlhdG9ycyB0aGF0IGRvIG5vdCByZXF1aXJlIG9wZXJhdGlv bnMgdG8gYmUgcHJvdGVjdGVkIGFjcm9zcw0KPiA+Pj4gICAgICAgIGluaXRpYXRvciBmYWlsdXJl cyAoYW5kIHN1YnNlcXVlbnQgaGFyZCByZXNldHMpLg0KPiA+Pj4NCj4gPj4+IEkgdGhpbmsgaXQn cyBmYWlyIHRvIHJlZ2FyZCBJX1QgTmV4dXMgbG9zcyBhcyBhbiBpbml0aWF0b3IgZmFpbHVyZS4g IEkNCj4gPj4+IGFsc28gZG9uJ3Qgc2VlIGFueSBnb29kIHdheSB0byBkbyBhIDNyZC1wYXJ0eSBy ZWxlYXNlIG9uIGEgUkVTRVJWRQ0KPiA+Pj4gcmVzZXJ2YXRpb24gdGhhdCBwZXJzaXN0cyBhZnRl ciBJX1QgTmV4dXMgbG9zcy4gIEkgYWxzbyBiZWxpZXZlIHRoYXQNCj4gPj4+IGluaXRpYXRvciBz b2Z0d2FyZSB0aGF0IHVzZXMgUmVzZXJ2ZS9SZWxlYXNlIGRvZXMgbm90IGV4cGVjdA0KPiA+Pj4g cmVzZXJ2YXRpb25zIHRvIHBlcnNpc3QgYWNyb3NzIElfVCBOZXh1cyBsb3NzLg0KPiA+Pj4NCj4g Pj4+IEFzIHRvIHdoYXQgbmVlZHMgdG8gYmUgc3RhdGVkIGluIHRoZSBpU0NTSSBjb25zb2xpZGF0 ZWQgZHJhZnQgLSB0aGUgU1BDDQo+ID4+PiByZWZlcmVuY2UgaGFzIGJlZW4gdXBkYXRlZCB0byBT UEMtMyBpbiB3aGljaCB0aGUgUmVzZXJ2ZS9SZWxlYXNlDQo+ID4+PiBtYW5hZ2VtZW50IG1ldGhv ZCBpcyBvYnNvbGV0ZSwgc28gdGhlcmUgaXNuJ3QgYW55IGFic29sdXRlIHJlcXVpcmVtZW50DQo+ ID4+PiB0byBjb3ZlciB0aGF0LiAgT1RPSCwgaXQgbWlnaHQgYmUgcmVhc29uYWJsZSB0byBhZGQg YSBzZW50ZW5jZSB0byBzYXkNCj4gPj4+IHRoYXQgUmVzZXJ2ZS9SZWxlYXNlIHN0YXRlIGlzIGlt cGxpY2l0IChlLmcuLCBhcyB0aGVyZSdzIG5vIHdheSBmb3IgYW4NCj4gPj4+IGluaXRpYXRvciB0 byBmaWd1cmUgb3V0IHdoaWNoIElfVCBOZXh1cyBob2xkcyBhIHJlc2VydmF0aW9uKSwgYW5kIHRo YXQNCj4gPj4+IHRyZWF0aW5nIFJlc2VydmUvUmVsZWFzZSBzdGF0ZSBhcyBpbmRlZmluaXRlbHkg cGVyc2lzdGVudCBhZnRlciBJX1QNCj4gPj4+IE5leHVzIExvc3MgaXMgYSBiYWQgaWRlYSAoInNo b3VsZCBub3QiLCBidXQgbG93ZXIgY2FzZSkgYXMgaXQgbWF5DQo+ID4+PiByZXF1aXJlIHRoZSBp bml0aWF0b3IgdG8gcmVzZXQgdGhlIHRhcmdldCBkZXZpY2UgaW4gb3JkZXIgdG8gY2xlYXIgb3V0 DQo+ID4+PiB0aGF0IHN0YXRlIChpc3N1aW5nIFJFTEVBU0UgY29tbWFuZHMgaXNuJ3QgZW5vdWdo KS4gIFRoaXMgc29ydCBvZiB0ZXh0DQo+ID4+PiB3b3VsZCByZXF1aXJlIGEgcmVmZXJlbmNlIHRv IFNQQy0yLg0KPiA+Pj4NCj4gPj4+IFRoYW5rcywNCj4gPj4+IC0tRGF2aWQNCj4gPj4+DQo+ID4+ Pj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPj4+PiBGcm9tOiBLbmlnaHQsIEZyZWRl cmljayBbbWFpbHRvOkZyZWRlcmljay5LbmlnaHRAbmV0YXBwLmNvbV0NCj4gPj4+PiBTZW50OiBN b25kYXksIFNlcHRlbWJlciAyNCwgMjAxMiA4OjU4IEFNDQo+ID4+Pj4gVG86IE1hbGxpa2FyanVu IENoYWRhbGFwYWthOyBCbGFjaywgRGF2aWQ7IHN0b3JtQGlldGYub3JnDQo+ID4+Pj4gU3ViamVj dDogUkU6IGlTQ1NJIGRyYWZ0cyAtIFQxMCBhbmQgVDExIHJlZmVyZW5jZXMNCj4gPj4+Pg0KPiA+ Pj4+IEZDUCBpcyBhbHNvIG5vdCBleHBsaWNpdCAoaXQgZGVmZXJzIHRvIHRoZSBTQ1NJIHNwZWNz IC0gd2hpY2ggZG8gbm90DQo+IHNheSkuDQo+ID4+Pj4gVGhlIGNsb3Nlc3Qgc3RhdGVtZW50cyBJ IGNhbiBmaW5kIHN0aWxsIHJlcXVpcmVzIGV4dHJhcG9sYXRpb246DQo+ID4+Pj4NCj4gPj4+PiBI ZXJlIGFyZSBzb21lIGtleSBwb2ludHMgKGZyb20gU0FNIDYuMy40IElfVCBuZXh1cyBsb3NzKToN Cj4gPj4+Pg0KPiA+Pj4+IDEpIElfVCBuZXh1cyBsb3NzIGNsZWFycyBhbGwgQUNBIGNvbmRpdGlv bnM7DQo+ID4+Pj4gMikgZXN0YWJsaXNoZWQgYSBVQSAtIG9uZSBvZiB0aGUgZm9sbG93aW5nIGlz IHN1Z2dlc3RlZDoNCj4gPj4+Pg0KPiA+Pj4+ICJJZiB0aGUgbG9naWNhbCB1bml0IHJldGFpbnMg c3RhdGUgaW5mb3JtYXRpb24gZm9yIHRoZSBJX1QgbmV4dXMgdGhhdCBpcw0KPiBsb3N0LA0KPiA+ Pj4+IGl0cyByZXNwb25zZSB0byB0aGUgc3Vic2VxdWVudCBJX1QgbmV4dXMNCj4gPj4+PiByZS1l c3RhYmxpc2htZW50IGZvciB0aGUgbG9naWNhbCB1bml0IHNob3VsZCBpbmNsdWRlIGVzdGFibGlz aGluZyBhIHVuaXQNCj4gPj4+PiBhdHRlbnRpb24gd2l0aCBhbiBhZGRpdGlvbmFsIHNlbnNlIGNv ZGUgc2V0DQo+ID4+Pj4gdG8gSV9UIE5FWFVTIExPU1MgT0NDVVJSRUQuDQo+ID4+Pj4NCj4gPj4+ PiBJZiB0aGUgbG9naWNhbCB1bml0IGRvZXMgbm90IHJldGFpbiBzdGF0ZSBpbmZvcm1hdGlvbiBm b3IgdGhlIElfVCBuZXh1cw0KPiB0aGF0DQo+ID4+Pj4gaXMgbG9zdCwgaXQgc2hhbGwgY29uc2lk ZXIgdGhlIHN1YnNlcXVlbnQNCj4gPj4+PiBJX1QgbmV4dXMgcmUtZXN0YWJsaXNobWVudCwgaWYg YW55LCBhcyB0aGUgZm9ybWF0aW9uIG9mIGEgbmV3IElfVCBuZXh1cw0KPiBmb3INCj4gPj4+PiB3 aGljaCB0aGVyZSBpcyBubyBwYXN0IGhpc3RvcnkgKGUuZy4sDQo+ID4+Pj4gZXN0YWJsaXNoIGEg dW5pdCBhdHRlbnRpb24gd2l0aCBhbiBhZGRpdGlvbmFsIHNlbnNlIGNvZGUgc2V0IHRvIFBPV0VS IE9ODQo+ID4+Pj4gT0NDVVJSRUQpLiINCj4gPj4+Pg0KPiA+Pj4+IEJ1dCwgdGhlcmUgaXMgbm8g ZGVmaW5pdGlvbiBvZiB3aGF0IGl0IG1lYW5zIHRvICJyZXRhaW4gc3RhdGUiLiAgVGhlDQo+ID4+ Pj4gY29uY2x1c2lvbiBJIGNvdWxkIGp1bXAgdG8gYmFzZWQgb24gIzEgYWJvdmUsIGlzIHRoYXQg UkVTRVJWRSBzdGF0ZSBpcw0KPiBhbHNvDQo+ID4+Pj4gY2xlYXJlZCAodGhpcyBhY3Rpb24gbGlu ZXMgdXAgbW9yZSB3aXRoIExVIFJFU0VUIHRoYW4gQ0xFQVIgVEFTSyBTRVQpOw0KPiBhbmQNCj4g Pj4+PiBiYXNlZCBvbiB0aGUgUE9XRVIgT04gQ09ORElUSU9OIFVBIG9mIDI5LzAxIChvciAyOS8w MCAtIGJvdGggYXJlIHBvd2VyIG9uDQo+ID4+Pj4gY29uZGl0aW9ucykgSSBjb3VsZCBhbHNvIGFz c3VtZSB0aGF0IHRoZSBSRVNFUlZFIHN0YXRlIGlzIGNsZWFyZWQ7IGJ1dA0KPiBiYXNlZA0KPiA+ Pj4+IG9uIHRoZSBVQSBvZiAyOS8wNyAocmV0YWluIHN0YXRlKSwgSSBjb3VsZCBqdW1wIHRvIHRo ZSBjb25jbHVzaW9uIHRoYXQNCj4gUkVTRVJWRQ0KPiA+Pj4+IHN0YXRlIGlzIHJldGFpbmVkLg0K PiA+Pj4+DQo+ID4+Pj4gQnV0IHRoZXNlIGFyZSBhbGwgY29uY2x1c2lvbnMgSSd2ZSBqdW1wIHRv IHdpdGggdmVyeSBsaW1pdGVkIGJhc2lzIGluIHRoZQ0KPiA+Pj4+IHRleHQuDQo+ID4+Pj4NCj4g Pj4+PiBCYXNlZCBvbiB3aGF0IFJGQzM3MjAgc2F5cyBhYm91dCBleHBsaWNpdCBzdGF0ZSAoZS5n LiwgbW9kZSBwYWdlcyksIEkNCj4gd291bGQNCj4gPj4+PiBhc3N1bWUgdGhhdCBob3N0cyB3b3Vs ZCBiZWxpZXZlIHRoYXQgZXZlcnkgY29tbWFuZCB0aGV5IGV4cGxpY2l0bHkgc2VuZA0KPiAobGlr ZQ0KPiA+Pj4+IE1PREUgU0VMRUNUIG9yIFJFU0VSVkUpIHdvdWxkIGhhdmUgY3JlYXRlZCBleHBs aWNpdCBzdGF0ZSBhbmQgdGhlcmVmb3JlDQo+IHRoZQ0KPiA+Pj4+IGhvc3Qgd291bGQgYXNzdW1l IFJFU0VSVkUgc3RhdGUgaXMgcmV0YWluZWQuICBJdCBkb2Vzbid0IG1ha2Ugc2Vuc2UgdG8NCj4g aGF2ZQ0KPiA+Pj4+IG1vZGUgcGFnZSB2YWx1ZXMgcmV0YWluZWQsIGJ1dCBoYXZlIFJFU0VSVkUg c3RhdGUgbG9zdC4NCj4gPj4+Pg0KPiA+Pj4+ICAgICAgQ2VydGFpbiBuZXh1cyByZWxhdGlvbnNo aXBzIGNvbnRhaW4gYW4gZXhwbGljaXQgc3RhdGUgKGUuZy4sDQo+ID4+Pj4gICAgICBpbml0aWF0 b3Itc3BlY2lmaWMgbW9kZSBwYWdlcykgdGhhdCBtYXkgbmVlZCB0byBiZSBwcmVzZXJ2ZWQgYnkN Cj4gPj4+PiAgICAgIHRoZSBkZXZpY2Ugc2VydmVyIFtTQU0yXSBpbiBhIGxvZ2ljYWwgdW5pdCB0 aHJvdWdoIGNoYW5nZXMgb3INCj4gPj4+PiAgICAgIGZhaWx1cmVzIGluIHRoZSBpU0NTSSBsYXll ciAoZS5nLiwgc2Vzc2lvbiBmYWlsdXJlcykuDQo+ID4+Pj4NCj4gPj4+PiBJdCB3aWxsIGJlIGlu dGVyZXN0aW5nIHRvIGhlYXIgd2hhdCBhIHJlYWwgaG9zdCBhc3N1bWVzIGFib3V0IHRoaXMNCj4g c2l0dWF0aW9uLg0KPiA+Pj4+DQo+ID4+Pj4gICAgICBGcmVkDQo+ID4+Pj4NCj4gPj4+PiAtLS0t LU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiA+Pj4+IEZyb206IE1hbGxpa2FyanVuIENoYWRhbGFw YWthIFttYWlsdG86Y2JtQGNoYWRhbGFwYWthLmNvbV0NCj4gPj4+PiBTZW50OiBTdW5kYXksIFNl cHRlbWJlciAyMywgMjAxMiA2OjM1IFBNDQo+ID4+Pj4gVG86IEtuaWdodCwgRnJlZGVyaWNrOyBk YXZpZC5ibGFja0BlbWMuY29tOyBzdG9ybUBpZXRmLm9yZw0KPiA+Pj4+IFN1YmplY3Q6IFJFOiBp U0NTSSBkcmFmdHMgLSBUMTAgYW5kIFQxMSByZWZlcmVuY2VzDQo+ID4+Pj4NCj4gPj4+PiBIaSBG cmVkLA0KPiA+Pj4+DQo+ID4+Pj4gU1BDLTMgZGlzY3Vzc2VzIGNsZWFyaW5nIGVmZmVjdHMgb2Yg SS1UIG5leHVzIGxvc3MgaW4gc2V2ZXJhbCBwbGFjZXMuDQo+ID4+Pj4NCj4gPj4+PiBTZWN0aW9u IDUuOC4yLjkgZm9yIEFMVUENCj4gPj4+PiBTZWN0aW9uIDUuNS4zLjIgZm9yIGZvcmVncm91bmQg c2VsZi10ZXN0IFNlY3Rpb24gNS41LjMuMyBmb3IgYmFja2dyb3VuZA0KPiBzZWxmLQ0KPiA+Pj4+ IHRlc3QgU2VjdGlvbiA1LjYuMSBvbiBwZXJzaXN0ZW50IHJlc2VydmF0aW9ucyBTZWN0aW9uIDUu MTMgb24gdGltZXN0YW1wcw0KPiA+Pj4+IC4uLi4uLg0KPiA+Pj4+DQo+ID4+Pj4gVGhhbmtzIGZv ciBwb2ludGluZyBvdXQgdGhlIGJyb2tlbiBjcm9zcy1yZWZlcmVuY2UgLSBBcHBlbmRpeCBGIGlu c3RlYWQNCj4gb2YgRS4NCj4gPj4+PiBJIGhhdmUganVzdCBmaXhlZCBpdCAtIGFuZCByZXZpZXdl ZCBhbGwgb3RoZXIgQXBwZW5kaXggcmVmZXJlbmNlcw0KPiBlbHNld2hlcmUgaW4NCj4gPj4+PiB0 aGUgZHJhZnQuIEhvd2V2ZXIsIEkgZG8gbm90IHNlZSBhIGNpcmN1bGFyIHJlZmVyZW5jZSBwcm9i bGVtIHBlciBzZSAtDQo+ID4+Pj4gQXBwZW5kaXggRSBpcyBjb3JyZWN0bHkgcmVmZXJlbmNpbmcg U2VjdGlvbiA2LjMuNSBmb3IgZGVmaW5pdGlvbiBvZg0KPiB0ZXJtcywNCj4gPj4+PiBlLmcuIHNl c3Npb24gY29udGludWF0aW9uLiBBbmQgU2VjdGlvbiA2LjMuNS4xIGlzIGNvcnJlY3RseSBkZWZl cnJpbmcgdG8NCj4gPj4+PiBBcHBlbmRpeCBFIHdoZW4gZGlzY3Vzc2luZyBjbGVhcmluZyBlZmZl Y3RzLiBMZXQgbWUga25vdyBpZiBJJ20NCj4gb3Zlcmxvb2tpbmcNCj4gPj4+PiBzb21ldGhpbmcg aGVyZS4NCj4gPj4+Pg0KPiA+Pj4+IE5vdywgdGhlIHJlc2VydmUvcmVsZWFzZSBtYW5hZ2VtZW50 IHN0YXRlLCA6KSA6IEkgIGFncmVlIHRoZSBmb3JtYWwgSV9UDQo+IG5leHVzDQo+ID4+Pj4gbG9z cyBldmVudCBkaWRuJ3QgZXhpc3QgaW4gU0FNLTIsIGhvd2V2ZXIgdGhlIG5vdGlvbiBvZiBhIG5l dyBJX1QgbmV4dXMNCj4gc3RhdGUNCj4gPj4+PiBkaWQgY28tZXhpc3Qgd2l0aCBhIHNlc3Npb24t b3JpZW50ZWQgSV9UIG5leHVzIHdlbGwgYmVmb3JlIGlTQ1NJIC0gRkNQDQo+IHByb2Nlc3MNCj4g Pj4+PiBsb2dpbiAoUFJMTykgY29tZXMgdG8gbWluZCBoZXJlLiBTbyBJJ2QgYmUgY3VyaW91cyBh Ym91dCB0cmFkaXRpb25hbCBGQ1ANCj4gPj4+PiBoYW5kbGluZyBvZiByZXNlcnZlL3JlbGVhc2Ug bWFuYWdlbWVudCBzdGF0ZS4gQW5kIEknbSBhbHNvIGNoZWNraW5nDQo+IGludGVybmFsbHkNCj4g Pj4+PiBoZXJlIGF0IE1TLiBGcm9tIHdoYXQgSSBrbm93IG5vdyB0aG91Z2gsIGl0IGRvZXMgKmFw cGVhciogdGhhdA0KPiByZXNlcnZlL3JlbGVhc2UNCj4gPj4+PiBtYW5hZ2VtZW50IHN0YXRlIHNo b3VsZCBub3QgYmUgY2xlYXJlZCB3aXRoIElfVCBuZXh1cyBsb3NzLg0KPiA+Pj4+DQo+ID4+Pj4g SGF2aW5nIHNhaWQgYWxsIHRoYXQsIG5vdGUgdGhhdCByZXNlcnZlL3JlbGVhc2UgbWFuYWdlbWVu dCBzdGF0ZSBpcw0KPiBzcXVhcmVseQ0KPiA+Pj4+IGluIHRoZSBTQ1NJIHRlcnJpdG9yeS4gaVND U0kgc3BlYyBoYXMgc28gZmFyIGJ5IGRlc2lnbiBkZWZlcnJlZCB0byBTQ1NJDQo+ID4+Pj4gc3Rh bmRhcmRzIGluIHNwZWNpZnlpbmcgY2xlYXJpbmcgZWZmZWN0cyBvbiBTQ1NJIG9iamVjdHMgLSBh cyBBcHBlbmRpeA0KPiBFLjINCj4gPj4+PiBtYWtlcyBpdCBjbGVhci4gV2hhdCB3ZSBkbyBzcGVj aWZ5IGlzIHRoZSBjbGVhcmluZyBlZmZlY3RzIG9mIFNDU0kNCj4gYWN0aW9ucywNCj4gPj4+PiBl LmcuIExVIHJlc2V0LCBvbiBpU0NTSSBvYmplY3RzLg0KPiA+Pj4+DQo+ID4+Pj4gVGhhbmtzLg0K PiA+Pj4+DQo+ID4+Pj4gTWFsbGlrYXJqdW4NCj4gPj4+Pg0KPiA+Pj4+DQo+ID4+Pj4NCj4gPj4+ Pg0KPiA+Pj4+DQo+ID4+Pj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPj4+PiBGcm9t OiBLbmlnaHQsIEZyZWRlcmljayBbbWFpbHRvOkZyZWRlcmljay5LbmlnaHRAbmV0YXBwLmNvbV0N Cj4gPj4+PiBTZW50OiBTdW5kYXksIFNlcHRlbWJlciAyMywgMjAxMiAzOjM4IEFNDQo+ID4+Pj4g VG86IE1hbGxpa2FyanVuIENoYWRhbGFwYWthOyBkYXZpZC5ibGFja0BlbWMuY29tOyBzdG9ybUBp ZXRmLm9yZw0KPiA+Pj4+IFN1YmplY3Q6IFJFOiBpU0NTSSBkcmFmdHMgLSBUMTAgYW5kIFQxMSBy ZWZlcmVuY2VzDQo+ID4+Pj4NCj4gPj4+PiBJbnRlcmVzdGluZywgSSBqdXN0IGNhbWUgYWNyb3Nz IGEgcXVlc3Rpb24gaW4gYSByZWxhdGVkIGFyZWEgKG9ubHkgZm9yDQo+IFNBTS0yKS4NCj4gPj4+ Pg0KPiA+Pj4+IFJGQzM3MjAgd2FzIHdyaXR0ZW4gaW4gYmV0d2VlbiBTQU0tMiBhbmQgU0FNLTM7 IHNvbWUgY29uY2VwdHMgd2VyZSBkcmF3bg0KPiBmcm9tDQo+ID4+Pj4gU0FNLTIsIGFuZCBzb21l IGZyb20gU0FNLTMuDQo+ID4+Pj4NCj4gPj4+PiBXZSBoYXZlIHJlZmVyZW5jZXMgdG8gSV9UIE5F WFVTIExPU1MgZXZlbnRzIGluIFJGQzM3MjAuICBUaGUgSV9UIE5leHVzDQo+IGxvc3MNCj4gPj4+ PiBjb25jZXB0IGRvZXMgbm90IGV4aXN0IGluIFNBTS0yLiAgU0FNLTIgZG9lcyBpbmNsdWRlIHJl ZmVyZW5jZXMgdG8NCj4gUkVTRVJWRSBhbmQNCj4gPj4+PiBSRUxFQVNFIGNvbW1hbmRzLiAgU0FN LTMgaW5jbHVkZXMgSV9UIE5leHVzIGxvc3MsIGJ1dCBpdCBoYXMgbm8gcmVmZXJlbmNlDQo+IHRv DQo+ID4+Pj4gUkVTRVJWRSBzdGF0ZS4gIFRoZXJlIGFyZSBhbGx1c2lvbnMgaW4gUkZDMzcyMCBh Ym91dCB0aGUgaW50ZXJzZWN0aW9uIG9mDQo+ID4+Pj4gUkVTRVJWRS9SRUxFQVNFIGFuZCBJX1Qg TkVYVVMgTE9TUywgYnV0IHRoZXJlIGlzIG5vdGhpbmcgZXhwbGljaXQuDQo+IFJGQzM3MjANCj4g Pj4+PiByZWZlcmVuY2VzIG90aGVyIGRvY3VtZW50cyAoU0FNMiwgU0FNMywgU0JDLCBTUEMzKSwg bm9uZSBvZiB3aGljaCBhZGRyZXNzDQo+IHRoZQ0KPiA+Pj4+IHNpdHVhdGlvbi4NCj4gPj4+Pg0K PiA+Pj4+IENvbnNpZGVyIHRleHQgc3VjaCBhcyB0aGUgZm9sbG93aW5nOg0KPiA+Pj4+IDQuNC4z LjEuIElfVCBOZXh1cyBTdGF0ZQ0KPiA+Pj4+DQo+ID4+Pj4gIENlcnRhaW4gbmV4dXMgcmVsYXRp b25zaGlwcyBjb250YWluIGFuIGV4cGxpY2l0IHN0YXRlIChlLmcuLA0KPiA+Pj4+ICBpbml0aWF0 b3Itc3BlY2lmaWMgbW9kZSBwYWdlcykgdGhhdCBtYXkgbmVlZCB0byBiZSBwcmVzZXJ2ZWQgYnkN Cj4gPj4+PiAgdGhlIGRldmljZSBzZXJ2ZXIgW1NBTTJdIGluIGEgbG9naWNhbCB1bml0IHRocm91 Z2ggY2hhbmdlcyBvcg0KPiA+Pj4+ICBmYWlsdXJlcyBpbiB0aGUgaVNDU0kgbGF5ZXIgKGUuZy4s IHNlc3Npb24gZmFpbHVyZXMpLiBJbiBvcmRlciBmb3INCj4gPj4+PiAgdGhhdCBzdGF0ZSB0byBi ZSByZXN0b3JlZCwgdGhlIGlTQ1NJIGluaXRpYXRvciBzaG91bGQgcmVlc3RhYmxpc2gNCj4gPj4+ PiAgaXRzIHNlc3Npb24gKHJlLWxvZ2luKSB0byB0aGUgc2FtZSBUYXJnZXQgUG9ydGFsIEdyb3Vw IHVzaW5nIHRoZQ0KPiA+Pj4+ICBwcmV2aW91cyBJU0lELiBUaGF0IGlzLCBpdCBzaG91bGQgcGVy Zm9ybSBzZXNzaW9uIHJlY292ZXJ5IGFzDQo+ID4+Pj4gIGRlc2NyaWJlZCBpbiBDaGFwdGVyIDYu IFRoaXMgaXMgYmVjYXVzZSB0aGUgU0NTSSBpbml0aWF0b3IgcG9ydA0KPiA+Pj4+ICBpZGVudGlm aWVyIGFuZCB0aGUgU0NTSSB0YXJnZXQgcG9ydCBpZGVudGlmaWVyIChvciByZWxhdGl2ZSB0YXJn ZXQNCj4gPj4+PiAgcG9ydCkgZm9ybSB0aGUgZGF0dW0gdGhhdCB0aGUgU0NTSSBsb2dpY2FsIHVu aXQgZGV2aWNlIHNlcnZlciB1c2VzDQo+ID4+Pj4gIHRvIGlkZW50aWZ5IHRoZSBJX1QgbmV4dXMu DQo+ID4+Pj4NCj4gPj4+PiBJcyBSRVNFUlZFIHN0YXRlIC0gZXhwbGljaXQgc3RhdGU/ICBXaGVu IGFuIElfVCBuZXh1cyBpcyByZWVzdGFibGlzaGVkLA0KPiA+Pj4+IGluaXRpYXRvci1zcGVjaWZp YyBtb2RlIHBhZ2VzIGFyZSBwcmVzZXJ2ZWQuICBUaGF0IGltcGxpZXMgdGhhdCBhbiBJX1QNCj4g bmV4dXMNCj4gPj4+PiBsb3NzIHNob3VsZCBiZSB0cmVhdGVkIGluIHRoZSB3YXkgdGhhdCBTQU0t MiBkZXNjcmliZXMgdmlhIENMRUFSIFRBU0sgU0VUDQo+ID4+Pj4gKHdoZXJlIG1vZGUgcGFnZXMg YW5kIFJFU0VSVkUgc3RhdGUgYXJlIHByZXNlcnZlZCkuICBJZiBhIHNlc3Npb24NCj4gPj4+PiBy ZWVzdGFibGlzaG1lbnQgY2F1c2VkIHNhdmVkIG1vZGUgcGFnZXMgdG8gYmUgcmVzdG9yZWQgcmF0 aGVyIHRoYW4gdGhlDQo+ID4+Pj4gZXN0YWJsaXNoZWQgaW5pdGlhdG9yLXNwZWNpZmljIG1vZGUg cGFnZXMgdG8gYmUgcmVzdG9yZWQsIHRoZW4gaXQgd291bGQNCj4gYmUNCj4gPj4+PiBsaWtlIGEg TFUgcmVzZXQgKHdoZXJlIHRoZSBSRVNFUlZFIHN0YXRlIGlzIGxvc3QpLiAgSXMgUkVTRVJWRSBz dGF0ZQ0KPiBleHBsaWNpdA0KPiA+Pj4+IHN0YXRlIHRoYXQgaXMgdG8gYmUgcHJlc2VydmVkPw0K PiA+Pj4+DQo+ID4+Pj4gQXBwZW5kaXggRSBjb3ZlcnMgY2xlYXJpbmcgZWZmZWN0cyBkdXJpbmcg dmFyaW91cyBldmVudHMsIGJ1dCBpdCBuZXZlcg0KPiA+Pj4+IG1lbnRpb25zIFJFU0VSVkUgc3Rh dGUuDQo+ID4+Pj4NCj4gPj4+PiBTZWN0aW9uIEUuMiBjb250YWlucyB0aGUgZm9sbG93aW5nIHRl eHQ6DQo+ID4+Pj4NCj4gPj4+PiAgVGhlIG9ubHkgaVNDU0kgcHJvdG9jb2wgYWN0aW9uIHRoYXQg Y2FuIGVmZmVjdCBjbGVhcmluZyBhY3Rpb25zIG9uDQo+ID4+Pj4gIFNDU0kgb2JqZWN0cyBpcyB0 aGUgIklfVCBuZXh1cyBsb3NzIiBub3RpZmljYXRpb24gKFNlY3Rpb24gNi4zLjUuMQ0KPiA+Pj4+ ICBMb3NzIG9mIE5leHVzIG5vdGlmaWNhdGlvbikuIFtTUEMzXSBkZXNjcmliZXMgdGhlIGNsZWFy aW5nIGVmZmVjdHMNCj4gPj4+PiAgb2YgdGhpcyBub3RpZmljYXRpb24gb24gYSB2YXJpZXR5IG9m IFNDU0kgYXR0cmlidXRlcy4gSW4gYWRkaXRpb24sDQo+ID4+Pj4gIFNDU0kgc3RhbmRhcmRzIGRv Y3VtZW50cyAoc3VjaCBhcyBbU0FNMl0gYW5kIFtTQkNdKSBkZWZpbmUNCj4gPj4+PiAgYWRkaXRp b25hbCBjbGVhcmluZyBhY3Rpb25zIHRoYXQgbWF5IHRha2UgcGxhY2UgZm9yIHNldmVyYWwgU0NT SQ0KPiA+Pj4+ICBvYmplY3RzIG9uIFNDU0kgZXZlbnRzIHN1Y2ggYXMgTFUgcmVzZXRzIGFuZCBw b3dlci1vbiByZXNldHMuDQo+ID4+Pj4NCj4gPj4+PiBJIGNvdWxkIG5vdCBmaW5kIGFueSB0ZXh0 IGluIFNQQzMgdGhhdCBkZXNjcmliZXMgdGhlc2UgY2xlYXJpbmcgZWZmZWN0cw0KPiAoSQ0KPiA+ Pj4+IGRvbid0IGtub3cgaWYgdGhpcyBTUEMzIHJlZmVyZW5jZSBpcyBpbmNvcnJlY3QsIGJ1dCBh dCBhIG1pbmltdW0sIGl0IGlzDQo+ID4+Pj4gaW5lZmZlY3RpdmUpLiAgVGhlIFNBTTIgdGV4dCBm b3IgTFUgcmVzZXRzIGFuZCBwb3dlci1vbiByZXNldHMgaXMgZWFzaWx5DQo+ID4+Pj4gZm91bmQs IGFuZCBjbGVhcmx5IHNheXMgdGhhdCBhbnkgUkVTRVJWRSBpcyByZWxlYXNlZCBpbiB0aG9zZSBj YXNlcy4gIEFzDQo+IGFuDQo+ID4+Pj4gYXNpZGUsIHRoZSByZWZlcmVuY2UgdG8gc2VjdGlvbiA2 LjMuNS4xIGlzIGEgY2lyY3VsYXIgcmVmZXJlbmNlKS4gIFRoZQ0KPiB0ZXh0IGluDQo+ID4+Pj4g RS4yIGdvZXMgb246DQo+ID4+Pj4NCj4gPj4+PiAgU2luY2UgaVNDU0kgZGVmaW5lcyBhIHRhcmdl dCBjb2xkIHJlc2V0IGFzIGEgcHJvdG9jb2wtZXF1aXZhbGVudA0KPiA+Pj4+ICB0byBhIHRhcmdl dCBwb3dlci1jeWNsZSwgdGhlIGlTQ1NJIHRhcmdldCBjb2xkIHJlc2V0IG11c3QgYWxzbyBiZQ0K PiA+Pj4+ICBjb25zaWRlcmVkIGFzIHRoZSBwb3dlci1vbiByZXNldCBldmVudCBpbiBpbnRlcnBy ZXRpbmcgdGhlIGFjdGlvbnMNCj4gPj4+PiAgZGVmaW5lZCBpbiB0aGUgU0NTSSBzdGFuZGFyZHMu DQo+ID4+Pj4NCj4gPj4+PiAgV2hlbiB0aGUgaVNDU0kgc2Vzc2lvbiBpcyByZWNvbnN0cnVjdGVk IChiZXR3ZWVuIHRoZSBzYW1lIFNDU0kNCj4gPj4+PiAgcG9ydHMgd2l0aCB0aGUgc2FtZSBuZXh1 cyBpZGVudGlmaWVyKSByZWVzdGFibGlzaGluZyB0aGUgc2FtZSBJX1QNCj4gPj4+PiAgbmV4dXMs IGFsbCBTQ1NJIG9iamVjdHMgdGhhdCBhcmUgZGVmaW5lZCB0byBub3QgY2xlYXIgb24gdGhlICJJ X1QNCj4gPj4+PiAgbmV4dXMgbG9zcyIgbm90aWZpY2F0aW9uIGV2ZW50LCBzdWNoIGFzIHBlcnNp c3RlbnQgcmVzZXJ2YXRpb25zLA0KPiA+Pj4+ICBhcmUgYXV0b21hdGljYWxseSBhc3NvY2lhdGVk IHRvIHRoaXMgbmV3IHNlc3Npb24uDQo+ID4+Pj4NCj4gPj4+PiBTbywgaXQgaXMgY2xlYXIgd2hh dCBoYXBwZW5zIHRvIFBFUlNJU1RFTlQgUkVTRVJWRSBzdGF0ZSwgYnV0IG5vdCBzbw0KPiBjbGVh cg0KPiA+Pj4+IGFib3V0IFJFU0VSVkUgc3RhdGUuICBJdCBhbHNvIHN0YXRlcyB0aGF0IGFsbCBT Q1NJIG9iamVjdHMgdGhhdCBhcmUNCj4gZGVmaW5lZCB0bw0KPiA+Pj4+IG5vdCBjbGVhciBvbiB0 aGUgSV9UIG5leHVzIGxvc3MgYXJlIHJldGFpbmVkLiAgQnV0IGFnYWluLCB0aGUgb25seSBsaXN0 IEkNCj4gY2FuDQo+ID4+Pj4gZmluZCBvZiB3aGF0IGlzIGNsZWFyZWQgaXMgaGVyZSBpbiB0aGlz IEFwcGVuZGl4LCBzaW5jZSBTQU0tMiwgU0FNLTMsDQo+IFNBTS00DQo+ID4+Pj4gYXJlIGFsbCBz aWxlbnQgb24gdGhlIGludGVyYWN0aW9uIGJldHdlZW4gUkVTRVJWRSBhbmQgSV9UIE5leHVzIGxv c3MuDQo+ID4+Pj4NCj4gPj4+PiBUMTAgaGFzIG5ldmVyIGhhZCB0byBkZWFsIHdpdGggdGhpcyBi ZWNhdXNlIHRoZXkgcmVtb3ZlZCBSRVNFUlZFIGJlZm9yZQ0KPiB0aGV5DQo+ID4+Pj4gYWRkZWQg SV9UIE5leHVzIGxvc3MuICBIb3dldmVyLCBpU0NTSSBkaWQgbm90IGRvIHRoYXQ7IHdlIGRpc2N1 c3MgQk9USCwNCj4gYnV0IHdlDQo+ID4+Pj4gZmFpbCB0byBkZXNjcmliZSBob3cgdGhleSBpbnRl cmFjdC4gIGlTQ1NJIGFsc28gaGFzIHRoZSBmZWF0dXJlIG9mDQo+ID4+Pj4gcmVlc3RhYmxpc2ht ZW50IG9mIGEgc2Vzc2lvbiB0aGF0IGRvZXMgbm90IGV4aXN0IGluIFNBTS0yIChTQU0tMiB3YXMN Cj4gc3RpbGwNCj4gPj4+PiBwYXJhbGxlbCBTQ1NJIGZvY3VzZWQsIHdoZXJlIGNvbm5lY3Rpb25z IGRpZG4ndCBleGlzdCkuICBTbyBhZ2FpbiwgaWYNCj4gc2Vzc2lvbg0KPiA+Pj4+IHJlZXN0YWJs aXNobWVudCBpcyBleGFtaW5lZCBpbiB0aGUgbGlnaHQgb2YgcGFyYWxsZWwgU0NTSSwgdGhlIGlu dGVudA0KPiB3b3VsZCBiZQ0KPiA+Pj4+IHRvIHJldGFpbiB0aGUgUkVTRVJWRSBzdGF0ZS4gIEhl cmUgaXMgYW5vdGhlciBzZWN0aW9uIG9uIHRoZSB0b3BpYzoNCj4gPj4+Pg0KPiA+Pj4+IDYuMy41 LjEuIExvc3Mgb2YgTmV4dXMgTm90aWZpY2F0aW9uDQo+ID4+Pj4NCj4gPj4+PiAgVGhlIGlTQ1NJ IGxheWVyIHByb3ZpZGVzIHRoZSBTQ1NJIGxheWVyIHdpdGggdGhlICJJX1QgbmV4dXMgbG9zcyIN Cj4gPj4+PiAgbm90aWZpY2F0aW9uIHdoZW4gYW55IG9uZSBvZiB0aGUgZm9sbG93aW5nIGV2ZW50 cyBoYXBwZW5zOg0KPiA+Pj4+DQo+ID4+Pj4gICAgIFN1Y2Nlc3NmdWwgY29tcGxldGlvbiBvZiBz ZXNzaW9uIHJlaW5zdGF0ZW1lbnQuDQo+ID4+Pj4gICAgIFNlc3Npb24gY2xvc3VyZSBldmVudC4N Cj4gPj4+PiAgICAgU2Vzc2lvbiB0aW1lb3V0IGV2ZW50Lg0KPiA+Pj4+DQo+ID4+Pj4gIENlcnRh aW4gU0NTSSBvYmplY3QgY2xlYXJpbmcgYWN0aW9ucyBtYXkgcmVzdWx0IGR1ZSB0byB0aGUNCj4g Pj4+PiAgbm90aWZpY2F0aW9uIGluIHRoZSBTQ1NJIGVuZCBub2RlcywgYXMgZG9jdW1lbnRlZCBp biBBcHBlbmRpeCBGLiAtDQo+ID4+Pj4gIENsZWFyaW5nIEVmZmVjdHMgb2YgVmFyaW91cyBFdmVu dHMgb24gVGFyZ2V0cy4NCj4gPj4+Pg0KPiA+Pj4+IEZXSVcgLSB0aGUgcmVmZXJlbmNlIGFib3Zl IGlzIGluY29ycmVjdCAtIHRoZXJlIGlzIG5vIEFwcGVuZGl4IEYgaW4gdGhlDQo+ID4+Pj4gY29u c29saWRhdGVkIGRyYWZ0IC0gdGhpcyBzaG91bGQgYmUgQXBwZW5kaXggRSAoYW5kIGFsc28sIEJU VyAtIHRoaXMgaXMgYQ0KPiA+Pj4+IGNpcmN1bGFyIHJlZmVyZW5jZSkuDQo+ID4+Pj4NCj4gPj4+ PiBTbyBhbnl3YXksIGl0IHNlZW1zIGxpa2UgdGhlIGludGVudCB3YXMgdGhhdCBzZXNzaW9uIHJl aW5zdGF0ZW1lbnQgd291bGQNCj4gPj4+PiBwcmVzZXJ2ZSB0aGUgUkVTRVJWRSBzdGF0ZSBhY3Jv c3MgdGhlIElfVCBOZXh1cyBsb3NzIGV2ZW50Lg0KPiA+Pj4+DQo+ID4+Pj4gU28gYXNpZGUgZnJv bSB0aGUgaW5jb3JyZWN0IHJlZmVyZW5jZSAoYW5kIGNpcmN1bGFyIG5hdHVyZSBvZiBpdCksIGFu eQ0KPiA+Pj4+IHRob3VnaHRzIG9uIHdoZXRoZXIgd2Ugc2hvdWxkIGdldCBleHBsaWNpdCBhYm91 dCB0aGlzIHNpdHVhdGlvbj8gIFNob3VsZA0KPiB3ZQ0KPiA+Pj4+IHJlbWFpbiBzaWxlbnQ7IHNo b3VsZCB3ZSB0d2VhayBBcHBlbmRpeCBFIGFsb25nIHRoZXNlIGxpbmVzIChvciBvdGhlcg0KPiBs aW5lcw0KPiA+Pj4+IHRoYXQgb3RoZXJzIG1pZ2h0IHN1Z2dlc3QpOg0KPiA+Pj4+DQo+ID4+Pj4g NC40LjMuMS4gSV9UIE5leHVzIFN0YXRlDQo+ID4+Pj4NCj4gPj4+PiAgQ2VydGFpbiBuZXh1cyBy ZWxhdGlvbnNoaXBzIGNvbnRhaW4gYW4gZXhwbGljaXQgc3RhdGUgKGUuZy4sDQo+ID4+Pj4gIGlu aXRpYXRvci1zcGVjaWZpYyBtb2RlIHBhZ2VzLCBhbmQgUkVTRVJWRSBzdGF0ZSkgdGhhdCBtYXkg bmVlZCB0byBiZQ0KPiA+Pj4+IHByZXNlcnZlZCBieQ0KPiA+Pj4+ICB0aGUgZGV2aWNlIHNlcnZl ciBbU0FNMl0gaW4gYSBsb2dpY2FsIHVuaXQgdGhyb3VnaCBjaGFuZ2VzIG9yDQo+ID4+Pj4gIGZh aWx1cmVzIGluIHRoZSBpU0NTSSBsYXllciAoZS5nLiwgc2Vzc2lvbiBmYWlsdXJlcykuLi4uDQo+ ID4+Pj4NCj4gPj4+PiAgICAgIEZyZWQgS25pZ2h0DQo+ID4+Pj4NCj4gPj4+Pg0KPiA+Pj4+DQo+ ID4+Pj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPj4+PiBGcm9tOiBzdG9ybS1ib3Vu Y2VzQGlldGYub3JnIFttYWlsdG86c3Rvcm0tYm91bmNlc0BpZXRmLm9yZ10gT24gQmVoYWxmIE9m DQo+ID4+Pj4gTWFsbGlrYXJqdW4gQ2hhZGFsYXBha2ENCj4gPj4+PiBTZW50OiBTdW5kYXksIFNl cHRlbWJlciAyMywgMjAxMiAxOjM0IEFNDQo+ID4+Pj4gVG86IGRhdmlkLmJsYWNrQGVtYy5jb207 IHN0b3JtQGlldGYub3JnDQo+ID4+Pj4gU3ViamVjdDogUmU6IFtzdG9ybV0gaVNDU0kgZHJhZnRz IC0gVDEwIGFuZCBUMTEgcmVmZXJlbmNlcw0KPiA+Pj4+DQo+ID4+Pj4gSGkgRGF2aWQsDQo+ID4+ Pj4NCj4gPj4+PiBJIGhhdmUgbm90aWNlZCB0aGlzIGNvbW1lbnQgYXMgSSBhbSB3b3JraW5nIHRo cm91Z2ggYWxsIExhc3QgQ2FsbA0KPiBjb21tZW50cy4NCj4gPj4+Pg0KPiA+Pj4+IFRoZSByZWZl cmVuY2VzIHRvIFNBTS0zIGFuZCBTQU0tNCBhcmUga2V5ZWQgdG8gdGhpcyBwYXJhZ3JhcGggaW4g U2VjdGlvbg0KPiAxMy45Og0KPiA+Pj4+DQo+ID4+Pj4gIltTQU0yXSBhbmQgW1NBTTNdIHNwZWNp ZmljYXRpb25zIG5vdGUgaW4gdGhlaXIgaW5mb3JtYXRpdmUgdGV4dCB0aGF0DQo+IFRQR1QNCj4g Pj4+PiB2YWx1ZSBzaG91bGQgYmUgbm9uLXplcm8sIG5vdGUgdGhhdCBpdCBpcyBpbmNvcnJlY3Qu ICBBIHplcm8gdmFsdWUgaXMNCj4gYWxsb3dlZA0KPiA+Pj4+IGFzIGEgbGVnYWwgdmFsdWUgZm9y IFRQR1QuIFRoaXMgZGlzY3JlcGFuY3kgY3VycmVudGx5IHN0YW5kcyBjb3JyZWN0ZWQgaW4NCj4g Pj4+PiBbU0FNNF0uIg0KPiA+Pj4+DQo+ID4+Pj4gQWRkaXRpb25hbGx5LCBTQU00IHJlZmVyZW5j ZSBpcyBhbHNvIHVzZWQgaW4gY2l0aW5nIHRoZSBuZXcgQVNDL0FTQ1ENCj4gY29kZXMNCj4gPj4+ PiB0aGF0IGl0IGRlZmluZXMsIGluIFNlY3Rpb24gMTEuMTQuNSBvZiB0aGUgZHJhZnQuDQo+ID4+ Pj4NCj4gPj4+PiBTbywgSSBiZWxpZXZlIHJlZmVyZW5jZXMgdG8gYWxsIHRocmVlIFNBTSBzcGVj cyB3b3VsZCBjb250aW51ZSB0byBiZQ0KPiA+Pj4+IGFwcHJvcHJpYXRlLg0KPiA+Pj4+DQo+ID4+ Pj4gVGhlIGN1cnJlbnQgZHJhZnQgcmVmZXJlbmNlcyBGQy1GUyBpbiBkaXNjdXNzaW5nIHRoZSBO QUEgZm9ybWF0IGluIGEgZmV3DQo+ID4+Pj4gcGxhY2VzLCBzbyB0aGUgY3VycmVudCByZWZlcmVu Y2UgYXQgdGhlIGVuZCBpcyBpbnRlbnRpb25hbGx5IHRoZSBzYW1lLiBJcw0KPiB0aGUNCj4gPj4+ PiBvbGRlciByZWZlcmVuY2UgYSBwcm9ibGVtPyBJZiBzbywgSSBzdXBwb3NlIHdlIGNhbiByZXBs YWNlIGFsbCByZWZlcmVuY2VzDQo+IHRvDQo+ID4+Pj4gRkMtRlMtNCwgYnV0IHRoYXQgc2VlbXMg dG8gYmUgc3RpbGwgaW4gZWFybHkgc3RhZ2VzKDAuMSBzcGVjIHBvc3RlZCBvbg0KPiBBcHJpbA0K PiA+Pj4+IDIwMTIpLCBzbyBwZXJoYXBzIHdlIHNob3VsZCB1c2UgRkMtRlMzPw0KPiA+Pj4+DQo+ ID4+Pj4gVGhhbmtzLg0KPiA+Pj4+DQo+ID4+Pj4gTWFsbGlrYXJqdW4NCj4gPj4+Pg0KPiA+Pj4+ DQo+ID4+Pj4NCj4gPj4+Pg0KPiA+Pj4+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+ID4+ Pj4gRnJvbTogc3Rvcm0tYm91bmNlc0BpZXRmLm9yZyBbbWFpbHRvOnN0b3JtLWJvdW5jZXNAaWV0 Zi5vcmddIE9uIEJlaGFsZiBPZg0KPiA+Pj4+IGRhdmlkLmJsYWNrQGVtYy5jb20NCj4gPj4+PiBT ZW50OiBUaHVyc2RheSwgSnVseSAyNiwgMjAxMiA2OjQ3IEFNDQo+ID4+Pj4gVG86IHN0b3JtQGll dGYub3JnDQo+ID4+Pj4gU3ViamVjdDogW3N0b3JtXSBpU0NTSSBkcmFmdHMgLSBUMTAgYW5kIFQx MSByZWZlcmVuY2VzDQo+ID4+Pj4NCj4gPj4+PiBJbiBtYWtpbmcgdXAgdGhlIGxpc3RzIG9mIFQx MCBhbmQgVDExIGRvY3VtZW50cyB0aGF0IGFyZSBjdXJyZW50bHkNCj4gcmVmZXJlbmNlZCwNCj4g Pj4+PiBzb21lIG9mIHRob3NlIHJlZmVyZW5jZXMgZG9uJ3QgbG9vayByaWdodCB0byBtZSwgZm9y IGV4YW1wbGU6DQo+ID4+Pj4NCj4gPj4+PiAtIFQxMDogSSdtIG5vdCBzdXJlIHdoZXRoZXIgd2Ug bmVlZCB0byByZWZlcmVuY2UgU0FNLTIsIC0zIGFuZCAtNC4NCj4gPj4+PiAtIFQxMTogVGhlIEZD LUZTIHJlZmVyZW5jZSBpcyBvdXQgb2YgZGF0ZS4NCj4gPj4+Pg0KPiA+Pj4+IEFzIHBhcnQgb2Yg ZGVhbGluZyB3aXRoIHRoZSBjb21tZW50cyBmcm9tIElFVEYgTGFzdCBDYWxsLCB3ZSdsbCBuZWVk IHRvDQo+IGdvDQo+ID4+Pj4gb3ZlciBhbGwgb2YgdGhlIFQxMCBhbmQgVDExIHJlZmVyZW5jZXMg KGFuZCBzaG91bGQgY2hlY2sgYWxsIHRoZQ0KPiByZWZlcmVuY2VzKQ0KPiA+Pj4+IHRvIG1ha2Ug c3VyZSB0aGF0IHRoZXkncmUgdXAgdG8gZGF0ZSBhbmQgbmVjZXNzYXJ5Lg0KPiA+Pj4+DQo+ID4+ Pj4gVGhhbmtzLA0KPiA+Pj4+IC0tRGF2aWQNCj4gPj4+PiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ID4+Pj4gRGF2aWQgTC4gQmxhY2ssIERp c3Rpbmd1aXNoZWQgRW5naW5lZXINCj4gPj4+PiBFTUMgQ29ycG9yYXRpb24sIDE3NiBTb3V0aCBT dC4sIEhvcGtpbnRvbiwgTUEgIDAxNzQ4DQo+ID4+Pj4gKzEgKDUwOCkgMjkzLTc5NTMgICAgICAg ICAgICAgRkFYOiArMSAoNTA4KSAyOTMtNzc4Ng0KPiA+Pj4+IGRhdmlkLmJsYWNrQGVtYy5jb20g ICAgICAgIE1vYmlsZTogKzEgKDk3OCkgMzk0LTc3NTQNCj4gPj4+PiAtLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ID4+Pj4NCj4gPj4+PiBfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiA+Pj4+IHN0b3Jt IG1haWxpbmcgbGlzdA0KPiA+Pj4+IHN0b3JtQGlldGYub3JnDQo+ID4+Pj4gaHR0cHM6Ly93d3cu aWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9zdG9ybQ0KPiA+Pj4+DQo+ID4+Pj4NCj4gPj4+PiBf X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiA+Pj4+IHN0 b3JtIG1haWxpbmcgbGlzdA0KPiA+Pj4+IHN0b3JtQGlldGYub3JnDQo+ID4+Pj4gaHR0cHM6Ly93 d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9zdG9ybQ0KPiA+Pj4+DQo+ID4+Pj4NCj4gPj4+ IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+ID4+PiBz dG9ybSBtYWlsaW5nIGxpc3QNCj4gPj4+IHN0b3JtQGlldGYub3JnDQo+ID4+PiBodHRwczovL3d3 dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3N0b3JtDQo+ID4+IF9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+ID4+IHN0b3JtIG1haWxpbmcgbGlzdA0K PiA+PiBzdG9ybUBpZXRmLm9yZw0KPiA+PiBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xp c3RpbmZvL3N0b3JtDQo+ID4NCj4gPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fXw0KPiA+IHN0b3JtIG1haWxpbmcgbGlzdA0KPiA+IHN0b3JtQGlldGYub3Jn DQo+ID4gaHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9zdG9ybQ0KPiBfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiBzdG9ybSBtYWls aW5nIGxpc3QNCj4gc3Rvcm1AaWV0Zi5vcmcNCj4gaHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1h bi9saXN0aW5mby9zdG9ybQ0K From roweber@ieee.org Tue Sep 25 07:13:28 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CE30E21F8543 for ; Tue, 25 Sep 2012 07:13:28 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bAXzVFGKW1Vu for ; Tue, 25 Sep 2012 07:13:27 -0700 (PDT) Received: from vms173005pub.verizon.net (vms173005pub.verizon.net [206.46.173.5]) by ietfa.amsl.com (Postfix) with ESMTP id 2CD2221F841B for ; Tue, 25 Sep 2012 07:13:27 -0700 (PDT) MIME-version: 1.0 Content-transfer-encoding: 8BIT Content-type: text/plain; charset=UTF-8; format=flowed Received: from [192.168.1.3] ([unknown] [96.226.101.225]) by vms173005.mailsrvcs.net (Sun Java(tm) System Messaging Server 7u2-7.02 32bit (built Apr 16 2009)) with ESMTPA id <0MAW007PISTWPT30@vms173005.mailsrvcs.net> for storm@ietf.org; Tue, 25 Sep 2012 09:13:09 -0500 (CDT) Message-id: <5061BBFA.1040002@ieee.org> Date: Tue, 25 Sep 2012 09:13:14 -0500 From: Ralph Weber User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120907 Thunderbird/15.0.1 To: "storm@ietf.org" References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE791D4@MX15A.corp.emc.com> <5061A45E.7010607@ieee.org> In-reply-to: Subject: Re: [storm] RESERVE state - clearing effects X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 14:13:28 -0000 Julian, Do you know what really grates on the nerves? ... You know you are old when you visit a computer museum and discover that the last half of its contents represent the story of your life. That is what happened to me about ten years ago. All the best, .Ralph On 9/25/2012 7:50 AM, Julian Satran wrote: > Ralph, > > I understand. I missed the fact that the discussion was exclusively abour reserve/release. And i was referring to VMs (even I am too young to remeber VMS - that was during the trojan war wasn't it?). > > Julo > > Sent from my iPad > > On 25 בספט 2012, at 14:32, Ralph Weber wrote: > >> Persistent Reservations were invented because Reserve/Release failed >> to provide the quality of reservations services demanded by clusters. >> Clusters that depend on Reserve/Release reservations get exactly >> what they deserve. >> >> As for Open VMS, my recollection is that it had only one minor >> dependence on reservations. The most I remember needing from >> reservations was a way to ensure that disks stopped processing >> I/Os from an expelled cluster member. As long as all commands are >> aborted at the same time a reservation is released, the VMS I >> remember will not suffer any adverse effects. >> >> In any case, VMS has had two decades to migrate to Persistent >> Reservations for whatever they needed or developed a dependence >> upon. Even for OS engineers, that should be long enough for >> them to have matched their software to the available services. >> >> All the best, >> >> .Ralph >> >> P.S. David and Fred, I am not venturing into the topics you are >> discussing because I see nothing upon which I can provide knowledgeable >> additional comment. Whatever agreement you all reach will be okay ... >> as far as I can tell. >> >> On 9/24/2012 11:55 PM, Julian Satran wrote: >>> Did you consideir clusters or migrating vms? With those usually one expects targets to retain some state accross IT nexus loss. To avoid third party reservation cleanup SCSI (t10) may want to introduce a timeout mechanism associated with nexus loss (state is preserved for a while after nexus loss and the timeout values can be set). >>> >>> julo >>> >>> Sent from my iPad >>> >>> On 24 בספט 2012, at 18:42, "Black, David" wrote: >>> >>>> Fred, >>>> >>>> This is interesting ... >>>> >>>>> Here are some key points (from SAM 6.3.4 I_T nexus loss): >>>>> >>>>> 1) I_T nexus loss clears all ACA conditions; >>>>> 2) established a UA - one of the following is suggested: >>>>> >>>>> "If the logical unit retains state information for the I_T nexus that >>>>> is lost, its response to the subsequent I_T nexus >>>>> re-establishment for the logical unit should include establishing a >>>>> unit attention with an additional sense code set >>>>> to I_T NEXUS LOSS OCCURRED. >>>>> >>>>> If the logical unit does not retain state information for the I_T nexus >>>>> that is lost, it shall consider the subsequent >>>>> I_T nexus re-establishment, if any, as the formation of a new I_T nexus >>>>> for which there is no past history (e.g., >>>>> establish a unit attention with an additional sense code set to POWER ON >>>>> OCCURRED)." >>>> That "state information" has to identify the endpoints of the I_T Nexus >>>> - for iSCSI that means at least the ISID and TPGT. If that's gone, I >>>> think the RESERVE state is gone as well, in contrast to PERSISTENT >>>> RESERVE state. On RESERVE state, SPC-2 has this to say in 5.5.2 "The >>>> Reserve/Release management method": >>>> >>>> The reserve/release management method commands, RESERVE(6), >>>> RESERVE(10), RELEASE(6), and RELEASE(10) are used among multiple >>>> initiators that do not require operations to be protected across >>>> initiator failures (and subsequent hard resets). >>>> >>>> I think it's fair to regard I_T Nexus loss as an initiator failure. I >>>> also don't see any good way to do a 3rd-party release on a RESERVE >>>> reservation that persists after I_T Nexus loss. I also believe that >>>> initiator software that uses Reserve/Release does not expect >>>> reservations to persist across I_T Nexus loss. >>>> >>>> As to what needs to be stated in the iSCSI consolidated draft - the SPC >>>> reference has been updated to SPC-3 in which the Reserve/Release >>>> management method is obsolete, so there isn't any absolute requirement >>>> to cover that. OTOH, it might be reasonable to add a sentence to say >>>> that Reserve/Release state is implicit (e.g., as there's no way for an >>>> initiator to figure out which I_T Nexus holds a reservation), and that >>>> treating Reserve/Release state as indefinitely persistent after I_T >>>> Nexus Loss is a bad idea ("should not", but lower case) as it may >>>> require the initiator to reset the target device in order to clear out >>>> that state (issuing RELEASE commands isn't enough). This sort of text >>>> would require a reference to SPC-2. >>>> >>>> Thanks, >>>> --David >>>> >>>>> -----Original Message----- >>>>> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] >>>>> Sent: Monday, September 24, 2012 8:58 AM >>>>> To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org >>>>> Subject: RE: iSCSI drafts - T10 and T11 references >>>>> >>>>> FCP is also not explicit (it defers to the SCSI specs - which do not say). >>>>> The closest statements I can find still requires extrapolation: >>>>> >>>>> Here are some key points (from SAM 6.3.4 I_T nexus loss): >>>>> >>>>> 1) I_T nexus loss clears all ACA conditions; >>>>> 2) established a UA - one of the following is suggested: >>>>> >>>>> "If the logical unit retains state information for the I_T nexus that is lost, >>>>> its response to the subsequent I_T nexus >>>>> re-establishment for the logical unit should include establishing a unit >>>>> attention with an additional sense code set >>>>> to I_T NEXUS LOSS OCCURRED. >>>>> >>>>> If the logical unit does not retain state information for the I_T nexus that >>>>> is lost, it shall consider the subsequent >>>>> I_T nexus re-establishment, if any, as the formation of a new I_T nexus for >>>>> which there is no past history (e.g., >>>>> establish a unit attention with an additional sense code set to POWER ON >>>>> OCCURRED)." >>>>> >>>>> But, there is no definition of what it means to "retain state". The >>>>> conclusion I could jump to based on #1 above, is that RESERVE state is also >>>>> cleared (this action lines up more with LU RESET than CLEAR TASK SET); and >>>>> based on the POWER ON CONDITION UA of 29/01 (or 29/00 - both are power on >>>>> conditions) I could also assume that the RESERVE state is cleared; but based >>>>> on the UA of 29/07 (retain state), I could jump to the conclusion that RESERVE >>>>> state is retained. >>>>> >>>>> But these are all conclusions I've jump to with very limited basis in the >>>>> text. >>>>> >>>>> Based on what RFC3720 says about explicit state (e.g., mode pages), I would >>>>> assume that hosts would believe that every command they explicitly send (like >>>>> MODE SELECT or RESERVE) would have created explicit state and therefore the >>>>> host would assume RESERVE state is retained. It doesn't make sense to have >>>>> mode page values retained, but have RESERVE state lost. >>>>> >>>>> Certain nexus relationships contain an explicit state (e.g., >>>>> initiator-specific mode pages) that may need to be preserved by >>>>> the device server [SAM2] in a logical unit through changes or >>>>> failures in the iSCSI layer (e.g., session failures). >>>>> >>>>> It will be interesting to hear what a real host assumes about this situation. >>>>> >>>>> Fred >>>>> >>>>> -----Original Message----- >>>>> From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] >>>>> Sent: Sunday, September 23, 2012 6:35 PM >>>>> To: Knight, Frederick; david.black@emc.com; storm@ietf.org >>>>> Subject: RE: iSCSI drafts - T10 and T11 references >>>>> >>>>> Hi Fred, >>>>> >>>>> SPC-3 discusses clearing effects of I-T nexus loss in several places. >>>>> >>>>> Section 5.8.2.9 for ALUA >>>>> Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for background self- >>>>> test Section 5.6.1 on persistent reservations Section 5.13 on timestamps >>>>> ...... >>>>> >>>>> Thanks for pointing out the broken cross-reference - Appendix F instead of E. >>>>> I have just fixed it - and reviewed all other Appendix references elsewhere in >>>>> the draft. However, I do not see a circular reference problem per se - >>>>> Appendix E is correctly referencing Section 6.3.5 for definition of terms, >>>>> e.g. session continuation. And Section 6.3.5.1 is correctly deferring to >>>>> Appendix E when discussing clearing effects. Let me know if I'm overlooking >>>>> something here. >>>>> >>>>> Now, the reserve/release management state, :) : I agree the formal I_T nexus >>>>> loss event didn't exist in SAM-2, however the notion of a new I_T nexus state >>>>> did co-exist with a session-oriented I_T nexus well before iSCSI - FCP process >>>>> login (PRLO) comes to mind here. So I'd be curious about traditional FCP >>>>> handling of reserve/release management state. And I'm also checking internally >>>>> here at MS. From what I know now though, it does *appear* that reserve/release >>>>> management state should not be cleared with I_T nexus loss. >>>>> >>>>> Having said all that, note that reserve/release management state is squarely >>>>> in the SCSI territory. iSCSI spec has so far by design deferred to SCSI >>>>> standards in specifying clearing effects on SCSI objects - as Appendix E.2 >>>>> makes it clear. What we do specify is the clearing effects of SCSI actions, >>>>> e.g. LU reset, on iSCSI objects. >>>>> >>>>> Thanks. >>>>> >>>>> Mallikarjun >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] >>>>> Sent: Sunday, September 23, 2012 3:38 AM >>>>> To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org >>>>> Subject: RE: iSCSI drafts - T10 and T11 references >>>>> >>>>> Interesting, I just came across a question in a related area (only for SAM-2). >>>>> >>>>> RFC3720 was written in between SAM-2 and SAM-3; some concepts were drawn from >>>>> SAM-2, and some from SAM-3. >>>>> >>>>> We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus loss >>>>> concept does not exist in SAM-2. SAM-2 does include references to RESERVE and >>>>> RELEASE commands. SAM-3 includes I_T Nexus loss, but it has no reference to >>>>> RESERVE state. There are allusions in RFC3720 about the intersection of >>>>> RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit. RFC3720 >>>>> references other documents (SAM2, SAM3, SBC, SPC3), none of which address the >>>>> situation. >>>>> >>>>> Consider text such as the following: >>>>> 4.4.3.1. I_T Nexus State >>>>> >>>>> Certain nexus relationships contain an explicit state (e.g., >>>>> initiator-specific mode pages) that may need to be preserved by >>>>> the device server [SAM2] in a logical unit through changes or >>>>> failures in the iSCSI layer (e.g., session failures). In order for >>>>> that state to be restored, the iSCSI initiator should reestablish >>>>> its session (re-login) to the same Target Portal Group using the >>>>> previous ISID. That is, it should perform session recovery as >>>>> described in Chapter 6. This is because the SCSI initiator port >>>>> identifier and the SCSI target port identifier (or relative target >>>>> port) form the datum that the SCSI logical unit device server uses >>>>> to identify the I_T nexus. >>>>> >>>>> Is RESERVE state - explicit state? When an I_T nexus is reestablished, >>>>> initiator-specific mode pages are preserved. That implies that an I_T nexus >>>>> loss should be treated in the way that SAM-2 describes via CLEAR TASK SET >>>>> (where mode pages and RESERVE state are preserved). If a session >>>>> reestablishment caused saved mode pages to be restored rather than the >>>>> established initiator-specific mode pages to be restored, then it would be >>>>> like a LU reset (where the RESERVE state is lost). Is RESERVE state explicit >>>>> state that is to be preserved? >>>>> >>>>> Appendix E covers clearing effects during various events, but it never >>>>> mentions RESERVE state. >>>>> >>>>> Section E.2 contains the following text: >>>>> >>>>> The only iSCSI protocol action that can effect clearing actions on >>>>> SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 >>>>> Loss of Nexus notification). [SPC3] describes the clearing effects >>>>> of this notification on a variety of SCSI attributes. In addition, >>>>> SCSI standards documents (such as [SAM2] and [SBC]) define >>>>> additional clearing actions that may take place for several SCSI >>>>> objects on SCSI events such as LU resets and power-on resets. >>>>> >>>>> I could not find any text in SPC3 that describes these clearing effects (I >>>>> don't know if this SPC3 reference is incorrect, but at a minimum, it is >>>>> ineffective). The SAM2 text for LU resets and power-on resets is easily >>>>> found, and clearly says that any RESERVE is released in those cases. As an >>>>> aside, the reference to section 6.3.5.1 is a circular reference). The text in >>>>> E.2 goes on: >>>>> >>>>> Since iSCSI defines a target cold reset as a protocol-equivalent >>>>> to a target power-cycle, the iSCSI target cold reset must also be >>>>> considered as the power-on reset event in interpreting the actions >>>>> defined in the SCSI standards. >>>>> >>>>> When the iSCSI session is reconstructed (between the same SCSI >>>>> ports with the same nexus identifier) reestablishing the same I_T >>>>> nexus, all SCSI objects that are defined to not clear on the "I_T >>>>> nexus loss" notification event, such as persistent reservations, >>>>> are automatically associated to this new session. >>>>> >>>>> So, it is clear what happens to PERSISTENT RESERVE state, but not so clear >>>>> about RESERVE state. It also states that all SCSI objects that are defined to >>>>> not clear on the I_T nexus loss are retained. But again, the only list I can >>>>> find of what is cleared is here in this Appendix, since SAM-2, SAM-3, SAM-4 >>>>> are all silent on the interaction between RESERVE and I_T Nexus loss. >>>>> >>>>> T10 has never had to deal with this because they removed RESERVE before they >>>>> added I_T Nexus loss. However, iSCSI did not do that; we discuss BOTH, but we >>>>> fail to describe how they interact. iSCSI also has the feature of >>>>> reestablishment of a session that does not exist in SAM-2 (SAM-2 was still >>>>> parallel SCSI focused, where connections didn't exist). So again, if session >>>>> reestablishment is examined in the light of parallel SCSI, the intent would be >>>>> to retain the RESERVE state. Here is another section on the topic: >>>>> >>>>> 6.3.5.1. Loss of Nexus Notification >>>>> >>>>> The iSCSI layer provides the SCSI layer with the "I_T nexus loss" >>>>> notification when any one of the following events happens: >>>>> >>>>> Successful completion of session reinstatement. >>>>> Session closure event. >>>>> Session timeout event. >>>>> >>>>> Certain SCSI object clearing actions may result due to the >>>>> notification in the SCSI end nodes, as documented in Appendix F. - >>>>> Clearing Effects of Various Events on Targets. >>>>> >>>>> FWIW - the reference above is incorrect - there is no Appendix F in the >>>>> consolidated draft - this should be Appendix E (and also, BTW - this is a >>>>> circular reference). >>>>> >>>>> So anyway, it seems like the intent was that session reinstatement would >>>>> preserve the RESERVE state across the I_T Nexus loss event. >>>>> >>>>> So aside from the incorrect reference (and circular nature of it), any >>>>> thoughts on whether we should get explicit about this situation? Should we >>>>> remain silent; should we tweak Appendix E along these lines (or other lines >>>>> that others might suggest): >>>>> >>>>> 4.4.3.1. I_T Nexus State >>>>> >>>>> Certain nexus relationships contain an explicit state (e.g., >>>>> initiator-specific mode pages, and RESERVE state) that may need to be >>>>> preserved by >>>>> the device server [SAM2] in a logical unit through changes or >>>>> failures in the iSCSI layer (e.g., session failures).... >>>>> >>>>> Fred Knight >>>>> >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of >>>>> Mallikarjun Chadalapaka >>>>> Sent: Sunday, September 23, 2012 1:34 AM >>>>> To: david.black@emc.com; storm@ietf.org >>>>> Subject: Re: [storm] iSCSI drafts - T10 and T11 references >>>>> >>>>> Hi David, >>>>> >>>>> I have noticed this comment as I am working through all Last Call comments. >>>>> >>>>> The references to SAM-3 and SAM-4 are keyed to this paragraph in Section 13.9: >>>>> >>>>> "[SAM2] and [SAM3] specifications note in their informative text that TPGT >>>>> value should be non-zero, note that it is incorrect. A zero value is allowed >>>>> as a legal value for TPGT. This discrepancy currently stands corrected in >>>>> [SAM4]." >>>>> >>>>> Additionally, SAM4 reference is also used in citing the new ASC/ASCQ codes >>>>> that it defines, in Section 11.14.5 of the draft. >>>>> >>>>> So, I believe references to all three SAM specs would continue to be >>>>> appropriate. >>>>> >>>>> The current draft references FC-FS in discussing the NAA format in a few >>>>> places, so the current reference at the end is intentionally the same. Is the >>>>> older reference a problem? If so, I suppose we can replace all references to >>>>> FC-FS-4, but that seems to be still in early stages(0.1 spec posted on April >>>>> 2012), so perhaps we should use FC-FS3? >>>>> >>>>> Thanks. >>>>> >>>>> Mallikarjun >>>>> >>>>> >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of >>>>> david.black@emc.com >>>>> Sent: Thursday, July 26, 2012 6:47 AM >>>>> To: storm@ietf.org >>>>> Subject: [storm] iSCSI drafts - T10 and T11 references >>>>> >>>>> In making up the lists of T10 and T11 documents that are currently referenced, >>>>> some of those references don't look right to me, for example: >>>>> >>>>> - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. >>>>> - T11: The FC-FS reference is out of date. >>>>> >>>>> As part of dealing with the comments from IETF Last Call, we'll need to go >>>>> over all of the T10 and T11 references (and should check all the references) >>>>> to make sure that they're up to date and necessary. >>>>> >>>>> Thanks, >>>>> --David >>>>> ---------------------------------------------------- >>>>> David L. Black, Distinguished Engineer >>>>> EMC Corporation, 176 South St., Hopkinton, MA 01748 >>>>> +1 (508) 293-7953 FAX: +1 (508) 293-7786 >>>>> david.black@emc.com Mobile: +1 (978) 394-7754 >>>>> ---------------------------------------------------- >>>>> >>>>> _______________________________________________ >>>>> storm mailing list >>>>> storm@ietf.org >>>>> https://www.ietf.org/mailman/listinfo/storm >>>>> >>>>> >>>>> _______________________________________________ >>>>> storm mailing list >>>>> storm@ietf.org >>>>> https://www.ietf.org/mailman/listinfo/storm >>>>> >>>>> >>>> _______________________________________________ >>>> storm mailing list >>>> storm@ietf.org >>>> https://www.ietf.org/mailman/listinfo/storm >>> _______________________________________________ >>> storm mailing list >>> storm@ietf.org >>> https://www.ietf.org/mailman/listinfo/storm >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm > From milton_scritsmier@alumni.hmc.edu Mon Sep 24 08:59:52 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 65BCE21F87E3 for ; Mon, 24 Sep 2012 08:59:52 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.437 X-Spam-Level: X-Spam-Status: No, score=-0.437 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_NET=0.611, RDNS_NONE=0.1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GiWSV0ZED2Ay for ; Mon, 24 Sep 2012 08:59:52 -0700 (PDT) Received: from qmta08.emeryville.ca.mail.comcast.net (qmta08.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:43:76:96:30:80]) by ietfa.amsl.com (Postfix) with ESMTP id F054321F8628 for ; Mon, 24 Sep 2012 08:59:51 -0700 (PDT) Received: from omta08.emeryville.ca.mail.comcast.net ([76.96.30.12]) by qmta08.emeryville.ca.mail.comcast.net with comcast id 33jq1k0070FhH24A83zrfN; Mon, 24 Sep 2012 15:59:51 +0000 Received: from [192.168.11.6] ([71.196.153.254]) by omta08.emeryville.ca.mail.comcast.net with comcast id 33up1k00n5VbR1z8U3urNe; Mon, 24 Sep 2012 15:54:51 +0000 Message-ID: <50608249.8080803@alumni.hmc.edu> Date: Mon, 24 Sep 2012 09:54:49 -0600 From: Milton Scritsmier User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20120907 Thunderbird/15.0.1 MIME-Version: 1.0 To: storm@ietf.org References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> In-Reply-To: X-Enigmail-Version: 1.4.4 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Tue, 25 Sep 2012 07:38:20 -0700 Subject: Re: [storm] iSCSI drafts - T10 and T11 references X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Sep 2012 15:59:52 -0000 On 9/23/2012 4:38 AM, Knight, Frederick wrote: > > We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus loss concept does not exist in SAM-2. SAM-2 does include references to RESERVE and RELEASE commands. SAM-3 includes I_T Nexus loss, but it has no reference to RESERVE state. There are allusions in RFC3720 about the intersection of RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit. RFC3720 references other documents (SAM2, SAM3, SBC, SPC3), none of which address the situation. > The concepts of I_T NEXUS and RESERVE/RELEASE go all the way back to SCSI-2 in the mid-80s, on which I did a little work (see the spec at http://www.mit.edu/afs/sipb.mit.edu/contrib/doc/specs/protocol/scsi-2/s2-r10l.txt ). In SCSI-2, the I_T NEXUS concept was created to deal with disconnect/reconnect on the parallel SCSI bus and the idea that a command could be active on both the host and target even though the bus was disconnected or the target was talking to another initiator. The RESERVE condition was meant to survive anything except a bus reset or a device power cycle (back then it was not assumed that devices had flash memory or any means of storing RESERVE/RELEASE status between power cycles or after a bus reset). So for example, if the initiator aborted a command, or persistent parity errors on the bus prevented communications, any RESERVE condition was still in effect. If a target was reset or power cycled, it would notify the host via CHECK CONDITION of the reset on the next command, which would implicitly tell the host that any RESERVE condition was no longer in effect. Section 9.2.12.1 of the SCSI-2 spec gives the conditions which clear a RESERVE condition, and it's clear that these are rather "catastrophic" conditions. Thus it was always the intent that the RESERVE condition was meant to survive an I_T NEXUS loss, although that concept is not well-defined in SCSI-2. I don't know if these concepts were carried through beyond parallel buses in SCSI-3, but that was the original intent. From david.black@emc.com Tue Sep 25 08:44:52 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2127421F86AD for ; Tue, 25 Sep 2012 08:44:52 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FTkTMFbjn5mu for ; Tue, 25 Sep 2012 08:44:51 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 60FDF21F87D3 for ; Tue, 25 Sep 2012 08:44:51 -0700 (PDT) Received: from hop04-l1d11-si01.isus.emc.com (HOP04-L1D11-SI01.isus.emc.com [10.254.111.54]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PFinDg026504 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 25 Sep 2012 11:44:49 -0400 Received: from mailhub.lss.emc.com (mailhubhoprd03.lss.emc.com [10.254.221.145]) by hop04-l1d11-si01.isus.emc.com (RSA Interceptor) for ; Tue, 25 Sep 2012 11:44:34 -0400 Received: from mxhub15.corp.emc.com (mxhub15.corp.emc.com [128.222.70.236]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PFiYvv021948 for ; Tue, 25 Sep 2012 11:44:34 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub15.corp.emc.com ([128.222.70.236]) with mapi; Tue, 25 Sep 2012 11:44:34 -0400 From: "Black, David" To: "storm@ietf.org" Date: Tue, 25 Sep 2012 11:44:32 -0400 Thread-Topic: SPC-2 reserve/release: Proposed text Thread-Index: Ac2bNKguGPJW+zTbTTC2uDGdWEV2OA== Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE7946D@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: [storm] SPC-2 reserve/release: Proposed text X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 15:44:52 -0000 Here's an initial attempt to propose text to deal with the reserve/ release topic in the consolidated iSCSI draft. Please comment, correct, suggest edits, etc. The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: 4.4.3.1. I_T Nexus State Certain nexus relationships contain an explicit state (e.g., initiator-specific mode pages) that may need to be preserved by the device server [SAM2] in a logical unit through changes or failures in the iSCSI layer (e.g., session failures). In order for that state to be restored, the iSCSI initiator should reestablish its session (re-login) to the same Target Portal Group using the previous ISID. That is, it should perform session recovery as described in Chapter 6. This is because the SCSI initiator port identifier and the SCSI target port identifier (or relative target port) form the datum that the SCSI logical unit device server uses to identify the I_T nexus. ------------------------------- First, for clarity, the following change should be made in the above 4.4.3.1 text to better identify the recovery mechanism: OLD That is, it should perform session recovery as described in Chapter 6. NEW That is, it should recover the session via connection reinstatement as described in Section 6.3.4. END I believe that a lower-case "should" remains appropriate for this text. ----------------------------------- NEW TEXT (first draft): 4.4.3.2. I_T Nexus State: Reservations The state of an I_T Nexus includes SCSI reservation state. There are two reservation management methods defined in the SCSI standards, reserve/release reservations, based on the RESERVE and RELEASE commands [SPC2], and persistent reservations, based on the PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be used; persistent reservations SHOULD be used instead. The I_T Nexus state for persistent reservations is generally required to persist through changes and failures at the iSCSI layer that result in I_T Nexus failures, see [SPC3] for details and specific requirements. In contrast, [SPC2] does not specify detailed persistence requirements for reserve/release reservation state - nonetheless, when reserve/release reservations are supported by an iSCSI target, the preferred implementation approach is to preserve reserve/release reservation state along with other I_T Nexus state for iSCSI session recovery or continuation via connection reinstatement. Two additional caveats apply to reserve/release reservations: - Retention of reserve/release reservation state for an extended period of time may require a hard reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that state when connection reinstatement is not performed. The DefaultTime2Wait value can be used to limit this retention time period (see section 13.15). - Reserve/release reservations may not behave as expected when persistent reservations are also used on the same logical unit; see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE behavior" in [SPC4]. ----------------------- Reference impacts: both [SPC2] and [SPC3] need to be normative references, but [SPC4] can be an informative reference. ------------------------- I deliberately used "the preferred iSCSI implementation approach" wording for reserve/release reservation state preservation requirements, as SPC-2 is vague on this topic and I'm rather uncomfortable with placing a requirement as strong as a "SHOULD" on an obsolete mechanism that "SHOULD NOT" be used. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0 =A0=A0=A0=A0 FAX: +1 (508) 293-77= 86 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- From roweber@ieee.org Tue Sep 25 08:55:03 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 63A6021F879C for ; Tue, 25 Sep 2012 08:55:03 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PA6aLYOoRbSH for ; Tue, 25 Sep 2012 08:55:02 -0700 (PDT) Received: from vms173011pub.verizon.net (vms173011pub.verizon.net [206.46.173.11]) by ietfa.amsl.com (Postfix) with ESMTP id 769B121F84F5 for ; Tue, 25 Sep 2012 08:55:02 -0700 (PDT) Received: from [192.168.1.3] ([unknown] [96.226.101.225]) by vms173011.mailsrvcs.net (Sun Java(tm) System Messaging Server 7u2-7.02 32bit (built Apr 16 2009)) with ESMTPA id <0MAW00M71XJ2TE10@vms173011.mailsrvcs.net> for storm@ietf.org; Tue, 25 Sep 2012 10:54:38 -0500 (CDT) Message-id: <5061D3C3.40602@ieee.org> Date: Tue, 25 Sep 2012 10:54:44 -0500 From: Ralph Weber User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120907 Thunderbird/15.0.1 MIME-version: 1.0 To: storm@ietf.org References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <50608249.8080803@alumni.hmc.edu> In-reply-to: <50608249.8080803@alumni.hmc.edu> Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7bit Subject: Re: [storm] iSCSI drafts - T10 and T11 references X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 15:55:03 -0000 All of this is more-or-less true, but the problem at hand is I_T NEXUS LOSS ... a concept created long after SCSI-2 had graduated to SCSI-3. The first serial protocol with any widespread usage was Fibre Channel Protocol (FCP), and that protocol included a Login/Logout concept, which has a much longer lifespan that Disconnect/Reconnect. One frequent mapping of I_T NEXUS LOSS is to an FCP Logout (either intentional or forced from the outside). This mapping is common, but far from universal. Thus, the question becomes where to install I_T NEXUS LOSS (a concept never envisioned by SCSI-2 because it is nonsense on the parallel bus) in the iSCSI pantheon. The global-effects ladder usually looks like this, in order of increasingly bigger hammers. ... {old} CLEAR TASK SET [new] I_T NEXUS LOSS {old} LOGICAL UNIT RESET ... CLEAR TASK SET preserves Reserve/Release reservations. LOGICAL UNIT RESET clears them. David and Fred are discussing which way I_T NEXUS LOSS should swing on this pendulum. As noted earlier today, where these two gentlemen are is quite clear. To one who lacks sufficient real-world knowledge of the applicable use cases, the only option is a coin toss. Fortunately, Fred and David are better equipped than this author. All the best, .Ralph On 9/24/2012 10:54 AM, Milton Scritsmier wrote: > On 9/23/2012 4:38 AM, Knight, Frederick wrote: >> We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus loss concept does not exist in SAM-2. SAM-2 does include references to RESERVE and RELEASE commands. SAM-3 includes I_T Nexus loss, but it has no reference to RESERVE state. There are allusions in RFC3720 about the intersection of RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit. RFC3720 references other documents (SAM2, SAM3, SBC, SPC3), none of which address the situation. >> > The concepts of I_T NEXUS and RESERVE/RELEASE go all the way back to > SCSI-2 in the mid-80s, on which I did a little work (see the spec at > http://www.mit.edu/afs/sipb.mit.edu/contrib/doc/specs/protocol/scsi-2/s2-r10l.txt > ). In SCSI-2, the I_T NEXUS concept was created to deal with > disconnect/reconnect on the parallel SCSI bus and the idea that a > command could be active on both the host and target even though the bus > was disconnected or the target was talking to another initiator. The > RESERVE condition was meant to survive anything except a bus reset or a > device power cycle (back then it was not assumed that devices had flash > memory or any means of storing RESERVE/RELEASE status between power > cycles or after a bus reset). > > So for example, if the initiator aborted a command, or persistent parity > errors on the bus prevented communications, any RESERVE condition was > still in effect. If a target was reset or power cycled, it would notify > the host via CHECK CONDITION of the reset on the next command, which > would implicitly tell the host that any RESERVE condition was no longer > in effect. Section 9.2.12.1 of the SCSI-2 spec gives the conditions > which clear a RESERVE condition, and it's clear that these are rather > "catastrophic" conditions. Thus it was always the intent that the > RESERVE condition was meant to survive an I_T NEXUS loss, although that > concept is not well-defined in SCSI-2. > > I don't know if these concepts were carried through beyond parallel > buses in SCSI-3, but that was the original intent. > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > From roweber@ieee.org Tue Sep 25 09:57:56 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 655F11F0C9C for ; Tue, 25 Sep 2012 09:57:56 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PeHBfElEcGcp for ; Tue, 25 Sep 2012 09:57:55 -0700 (PDT) Received: from vms173021pub.verizon.net (vms173021pub.verizon.net [206.46.173.21]) by ietfa.amsl.com (Postfix) with ESMTP id 04A4D1F0C86 for ; Tue, 25 Sep 2012 09:57:55 -0700 (PDT) Received: from [192.168.1.3] ([unknown] [96.226.101.225]) by vms173021.mailsrvcs.net (Sun Java(tm) System Messaging Server 7u2-7.02 32bit (built Apr 16 2009)) with ESMTPA id <0MAX001FZ0FSV7EE@vms173021.mailsrvcs.net> for storm@ietf.org; Tue, 25 Sep 2012 11:57:29 -0500 (CDT) Message-id: <5061E27A.7000109@ieee.org> Date: Tue, 25 Sep 2012 11:57:30 -0500 From: Ralph Weber User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120907 Thunderbird/15.0.1 MIME-version: 1.0 To: storm@ietf.org References: <8D3D17ACE214DC429325B2B98F3AE7120DE7946D@MX15A.corp.emc.com> In-reply-to: <8D3D17ACE214DC429325B2B98F3AE7120DE7946D@MX15A.corp.emc.com> Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7bit Subject: Re: [storm] SPC-2 reserve/release: Proposed text X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 16:57:56 -0000 David, I believe a paragraph break is needed as shown in the following: > Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be > used; persistent reservations SHOULD be used instead. [break] The > I_T Nexus > state for persistent reservations is generally required ... The chances of confusion are high when a paragraph that begins with a discussion of reserve/release reservations continues into the persistent reservations territory. Also please be aware that Annex B of SPC-4 describes the means by which the use of reserve/release reservations can be adapted to employ persistent reservations. I am not sure this an appropriate informative reference for an RFC, but it seems worthy of mention on this reflector. All the best, .Ralph On 9/25/2012 10:44 AM, Black, David wrote: > > > Here's an initial attempt to propose text to deal with the reserve/ > release topic in the consolidated iSCSI draft. Please comment, > correct, suggest edits, etc. > > The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: > > 4.4.3.1. I_T Nexus State > > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages) that may need to be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures). In order for > that state to be restored, the iSCSI initiator should reestablish > its session (re-login) to the same Target Portal Group using the > previous ISID. That is, it should perform session recovery as > described in Chapter 6. This is because the SCSI initiator port > identifier and the SCSI target port identifier (or relative target > port) form the datum that the SCSI logical unit device server uses > to identify the I_T nexus. > > ------------------------------- > > First, for clarity, the following change should be made in the > above 4.4.3.1 text to better identify the recovery mechanism: > > OLD > That is, it should perform session recovery as > described in Chapter 6. > NEW > That is, it should recover the session via connection > reinstatement as described in Section 6.3.4. > END > > I believe that a lower-case "should" remains appropriate for this text. > > ----------------------------------- > > NEW TEXT (first draft): > > 4.4.3.2. I_T Nexus State: Reservations > > The state of an I_T Nexus includes SCSI reservation state. There > are two reservation management methods defined in the SCSI standards, > reserve/release reservations, based on the RESERVE and RELEASE > commands [SPC2], and persistent reservations, based on the PERSISTENT > RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. > > Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be > used; persistent reservations SHOULD be used instead. The I_T Nexus > state for persistent reservations is generally required to persist > through changes and failures at the iSCSI layer that result in I_T > Nexus failures, see [SPC3] for details and specific requirements. > > In contrast, [SPC2] does not specify detailed persistence > requirements for reserve/release reservation state - nonetheless, > when reserve/release reservations are supported by an iSCSI target, > the preferred implementation approach is to preserve reserve/release > reservation state along with other I_T Nexus state for iSCSI session > recovery or continuation via connection reinstatement. > > Two additional caveats apply to reserve/release reservations: > > - Retention of reserve/release reservation state for an extended > period of time may require a hard reset (e.g., LOGICAL UNIT RESET, > see section 11.5) in order to remove that state when connection > reinstatement is not performed. The DefaultTime2Wait value > can be used to limit this retention time period (see section 13.15). > > - Reserve/release reservations may not behave as expected when > persistent reservations are also used on the same logical unit; > see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE > behavior" in [SPC4]. > > ----------------------- > > Reference impacts: both [SPC2] and [SPC3] need to be normative > references, but [SPC4] can be an informative reference. > > ------------------------- > > I deliberately used "the preferred iSCSI implementation approach" > wording for reserve/release reservation state preservation requirements, > as SPC-2 is vague on this topic and I'm rather uncomfortable with placing > a requirement as strong as a "SHOULD" on an obsolete mechanism that > "SHOULD NOT" be used. > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > david.black@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > From cbm@chadalapaka.com Tue Sep 25 10:34:38 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3476B21F88C3 for ; Tue, 25 Sep 2012 10:34:38 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FEn32SRFIMqA for ; Tue, 25 Sep 2012 10:34:37 -0700 (PDT) Received: from va3outboundpool.messaging.microsoft.com (va3ehsobe002.messaging.microsoft.com [216.32.180.12]) by ietfa.amsl.com (Postfix) with ESMTP id B084221F887B for ; Tue, 25 Sep 2012 10:34:36 -0700 (PDT) Received: from mail48-va3-R.bigfish.com (10.7.14.248) by VA3EHSOBE009.bigfish.com (10.7.40.29) with Microsoft SMTP Server id 14.1.225.23; Tue, 25 Sep 2012 17:34:35 +0000 Received: from mail48-va3 (localhost [127.0.0.1]) by mail48-va3-R.bigfish.com (Postfix) with ESMTP id 794F5A009E; Tue, 25 Sep 2012 17:34:35 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.240.117; KIP:(null); UIP:(null); IPV:NLI; H:BL2PRD0610HT004.namprd06.prod.outlook.com; RD:none; EFVD:NLI X-SpamScore: -29 X-BigFish: PS-29(zz9371I146fI542M1432I4015I1447Id6f1izz1202h1d1ah1d2ahzz1033IL17326ah8275bh8275dhz2fh2a8h668h839hd25hf0ah107ah1288h12a5h12a9h12bdh137ah13b6h1155h) Received-SPF: pass (mail48-va3: domain of chadalapaka.com designates 157.56.240.117 as permitted sender) client-ip=157.56.240.117; envelope-from=cbm@chadalapaka.com; helo=BL2PRD0610HT004.namprd06.prod.outlook.com ; .outlook.com ; Received: from mail48-va3 (localhost.localdomain [127.0.0.1]) by mail48-va3 (MessageSwitch) id 1348594472428924_24040; Tue, 25 Sep 2012 17:34:32 +0000 (UTC) Received: from VA3EHSMHS028.bigfish.com (unknown [10.7.14.253]) by mail48-va3.bigfish.com (Postfix) with ESMTP id 62DF9140047; Tue, 25 Sep 2012 17:34:32 +0000 (UTC) Received: from BL2PRD0610HT004.namprd06.prod.outlook.com (157.56.240.117) by VA3EHSMHS028.bigfish.com (10.7.99.38) with Microsoft SMTP Server (TLS) id 14.1.225.23; Tue, 25 Sep 2012 17:34:30 +0000 Received: from BL2PRD0610MB361.namprd06.prod.outlook.com ([169.254.12.31]) by BL2PRD0610HT004.namprd06.prod.outlook.com ([10.255.101.39]) with mapi id 14.16.0190.008; Tue, 25 Sep 2012 17:34:29 +0000 From: Mallikarjun Chadalapaka To: "Black, David" , "Knight, Frederick" , "storm@ietf.org" Thread-Topic: iSCSI drafts - T10 and T11 references Thread-Index: Ac1rNRGDFlJk/LPPRq2a5Pa224AxSwuFP5jwAAlZtBAAPXLRYAA3F3tA Date: Tue, 25 Sep 2012 17:34:28 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79192@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE79192@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [131.107.174.24] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: chadalapaka.com Subject: Re: [storm] iSCSI drafts - T10 and T11 references X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 17:34:38 -0000 David, Thanks for the suggestions. I am largely in agreement. A couple of questions: >2) [SAM2] and [SAM3] ought to be normative references Don't know the related T10 history here, but I no longer see SAM-3 spec on = the T10 web site. Any insights? >[SAM-4] ought to be an informative reference Was the concept of I_T nexus loss notification introduced in SAM-3 or SAM-4= ? (I can't tell because I don't see SAM-3... If the notification were intro= duced in SAM-4, that could make SAM-4 a normative reference. SAM-2 does not formally discuss I_T nexus loss notification, other than an = oblique reference to the related check condition in clause 5.3.2, status pr= ecedence discussion. Thanks. Mallikarjun -----Original Message----- From: Black, David [mailto:david.black@emc.com]=20 Sent: Monday, September 24, 2012 8:37 AM To: Knight, Frederick; Mallikarjun Chadalapaka; storm@ietf.org Subject: RE: iSCSI drafts - T10 and T11 references Fred, Thank you for the discussion, particularly around the dependence on SAM-3. In light of that, and a few other things (e.g., a strong preference that th= e iSCSI consolidated draft not reference work in progress), my recommendati= ons on the T10 and T11 references are: --- T10 (SCSI) --- 1) [SBC2] and [SPC3] are the current published standards; those should be t= he correct references for the iSCSI consolidated draft. [SPC3] is clearly = normative, but I think [SBC2] is (and ought to be) informative - [SBC2] is = only cited once in what looks like an informative fashion in Appendix E.2: In addition, SCSI standards documents (such as [SAM2] and [SBC2]) define additional clearing actions that may take place for several SCSI objects on SCSI events such as LU resets and power-on resets. Moreover, there is no requirement to implement the SBC command set for all = iSCSI implementations, whereas the SPC command set is a requirement for obv= ious reasons (P =3D Primary). 2) [SAM2] and [SAM3] ought to be normative references, in that we're drawin= g some concepts from [SAM-3]. OTOH, [SAM-4] ought to be an informative ref= erence, as the TPGT text that is referred to in SAM-4: > "[SAM2] and [SAM3] specifications note in their informative text that=20 > TPGT value should be non-zero, note that it is incorrect. A zero=20 > value is allowed as a legal value for TPGT. This discrepancy currently=20 > stands corrected in [SAM4]." is an informative table footnote in an annex. Further, > Additionally, SAM4 reference is also used in citing the new ASC/ASCQ=20 > codes that it defines, in Section 11.14.5 of the draft. That new ASC/Q code's name does not occur in the body of SAM-4, so I would = remove that second citation of SAM-4. 3) The [SAS] reference needs to be expanded ;-). The current informative [= SAS] reference needs to be replaced by references to SAS-2.1, SPL and SPL A= mendment 1, see: http://www.t10.org/drafts.htm#SCSI3_SAS 4) OTOH, I recommend removing the [SRP] reference and related material, 'na= mely the definition of "SCSI RDMA Protocol" and the alternative expansion o= f the SRP acronym. The SRP-2 standards project to update SRP was abandoned= by T10 in 2005 due to lack of interest (see T10/05-097r0), leaving us with= a vintage-2002 SRP document. --- T11 (Fibre Channel) --- The FC-FS reference ought to be replaced by a reference to the current publ= ished FS-FS-3 standard. Thanks, --David > -----Original Message----- > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > Sent: Sunday, September 23, 2012 6:38 AM > To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references >=20 > Interesting, I just came across a question in a related area (only for SA= M-2). >=20 > RFC3720 was written in between SAM-2 and SAM-3; some concepts were=20 > drawn from SAM-2, and some from SAM-3. >=20 > We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus=20 > loss concept does not exist in SAM-2. SAM-2 does include references=20 > to RESERVE and RELEASE commands. SAM-3 includes I_T Nexus loss, but=20 > it has no reference to RESERVE state. There are allusions in RFC3720=20 > about the intersection of RESERVE/RELEASE and I_T NEXUS LOSS, but=20 > there is nothing explicit. RFC3720 references other documents (SAM2,=20 > SAM3, SBC, SPC3), none of which address the situation. >=20 > Consider text such as the following: > 4.4.3.1. I_T Nexus State >=20 > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages) that may need to be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures). In order for > that state to be restored, the iSCSI initiator should reestablish > its session (re-login) to the same Target Portal Group using the > previous ISID. That is, it should perform session recovery as > described in Chapter 6. This is because the SCSI initiator port > identifier and the SCSI target port identifier (or relative target > port) form the datum that the SCSI logical unit device server uses > to identify the I_T nexus. >=20 > Is RESERVE state - explicit state? When an I_T nexus is=20 > reestablished, initiator-specific mode pages are preserved. That=20 > implies that an I_T nexus loss should be treated in the way that SAM-2=20 > describes via CLEAR TASK SET (where mode pages and RESERVE state are=20 > preserved). If a session reestablishment caused saved mode pages to=20 > be restored rather than the established initiator-specific mode pages=20 > to be restored, then it would be like a LU reset (where the RESERVE=20 > state is lost). Is RESERVE state explicit state that is to be preserved? >=20 > Appendix E covers clearing effects during various events, but it never=20 > mentions RESERVE state. >=20 > Section E.2 contains the following text: >=20 > The only iSCSI protocol action that can effect clearing actions on > SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 > Loss of Nexus notification). [SPC3] describes the clearing effects > of this notification on a variety of SCSI attributes. In addition, > SCSI standards documents (such as [SAM2] and [SBC]) define > additional clearing actions that may take place for several SCSI > objects on SCSI events such as LU resets and power-on resets. >=20 > I could not find any text in SPC3 that describes these clearing=20 > effects (I don't know if this SPC3 reference is incorrect, but at a=20 > minimum, it is ineffective). The SAM2 text for LU resets and power-on=20 > resets is easily found, and clearly says that any RESERVE is released=20 > in those cases. As an aside, the reference to section 6.3.5.1 is a=20 > circular reference). The text in > E.2 goes on: >=20 > Since iSCSI defines a target cold reset as a protocol-equivalent > to a target power-cycle, the iSCSI target cold reset must also be > considered as the power-on reset event in interpreting the actions > defined in the SCSI standards. >=20 > When the iSCSI session is reconstructed (between the same SCSI > ports with the same nexus identifier) reestablishing the same I_T > nexus, all SCSI objects that are defined to not clear on the "I_T > nexus loss" notification event, such as persistent reservations, > are automatically associated to this new session. >=20 > So, it is clear what happens to PERSISTENT RESERVE state, but not so=20 > clear about RESERVE state. It also states that all SCSI objects that=20 > are defined to not clear on the I_T nexus loss are retained. But=20 > again, the only list I can find of what is cleared is here in this=20 > Appendix, since SAM-2, SAM-3, SAM-4 are all silent on the interaction bet= ween RESERVE and I_T Nexus loss. >=20 > T10 has never had to deal with this because they removed RESERVE=20 > before they added I_T Nexus loss. However, iSCSI did not do that; we=20 > discuss BOTH, but we fail to describe how they interact. iSCSI also=20 > has the feature of reestablishment of a session that does not exist in=20 > SAM-2 (SAM-2 was still parallel SCSI focused, where connections didn't=20 > exist). So again, if session reestablishment is examined in the light=20 > of parallel SCSI, the intent would be to retain the RESERVE state. Here = is another section on the topic: >=20 > 6.3.5.1. Loss of Nexus Notification >=20 > The iSCSI layer provides the SCSI layer with the "I_T nexus loss" > notification when any one of the following events happens: >=20 > Successful completion of session reinstatement. > Session closure event. > Session timeout event. >=20 > Certain SCSI object clearing actions may result due to the > notification in the SCSI end nodes, as documented in Appendix F. - > Clearing Effects of Various Events on Targets. >=20 > FWIW - the reference above is incorrect - there is no Appendix F in=20 > the consolidated draft - this should be Appendix E (and also, BTW -=20 > this is a circular reference). >=20 > So anyway, it seems like the intent was that session reinstatement=20 > would preserve the RESERVE state across the I_T Nexus loss event. >=20 > So aside from the incorrect reference (and circular nature of it), any=20 > thoughts on whether we should get explicit about this situation? =20 > Should we remain silent; should we tweak Appendix E along these lines=20 > (or other lines that others might suggest): >=20 > 4.4.3.1. I_T Nexus State >=20 > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages, and RESERVE state) that may need to=20 > be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures).... >=20 > Fred Knight >=20 >=20 >=20 > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf=20 > Of Mallikarjun Chadalapaka > Sent: Sunday, September 23, 2012 1:34 AM > To: david.black@emc.com; storm@ietf.org > Subject: Re: [storm] iSCSI drafts - T10 and T11 references >=20 > Hi David, >=20 > I have noticed this comment as I am working through all Last Call comment= s. >=20 > The references to SAM-3 and SAM-4 are keyed to this paragraph in Section = 13.9: >=20 > "[SAM2] and [SAM3] specifications note in their informative text that=20 > TPGT value should be non-zero, note that it is incorrect. A zero=20 > value is allowed as a legal value for TPGT. This discrepancy currently=20 > stands corrected in [SAM4]." >=20 > Additionally, SAM4 reference is also used in citing the new ASC/ASCQ=20 > codes that it defines, in Section 11.14.5 of the draft. >=20 > So, I believe references to all three SAM specs would continue to be=20 > appropriate. >=20 > The current draft references FC-FS in discussing the NAA format in a=20 > few places, so the current reference at the end is intentionally the=20 > same. Is the older reference a problem? If so, I suppose we can=20 > replace all references to FC-FS-4, but that seems to be still in early=20 > stages(0.1 spec posted on April 2012), so perhaps we should use FC-FS3? >=20 > Thanks. >=20 > Mallikarjun >=20 >=20 >=20 >=20 > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf=20 > Of david.black@emc.com > Sent: Thursday, July 26, 2012 6:47 AM > To: storm@ietf.org > Subject: [storm] iSCSI drafts - T10 and T11 references >=20 > In making up the lists of T10 and T11 documents that are currently=20 > referenced, some of those references don't look right to me, for example: >=20 > - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. > - T11: The FC-FS reference is out of date. >=20 > As part of dealing with the comments from IETF Last Call, we'll need=20 > to go over all of the T10 and T11 references (and should check all the=20 > references) to make sure that they're up to date and necessary. >=20 > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer EMC Corporation, 176 South St.,=20 > Hopkinton, MA=A0 01748 > +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-7= 786 > david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 > ---------------------------------------------------- >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm >=20 >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From david.black@emc.com Tue Sep 25 11:29:12 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4A5AA1F0CD1 for ; Tue, 25 Sep 2012 11:29:12 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HS7Dr+mwdxrK for ; Tue, 25 Sep 2012 11:29:10 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 911051F0CCA for ; Tue, 25 Sep 2012 11:29:10 -0700 (PDT) Received: from hop04-l1d11-si03.isus.emc.com (HOP04-L1D11-SI03.isus.emc.com [10.254.111.23]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PIT7m4003393 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 25 Sep 2012 14:29:08 -0400 Received: from mailhub.lss.emc.com (mailhubhoprd03.lss.emc.com [10.254.221.145]) by hop04-l1d11-si03.isus.emc.com (RSA Interceptor); Tue, 25 Sep 2012 14:28:51 -0400 Received: from mxhub16.corp.emc.com (mxhub16.corp.emc.com [128.222.70.237]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PISosC012994; Tue, 25 Sep 2012 14:28:50 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub16.corp.emc.com ([128.222.70.237]) with mapi; Tue, 25 Sep 2012 14:28:50 -0400 From: "Black, David" To: Mallikarjun Chadalapaka , "Knight, Frederick" , "storm@ietf.org" Date: Tue, 25 Sep 2012 14:28:49 -0400 Thread-Topic: iSCSI drafts - T10 and T11 references Thread-Index: Ac1rNRGDFlJk/LPPRq2a5Pa224AxSwuFP5jwAAlZtBAAPXLRYAA3F3tAAAJtUOA= Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE794F3@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79192@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: Re: [storm] iSCSI drafts - T10 and T11 references X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 18:29:12 -0000 > Was the concept of I_T nexus loss notification introduced in SAM-3 or SAM= -4? > (I can't tell because I don't see SAM-3... If the notification were intro= duced > in SAM-4, that could make SAM-4 a normative reference. It's a defined term in SAM-3 (3.1.44) and specified in clause 6 of SAM-3. Thanks, --David > -----Original Message----- > From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] > Sent: Tuesday, September 25, 2012 1:34 PM > To: Black, David; Knight, Frederick; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references >=20 > David, >=20 > Thanks for the suggestions. I am largely in agreement. >=20 > A couple of questions: >=20 > >2) [SAM2] and [SAM3] ought to be normative references >=20 > Don't know the related T10 history here, but I no longer see SAM-3 spec o= n the > T10 web site. Any insights? >=20 > >[SAM-4] ought to be an informative reference >=20 > Was the concept of I_T nexus loss notification introduced in SAM-3 or SAM= -4? > (I can't tell because I don't see SAM-3... If the notification were intro= duced > in SAM-4, that could make SAM-4 a normative reference. >=20 > SAM-2 does not formally discuss I_T nexus loss notification, other than a= n > oblique reference to the related check condition in clause 5.3.2, status > precedence discussion. >=20 > Thanks. >=20 > Mallikarjun >=20 >=20 >=20 >=20 > -----Original Message----- > From: Black, David [mailto:david.black@emc.com] > Sent: Monday, September 24, 2012 8:37 AM > To: Knight, Frederick; Mallikarjun Chadalapaka; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references >=20 > Fred, >=20 > Thank you for the discussion, particularly around the dependence on SAM-3= . >=20 > In light of that, and a few other things (e.g., a strong preference that = the > iSCSI consolidated draft not reference work in progress), my recommendati= ons > on the T10 and T11 references are: >=20 > --- T10 (SCSI) --- >=20 > 1) [SBC2] and [SPC3] are the current published standards; those should be= the > correct references for the iSCSI consolidated draft. [SPC3] is clearly > normative, but I think [SBC2] is (and ought to be) informative - [SBC2] i= s > only cited once in what looks like an informative fashion in Appendix E.2= : >=20 > In addition, SCSI standards documents (such as [SAM2] and [SBC2]) > define additional clearing actions that may take place for several > SCSI objects on SCSI events such as LU resets and power-on resets. >=20 > Moreover, there is no requirement to implement the SBC command set for al= l > iSCSI implementations, whereas the SPC command set is a requirement for > obvious reasons (P =3D Primary). >=20 > 2) [SAM2] and [SAM3] ought to be normative references, in that we're draw= ing > some concepts from [SAM-3]. OTOH, [SAM-4] ought to be an informative > reference, as the TPGT text that is referred to in SAM-4: >=20 > > "[SAM2] and [SAM3] specifications note in their informative text that > > TPGT value should be non-zero, note that it is incorrect. A zero > > value is allowed as a legal value for TPGT. This discrepancy currently > > stands corrected in [SAM4]." >=20 > is an informative table footnote in an annex. Further, >=20 > > Additionally, SAM4 reference is also used in citing the new ASC/ASCQ > > codes that it defines, in Section 11.14.5 of the draft. >=20 > That new ASC/Q code's name does not occur in the body of SAM-4, so I woul= d > remove that second citation of SAM-4. >=20 > 3) The [SAS] reference needs to be expanded ;-). The current informative > [SAS] reference needs to be replaced by references to SAS-2.1, SPL and SP= L > Amendment 1, see: >=20 > http://www.t10.org/drafts.htm#SCSI3_SAS >=20 > 4) OTOH, I recommend removing the [SRP] reference and related material, > 'namely the definition of "SCSI RDMA Protocol" and the alternative expans= ion > of the SRP acronym. The SRP-2 standards project to update SRP was abando= ned > by T10 in 2005 due to lack of interest (see T10/05-097r0), leaving us wit= h a > vintage-2002 SRP document. >=20 > --- T11 (Fibre Channel) --- >=20 > The FC-FS reference ought to be replaced by a reference to the current > published FS-FS-3 standard. >=20 > Thanks, > --David >=20 > > -----Original Message----- > > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > > Sent: Sunday, September 23, 2012 6:38 AM > > To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org > > Subject: RE: iSCSI drafts - T10 and T11 references > > > > Interesting, I just came across a question in a related area (only for = SAM- > 2). > > > > RFC3720 was written in between SAM-2 and SAM-3; some concepts were > > drawn from SAM-2, and some from SAM-3. > > > > We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus > > loss concept does not exist in SAM-2. SAM-2 does include references > > to RESERVE and RELEASE commands. SAM-3 includes I_T Nexus loss, but > > it has no reference to RESERVE state. There are allusions in RFC3720 > > about the intersection of RESERVE/RELEASE and I_T NEXUS LOSS, but > > there is nothing explicit. RFC3720 references other documents (SAM2, > > SAM3, SBC, SPC3), none of which address the situation. > > > > Consider text such as the following: > > 4.4.3.1. I_T Nexus State > > > > Certain nexus relationships contain an explicit state (e.g., > > initiator-specific mode pages) that may need to be preserved by > > the device server [SAM2] in a logical unit through changes or > > failures in the iSCSI layer (e.g., session failures). In order for > > that state to be restored, the iSCSI initiator should reestablish > > its session (re-login) to the same Target Portal Group using the > > previous ISID. That is, it should perform session recovery as > > described in Chapter 6. This is because the SCSI initiator port > > identifier and the SCSI target port identifier (or relative target > > port) form the datum that the SCSI logical unit device server uses > > to identify the I_T nexus. > > > > Is RESERVE state - explicit state? When an I_T nexus is > > reestablished, initiator-specific mode pages are preserved. That > > implies that an I_T nexus loss should be treated in the way that SAM-2 > > describes via CLEAR TASK SET (where mode pages and RESERVE state are > > preserved). If a session reestablishment caused saved mode pages to > > be restored rather than the established initiator-specific mode pages > > to be restored, then it would be like a LU reset (where the RESERVE > > state is lost). Is RESERVE state explicit state that is to be preserve= d? > > > > Appendix E covers clearing effects during various events, but it never > > mentions RESERVE state. > > > > Section E.2 contains the following text: > > > > The only iSCSI protocol action that can effect clearing actions on > > SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 > > Loss of Nexus notification). [SPC3] describes the clearing effects > > of this notification on a variety of SCSI attributes. In addition, > > SCSI standards documents (such as [SAM2] and [SBC]) define > > additional clearing actions that may take place for several SCSI > > objects on SCSI events such as LU resets and power-on resets. > > > > I could not find any text in SPC3 that describes these clearing > > effects (I don't know if this SPC3 reference is incorrect, but at a > > minimum, it is ineffective). The SAM2 text for LU resets and power-on > > resets is easily found, and clearly says that any RESERVE is released > > in those cases. As an aside, the reference to section 6.3.5.1 is a > > circular reference). The text in > > E.2 goes on: > > > > Since iSCSI defines a target cold reset as a protocol-equivalent > > to a target power-cycle, the iSCSI target cold reset must also be > > considered as the power-on reset event in interpreting the actions > > defined in the SCSI standards. > > > > When the iSCSI session is reconstructed (between the same SCSI > > ports with the same nexus identifier) reestablishing the same I_T > > nexus, all SCSI objects that are defined to not clear on the "I_T > > nexus loss" notification event, such as persistent reservations, > > are automatically associated to this new session. > > > > So, it is clear what happens to PERSISTENT RESERVE state, but not so > > clear about RESERVE state. It also states that all SCSI objects that > > are defined to not clear on the I_T nexus loss are retained. But > > again, the only list I can find of what is cleared is here in this > > Appendix, since SAM-2, SAM-3, SAM-4 are all silent on the interaction > between RESERVE and I_T Nexus loss. > > > > T10 has never had to deal with this because they removed RESERVE > > before they added I_T Nexus loss. However, iSCSI did not do that; we > > discuss BOTH, but we fail to describe how they interact. iSCSI also > > has the feature of reestablishment of a session that does not exist in > > SAM-2 (SAM-2 was still parallel SCSI focused, where connections didn't > > exist). So again, if session reestablishment is examined in the light > > of parallel SCSI, the intent would be to retain the RESERVE state. Her= e is > another section on the topic: > > > > 6.3.5.1. Loss of Nexus Notification > > > > The iSCSI layer provides the SCSI layer with the "I_T nexus loss" > > notification when any one of the following events happens: > > > > Successful completion of session reinstatement. > > Session closure event. > > Session timeout event. > > > > Certain SCSI object clearing actions may result due to the > > notification in the SCSI end nodes, as documented in Appendix F. - > > Clearing Effects of Various Events on Targets. > > > > FWIW - the reference above is incorrect - there is no Appendix F in > > the consolidated draft - this should be Appendix E (and also, BTW - > > this is a circular reference). > > > > So anyway, it seems like the intent was that session reinstatement > > would preserve the RESERVE state across the I_T Nexus loss event. > > > > So aside from the incorrect reference (and circular nature of it), any > > thoughts on whether we should get explicit about this situation? > > Should we remain silent; should we tweak Appendix E along these lines > > (or other lines that others might suggest): > > > > 4.4.3.1. I_T Nexus State > > > > Certain nexus relationships contain an explicit state (e.g., > > initiator-specific mode pages, and RESERVE state) that may need to > > be preserved by > > the device server [SAM2] in a logical unit through changes or > > failures in the iSCSI layer (e.g., session failures).... > > > > Fred Knight > > > > > > > > -----Original Message----- > > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf > > Of Mallikarjun Chadalapaka > > Sent: Sunday, September 23, 2012 1:34 AM > > To: david.black@emc.com; storm@ietf.org > > Subject: Re: [storm] iSCSI drafts - T10 and T11 references > > > > Hi David, > > > > I have noticed this comment as I am working through all Last Call comme= nts. > > > > The references to SAM-3 and SAM-4 are keyed to this paragraph in Sectio= n > 13.9: > > > > "[SAM2] and [SAM3] specifications note in their informative text that > > TPGT value should be non-zero, note that it is incorrect. A zero > > value is allowed as a legal value for TPGT. This discrepancy currently > > stands corrected in [SAM4]." > > > > Additionally, SAM4 reference is also used in citing the new ASC/ASCQ > > codes that it defines, in Section 11.14.5 of the draft. > > > > So, I believe references to all three SAM specs would continue to be > > appropriate. > > > > The current draft references FC-FS in discussing the NAA format in a > > few places, so the current reference at the end is intentionally the > > same. Is the older reference a problem? If so, I suppose we can > > replace all references to FC-FS-4, but that seems to be still in early > > stages(0.1 spec posted on April 2012), so perhaps we should use FC-FS3? > > > > Thanks. > > > > Mallikarjun > > > > > > > > > > -----Original Message----- > > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf > > Of david.black@emc.com > > Sent: Thursday, July 26, 2012 6:47 AM > > To: storm@ietf.org > > Subject: [storm] iSCSI drafts - T10 and T11 references > > > > In making up the lists of T10 and T11 documents that are currently > > referenced, some of those references don't look right to me, for exampl= e: > > > > - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. > > - T11: The FC-FS reference is out of date. > > > > As part of dealing with the comments from IETF Last Call, we'll need > > to go over all of the T10 and T11 references (and should check all the > > references) to make sure that they're up to date and necessary. > > > > Thanks, > > --David > > ---------------------------------------------------- > > David L. Black, Distinguished Engineer EMC Corporation, 176 South St., > > Hopkinton, MA=A0 01748 > > +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293= -7786 > > david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 > > ---------------------------------------------------- > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > > > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm >=20 >=20 From Frederick.Knight@netapp.com Tue Sep 25 12:18:49 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B40FE21F8858 for ; Tue, 25 Sep 2012 12:18:49 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HOdIK+wA8XGw for ; Tue, 25 Sep 2012 12:18:48 -0700 (PDT) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id 52B1C21F855A for ; Tue, 25 Sep 2012 12:18:48 -0700 (PDT) X-IronPort-AV: E=Sophos;i="4.80,484,1344236400"; d="scan'208";a="694006044" Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx2-out.netapp.com with ESMTP; 25 Sep 2012 12:18:48 -0700 Received: from vmwexceht04-prd.hq.netapp.com (vmwexceht04-prd.hq.netapp.com [10.106.77.34]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8PJIlT3007453; Tue, 25 Sep 2012 12:18:48 -0700 (PDT) Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.102]) by vmwexceht04-prd.hq.netapp.com ([10.106.77.34]) with mapi id 14.02.0309.002; Tue, 25 Sep 2012 12:18:47 -0700 From: "Knight, Frederick" To: Mallikarjun Chadalapaka , "Black, David" , "storm@ietf.org" Thread-Topic: iSCSI drafts - T10 and T11 references Thread-Index: Ac1rNRGDFlJk/LPPRq2a5Pa224AxSwuFP5jwAAlZtBAAPXLRYAA3F3tAAAQmzpA= Date: Tue, 25 Sep 2012 19:18:46 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79192@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.106.53.53] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [storm] iSCSI drafts - T10 and T11 references X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 19:18:49 -0000 FK> INLINE -----Original Message----- From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]=20 Sent: Tuesday, September 25, 2012 1:34 PM To: Black, David; Knight, Frederick; storm@ietf.org Subject: RE: iSCSI drafts - T10 and T11 references David, Thanks for the suggestions. I am largely in agreement. A couple of questions: >2) [SAM2] and [SAM3] ought to be normative references Don't know the related T10 history here, but I no longer see SAM-3 spec on = the T10 web site. Any insights? FK> Look down under the withdrawn standards. BTW - should we really normat= ively referencing something that INCITS/T10 withdrew? >[SAM-4] ought to be an informative reference Was the concept of I_T nexus loss notification introduced in SAM-3 or SAM-4= ? (I can't tell because I don't see SAM-3... If the notification were intro= duced in SAM-4, that could make SAM-4 a normative reference. FK> It was introduced in SAM-3. Fred SAM-2 does not formally discuss I_T nexus loss notification, other than an = oblique reference to the related check condition in clause 5.3.2, status pr= ecedence discussion. Thanks. Mallikarjun -----Original Message----- From: Black, David [mailto:david.black@emc.com] Sent: Monday, September 24, 2012 8:37 AM To: Knight, Frederick; Mallikarjun Chadalapaka; storm@ietf.org Subject: RE: iSCSI drafts - T10 and T11 references Fred, Thank you for the discussion, particularly around the dependence on SAM-3. In light of that, and a few other things (e.g., a strong preference that th= e iSCSI consolidated draft not reference work in progress), my recommendati= ons on the T10 and T11 references are: --- T10 (SCSI) --- 1) [SBC2] and [SPC3] are the current published standards; those should be t= he correct references for the iSCSI consolidated draft. [SPC3] is clearly = normative, but I think [SBC2] is (and ought to be) informative - [SBC2] is = only cited once in what looks like an informative fashion in Appendix E.2: In addition, SCSI standards documents (such as [SAM2] and [SBC2]) define additional clearing actions that may take place for several SCSI objects on SCSI events such as LU resets and power-on resets. Moreover, there is no requirement to implement the SBC command set for all = iSCSI implementations, whereas the SPC command set is a requirement for obv= ious reasons (P =3D Primary). 2) [SAM2] and [SAM3] ought to be normative references, in that we're drawin= g some concepts from [SAM-3]. OTOH, [SAM-4] ought to be an informative ref= erence, as the TPGT text that is referred to in SAM-4: > "[SAM2] and [SAM3] specifications note in their informative text that=20 > TPGT value should be non-zero, note that it is incorrect. A zero=20 > value is allowed as a legal value for TPGT. This discrepancy currently=20 > stands corrected in [SAM4]." is an informative table footnote in an annex. Further, > Additionally, SAM4 reference is also used in citing the new ASC/ASCQ=20 > codes that it defines, in Section 11.14.5 of the draft. That new ASC/Q code's name does not occur in the body of SAM-4, so I would = remove that second citation of SAM-4. 3) The [SAS] reference needs to be expanded ;-). The current informative [= SAS] reference needs to be replaced by references to SAS-2.1, SPL and SPL A= mendment 1, see: http://www.t10.org/drafts.htm#SCSI3_SAS 4) OTOH, I recommend removing the [SRP] reference and related material, 'na= mely the definition of "SCSI RDMA Protocol" and the alternative expansion o= f the SRP acronym. The SRP-2 standards project to update SRP was abandoned= by T10 in 2005 due to lack of interest (see T10/05-097r0), leaving us with= a vintage-2002 SRP document. --- T11 (Fibre Channel) --- The FC-FS reference ought to be replaced by a reference to the current publ= ished FS-FS-3 standard. Thanks, --David > -----Original Message----- > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > Sent: Sunday, September 23, 2012 6:38 AM > To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references >=20 > Interesting, I just came across a question in a related area (only for SA= M-2). >=20 > RFC3720 was written in between SAM-2 and SAM-3; some concepts were=20 > drawn from SAM-2, and some from SAM-3. >=20 > We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus=20 > loss concept does not exist in SAM-2. SAM-2 does include references=20 > to RESERVE and RELEASE commands. SAM-3 includes I_T Nexus loss, but=20 > it has no reference to RESERVE state. There are allusions in RFC3720=20 > about the intersection of RESERVE/RELEASE and I_T NEXUS LOSS, but=20 > there is nothing explicit. RFC3720 references other documents (SAM2,=20 > SAM3, SBC, SPC3), none of which address the situation. >=20 > Consider text such as the following: > 4.4.3.1. I_T Nexus State >=20 > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages) that may need to be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures). In order for > that state to be restored, the iSCSI initiator should reestablish > its session (re-login) to the same Target Portal Group using the > previous ISID. That is, it should perform session recovery as > described in Chapter 6. This is because the SCSI initiator port > identifier and the SCSI target port identifier (or relative target > port) form the datum that the SCSI logical unit device server uses > to identify the I_T nexus. >=20 > Is RESERVE state - explicit state? When an I_T nexus is=20 > reestablished, initiator-specific mode pages are preserved. That=20 > implies that an I_T nexus loss should be treated in the way that SAM-2=20 > describes via CLEAR TASK SET (where mode pages and RESERVE state are=20 > preserved). If a session reestablishment caused saved mode pages to=20 > be restored rather than the established initiator-specific mode pages=20 > to be restored, then it would be like a LU reset (where the RESERVE=20 > state is lost). Is RESERVE state explicit state that is to be preserved? >=20 > Appendix E covers clearing effects during various events, but it never=20 > mentions RESERVE state. >=20 > Section E.2 contains the following text: >=20 > The only iSCSI protocol action that can effect clearing actions on > SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 > Loss of Nexus notification). [SPC3] describes the clearing effects > of this notification on a variety of SCSI attributes. In addition, > SCSI standards documents (such as [SAM2] and [SBC]) define > additional clearing actions that may take place for several SCSI > objects on SCSI events such as LU resets and power-on resets. >=20 > I could not find any text in SPC3 that describes these clearing=20 > effects (I don't know if this SPC3 reference is incorrect, but at a=20 > minimum, it is ineffective). The SAM2 text for LU resets and power-on=20 > resets is easily found, and clearly says that any RESERVE is released=20 > in those cases. As an aside, the reference to section 6.3.5.1 is a=20 > circular reference). The text in > E.2 goes on: >=20 > Since iSCSI defines a target cold reset as a protocol-equivalent > to a target power-cycle, the iSCSI target cold reset must also be > considered as the power-on reset event in interpreting the actions > defined in the SCSI standards. >=20 > When the iSCSI session is reconstructed (between the same SCSI > ports with the same nexus identifier) reestablishing the same I_T > nexus, all SCSI objects that are defined to not clear on the "I_T > nexus loss" notification event, such as persistent reservations, > are automatically associated to this new session. >=20 > So, it is clear what happens to PERSISTENT RESERVE state, but not so=20 > clear about RESERVE state. It also states that all SCSI objects that=20 > are defined to not clear on the I_T nexus loss are retained. But=20 > again, the only list I can find of what is cleared is here in this=20 > Appendix, since SAM-2, SAM-3, SAM-4 are all silent on the interaction bet= ween RESERVE and I_T Nexus loss. >=20 > T10 has never had to deal with this because they removed RESERVE=20 > before they added I_T Nexus loss. However, iSCSI did not do that; we=20 > discuss BOTH, but we fail to describe how they interact. iSCSI also=20 > has the feature of reestablishment of a session that does not exist in > SAM-2 (SAM-2 was still parallel SCSI focused, where connections didn't=20 > exist). So again, if session reestablishment is examined in the light=20 > of parallel SCSI, the intent would be to retain the RESERVE state. Here = is another section on the topic: >=20 > 6.3.5.1. Loss of Nexus Notification >=20 > The iSCSI layer provides the SCSI layer with the "I_T nexus loss" > notification when any one of the following events happens: >=20 > Successful completion of session reinstatement. > Session closure event. > Session timeout event. >=20 > Certain SCSI object clearing actions may result due to the > notification in the SCSI end nodes, as documented in Appendix F. - > Clearing Effects of Various Events on Targets. >=20 > FWIW - the reference above is incorrect - there is no Appendix F in=20 > the consolidated draft - this should be Appendix E (and also, BTW -=20 > this is a circular reference). >=20 > So anyway, it seems like the intent was that session reinstatement=20 > would preserve the RESERVE state across the I_T Nexus loss event. >=20 > So aside from the incorrect reference (and circular nature of it), any=20 > thoughts on whether we should get explicit about this situation? > Should we remain silent; should we tweak Appendix E along these lines=20 > (or other lines that others might suggest): >=20 > 4.4.3.1. I_T Nexus State >=20 > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages, and RESERVE state) that may need to=20 > be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures).... >=20 > Fred Knight >=20 >=20 >=20 > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf=20 > Of Mallikarjun Chadalapaka > Sent: Sunday, September 23, 2012 1:34 AM > To: david.black@emc.com; storm@ietf.org > Subject: Re: [storm] iSCSI drafts - T10 and T11 references >=20 > Hi David, >=20 > I have noticed this comment as I am working through all Last Call comment= s. >=20 > The references to SAM-3 and SAM-4 are keyed to this paragraph in Section = 13.9: >=20 > "[SAM2] and [SAM3] specifications note in their informative text that=20 > TPGT value should be non-zero, note that it is incorrect. A zero=20 > value is allowed as a legal value for TPGT. This discrepancy currently=20 > stands corrected in [SAM4]." >=20 > Additionally, SAM4 reference is also used in citing the new ASC/ASCQ=20 > codes that it defines, in Section 11.14.5 of the draft. >=20 > So, I believe references to all three SAM specs would continue to be=20 > appropriate. >=20 > The current draft references FC-FS in discussing the NAA format in a=20 > few places, so the current reference at the end is intentionally the=20 > same. Is the older reference a problem? If so, I suppose we can=20 > replace all references to FC-FS-4, but that seems to be still in early > stages(0.1 spec posted on April 2012), so perhaps we should use FC-FS3? >=20 > Thanks. >=20 > Mallikarjun >=20 >=20 >=20 >=20 > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf=20 > Of david.black@emc.com > Sent: Thursday, July 26, 2012 6:47 AM > To: storm@ietf.org > Subject: [storm] iSCSI drafts - T10 and T11 references >=20 > In making up the lists of T10 and T11 documents that are currently=20 > referenced, some of those references don't look right to me, for example: >=20 > - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. > - T11: The FC-FS reference is out of date. >=20 > As part of dealing with the comments from IETF Last Call, we'll need=20 > to go over all of the T10 and T11 references (and should check all the > references) to make sure that they're up to date and necessary. >=20 > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer EMC Corporation, 176 South St.,=20 > Hopkinton, MA=A0 01748 > +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-7= 786 > david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 > ---------------------------------------------------- >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm >=20 >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From david.black@emc.com Tue Sep 25 12:20:55 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D937421F8870 for ; Tue, 25 Sep 2012 12:20:55 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PJvgyghNBnX1 for ; Tue, 25 Sep 2012 12:20:55 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id D031521F886E for ; Tue, 25 Sep 2012 12:20:54 -0700 (PDT) Received: from hop04-l1d11-si01.isus.emc.com (HOP04-L1D11-SI01.isus.emc.com [10.254.111.54]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PJKrnq006942 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 25 Sep 2012 15:20:54 -0400 Received: from mailhub.lss.emc.com (mailhubhoprd03.lss.emc.com [10.254.221.145]) by hop04-l1d11-si01.isus.emc.com (RSA Interceptor) for ; Tue, 25 Sep 2012 15:20:36 -0400 Received: from mxhub23.corp.emc.com (mxhub23.corp.emc.com [128.222.70.135]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PJKa3N030673 for ; Tue, 25 Sep 2012 15:20:36 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub23.corp.emc.com ([128.222.70.135]) with mapi; Tue, 25 Sep 2012 15:20:35 -0400 From: "Black, David" To: "storm@ietf.org" Date: Tue, 25 Sep 2012 15:20:34 -0400 Thread-Topic: SPC-2 reserve/release: Proposed text - version 2 Thread-Index: Ac2bNKguGPJW+zTbTTC2uDGdWEV2OAAGl/vw Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE79512@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: [storm] SPC-2 reserve/release: Proposed text - version 2 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 19:20:56 -0000 I moved a paragraph break in response to Ralph's comments, and rewrote the first additional consideration to reference the connection timeout values in general. I also found some minor problems that need correction in Section 4.6.3.3 (see end of this message). Here's an initial attempt to propose text to deal with the reserve/ release topic in the consolidated iSCSI draft. Please comment, correct, suggest edits, etc. The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: 4.4.3.1. I_T Nexus State Certain nexus relationships contain an explicit state (e.g., initiator-specific mode pages) that may need to be preserved by the device server [SAM2] in a logical unit through changes or failures in the iSCSI layer (e.g., session failures). In order for that state to be restored, the iSCSI initiator should reestablish its session (re-login) to the same Target Portal Group using the previous ISID. That is, it should perform session recovery as described in Chapter 6. This is because the SCSI initiator port identifier and the SCSI target port identifier (or relative target port) form the datum that the SCSI logical unit device server uses to identify the I_T nexus. ------------------------------- First, for clarity, the following change should be made in the above 4.4.3.1 text to better identify the recovery mechanism: OLD That is, it should perform session recovery as described in Chapter 6. NEW That is, it should recover the session via connection reinstatement as described in Section 6.3.4. END I believe that a lower-case "should" remains appropriate for this text. ----------------------------------- NEW TEXT (second draft): 4.4.3.2. I_T Nexus State: Reservations The state of an I_T Nexus includes SCSI reservation state. There are two reservation management methods defined in the SCSI standards, reserve/release reservations, based on the RESERVE and RELEASE commands [SPC2], and persistent reservations, based on the PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be used; persistent reservations SHOULD be used instead. The I_T Nexus state for persistent reservations is generally required to persist through changes and failures at the iSCSI layer that result in I_T Nexus failures, see [SPC3] for details and specific requirements. In contrast, [SPC2] does not specify detailed persistence requirements for reserve/release reservation state - nonetheless, when reserve/release reservations are supported by an iSCSI target, the preferred implementation approach is to preserve reserve/release reservation state along with other I_T Nexus state for iSCSI session recovery or continuation via connection reinstatement. Two additional caveats apply to reserve/release reservations: - Retention of reserve/release reservation state for an extended period of time after connection failure may require a hard reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state when connection reinstatement is not performed. When reserve/release reservation state is retained after connection failure, the need for such hard resets may be reduced via suitable selection of the Time2Wait and Time2Retain timeout values that control the duration of that retention (see Section 7.5). =20 - Reserve/release reservations may not behave as expected when persistent reservations are also used on the same logical unit; see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE behavior" in [SPC4]. ----------------------- Reference impacts: both [SPC2] and [SPC3] need to be normative references, but [SPC4] can be an informative reference. ------------------------- I deliberately used "the preferred iSCSI implementation approach" wording for reserve/release reservation state preservation requirements, as SPC-2 is vague on this topic and I'm rather uncomfortable with placing a requirement as strong as a "SHOULD" on an obsolete mechanism that "SHOULD NOT" be used. --------------- New problem in Section 4.6.3.3 - in this text: The Logout response indicates that the connection or session cleanup is completed and no other responses will arrive on the connection (if received on the logging out connection). In addition, the Logout Response indicates how long the target will continue to hold resources for recovery (e.g., command execution that continues on a new connection) in the text key Time2Retain and how long the initiator must wait before proceeding with recovery in the text key Time2Wait. Time2Wait and Time2Retain are not text keys - they're binary fields in the Logout Response PDU. Two changes are in order: - "text key Time2Retain" -> "Time2Retain field" - "text key Time2Wait" -> "Time2Wait field" Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0 =A0=A0=A0=A0 FAX: +1 (508) 293-77= 86 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From cbm@chadalapaka.com Tue Sep 25 12:22:33 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 81D3221F88A3 for ; Tue, 25 Sep 2012 12:22:33 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6Mr+D54OXhIi for ; Tue, 25 Sep 2012 12:22:32 -0700 (PDT) Received: from am1outboundpool.messaging.microsoft.com (am1ehsobe006.messaging.microsoft.com [213.199.154.209]) by ietfa.amsl.com (Postfix) with ESMTP id 8780421F889C for ; Tue, 25 Sep 2012 12:22:24 -0700 (PDT) Received: from mail4-am1-R.bigfish.com (10.3.201.242) by AM1EHSOBE008.bigfish.com (10.3.204.28) with Microsoft SMTP Server id 14.1.225.23; Tue, 25 Sep 2012 19:22:20 +0000 Received: from mail4-am1 (localhost [127.0.0.1]) by mail4-am1-R.bigfish.com (Postfix) with ESMTP id A22F72A00F9; Tue, 25 Sep 2012 19:22:20 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.240.117; KIP:(null); UIP:(null); IPV:NLI; H:BL2PRD0610HT001.namprd06.prod.outlook.com; RD:none; EFVD:NLI X-SpamScore: -35 X-BigFish: PS-35(zzbb2dI98dI9371I542M1432I103eM4015I1447Id6f1izz1202h1d1ah1d2ahzz1033IL8275bh8275dhz2fh2a8h668h839h944hd25hf0ah107ah1220h1288h12a5h12a9h12bdh137ah13b6h1155h) Received-SPF: pass (mail4-am1: domain of chadalapaka.com designates 157.56.240.117 as permitted sender) client-ip=157.56.240.117; envelope-from=cbm@chadalapaka.com; helo=BL2PRD0610HT001.namprd06.prod.outlook.com ; .outlook.com ; Received: from mail4-am1 (localhost.localdomain [127.0.0.1]) by mail4-am1 (MessageSwitch) id 134860093848309_7841; Tue, 25 Sep 2012 19:22:18 +0000 (UTC) Received: from AM1EHSMHS013.bigfish.com (unknown [10.3.201.248]) by mail4-am1.bigfish.com (Postfix) with ESMTP id F2F0540061; Tue, 25 Sep 2012 19:22:17 +0000 (UTC) Received: from BL2PRD0610HT001.namprd06.prod.outlook.com (157.56.240.117) by AM1EHSMHS013.bigfish.com (10.3.207.151) with Microsoft SMTP Server (TLS) id 14.1.225.23; Tue, 25 Sep 2012 19:22:16 +0000 Received: from BL2PRD0610MB361.namprd06.prod.outlook.com ([169.254.12.31]) by BL2PRD0610HT001.namprd06.prod.outlook.com ([10.255.101.36]) with mapi id 14.16.0190.008; Tue, 25 Sep 2012 19:22:16 +0000 From: Mallikarjun Chadalapaka To: Ralph Weber , "storm@ietf.org" Thread-Topic: [storm] SPC-2 reserve/release: Proposed text Thread-Index: Ac2bNKguGPJW+zTbTTC2uDGdWEV2OAACjEkAAALD63A= Date: Tue, 25 Sep 2012 19:22:15 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE7120DE7946D@MX15A.corp.emc.com> <5061E27A.7000109@ieee.org> In-Reply-To: <5061E27A.7000109@ieee.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [131.107.174.24] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: chadalapaka.com Subject: Re: [storm] SPC-2 reserve/release: Proposed text X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 19:22:33 -0000 Hi David, thanks for the first draft. I have a few quick comments. 1) IMHO, opening sentence of "I_T nexus state includes reservation state" i= s a little too broad, particularly since you go on to spell out two kinds o= f reservation state. So it can easily be misconstrued as suggesting that I_= T nexus state includes persistent reservation state - however the following= text correctly states that persistent reservations are unaffected by I_T n= exus loss. Persistent reservation state is a device server state, but it is= held by one or more I_T nexus(es). 2) Hard reset is not an LU reset - it is either an iSCSI Target Warm Reset,= or (more likely) an iSCSI Target Cold Reset. In either case, any of the th= ree resets should clear the reserve/release state. 3) Should use Time2Retain+Time2Wait, not Default Time2Wait 4) Session recovery does not necessarily preserve the I_T nexus state (reco= very may not occur within Time2Wait+Time2Retain, and may not use the same I= SID when it occurs), but session reinstatement and session continuation cer= tainly should. 5) Should articulate the underlying linkage between Time2Retain (an iSCSI c= oncept), and the reserve/release state (a SCSI state). Thanks. Mallikarjun Here is the revised text addressing the above comments: There are two reservation management methods defined in the SCSI standard= s, reserve/release reservations, based on the RESERVE and RELEASE commands [SPC2], and persistent reservations, based on the PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. Reserve/release reservations are obsolete in [SPC3] and SHOULD NOT be used; persistent reservations SHOULD be used instead. =20 The reservation state for persistent reservations is required to persist through changes and failures at the iSCSI layer that result in I_T nexus failures, see [SPC3] for details and specific requirements. In contrast, [SPC2] does not specify detailed persistence requirements=20 for reserve/release reservation state - nonetheless, when reserve/release= =20 reservations are supported by an iSCSI target implementation,=20 the implementations should consider the I_T nexus state to include SCSI r= eserve/release=20 reservation state. The preferred implementation approach is to preserve = reserve/release reservation state along with other I_T nexus state until iSCSI session reinstatement (Section 6.3.5) or continuation via session continuation (S= ection 6.3.6). Two additional considerations are applicable to reserve/release reservati= ons: - When the original iSCSI session, i.e. I_T nexus, is not reinstated=20 or continued by the initiator and if the target retains reserve/releas= e=20 reservation state for an extended period of time, initiator may be=20 forced to issue a Target Cold Reset, or a Target Warm Reset,=20 or a Logical Unit Reset action (Section 11.5) in order to remove that= =20 reservation state. So targets should apply the session-global Time2Wai= t and Time2Retain timeout values (Section 7.5) to manage the reserve/rel= ease=20 reservation state along with the rest of the iSCSI session state. - Reserve/release reservations may not behave as expected when persistent reservations are also used on the same logical unit; see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE behavior" in [SPC4], and the discussion in Annex B regarding how to a= pply=20 persistent reservations to realize the reserve/release reservation se= mantics. -----Original Message----- From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of R= alph Weber Sent: Tuesday, September 25, 2012 9:58 AM To: storm@ietf.org Subject: Re: [storm] SPC-2 reserve/release: Proposed text David, I believe a paragraph break is needed as shown in the following: > Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be > used; persistent reservations SHOULD be used instead. [break] The=20 > I_T Nexus > state for persistent reservations is generally required ... The chances of confusion are high when a paragraph that begins with a discu= ssion of reserve/release reservations continues into the persistent reserva= tions territory. Also please be aware that Annex B of SPC-4 describes the means by which the= use of reserve/release reservations can be adapted to employ persistent re= servations. I am not sure this an appropriate informative reference for an = RFC, but it seems worthy of mention on this reflector. All the best, .Ralph On 9/25/2012 10:44 AM, Black, David wrote: > > > Here's an initial attempt to propose text to deal with the reserve/=20 > release topic in the consolidated iSCSI draft. Please comment,=20 > correct, suggest edits, etc. > > The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: > > 4.4.3.1. I_T Nexus State > > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages) that may need to be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures). In order for > that state to be restored, the iSCSI initiator should reestablish > its session (re-login) to the same Target Portal Group using the > previous ISID. That is, it should perform session recovery as > described in Chapter 6. This is because the SCSI initiator port > identifier and the SCSI target port identifier (or relative target > port) form the datum that the SCSI logical unit device server uses > to identify the I_T nexus. > > ------------------------------- > > First, for clarity, the following change should be made in the above=20 > 4.4.3.1 text to better identify the recovery mechanism: > > OLD > That is, it should perform session recovery as > described in Chapter 6. > NEW > That is, it should recover the session via connection > reinstatement as described in Section 6.3.4. > END > > I believe that a lower-case "should" remains appropriate for this text. > > ----------------------------------- > > NEW TEXT (first draft): > > 4.4.3.2. I_T Nexus State: Reservations > > The state of an I_T Nexus includes SCSI reservation state. There > are two reservation management methods defined in the SCSI standards, > reserve/release reservations, based on the RESERVE and RELEASE > commands [SPC2], and persistent reservations, based on the PERSISTENT > RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. > > Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be > used; persistent reservations SHOULD be used instead. The I_T Nexus > state for persistent reservations is generally required to persist > through changes and failures at the iSCSI layer that result in I_T > Nexus failures, see [SPC3] for details and specific requirements. > > In contrast, [SPC2] does not specify detailed persistence > requirements for reserve/release reservation state - nonetheless, > when reserve/release reservations are supported by an iSCSI target, > the preferred implementation approach is to preserve reserve/release > reservation state along with other I_T Nexus state for iSCSI session > recovery or continuation via connection reinstatement. > > Two additional caveats apply to reserve/release reservations: > > - Retention of reserve/release reservation state for an extended > period of time may require a hard reset (e.g., LOGICAL UNIT RESET, > see section 11.5) in order to remove that state when connection > reinstatement is not performed. The DefaultTime2Wait value > can be used to limit this retention time period (see section 13.15= ). > > - Reserve/release reservations may not behave as expected when > persistent reservations are also used on the same logical unit; > see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE > behavior" in [SPC4]. > > ----------------------- > > Reference impacts: both [SPC2] and [SPC3] need to be normative=20 > references, but [SPC4] can be an informative reference. > > ------------------------- > > I deliberately used "the preferred iSCSI implementation approach" > wording for reserve/release reservation state preservation=20 > requirements, as SPC-2 is vague on this topic and I'm rather=20 > uncomfortable with placing a requirement as strong as a "SHOULD" on an=20 > obsolete mechanism that "SHOULD NOT" be used. > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer EMC Corporation, 176 South St.,=20 > Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > david.black@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From cbm@chadalapaka.com Tue Sep 25 12:25:10 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D406B21F8606 for ; Tue, 25 Sep 2012 12:25:10 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id S7gLo4BENFlA for ; Tue, 25 Sep 2012 12:25:09 -0700 (PDT) Received: from co1outboundpool.messaging.microsoft.com (co1ehsobe005.messaging.microsoft.com [216.32.180.188]) by ietfa.amsl.com (Postfix) with ESMTP id 2D41C21F8625 for ; Tue, 25 Sep 2012 12:25:09 -0700 (PDT) Received: from mail85-co1-R.bigfish.com (10.243.78.233) by CO1EHSOBE014.bigfish.com (10.243.66.77) with Microsoft SMTP Server id 14.1.225.23; Tue, 25 Sep 2012 19:25:08 +0000 Received: from mail85-co1 (localhost [127.0.0.1]) by mail85-co1-R.bigfish.com (Postfix) with ESMTP id 96CF21800F3; Tue, 25 Sep 2012 19:25:08 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.240.117; KIP:(null); UIP:(null); IPV:NLI; H:BL2PRD0610HT003.namprd06.prod.outlook.com; RD:none; EFVD:NLI X-SpamScore: -29 X-BigFish: PS-29(zz9371I146fI542M1432I4015I1447Id6f1izz1202h1d1ah1d2ahzz1033IL17326ah8275bh8275dhz2fh2a8h668h839hd25hf0ah107ah1288h12a5h12a9h12bdh137ah13b6h1155h) Received-SPF: pass (mail85-co1: domain of chadalapaka.com designates 157.56.240.117 as permitted sender) client-ip=157.56.240.117; envelope-from=cbm@chadalapaka.com; helo=BL2PRD0610HT003.namprd06.prod.outlook.com ; .outlook.com ; Received: from mail85-co1 (localhost.localdomain [127.0.0.1]) by mail85-co1 (MessageSwitch) id 134860110658571_26079; Tue, 25 Sep 2012 19:25:06 +0000 (UTC) Received: from CO1EHSMHS029.bigfish.com (unknown [10.243.78.249]) by mail85-co1.bigfish.com (Postfix) with ESMTP id 0913C2E0047; Tue, 25 Sep 2012 19:25:06 +0000 (UTC) Received: from BL2PRD0610HT003.namprd06.prod.outlook.com (157.56.240.117) by CO1EHSMHS029.bigfish.com (10.243.66.39) with Microsoft SMTP Server (TLS) id 14.1.225.23; Tue, 25 Sep 2012 19:25:05 +0000 Received: from BL2PRD0610MB361.namprd06.prod.outlook.com ([169.254.12.31]) by BL2PRD0610HT003.namprd06.prod.outlook.com ([10.255.101.38]) with mapi id 14.16.0190.008; Tue, 25 Sep 2012 19:25:04 +0000 From: Mallikarjun Chadalapaka To: "Knight, Frederick" , "Black, David" , "storm@ietf.org" Thread-Topic: iSCSI drafts - T10 and T11 references Thread-Index: Ac1rNRGDFlJk/LPPRq2a5Pa224AxSwuFP5jwAAlZtBAAPXLRYAA3F3tAAAQmzpAAADrtIA== Date: Tue, 25 Sep 2012 19:25:03 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79192@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [131.107.174.24] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: chadalapaka.com Subject: Re: [storm] iSCSI drafts - T10 and T11 references X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 19:25:10 -0000 Yes, thanks Fred for pointing this out, I too had realized that SAM-3 was w= ithdrawn after I sent my previous note. IMO, it does not make sense to normatively reference a withdrawn T10 standa= rd in a new RFC. So my default plan now is to replace all SAM-3 references = to SAM-4, and leave the latter as a normative reference. David? Thanks. Mallikarjun -----Original Message----- From: Knight, Frederick [mailto:Frederick.Knight@netapp.com]=20 Sent: Tuesday, September 25, 2012 12:19 PM To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org Subject: RE: iSCSI drafts - T10 and T11 references FK> INLINE -----Original Message----- From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] Sent: Tuesday, September 25, 2012 1:34 PM To: Black, David; Knight, Frederick; storm@ietf.org Subject: RE: iSCSI drafts - T10 and T11 references David, Thanks for the suggestions. I am largely in agreement. A couple of questions: >2) [SAM2] and [SAM3] ought to be normative references Don't know the related T10 history here, but I no longer see SAM-3 spec on = the T10 web site. Any insights? FK> Look down under the withdrawn standards. BTW - should we really normat= ively referencing something that INCITS/T10 withdrew? >[SAM-4] ought to be an informative reference Was the concept of I_T nexus loss notification introduced in SAM-3 or SAM-4= ? (I can't tell because I don't see SAM-3... If the notification were intro= duced in SAM-4, that could make SAM-4 a normative reference. FK> It was introduced in SAM-3. Fred SAM-2 does not formally discuss I_T nexus loss notification, other than an = oblique reference to the related check condition in clause 5.3.2, status pr= ecedence discussion. Thanks. Mallikarjun -----Original Message----- From: Black, David [mailto:david.black@emc.com] Sent: Monday, September 24, 2012 8:37 AM To: Knight, Frederick; Mallikarjun Chadalapaka; storm@ietf.org Subject: RE: iSCSI drafts - T10 and T11 references Fred, Thank you for the discussion, particularly around the dependence on SAM-3. In light of that, and a few other things (e.g., a strong preference that th= e iSCSI consolidated draft not reference work in progress), my recommendati= ons on the T10 and T11 references are: --- T10 (SCSI) --- 1) [SBC2] and [SPC3] are the current published standards; those should be t= he correct references for the iSCSI consolidated draft. [SPC3] is clearly = normative, but I think [SBC2] is (and ought to be) informative - [SBC2] is = only cited once in what looks like an informative fashion in Appendix E.2: In addition, SCSI standards documents (such as [SAM2] and [SBC2]) define additional clearing actions that may take place for several SCSI objects on SCSI events such as LU resets and power-on resets. Moreover, there is no requirement to implement the SBC command set for all = iSCSI implementations, whereas the SPC command set is a requirement for obv= ious reasons (P =3D Primary). 2) [SAM2] and [SAM3] ought to be normative references, in that we're drawin= g some concepts from [SAM-3]. OTOH, [SAM-4] ought to be an informative ref= erence, as the TPGT text that is referred to in SAM-4: > "[SAM2] and [SAM3] specifications note in their informative text that=20 > TPGT value should be non-zero, note that it is incorrect. A zero=20 > value is allowed as a legal value for TPGT. This discrepancy currently=20 > stands corrected in [SAM4]." is an informative table footnote in an annex. Further, > Additionally, SAM4 reference is also used in citing the new ASC/ASCQ=20 > codes that it defines, in Section 11.14.5 of the draft. That new ASC/Q code's name does not occur in the body of SAM-4, so I would = remove that second citation of SAM-4. 3) The [SAS] reference needs to be expanded ;-). The current informative [= SAS] reference needs to be replaced by references to SAS-2.1, SPL and SPL A= mendment 1, see: http://www.t10.org/drafts.htm#SCSI3_SAS 4) OTOH, I recommend removing the [SRP] reference and related material, 'na= mely the definition of "SCSI RDMA Protocol" and the alternative expansion o= f the SRP acronym. The SRP-2 standards project to update SRP was abandoned= by T10 in 2005 due to lack of interest (see T10/05-097r0), leaving us with= a vintage-2002 SRP document. --- T11 (Fibre Channel) --- The FC-FS reference ought to be replaced by a reference to the current publ= ished FS-FS-3 standard. Thanks, --David > -----Original Message----- > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > Sent: Sunday, September 23, 2012 6:38 AM > To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references >=20 > Interesting, I just came across a question in a related area (only for SA= M-2). >=20 > RFC3720 was written in between SAM-2 and SAM-3; some concepts were=20 > drawn from SAM-2, and some from SAM-3. >=20 > We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus=20 > loss concept does not exist in SAM-2. SAM-2 does include references=20 > to RESERVE and RELEASE commands. SAM-3 includes I_T Nexus loss, but=20 > it has no reference to RESERVE state. There are allusions in RFC3720=20 > about the intersection of RESERVE/RELEASE and I_T NEXUS LOSS, but=20 > there is nothing explicit. RFC3720 references other documents (SAM2,=20 > SAM3, SBC, SPC3), none of which address the situation. >=20 > Consider text such as the following: > 4.4.3.1. I_T Nexus State >=20 > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages) that may need to be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures). In order for > that state to be restored, the iSCSI initiator should reestablish > its session (re-login) to the same Target Portal Group using the > previous ISID. That is, it should perform session recovery as > described in Chapter 6. This is because the SCSI initiator port > identifier and the SCSI target port identifier (or relative target > port) form the datum that the SCSI logical unit device server uses > to identify the I_T nexus. >=20 > Is RESERVE state - explicit state? When an I_T nexus is=20 > reestablished, initiator-specific mode pages are preserved. That=20 > implies that an I_T nexus loss should be treated in the way that SAM-2=20 > describes via CLEAR TASK SET (where mode pages and RESERVE state are=20 > preserved). If a session reestablishment caused saved mode pages to=20 > be restored rather than the established initiator-specific mode pages=20 > to be restored, then it would be like a LU reset (where the RESERVE=20 > state is lost). Is RESERVE state explicit state that is to be preserved? >=20 > Appendix E covers clearing effects during various events, but it never=20 > mentions RESERVE state. >=20 > Section E.2 contains the following text: >=20 > The only iSCSI protocol action that can effect clearing actions on > SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 > Loss of Nexus notification). [SPC3] describes the clearing effects > of this notification on a variety of SCSI attributes. In addition, > SCSI standards documents (such as [SAM2] and [SBC]) define > additional clearing actions that may take place for several SCSI > objects on SCSI events such as LU resets and power-on resets. >=20 > I could not find any text in SPC3 that describes these clearing=20 > effects (I don't know if this SPC3 reference is incorrect, but at a=20 > minimum, it is ineffective). The SAM2 text for LU resets and power-on=20 > resets is easily found, and clearly says that any RESERVE is released=20 > in those cases. As an aside, the reference to section 6.3.5.1 is a=20 > circular reference). The text in > E.2 goes on: >=20 > Since iSCSI defines a target cold reset as a protocol-equivalent > to a target power-cycle, the iSCSI target cold reset must also be > considered as the power-on reset event in interpreting the actions > defined in the SCSI standards. >=20 > When the iSCSI session is reconstructed (between the same SCSI > ports with the same nexus identifier) reestablishing the same I_T > nexus, all SCSI objects that are defined to not clear on the "I_T > nexus loss" notification event, such as persistent reservations, > are automatically associated to this new session. >=20 > So, it is clear what happens to PERSISTENT RESERVE state, but not so=20 > clear about RESERVE state. It also states that all SCSI objects that=20 > are defined to not clear on the I_T nexus loss are retained. But=20 > again, the only list I can find of what is cleared is here in this=20 > Appendix, since SAM-2, SAM-3, SAM-4 are all silent on the interaction bet= ween RESERVE and I_T Nexus loss. >=20 > T10 has never had to deal with this because they removed RESERVE=20 > before they added I_T Nexus loss. However, iSCSI did not do that; we=20 > discuss BOTH, but we fail to describe how they interact. iSCSI also=20 > has the feature of reestablishment of a session that does not exist in > SAM-2 (SAM-2 was still parallel SCSI focused, where connections didn't=20 > exist). So again, if session reestablishment is examined in the light=20 > of parallel SCSI, the intent would be to retain the RESERVE state. Here = is another section on the topic: >=20 > 6.3.5.1. Loss of Nexus Notification >=20 > The iSCSI layer provides the SCSI layer with the "I_T nexus loss" > notification when any one of the following events happens: >=20 > Successful completion of session reinstatement. > Session closure event. > Session timeout event. >=20 > Certain SCSI object clearing actions may result due to the > notification in the SCSI end nodes, as documented in Appendix F. - > Clearing Effects of Various Events on Targets. >=20 > FWIW - the reference above is incorrect - there is no Appendix F in=20 > the consolidated draft - this should be Appendix E (and also, BTW -=20 > this is a circular reference). >=20 > So anyway, it seems like the intent was that session reinstatement=20 > would preserve the RESERVE state across the I_T Nexus loss event. >=20 > So aside from the incorrect reference (and circular nature of it), any=20 > thoughts on whether we should get explicit about this situation? > Should we remain silent; should we tweak Appendix E along these lines=20 > (or other lines that others might suggest): >=20 > 4.4.3.1. I_T Nexus State >=20 > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages, and RESERVE state) that may need to=20 > be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures).... >=20 > Fred Knight >=20 >=20 >=20 > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf=20 > Of Mallikarjun Chadalapaka > Sent: Sunday, September 23, 2012 1:34 AM > To: david.black@emc.com; storm@ietf.org > Subject: Re: [storm] iSCSI drafts - T10 and T11 references >=20 > Hi David, >=20 > I have noticed this comment as I am working through all Last Call comment= s. >=20 > The references to SAM-3 and SAM-4 are keyed to this paragraph in Section = 13.9: >=20 > "[SAM2] and [SAM3] specifications note in their informative text that=20 > TPGT value should be non-zero, note that it is incorrect. A zero=20 > value is allowed as a legal value for TPGT. This discrepancy currently=20 > stands corrected in [SAM4]." >=20 > Additionally, SAM4 reference is also used in citing the new ASC/ASCQ=20 > codes that it defines, in Section 11.14.5 of the draft. >=20 > So, I believe references to all three SAM specs would continue to be=20 > appropriate. >=20 > The current draft references FC-FS in discussing the NAA format in a=20 > few places, so the current reference at the end is intentionally the=20 > same. Is the older reference a problem? If so, I suppose we can=20 > replace all references to FC-FS-4, but that seems to be still in early > stages(0.1 spec posted on April 2012), so perhaps we should use FC-FS3? >=20 > Thanks. >=20 > Mallikarjun >=20 >=20 >=20 >=20 > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf=20 > Of david.black@emc.com > Sent: Thursday, July 26, 2012 6:47 AM > To: storm@ietf.org > Subject: [storm] iSCSI drafts - T10 and T11 references >=20 > In making up the lists of T10 and T11 documents that are currently=20 > referenced, some of those references don't look right to me, for example: >=20 > - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. > - T11: The FC-FS reference is out of date. >=20 > As part of dealing with the comments from IETF Last Call, we'll need=20 > to go over all of the T10 and T11 references (and should check all the > references) to make sure that they're up to date and necessary. >=20 > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer EMC Corporation, 176 South St.,=20 > Hopkinton, MA=A0 01748 > +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-7= 786 > david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 > ---------------------------------------------------- >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm >=20 >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From david.black@emc.com Tue Sep 25 12:37:09 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7DD4521F897A for ; Tue, 25 Sep 2012 12:37:09 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 75aGD8TFg3yB for ; Tue, 25 Sep 2012 12:37:08 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id DAE2321F8976 for ; Tue, 25 Sep 2012 12:37:07 -0700 (PDT) Received: from hop04-l1d11-si01.isus.emc.com (HOP04-L1D11-SI01.isus.emc.com [10.254.111.54]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PJb4TM025454 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 25 Sep 2012 15:37:05 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.222.130]) by hop04-l1d11-si01.isus.emc.com (RSA Interceptor); Tue, 25 Sep 2012 15:36:55 -0400 Received: from mxhub21.corp.emc.com (mxhub21.corp.emc.com [128.222.70.133]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PJasTU022642; Tue, 25 Sep 2012 15:36:54 -0400 Received: from mxhub37.corp.emc.com (128.222.70.104) by mxhub21.corp.emc.com (128.222.70.133) with Microsoft SMTP Server (TLS) id 8.3.213.0; Tue, 25 Sep 2012 15:36:54 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub37.corp.emc.com ([128.222.70.104]) with mapi; Tue, 25 Sep 2012 15:36:54 -0400 From: "Black, David" To: Mallikarjun Chadalapaka , "Knight, Frederick" , "storm@ietf.org" Date: Tue, 25 Sep 2012 15:36:52 -0400 Thread-Topic: iSCSI drafts - T10 and T11 references Thread-Index: Ac1rNRGDFlJk/LPPRq2a5Pa224AxSwuFP5jwAAlZtBAAPXLRYAA3F3tAAAQmzpAAADrtIAAAeVyw Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE79521@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79192@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: Re: [storm] iSCSI drafts - T10 and T11 references X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 19:37:09 -0000 Yes, that's the right answer, as I had also missed T10's withdrawal of SAM-= 3. Thanks, --David > -----Original Message----- > From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] > Sent: Tuesday, September 25, 2012 3:25 PM > To: Knight, Frederick; Black, David; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references >=20 > Yes, thanks Fred for pointing this out, I too had realized that SAM-3 was > withdrawn after I sent my previous note. >=20 > IMO, it does not make sense to normatively reference a withdrawn T10 stan= dard > in a new RFC. So my default plan now is to replace all SAM-3 references t= o > SAM-4, and leave the latter as a normative reference. David? >=20 > Thanks. >=20 > Mallikarjun >=20 >=20 > -----Original Message----- > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > Sent: Tuesday, September 25, 2012 12:19 PM > To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references >=20 > FK> INLINE >=20 > -----Original Message----- > From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] > Sent: Tuesday, September 25, 2012 1:34 PM > To: Black, David; Knight, Frederick; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references >=20 > David, >=20 > Thanks for the suggestions. I am largely in agreement. >=20 > A couple of questions: >=20 > >2) [SAM2] and [SAM3] ought to be normative references >=20 > Don't know the related T10 history here, but I no longer see SAM-3 spec o= n the > T10 web site. Any insights? >=20 > FK> Look down under the withdrawn standards. BTW - should we really > normatively referencing something that INCITS/T10 withdrew? >=20 > >[SAM-4] ought to be an informative reference >=20 > Was the concept of I_T nexus loss notification introduced in SAM-3 or SAM= -4? > (I can't tell because I don't see SAM-3... If the notification were intro= duced > in SAM-4, that could make SAM-4 a normative reference. >=20 > FK> It was introduced in SAM-3. >=20 > Fred >=20 > SAM-2 does not formally discuss I_T nexus loss notification, other than a= n > oblique reference to the related check condition in clause 5.3.2, status > precedence discussion. >=20 > Thanks. >=20 > Mallikarjun >=20 >=20 >=20 >=20 > -----Original Message----- > From: Black, David [mailto:david.black@emc.com] > Sent: Monday, September 24, 2012 8:37 AM > To: Knight, Frederick; Mallikarjun Chadalapaka; storm@ietf.org > Subject: RE: iSCSI drafts - T10 and T11 references >=20 > Fred, >=20 > Thank you for the discussion, particularly around the dependence on SAM-3= . >=20 > In light of that, and a few other things (e.g., a strong preference that = the > iSCSI consolidated draft not reference work in progress), my recommendati= ons > on the T10 and T11 references are: >=20 > --- T10 (SCSI) --- >=20 > 1) [SBC2] and [SPC3] are the current published standards; those should be= the > correct references for the iSCSI consolidated draft. [SPC3] is clearly > normative, but I think [SBC2] is (and ought to be) informative - [SBC2] i= s > only cited once in what looks like an informative fashion in Appendix E.2= : >=20 > In addition, SCSI standards documents (such as [SAM2] and [SBC2]) > define additional clearing actions that may take place for several > SCSI objects on SCSI events such as LU resets and power-on resets. >=20 > Moreover, there is no requirement to implement the SBC command set for al= l > iSCSI implementations, whereas the SPC command set is a requirement for > obvious reasons (P =3D Primary). >=20 > 2) [SAM2] and [SAM3] ought to be normative references, in that we're draw= ing > some concepts from [SAM-3]. OTOH, [SAM-4] ought to be an informative > reference, as the TPGT text that is referred to in SAM-4: >=20 > > "[SAM2] and [SAM3] specifications note in their informative text that > > TPGT value should be non-zero, note that it is incorrect. A zero > > value is allowed as a legal value for TPGT. This discrepancy currently > > stands corrected in [SAM4]." >=20 > is an informative table footnote in an annex. Further, >=20 > > Additionally, SAM4 reference is also used in citing the new ASC/ASCQ > > codes that it defines, in Section 11.14.5 of the draft. >=20 > That new ASC/Q code's name does not occur in the body of SAM-4, so I woul= d > remove that second citation of SAM-4. >=20 > 3) The [SAS] reference needs to be expanded ;-). The current informative > [SAS] reference needs to be replaced by references to SAS-2.1, SPL and SP= L > Amendment 1, see: >=20 > http://www.t10.org/drafts.htm#SCSI3_SAS >=20 > 4) OTOH, I recommend removing the [SRP] reference and related material, > 'namely the definition of "SCSI RDMA Protocol" and the alternative expans= ion > of the SRP acronym. The SRP-2 standards project to update SRP was abando= ned > by T10 in 2005 due to lack of interest (see T10/05-097r0), leaving us wit= h a > vintage-2002 SRP document. >=20 > --- T11 (Fibre Channel) --- >=20 > The FC-FS reference ought to be replaced by a reference to the current > published FS-FS-3 standard. >=20 > Thanks, > --David >=20 > > -----Original Message----- > > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > > Sent: Sunday, September 23, 2012 6:38 AM > > To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org > > Subject: RE: iSCSI drafts - T10 and T11 references > > > > Interesting, I just came across a question in a related area (only for = SAM- > 2). > > > > RFC3720 was written in between SAM-2 and SAM-3; some concepts were > > drawn from SAM-2, and some from SAM-3. > > > > We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus > > loss concept does not exist in SAM-2. SAM-2 does include references > > to RESERVE and RELEASE commands. SAM-3 includes I_T Nexus loss, but > > it has no reference to RESERVE state. There are allusions in RFC3720 > > about the intersection of RESERVE/RELEASE and I_T NEXUS LOSS, but > > there is nothing explicit. RFC3720 references other documents (SAM2, > > SAM3, SBC, SPC3), none of which address the situation. > > > > Consider text such as the following: > > 4.4.3.1. I_T Nexus State > > > > Certain nexus relationships contain an explicit state (e.g., > > initiator-specific mode pages) that may need to be preserved by > > the device server [SAM2] in a logical unit through changes or > > failures in the iSCSI layer (e.g., session failures). In order for > > that state to be restored, the iSCSI initiator should reestablish > > its session (re-login) to the same Target Portal Group using the > > previous ISID. That is, it should perform session recovery as > > described in Chapter 6. This is because the SCSI initiator port > > identifier and the SCSI target port identifier (or relative target > > port) form the datum that the SCSI logical unit device server uses > > to identify the I_T nexus. > > > > Is RESERVE state - explicit state? When an I_T nexus is > > reestablished, initiator-specific mode pages are preserved. That > > implies that an I_T nexus loss should be treated in the way that SAM-2 > > describes via CLEAR TASK SET (where mode pages and RESERVE state are > > preserved). If a session reestablishment caused saved mode pages to > > be restored rather than the established initiator-specific mode pages > > to be restored, then it would be like a LU reset (where the RESERVE > > state is lost). Is RESERVE state explicit state that is to be preserve= d? > > > > Appendix E covers clearing effects during various events, but it never > > mentions RESERVE state. > > > > Section E.2 contains the following text: > > > > The only iSCSI protocol action that can effect clearing actions on > > SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1 > > Loss of Nexus notification). [SPC3] describes the clearing effects > > of this notification on a variety of SCSI attributes. In addition, > > SCSI standards documents (such as [SAM2] and [SBC]) define > > additional clearing actions that may take place for several SCSI > > objects on SCSI events such as LU resets and power-on resets. > > > > I could not find any text in SPC3 that describes these clearing > > effects (I don't know if this SPC3 reference is incorrect, but at a > > minimum, it is ineffective). The SAM2 text for LU resets and power-on > > resets is easily found, and clearly says that any RESERVE is released > > in those cases. As an aside, the reference to section 6.3.5.1 is a > > circular reference). The text in > > E.2 goes on: > > > > Since iSCSI defines a target cold reset as a protocol-equivalent > > to a target power-cycle, the iSCSI target cold reset must also be > > considered as the power-on reset event in interpreting the actions > > defined in the SCSI standards. > > > > When the iSCSI session is reconstructed (between the same SCSI > > ports with the same nexus identifier) reestablishing the same I_T > > nexus, all SCSI objects that are defined to not clear on the "I_T > > nexus loss" notification event, such as persistent reservations, > > are automatically associated to this new session. > > > > So, it is clear what happens to PERSISTENT RESERVE state, but not so > > clear about RESERVE state. It also states that all SCSI objects that > > are defined to not clear on the I_T nexus loss are retained. But > > again, the only list I can find of what is cleared is here in this > > Appendix, since SAM-2, SAM-3, SAM-4 are all silent on the interaction > between RESERVE and I_T Nexus loss. > > > > T10 has never had to deal with this because they removed RESERVE > > before they added I_T Nexus loss. However, iSCSI did not do that; we > > discuss BOTH, but we fail to describe how they interact. iSCSI also > > has the feature of reestablishment of a session that does not exist in > > SAM-2 (SAM-2 was still parallel SCSI focused, where connections didn't > > exist). So again, if session reestablishment is examined in the light > > of parallel SCSI, the intent would be to retain the RESERVE state. Her= e is > another section on the topic: > > > > 6.3.5.1. Loss of Nexus Notification > > > > The iSCSI layer provides the SCSI layer with the "I_T nexus loss" > > notification when any one of the following events happens: > > > > Successful completion of session reinstatement. > > Session closure event. > > Session timeout event. > > > > Certain SCSI object clearing actions may result due to the > > notification in the SCSI end nodes, as documented in Appendix F. - > > Clearing Effects of Various Events on Targets. > > > > FWIW - the reference above is incorrect - there is no Appendix F in > > the consolidated draft - this should be Appendix E (and also, BTW - > > this is a circular reference). > > > > So anyway, it seems like the intent was that session reinstatement > > would preserve the RESERVE state across the I_T Nexus loss event. > > > > So aside from the incorrect reference (and circular nature of it), any > > thoughts on whether we should get explicit about this situation? > > Should we remain silent; should we tweak Appendix E along these lines > > (or other lines that others might suggest): > > > > 4.4.3.1. I_T Nexus State > > > > Certain nexus relationships contain an explicit state (e.g., > > initiator-specific mode pages, and RESERVE state) that may need to > > be preserved by > > the device server [SAM2] in a logical unit through changes or > > failures in the iSCSI layer (e.g., session failures).... > > > > Fred Knight > > > > > > > > -----Original Message----- > > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf > > Of Mallikarjun Chadalapaka > > Sent: Sunday, September 23, 2012 1:34 AM > > To: david.black@emc.com; storm@ietf.org > > Subject: Re: [storm] iSCSI drafts - T10 and T11 references > > > > Hi David, > > > > I have noticed this comment as I am working through all Last Call comme= nts. > > > > The references to SAM-3 and SAM-4 are keyed to this paragraph in Sectio= n > 13.9: > > > > "[SAM2] and [SAM3] specifications note in their informative text that > > TPGT value should be non-zero, note that it is incorrect. A zero > > value is allowed as a legal value for TPGT. This discrepancy currently > > stands corrected in [SAM4]." > > > > Additionally, SAM4 reference is also used in citing the new ASC/ASCQ > > codes that it defines, in Section 11.14.5 of the draft. > > > > So, I believe references to all three SAM specs would continue to be > > appropriate. > > > > The current draft references FC-FS in discussing the NAA format in a > > few places, so the current reference at the end is intentionally the > > same. Is the older reference a problem? If so, I suppose we can > > replace all references to FC-FS-4, but that seems to be still in early > > stages(0.1 spec posted on April 2012), so perhaps we should use FC-FS3? > > > > Thanks. > > > > Mallikarjun > > > > > > > > > > -----Original Message----- > > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf > > Of david.black@emc.com > > Sent: Thursday, July 26, 2012 6:47 AM > > To: storm@ietf.org > > Subject: [storm] iSCSI drafts - T10 and T11 references > > > > In making up the lists of T10 and T11 documents that are currently > > referenced, some of those references don't look right to me, for exampl= e: > > > > - T10: I'm not sure whether we need to reference SAM-2, -3 and -4. > > - T11: The FC-FS reference is out of date. > > > > As part of dealing with the comments from IETF Last Call, we'll need > > to go over all of the T10 and T11 references (and should check all the > > references) to make sure that they're up to date and necessary. > > > > Thanks, > > --David > > ---------------------------------------------------- > > David L. Black, Distinguished Engineer EMC Corporation, 176 South St., > > Hopkinton, MA=A0 01748 > > +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293= -7786 > > david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 > > ---------------------------------------------------- > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > > > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm >=20 >=20 >=20 >=20 From cbm@chadalapaka.com Tue Sep 25 12:40:32 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AE4421F0C86 for ; Tue, 25 Sep 2012 12:40:32 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kpZXhnXUJiWX for ; Tue, 25 Sep 2012 12:40:31 -0700 (PDT) Received: from va3outboundpool.messaging.microsoft.com (va3ehsobe002.messaging.microsoft.com [216.32.180.12]) by ietfa.amsl.com (Postfix) with ESMTP id 62EB521F8873 for ; Tue, 25 Sep 2012 12:40:31 -0700 (PDT) Received: from mail227-va3-R.bigfish.com (10.7.14.236) by VA3EHSOBE004.bigfish.com (10.7.40.24) with Microsoft SMTP Server id 14.1.225.23; Tue, 25 Sep 2012 19:40:30 +0000 Received: from mail227-va3 (localhost [127.0.0.1]) by mail227-va3-R.bigfish.com (Postfix) with ESMTP id DFCACA80130 for ; Tue, 25 Sep 2012 19:40:30 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.240.117; KIP:(null); UIP:(null); IPV:NLI; H:BL2PRD0610HT004.namprd06.prod.outlook.com; RD:none; EFVD:NLI X-SpamScore: -26 X-BigFish: PS-26(zz9371I542M4015Id6f1izz1202h1d1ah1d2ahzz1033IL8275dhz2fh2a8h668h839hd25hf0ah107ah1288h12a5h12a9h12bdh137ah13b6h1155h) Received-SPF: pass (mail227-va3: domain of chadalapaka.com designates 157.56.240.117 as permitted sender) client-ip=157.56.240.117; envelope-from=cbm@chadalapaka.com; helo=BL2PRD0610HT004.namprd06.prod.outlook.com ; .outlook.com ; Received: from mail227-va3 (localhost.localdomain [127.0.0.1]) by mail227-va3 (MessageSwitch) id 1348602027715733_17539; Tue, 25 Sep 2012 19:40:27 +0000 (UTC) Received: from VA3EHSMHS002.bigfish.com (unknown [10.7.14.243]) by mail227-va3.bigfish.com (Postfix) with ESMTP id A304E6C005D; Tue, 25 Sep 2012 19:40:27 +0000 (UTC) Received: from BL2PRD0610HT004.namprd06.prod.outlook.com (157.56.240.117) by VA3EHSMHS002.bigfish.com (10.7.99.12) with Microsoft SMTP Server (TLS) id 14.1.225.23; Tue, 25 Sep 2012 19:40:25 +0000 Received: from BL2PRD0610MB361.namprd06.prod.outlook.com ([169.254.12.31]) by BL2PRD0610HT004.namprd06.prod.outlook.com ([10.255.101.39]) with mapi id 14.16.0190.008; Tue, 25 Sep 2012 19:40:25 +0000 From: Mallikarjun Chadalapaka To: "Black, David" , "storm@ietf.org" Thread-Topic: SPC-2 reserve/release: Proposed text - version 2 Thread-Index: Ac2bNKguGPJW+zTbTTC2uDGdWEV2OAAGl/vwAAE5wPA= Date: Tue, 25 Sep 2012 19:40:23 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE7120DE79512@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE79512@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [131.107.174.24] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: chadalapaka.com Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 2 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 19:40:32 -0000 Hi David, Good catch on Section 4.6.3.3 - I have now fixed it. Please see my comments and revised text that I just sent in response to you= r first email. Your reference impact summary looks appropriate to me. Thanks. Mallikarjun -----Original Message----- From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of B= lack, David Sent: Tuesday, September 25, 2012 12:21 PM To: storm@ietf.org Subject: [storm] SPC-2 reserve/release: Proposed text - version 2 I moved a paragraph break in response to Ralph's comments, and rewrote the = first additional consideration to reference the connection timeout values i= n general. I also found some minor problems that need correction in Sectio= n 4.6.3.3 (see end of this message). Here's an initial attempt to propose text to deal with the reserve/ release= topic in the consolidated iSCSI draft. Please comment, correct, suggest e= dits, etc. The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: 4.4.3.1. I_T Nexus State Certain nexus relationships contain an explicit state (e.g., initiator-specific mode pages) that may need to be preserved by the device server [SAM2] in a logical unit through changes or failures in the iSCSI layer (e.g., session failures). In order for that state to be restored, the iSCSI initiator should reestablish its session (re-login) to the same Target Portal Group using the previous ISID. That is, it should perform session recovery as described in Chapter 6. This is because the SCSI initiator port identifier and the SCSI target port identifier (or relative target port) form the datum that the SCSI logical unit device server uses to identify the I_T nexus. ------------------------------- First, for clarity, the following change should be made in the above 4.4.3.= 1 text to better identify the recovery mechanism: OLD That is, it should perform session recovery as described in Chapter 6. NEW That is, it should recover the session via connection reinstatement as described in Section 6.3.4. END I believe that a lower-case "should" remains appropriate for this text. ----------------------------------- NEW TEXT (second draft): 4.4.3.2. I_T Nexus State: Reservations The state of an I_T Nexus includes SCSI reservation state. There are two reservation management methods defined in the SCSI standards, reserve/release reservations, based on the RESERVE and RELEASE commands [SPC2], and persistent reservations, based on the PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be used; persistent reservations SHOULD be used instead. The I_T Nexus state for persistent reservations is generally required to persist through changes and failures at the iSCSI layer that result in I_T Nexus failures, see [SPC3] for details and specific requirements. In contrast, [SPC2] does not specify detailed persistence requirements for reserve/release reservation state - nonetheless, when reserve/release reservations are supported by an iSCSI target, the preferred implementation approach is to preserve reserve/release reservation state along with other I_T Nexus state for iSCSI session recovery or continuation via connection reinstatement. Two additional caveats apply to reserve/release reservations: - Retention of reserve/release reservation state for an extended period of time after connection failure may require a hard reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state when connection reinstatement is not performed. When reserve/release reservation state is retained after connection failure, the need for such hard resets may be reduced via suitable selection of the Time2Wait and Time2Retain timeout values that control the duration of that retention (see Section 7.5). =20 - Reserve/release reservations may not behave as expected when persistent reservations are also used on the same logical unit; see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE behavior" in [SPC4]. ----------------------- Reference impacts: both [SPC2] and [SPC3] need to be normative references, = but [SPC4] can be an informative reference. ------------------------- I deliberately used "the preferred iSCSI implementation approach" wording for reserve/release reservation state preservation requirements, as= SPC-2 is vague on this topic and I'm rather uncomfortable with placing a r= equirement as strong as a "SHOULD" on an obsolete mechanism that "SHOULD NO= T" be used. --------------- New problem in Section 4.6.3.3 - in this text: The Logout response indicates that the connection or session cleanup is completed and no other responses will arrive on the connection (if received on the logging out connection). In addition, the Logout Response indicates how long the target will continue to hold resources for recovery (e.g., command execution that continues on a new connection) in the text key Time2Retain and how long the initiator must wait before proceeding with recovery in the text key Time2Wait. Time2Wait and Time2Retain are not text keys - they're binary fields in the = Logout Response PDU. Two changes are in order: - "text key Time2Retain" -> "Time2Retain field" - "text key Time2Wait" -> "Time2Wait field" Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0 =A0=A0=A0=A0 FAX: +1 (508) 293-77= 86 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From david.black@emc.com Tue Sep 25 12:53:58 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5838A21F85AF for ; Tue, 25 Sep 2012 12:53:58 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mxEbPIkbw-Xe for ; Tue, 25 Sep 2012 12:53:57 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 6699821F85AC for ; Tue, 25 Sep 2012 12:53:57 -0700 (PDT) Received: from hop04-l1d11-si04.isus.emc.com (HOP04-L1D11-SI04.isus.emc.com [10.254.111.24]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PJrudJ012604 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 25 Sep 2012 15:53:56 -0400 Received: from mailhub.lss.emc.com (mailhubhoprd01.lss.emc.com [10.254.221.251]) by hop04-l1d11-si04.isus.emc.com (RSA Interceptor) for ; Tue, 25 Sep 2012 15:53:42 -0400 Received: from mxhub16.corp.emc.com (mxhub16.corp.emc.com [128.222.70.237]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PJre2a016532 for ; Tue, 25 Sep 2012 15:53:40 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub16.corp.emc.com ([128.222.70.237]) with mapi; Tue, 25 Sep 2012 15:53:40 -0400 From: "Black, David" To: "storm@ietf.org" Date: Tue, 25 Sep 2012 15:53:39 -0400 Thread-Topic: SPC-2 reserve/release: Proposed text - version 3 Thread-Index: Ac2bNKguGPJW+zTbTTC2uDGdWEV2OAAGl/vwAAGNYXA= Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE7952E@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: [storm] SPC-2 reserve/release: Proposed text - version 3 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 19:53:58 -0000 version 3: Responding to Mallikarjun's comments (plus some more edits) - > 1) IMHO, opening sentence of "I_T nexus state includes reservation state" > is a little too broad, I removed that sentence, removed I_T Nexus from new section title and tweak= ed the persistent reservation text to talk about state in general as opposed t= o I_T Nexus state in particular. > 2) Hard reset is not an LU reset - it is either an iSCSI Target Warm Rese= t, > or (more likely) an iSCSI Target Cold Reset. In either case, any of the > three resets should clear the reserve/release state. I removed the word "hard" - as you noted, an LU reset is sufficient to remo= ve a reserve/release reservation. > 3) Should use Time2Retain+Time2Wait, not Default Time2Wait Brilliant minds think alike - I got to that one in version 2 ;-). > 4) Session recovery does not necessarily preserve the I_T nexus state > (recovery may not occur within Time2Wait+Time2Retain, and may not use > the same ISID when it occurs), but session reinstatement and session > continuation certainly should.=20 I changed "session recovery" to "session reinstatement". > 5) Should articulate the underlying linkage between Time2Retain > (an iSCSI concept), and the reserve/release state (a SCSI state). I believe I also got to that one in version 2. version 2: I moved a paragraph break in response to Ralph's comments, and rewrote the first additional consideration to reference the connection timeout values in general. I also found some minor problems that need correction in Section 4.6.3.3 (see end of this message). Here's an initial attempt to propose text to deal with the reserve/ release topic in the consolidated iSCSI draft. Please comment, correct, suggest edits, etc. The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: 4.4.3.1. I_T Nexus State Certain nexus relationships contain an explicit state (e.g., initiator-specific mode pages) that may need to be preserved by the device server [SAM2] in a logical unit through changes or failures in the iSCSI layer (e.g., session failures). In order for that state to be restored, the iSCSI initiator should reestablish its session (re-login) to the same Target Portal Group using the previous ISID. That is, it should perform session recovery as described in Chapter 6. This is because the SCSI initiator port identifier and the SCSI target port identifier (or relative target port) form the datum that the SCSI logical unit device server uses to identify the I_T nexus. ------------------------------- First, for clarity, the following change should be made in the above 4.4.3.1 text to better identify the recovery mechanism: OLD That is, it should perform session recovery as described in Chapter 6. NEW That is, it should recover the session via connection reinstatement as described in Section 6.3.4. END I believe that a lower-case "should" remains appropriate for this text. ----------------------------------- NEW TEXT (second draft): 4.4.3.2. Reservations There are two reservation management methods defined in the SCSI standards, reserve/release reservations, based on the RESERVE and RELEASE commands [SPC2], and persistent reservations, based on the PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be used; persistent reservations SHOULD be used instead. State for persistent reservations is generally required to persist through changes and failures at the iSCSI layer that result in I_T Nexus failures, see [SPC3] for details and specific requirements. In contrast, [SPC2] does not specify detailed persistence requirements for reserve/release reservation state after an I_T Nexus failure. Nonetheless, when reserve/release reservations are supported by an iSCSI target, the preferred implementation approach is to preserve reserve/release reservation state along with other I_T Nexus state for iSCSI session reinstatement or session continuation via connection reinstatement. Two additional caveats apply to reserve/release reservations: - Retention of reserve/release reservation state for an extended period of time after connection failure may require a reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state when connection reinstatement is not performed. When reserve/release reservation state is retained after connection failure, the need for such resets may be reduced via suitable selection of the Time2Wait and Time2Retain timeout values that control connection state retention duration (see Section 7.5). =20 - Reserve/release reservations may not behave as expected when persistent reservations are also used on the same logical unit; see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE behavior" in [SPC4]. ----------------------- Reference impacts: both [SPC2] and [SPC3] need to be normative references, but [SPC4] can be an informative reference. ------------------------- I deliberately used "the preferred iSCSI implementation approach" wording for reserve/release reservation state preservation requirements, as SPC-2 is vague on this topic and I'm rather uncomfortable with placing a requirement as strong as a "SHOULD" on an obsolete mechanism that "SHOULD NOT" be used. --------------- New problem in Section 4.6.3.3 - in this text: The Logout response indicates that the connection or session cleanup is completed and no other responses will arrive on the connection (if received on the logging out connection). In addition, the Logout Response indicates how long the target will continue to hold resources for recovery (e.g., command execution that continues on a new connection) in the text key Time2Retain and how long the initiator must wait before proceeding with recovery in the text key Time2Wait. Time2Wait and Time2Retain are not text keys - they're binary fields in the Logout Response PDU. Two changes are in order: - "text key Time2Retain" -> "Time2Retain field" - "text key Time2Wait" -> "Time2Wait field" Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0 =A0=A0=A0=A0 FAX: +1 (508) 293-77= 86 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From Mark_Bakke@dell.com Tue Sep 25 12:54:48 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5995A21F85C7 for ; Tue, 25 Sep 2012 12:54:48 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.599 X-Spam-Level: X-Spam-Status: No, score=-106.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kv5MVqmiW614 for ; Tue, 25 Sep 2012 12:54:47 -0700 (PDT) Received: from aussmtpmrkps320.us.dell.com (aussmtpmrkps320.us.dell.com [143.166.224.254]) by ietfa.amsl.com (Postfix) with ESMTP id 64FA621F85F7 for ; Tue, 25 Sep 2012 12:54:47 -0700 (PDT) X-Loopcount0: from 64.238.244.148 X-IronPort-AV: E=Sophos;i="4.80,484,1344229200"; d="scan'208";a="3941980" From: Mark Bakke To: "Black, David" , "storm@ietf.org" Date: Tue, 25 Sep 2012 14:47:08 -0500 Thread-Topic: iSCSI MIB: Functional enhancements proposal Thread-Index: Ac2S3k0VAOHDeEHlQP66E4/tbPSxkAFVDrNQAMjXDKA= Message-ID: <975552A94CBC0F4DA60ED7B36C949CBA0414ADF805@shandy.Beer.Town> References: <8D3D17ACE214DC429325B2B98F3AE7120DCD1567@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE78FD8@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE78FD8@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [storm] iSCSI MIB: Functional enhancements proposal X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 19:54:48 -0000 David, this makes sense to me. I haven't heard any other comments asking f= or more than this. We will publish a new draft soon. Mark -----Original Message----- From: Black, David [mailto:david.black@emc.com]=20 Sent: Friday, September 21, 2012 2:55 PM To: Black, David; storm@ietf.org Subject: Re: [storm] iSCSI MIB: Functional enhancements proposal Importance: High In order to move forward, I suggest that the authors make the functional ch= anges [1] - [6], not make changes [A] - [F] and [I}, and use their best jud= gment on what (if anything) to do about [G] and [H] > [G] Additional session reporting, including digest, error recovery level,= session type > and additional counters for missing PDUs, data in PDUs, data out PDUs, a= sync PDUs. > [H] Report initial remote IP address and port for connections in addition= to actual > IP address and port - this would be for situations in which a redirectio= n has > occurred. Please submit a revised version of the MIB draft soon. Thanks, --David (storm WG co-chair). > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf=20 > Of Black, David > Sent: Friday, September 14, 2012 9:06 PM > To: storm@ietf.org > Subject: [storm] iSCSI MIB: Functional enhancements proposal >=20 > The iSCSI MIB has been waiting for a WG decision about what to=20 > incorporate from the list of functional enhancements suggested by the MIB= Doctor review. >=20 > With , based on consulting with the MIB draft=20 > editors, I offer the following initial proposal for the WG's=20 > consideration ... in the hope of getting to a decision. >=20 > --- Make the following functional enhancements: >=20 > [1] Add NOP counters at iSCSI session scope for heartbeat tracking [2]=20 > Add a port number to the iscsiTgtLoginFailure and iscsiIntrLoginFailure n= otifications, > and to the last failure info in iscsiInitiatorAttributesEntry. > [3] Add a description string to the iSCSI portal: >=20 > iscsiPortalDescr OBJECT-TYPE > SYNTAX SnmpAdminString > MAX-ACCESS read-only > STATUS current > DESCRIPTION > "A UTF-8 string, determined by the implementation to > describe the iSCSI portal. When only a single instance > is present, this object may be set to the zero-length > string; with multiple iSCSI portals, it may be used in > an implementation-dependent manner to describe the > respective portal, and could include information such as > HBA model, description and version or software driver and > version." >=20 > [4] Support "Target Unmapped" session failure reporting in the=20 > iscsiInstSessionFailure notification: >=20 > Editors: FailureType is an OID pointing to a stat in=20 > iscsiInstSsnErrorStatsTable, so we would need to add this: >=20 > iscsiInstSsnTgtUnmappedErrors Counter32 >=20 > [5] Add a port number to iscsiPortalAttributesEntry >=20 > [6] Add a timeout counter to iscsiInitiatorLogoutStatsEntry to distinguis= h implicit > logouts caused by timeouts from implicit logouts caused by other reasons= . >=20 > --- Do not make any other functional enhancements, including the=20 > following >=20 > [A] Summary notifications for large numbers of targets or initiators. (A= t most > a single summary with a count would be appropriate for each notification= type). > [B] RowPointer or other connection to transport state as part of > iscsiInstanceSsnErrorStatsEntry (can be found indirectly in current stru= cture). > [C] Support for reporting more than two digest algorithms (only one is st= andardized). > [D] Additional list of authentication methods for negotiation (already av= ailable based > on reporting of authorized entities). > [E] Additional support for mutual authentication (check the MIBs at both = ends instead). > [F] Support for digest provisioning at the iSCSI node level (use iSCSI po= rtal instead). > [G] Additional session reporting, including digest, error recovery level,= session type > and additional counters for missing PDUs, data in PDUs, data out PDUs, a= sync PDUs. > [H] Report initial remote IP address and port for connections in addition= to actual > IP address and port - this would be for situations in which a redirectio= n has > occurred. > [I] Report list of discovered targets (out of scope for this MIB). >=20 > -------------------- >=20 > Of the above list, I'd be particularly interested in comments on [A],=20 > [G] and [H] - if these were added to the MIB, would they be likely to=20 > be implemented and used? >=20 > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer EMC Corporation, 176 South St.,=20 > Hopkinton, MA=A0 01748 > +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-7= 786 > david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 > ---------------------------------------------------- >=20 >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From david.black@emc.com Tue Sep 25 14:39:10 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 18C741F0CBC for ; Tue, 25 Sep 2012 14:39:10 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XWLL3CiUufRf for ; Tue, 25 Sep 2012 14:39:09 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 318561F0C72 for ; Tue, 25 Sep 2012 14:39:08 -0700 (PDT) Received: from hop04-l1d11-si02.isus.emc.com (HOP04-L1D11-SI02.isus.emc.com [10.254.111.55]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PLd8OF008147 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 25 Sep 2012 17:39:08 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.221.253]) by hop04-l1d11-si02.isus.emc.com (RSA Interceptor) for ; Tue, 25 Sep 2012 17:38:46 -0400 Received: from mxhub12.corp.emc.com (mxhub12.corp.emc.com [10.254.92.107]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PLck1c028438 for ; Tue, 25 Sep 2012 17:38:46 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub12.corp.emc.com ([10.254.92.107]) with mapi; Tue, 25 Sep 2012 17:38:46 -0400 From: "Black, David" To: "storm@ietf.org" Date: Tue, 25 Sep 2012 17:38:44 -0400 Thread-Topic: SPC-2 reserve/release: Proposed text - version 4 Thread-Index: Ac2bNKguGPJW+zTbTTC2uDGdWEV2OAAGl/vwAAGNYXAAA3tHQA== Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE79578@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: [storm] FW: SPC-2 reserve/release: Proposed text - version 4 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 21:39:10 -0000 version 4 changes: More changes for Mallikarjun's comments, mostly a rewrite of the first additional caveat bullet. I've chosen not to reference Annex B of SPC-4 as that seems a bit far afield for a SCSI transport standard like iSCSI. version 3 changes: Responding to Mallikarjun's comments (plus some more edits) - > 1) IMHO, opening sentence of "I_T nexus state includes reservation state" > is a little too broad, I removed that sentence, removed I_T Nexus from new section title and tweak= ed the persistent reservation text to talk about state in general as opposed t= o I_T Nexus state in particular. > 2) Hard reset is not an LU reset - it is either an iSCSI Target Warm Rese= t, > or (more likely) an iSCSI Target Cold Reset. In either case, any of the > three resets should clear the reserve/release state. I removed the word "hard" - as you noted, an LU reset is sufficient to remo= ve a reserve/release reservation. > 3) Should use Time2Retain+Time2Wait, not Default Time2Wait Brilliant minds think alike - I got to that one in version 2 ;-). > 4) Session recovery does not necessarily preserve the I_T nexus state > (recovery may not occur within Time2Wait+Time2Retain, and may not use > the same ISID when it occurs), but session reinstatement and session > continuation certainly should.=20 I changed "session recovery" to "session reinstatement". > 5) Should articulate the underlying linkage between Time2Retain > (an iSCSI concept), and the reserve/release state (a SCSI state). I believe I also got to that one in version 2. version 2 changes: I moved a paragraph break in response to Ralph's comments, and rewrote the first additional consideration to reference the connection timeout values in general. I also found some minor problems that need correction in Section 4.6.3.3 (see end of this message). --- version 4 follows --- Here's an initial attempt to propose text to deal with the reserve/ release topic in the consolidated iSCSI draft. Please comment, correct, suggest edits, etc. The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: 4.4.3.1. I_T Nexus State Certain nexus relationships contain an explicit state (e.g., initiator-specific mode pages) that may need to be preserved by the device server [SAM2] in a logical unit through changes or failures in the iSCSI layer (e.g., session failures). In order for that state to be restored, the iSCSI initiator should reestablish its session (re-login) to the same Target Portal Group using the previous ISID. That is, it should perform session recovery as described in Chapter 6. This is because the SCSI initiator port identifier and the SCSI target port identifier (or relative target port) form the datum that the SCSI logical unit device server uses to identify the I_T nexus. ------------------------------- First, for clarity, the following change should be made in the above 4.4.3.1 text to better identify the recovery mechanism: OLD That is, it should perform session recovery as described in Chapter 6. NEW That is, it should recover the session via connection reinstatement as described in Section 6.3.4. END I believe that a lower-case "should" remains appropriate for this text. ----------------------------------- NEW TEXT (second draft): 4.4.3.2. Reservations There are two reservation management methods defined in the SCSI standards, reserve/release reservations, based on the RESERVE and RELEASE commands [SPC2], and persistent reservations, based on the PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be used; persistent reservations SHOULD be used instead. State for persistent reservations is required to persist through changes and failures at the iSCSI layer that result in I_T Nexus failures, see [SPC3] for details and specific requirements. In contrast, [SPC2] does not specify detailed persistence requirements for reserve/release reservation state after an I_T Nexus failure. Nonetheless, when reserve/release reservations are supported by an iSCSI target, the preferred implementation approach is to preserve reserve/release reservation state as part of the I_T Nexus state for iSCSI session reinstatement (see Section 6.3.5) or session continuation (see Section 6.3.6). Two additional caveats apply to reserve/release reservations: - When connection failure causes the iSCSI session to fail, and the session is not reinstated or continued, target retention of that session's reserve/release reservation state for an extended period of time may require the initiator to issue a reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state. The Time2Wait and Time2Retain timeout values (see Section 7.55) apply to retention of reserve/relea= se reservation state after iSCSI session failure because that state is part of the I_T Nexus state. The need for resets in this scenario may be reduced by suitable selection of values for these two timeouts. - Reserve/release reservations may not behave as expected when persistent reservations are also used on the same logical unit; see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE behavior" in [SPC4]. ----------------------- Reference impacts: both [SPC2] and [SPC3] need to be normative references, but [SPC4] can be an informative reference. ------------------------- I deliberately used "the preferred iSCSI implementation approach" wording for reserve/release reservation state preservation requirements, as SPC-2 is vague on this topic and I'm rather uncomfortable with placing a requirement as strong as a "SHOULD" on an obsolete mechanism that "SHOULD NOT" be used. --------------- New problem in Section 4.6.3.3 - in this text: The Logout response indicates that the connection or session cleanup is completed and no other responses will arrive on the connection (if received on the logging out connection). In addition, the Logout Response indicates how long the target will continue to hold resources for recovery (e.g., command execution that continues on a new connection) in the text key Time2Retain and how long the initiator must wait before proceeding with recovery in the text key Time2Wait. Time2Wait and Time2Retain are not text keys - they're binary fields in the Logout Response PDU. Two changes are in order: - "text key Time2Retain" -> "Time2Retain field" - "text key Time2Wait" -> "Time2Wait field" Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0 =A0=A0=A0=A0 FAX: +1 (508) 293-77= 86 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From cbm@chadalapaka.com Tue Sep 25 18:39:11 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 846E611E808D for ; Tue, 25 Sep 2012 18:39:11 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id POvlj4Aqx6ej for ; Tue, 25 Sep 2012 18:39:10 -0700 (PDT) Received: from am1outboundpool.messaging.microsoft.com (am1ehsobe002.messaging.microsoft.com [213.199.154.205]) by ietfa.amsl.com (Postfix) with ESMTP id E056E11E808A for ; Tue, 25 Sep 2012 18:39:09 -0700 (PDT) Received: from mail43-am1-R.bigfish.com (10.3.201.253) by AM1EHSOBE010.bigfish.com (10.3.204.30) with Microsoft SMTP Server id 14.1.225.23; Wed, 26 Sep 2012 01:39:08 +0000 Received: from mail43-am1 (localhost [127.0.0.1]) by mail43-am1-R.bigfish.com (Postfix) with ESMTP id 933B7C00C9 for ; Wed, 26 Sep 2012 01:39:08 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.240.117; KIP:(null); UIP:(null); IPV:NLI; H:BL2PRD0610HT004.namprd06.prod.outlook.com; RD:none; EFVD:NLI X-SpamScore: -26 X-BigFish: PS-26(zz9371I542M4015Id6f1izz1202h1d1ah1d2ahzz1033IL8275dhz2fh2a8h668h839hd25hf0ah107ah1288h12a5h12a9h12bdh137ah13b6h1155h) Received-SPF: pass (mail43-am1: domain of chadalapaka.com designates 157.56.240.117 as permitted sender) client-ip=157.56.240.117; envelope-from=cbm@chadalapaka.com; helo=BL2PRD0610HT004.namprd06.prod.outlook.com ; .outlook.com ; Received: from mail43-am1 (localhost.localdomain [127.0.0.1]) by mail43-am1 (MessageSwitch) id 1348623546773899_13573; Wed, 26 Sep 2012 01:39:06 +0000 (UTC) Received: from AM1EHSMHS012.bigfish.com (unknown [10.3.201.229]) by mail43-am1.bigfish.com (Postfix) with ESMTP id AF8131E022F; Wed, 26 Sep 2012 01:39:06 +0000 (UTC) Received: from BL2PRD0610HT004.namprd06.prod.outlook.com (157.56.240.117) by AM1EHSMHS012.bigfish.com (10.3.207.112) with Microsoft SMTP Server (TLS) id 14.1.225.23; Wed, 26 Sep 2012 01:39:01 +0000 Received: from BL2PRD0610MB361.namprd06.prod.outlook.com ([169.254.12.31]) by BL2PRD0610HT004.namprd06.prod.outlook.com ([10.255.101.39]) with mapi id 14.16.0190.008; Wed, 26 Sep 2012 01:39:00 +0000 From: Mallikarjun Chadalapaka To: "Black, David" , "storm@ietf.org" Thread-Topic: SPC-2 reserve/release: Proposed text - version 4 Thread-Index: Ac2bNKguGPJW+zTbTTC2uDGdWEV2OAAGl/vwAAGNYXAAA3tHQAAIrqHw Date: Wed, 26 Sep 2012 01:39:00 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE7120DE79578@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE79578@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [131.107.174.24] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: chadalapaka.com Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2012 01:39:11 -0000 Looks good to me overall. Two minor comments: 1) For 4.4.3.1, I have replaced the current sentence with the following one= (instead of the proposed): That is, it should reinstate the session via iSCSI session reinstatement (S= ection 6.3.5) or continue via session continuation (Section 6.3.6). 2) For proposed new text, I have tweaked the version 4 sentence below, keep= ing the existing implementations in view: VERSION 4: The Time2Wait and Time2Retain timeout values (see Section 7.55) apply to retention of reserve/relea= se reservation state after iSCSI session failure because that state is part of the I_T Nexus state. VERSION 5: Instead, implementations are strongly encouraged to apply the=20 Time2Wait and Time2Retain timeout values (see Section 7.5)=20 to manage retention of reserve/release reservation state after=20 iSCSI session failure because that state is part of the I_T nexus stat= e. Thanks. Mallikarjun -----Original Message----- From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of B= lack, David Sent: Tuesday, September 25, 2012 2:39 PM To: storm@ietf.org Subject: [storm] FW: SPC-2 reserve/release: Proposed text - version 4 version 4 changes: More changes for Mallikarjun's comments, mostly a rewrite of the first addi= tional caveat bullet. I've chosen not to reference Annex B of SPC-4 as that seems a bit far afield for a SCSI transport standard like iSC= SI. version 3 changes: Responding to Mallikarjun's comments (plus some more edits) - > 1) IMHO, opening sentence of "I_T nexus state includes reservation state" > is a little too broad, I removed that sentence, removed I_T Nexus from new section title and tweak= ed the persistent reservation text to talk about state in general as oppose= d to I_T Nexus state in particular. > 2) Hard reset is not an LU reset - it is either an iSCSI Target Warm=20 > Reset, or (more likely) an iSCSI Target Cold Reset. In either case,=20 > any of the three resets should clear the reserve/release state. I removed the word "hard" - as you noted, an LU reset is sufficient to remo= ve a reserve/release reservation. > 3) Should use Time2Retain+Time2Wait, not Default Time2Wait Brilliant minds think alike - I got to that one in version 2 ;-). > 4) Session recovery does not necessarily preserve the I_T nexus state=20 > (recovery may not occur within Time2Wait+Time2Retain, and may not use=20 > the same ISID when it occurs), but session reinstatement and session=20 > continuation certainly should. I changed "session recovery" to "session reinstatement". > 5) Should articulate the underlying linkage between Time2Retain (an=20 > iSCSI concept), and the reserve/release state (a SCSI state). I believe I also got to that one in version 2. version 2 changes: I moved a paragraph break in response to Ralph's comments, and rewrote the = first additional consideration to reference the connection timeout values i= n general. I also found some minor problems that need correction in Sectio= n 4.6.3.3 (see end of this message). --- version 4 follows --- Here's an initial attempt to propose text to deal with the reserve/ release= topic in the consolidated iSCSI draft. Please comment, correct, suggest e= dits, etc. The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: 4.4.3.1. I_T Nexus State Certain nexus relationships contain an explicit state (e.g., initiator-specific mode pages) that may need to be preserved by the device server [SAM2] in a logical unit through changes or failures in the iSCSI layer (e.g., session failures). In order for that state to be restored, the iSCSI initiator should reestablish its session (re-login) to the same Target Portal Group using the previous ISID. That is, it should perform session recovery as described in Chapter 6. This is because the SCSI initiator port identifier and the SCSI target port identifier (or relative target port) form the datum that the SCSI logical unit device server uses to identify the I_T nexus. ------------------------------- First, for clarity, the following change should be made in the above 4.4.3.= 1 text to better identify the recovery mechanism: OLD That is, it should perform session recovery as described in Chapter 6. NEW That is, it should recover the session via connection reinstatement as described in Section 6.3.4. END I believe that a lower-case "should" remains appropriate for this text. ----------------------------------- NEW TEXT (second draft): 4.4.3.2. Reservations There are two reservation management methods defined in the SCSI standards, reserve/release reservations, based on the RESERVE and RELEASE commands [SPC2], and persistent reservations, based on the PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be used; persistent reservations SHOULD be used instead. State for persistent reservations is required to persist through changes and failures at the iSCSI layer that result in I_T Nexus failures, see [SPC3] for details and specific requirements. In contrast, [SPC2] does not specify detailed persistence requirements for reserve/release reservation state after an I_T Nexus failure. Nonetheless, when reserve/release reservations are supported by an iSCSI target, the preferred implementation approach is to preserve reserve/release reservation state as part of the I_T Nexus state for iSCSI session reinstatement (see Section 6.3.5) or session continuation (see Section 6.3.6). Two additional caveats apply to reserve/release reservations: - When connection failure causes the iSCSI session to fail, and the session is not reinstated or continued, target retention of that session's reserve/release reservation state for an extended period of time may require the initiator to issue a reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state. The Time2Wait and Time2Retain timeout values (see Section 7.55) apply to retention of reserve/relea= se reservation state after iSCSI session failure because that state is part of the I_T Nexus state. The need for resets in this scenario may be reduced by suitable selection of values for these two timeouts. - Reserve/release reservations may not behave as expected when persistent reservations are also used on the same logical unit; see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE behavior" in [SPC4]. ----------------------- Reference impacts: both [SPC2] and [SPC3] need to be normative references, = but [SPC4] can be an informative reference. ------------------------- I deliberately used "the preferred iSCSI implementation approach" wording for reserve/release reservation state preservation requirements, as= SPC-2 is vague on this topic and I'm rather uncomfortable with placing a r= equirement as strong as a "SHOULD" on an obsolete mechanism that "SHOULD NO= T" be used. --------------- New problem in Section 4.6.3.3 - in this text: The Logout response indicates that the connection or session cleanup is completed and no other responses will arrive on the connection (if received on the logging out connection). In addition, the Logout Response indicates how long the target will continue to hold resources for recovery (e.g., command execution that continues on a new connection) in the text key Time2Retain and how long the initiator must wait before proceeding with recovery in the text key Time2Wait. Time2Wait and Time2Retain are not text keys - they're binary fields in the = Logout Response PDU. Two changes are in order: - "text key Time2Retain" -> "Time2Retain field" - "text key Time2Wait" -> "Time2Wait field" Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0 =A0=A0=A0=A0 FAX: +1 (508) 293-77= 86 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From Frederick.Knight@netapp.com Wed Sep 26 05:39:20 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 14EE921F8817 for ; Wed, 26 Sep 2012 05:39:20 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rqxMgva4g0lF for ; Wed, 26 Sep 2012 05:39:18 -0700 (PDT) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id BD2D721F86D1 for ; Wed, 26 Sep 2012 05:39:18 -0700 (PDT) X-IronPort-AV: E=Sophos;i="4.80,490,1344236400"; d="scan'208";a="694299660" Received: from smtp2.corp.netapp.com ([10.57.159.114]) by mx2-out.netapp.com with ESMTP; 26 Sep 2012 05:39:18 -0700 Received: from vmwexceht02-prd.hq.netapp.com (vmwexceht02-prd.hq.netapp.com [10.106.76.240]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8QCdHW6012739; Wed, 26 Sep 2012 05:39:17 -0700 (PDT) Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.102]) by vmwexceht02-prd.hq.netapp.com ([10.106.76.240]) with mapi id 14.02.0309.002; Wed, 26 Sep 2012 05:39:16 -0700 From: "Knight, Frederick" To: Mallikarjun Chadalapaka , "Black, David" , "storm@ietf.org" Thread-Topic: SPC-2 reserve/release: Proposed text - version 4 Thread-Index: Ac2bNKguGPJW+zTbTTC2uDGdWEV2OAAGl/vwAAGNYXAAA3tHQAAIrqHwABYKaSA= Date: Wed, 26 Sep 2012 12:39:15 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE7120DE79578@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.104.60.116] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2012 12:39:20 -0000 I'm concerned about this new link between Time2Wait and RESERVE state. Suc= h a link has not previously existed. I don't think we should create such a= linkage. Section 4.6.3.3 includes: .... In addition, the Logout Response indicates how long the target will co= ntinue to hold resources for recovery (e.g., command execution that continu= es on a new connection).... This and many other places where Time2Wait and Time2Retain are discussed, t= he points all have to do with command continuation; no mention of I_T state= , or RESERVE state, or mode pages, or anything else. Section 7.5 says: .... Time2Wait is the initial "respite time" before attempting an explicit/= implicit Logout for the CID in question or task reassignment for the affect= ed tasks (if any).... Or, task reassignment of affected tasks; with no mention of I_T state. Section 7.6 ties the termination of tasks to the Time2Wait / Time2Retain ti= mers. Section 11.14.5 makes the same statements about the task termination= linkage to those timers. Section 11.15.3 says those timers are "...the minimum amount of time, in se= conds, to wait before attempting task reassignment.". Section 8.3 defines Session State. And there is one sentence that I can fi= nd in all 345 pages that associates that session state with these timers. = It is in section 11.15.4 where it says: If it is the last connection of a s= ession, the whole session state is discarded after Time2Retain. I don't see anywhere that these timers are associated with any SCSI state, = or with the RESERVE state. Attempting to tie RESERVE state to these timers= is a NEW requirement. Our goal here was specifically to NOT create any ne= w requirements. We do not want to impact existing iSCSI implementations. Could we associat= e the action that occurred with the observable behavior the host sees? I d= on't like putting SCSI layer stuff in a transport layer spec, but it should= just be informative. Something like: If the initiator receives a UA: 29/00 or 29/01 or 29/04 then the RESERVE st= ate has been lost. (Maybe 29/04 belongs in a RESERVE state unknown list). If the initiator receives a UA: 29/07 then the RESERVE state has been prese= rved. I didn't list 29/02 or 29/03 or 29/05 or 29/06 since I view those as Parall= el SCSI specific. We can debate which items belong in which list, but woul= d this kind of observable behavior approach be better than a new requiremen= t? FYI - I didn't invent this; I copied it from SAM-5r11 (sub-clause 6.3.4= ) where it states it as a requirement on the target - that when an I_T NEX= US LOSS event occurs, if the state is retained, 29/07 is returned, and if t= he state is not retained, then 29/01 is returned. We also don't want to create any new requirement now in this consolidated d= raft that conflicts with this SAM-5 text, since we will eventually have to = comply with SAM-5 and its follow-ons. Another approach would be to ignore this for the consolidated draft, and fo= rce it into the iSCSI-SAM draft - which references SAM-4 and SAM-5 where I_= T Nexus loss is already defined and the above text already exists. Fred -----Original Message----- From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of M= allikarjun Chadalapaka Sent: Tuesday, September 25, 2012 9:39 PM To: Black, David; storm@ietf.org Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 Looks good to me overall. Two minor comments: 1) For 4.4.3.1, I have replaced the current sentence with the following one= (instead of the proposed): That is, it should reinstate the session via iSCSI session reinstatement (S= ection 6.3.5) or continue via session continuation (Section 6.3.6). 2) For proposed new text, I have tweaked the version 4 sentence below, keep= ing the existing implementations in view: VERSION 4: The Time2Wait and Time2Retain timeout values (see Section 7.55) apply to retention of reserve/relea= se reservation state after iSCSI session failure because that state is part of the I_T Nexus state. VERSION 5: Instead, implementations are strongly encouraged to apply the=20 Time2Wait and Time2Retain timeout values (see Section 7.5)=20 to manage retention of reserve/release reservation state after=20 iSCSI session failure because that state is part of the I_T nexus stat= e. Thanks. Mallikarjun -----Original Message----- From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of B= lack, David Sent: Tuesday, September 25, 2012 2:39 PM To: storm@ietf.org Subject: [storm] FW: SPC-2 reserve/release: Proposed text - version 4 version 4 changes: More changes for Mallikarjun's comments, mostly a rewrite of the first addi= tional caveat bullet. I've chosen not to reference Annex B of SPC-4 as that seems a bit far afield for a SCSI transport standard like iSC= SI. version 3 changes: Responding to Mallikarjun's comments (plus some more edits) - > 1) IMHO, opening sentence of "I_T nexus state includes reservation state" > is a little too broad, I removed that sentence, removed I_T Nexus from new section title and tweak= ed the persistent reservation text to talk about state in general as oppose= d to I_T Nexus state in particular. > 2) Hard reset is not an LU reset - it is either an iSCSI Target Warm=20 > Reset, or (more likely) an iSCSI Target Cold Reset. In either case,=20 > any of the three resets should clear the reserve/release state. I removed the word "hard" - as you noted, an LU reset is sufficient to remo= ve a reserve/release reservation. > 3) Should use Time2Retain+Time2Wait, not Default Time2Wait Brilliant minds think alike - I got to that one in version 2 ;-). > 4) Session recovery does not necessarily preserve the I_T nexus state=20 > (recovery may not occur within Time2Wait+Time2Retain, and may not use=20 > the same ISID when it occurs), but session reinstatement and session=20 > continuation certainly should. I changed "session recovery" to "session reinstatement". > 5) Should articulate the underlying linkage between Time2Retain (an=20 > iSCSI concept), and the reserve/release state (a SCSI state). I believe I also got to that one in version 2. version 2 changes: I moved a paragraph break in response to Ralph's comments, and rewrote the = first additional consideration to reference the connection timeout values i= n general. I also found some minor problems that need correction in Sectio= n 4.6.3.3 (see end of this message). --- version 4 follows --- Here's an initial attempt to propose text to deal with the reserve/ release= topic in the consolidated iSCSI draft. Please comment, correct, suggest e= dits, etc. The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: 4.4.3.1. I_T Nexus State Certain nexus relationships contain an explicit state (e.g., initiator-specific mode pages) that may need to be preserved by the device server [SAM2] in a logical unit through changes or failures in the iSCSI layer (e.g., session failures). In order for that state to be restored, the iSCSI initiator should reestablish its session (re-login) to the same Target Portal Group using the previous ISID. That is, it should perform session recovery as described in Chapter 6. This is because the SCSI initiator port identifier and the SCSI target port identifier (or relative target port) form the datum that the SCSI logical unit device server uses to identify the I_T nexus. ------------------------------- First, for clarity, the following change should be made in the above 4.4.3.= 1 text to better identify the recovery mechanism: OLD That is, it should perform session recovery as described in Chapter 6. NEW That is, it should recover the session via connection reinstatement as described in Section 6.3.4. END I believe that a lower-case "should" remains appropriate for this text. ----------------------------------- NEW TEXT (second draft): 4.4.3.2. Reservations There are two reservation management methods defined in the SCSI standards, reserve/release reservations, based on the RESERVE and RELEASE commands [SPC2], and persistent reservations, based on the PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be used; persistent reservations SHOULD be used instead. State for persistent reservations is required to persist through changes and failures at the iSCSI layer that result in I_T Nexus failures, see [SPC3] for details and specific requirements. In contrast, [SPC2] does not specify detailed persistence requirements for reserve/release reservation state after an I_T Nexus failure. Nonetheless, when reserve/release reservations are supported by an iSCSI target, the preferred implementation approach is to preserve reserve/release reservation state as part of the I_T Nexus state for iSCSI session reinstatement (see Section 6.3.5) or session continuation (see Section 6.3.6). Two additional caveats apply to reserve/release reservations: - When connection failure causes the iSCSI session to fail, and the session is not reinstated or continued, target retention of that session's reserve/release reservation state for an extended period of time may require the initiator to issue a reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state. The Time2Wait and Time2Retain timeout values (see Section 7.55) apply to retention of reserve/relea= se reservation state after iSCSI session failure because that state is part of the I_T Nexus state. The need for resets in this scenario may be reduced by suitable selection of values for these two timeouts. - Reserve/release reservations may not behave as expected when persistent reservations are also used on the same logical unit; see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE behavior" in [SPC4]. ----------------------- Reference impacts: both [SPC2] and [SPC3] need to be normative references, = but [SPC4] can be an informative reference. ------------------------- I deliberately used "the preferred iSCSI implementation approach" wording for reserve/release reservation state preservation requirements, as= SPC-2 is vague on this topic and I'm rather uncomfortable with placing a r= equirement as strong as a "SHOULD" on an obsolete mechanism that "SHOULD NO= T" be used. --------------- New problem in Section 4.6.3.3 - in this text: The Logout response indicates that the connection or session cleanup is completed and no other responses will arrive on the connection (if received on the logging out connection). In addition, the Logout Response indicates how long the target will continue to hold resources for recovery (e.g., command execution that continues on a new connection) in the text key Time2Retain and how long the initiator must wait before proceeding with recovery in the text key Time2Wait. Time2Wait and Time2Retain are not text keys - they're binary fields in the = Logout Response PDU. Two changes are in order: - "text key Time2Retain" -> "Time2Retain field" - "text key Time2Wait" -> "Time2Wait field" Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0 =A0=A0=A0=A0 FAX: +1 (508) 293-77= 86 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From david.black@emc.com Wed Sep 26 07:51:37 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E5CB321F861D for ; Wed, 26 Sep 2012 07:51:37 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZPtmLiSeM9h0 for ; Wed, 26 Sep 2012 07:51:36 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 2E91721F85F4 for ; Wed, 26 Sep 2012 07:51:35 -0700 (PDT) Received: from hop04-l1d11-si01.isus.emc.com (HOP04-L1D11-SI01.isus.emc.com [10.254.111.54]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8QEpQva002271 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 26 Sep 2012 10:51:33 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.221.251]) by hop04-l1d11-si01.isus.emc.com (RSA Interceptor); Wed, 26 Sep 2012 10:51:08 -0400 Received: from mxhub20.corp.emc.com (mxhub20.corp.emc.com [10.254.93.49]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8QEp61E029895; Wed, 26 Sep 2012 10:51:06 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub20.corp.emc.com ([10.254.93.49]) with mapi; Wed, 26 Sep 2012 10:51:05 -0400 From: "Black, David" To: "Knight, Frederick" , "storm@ietf.org" Date: Wed, 26 Sep 2012 10:51:04 -0400 Thread-Topic: SPC-2 reserve/release: Proposed text - version 4 Thread-Index: Ac2bNKguGPJW+zTbTTC2uDGdWEV2OAAGl/vwAAGNYXAAA3tHQAAIrqHwABYKaSAAA1Dz0A== Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE7969E@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE7120DE79578@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2012 14:51:38 -0000 Fred, > I don't see anywhere that these timers are associated with any SCSI state= , or > with the RESERVE state. Attempting to tie RESERVE state to these timers = is a > NEW requirement. Our goal here was specifically to NOT create any new > requirements. I think you have a point that this is too specific. What if we just delete mention of those timers? The bullet in question could then simply say: - When connection failure causes the iSCSI session to fail, and the session is not reinstated or continued, target retention of that session's reserve/release reservation state for an extended period of time may require the initiator to issue a reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state. That captures the primary concern that I wanted to alert implementers to. > We do not want to impact existing iSCSI implementations. Could we associ= ate > the action that occurred with the observable behavior the host sees? I d= on't > like putting SCSI layer stuff in a transport layer spec, but it should ju= st be > informative. The text that follows explicitly lists the SCSI ASC/Q values (e.g., 29/07, = in hex). That's something I'd prefer to avoid, e.g., as T10 could easily add additional ASC/Q values that are relevant to this situation. As an alternative, I suggest that we state the principle and point to SAM-4 for the details, e.g., add the following text to the end of 4.4.3.1: The specific Unit Attention condition that results from session re-establishment indicates whether or not nexus state was preserved, see Section 6.3.4 of [SAM4]. Would that be ok? Thanks, --David > -----Original Message----- > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > Sent: Wednesday, September 26, 2012 8:39 AM > To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org > Subject: RE: SPC-2 reserve/release: Proposed text - version 4 > > I'm concerned about this new link between Time2Wait and RESERVE state. S= uch a > link has not previously existed. I don't think we should create such a > linkage. > > Section 4.6.3.3 includes: > .... In addition, the Logout Response indicates how long the target will > continue to hold resources for recovery (e.g., command execution that > continues on a new connection).... > > This and many other places where Time2Wait and Time2Retain are discussed,= the > points all have to do with command continuation; no mention of I_T state,= or > RESERVE state, or mode pages, or anything else. > > Section 7.5 says: > .... Time2Wait is the initial "respite time" before attempting an > explicit/implicit Logout for the CID in question or task reassignment for= the > affected tasks (if any).... > > Or, task reassignment of affected tasks; with no mention of I_T state. > > Section 7.6 ties the termination of tasks to the Time2Wait / Time2Retain > timers. Section 11.14.5 makes the same statements about the task termina= tion > linkage to those timers. > > Section 11.15.3 says those timers are "...the minimum amount of time, in > seconds, to wait before attempting task reassignment.". > > Section 8.3 defines Session State. And there is one sentence that I can = find > in all 345 pages that associates that session state with these timers. I= t is > in section 11.15.4 where it says: If it is the last connection of a sessi= on, > the whole session state is discarded after Time2Retain. > > I don't see anywhere that these timers are associated with any SCSI state= , or > with the RESERVE state. Attempting to tie RESERVE state to these timers = is a > NEW requirement. Our goal here was specifically to NOT create any new > requirements. > > We do not want to impact existing iSCSI implementations. Could we associ= ate > the action that occurred with the observable behavior the host sees? I d= on't > like putting SCSI layer stuff in a transport layer spec, but it should ju= st be > informative. Something like: > > If the initiator receives a UA: 29/00 or 29/01 or 29/04 then the RESERVE = state > has been lost. (Maybe 29/04 belongs in a RESERVE state unknown list). > > If the initiator receives a UA: 29/07 then the RESERVE state has been > preserved. > > I didn't list 29/02 or 29/03 or 29/05 or 29/06 since I view those as Para= llel > SCSI specific. We can debate which items belong in which list, but would= this > kind of observable behavior approach be better than a new requirement? F= YI - > I didn't invent this; I copied it from SAM-5r11 (sub-clause 6.3.4) where = it > states it as a requirement on the target - that when an I_T NEXUS LOSS e= vent > occurs, if the state is retained, 29/07 is returned, and if the state is = not > retained, then 29/01 is returned. > > We also don't want to create any new requirement now in this consolidated > draft that conflicts with this SAM-5 text, since we will eventually have = to > comply with SAM-5 and its follow-ons. > > Another approach would be to ignore this for the consolidated draft, and = force > it into the iSCSI-SAM draft - which references SAM-4 and SAM-5 where I_T = Nexus > loss is already defined and the above text already exists. > > Fred > > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of > Mallikarjun Chadalapaka > Sent: Tuesday, September 25, 2012 9:39 PM > To: Black, David; storm@ietf.org > Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 > > Looks good to me overall. Two minor comments: > > 1) For 4.4.3.1, I have replaced the current sentence with the following o= ne > (instead of the proposed): > > That is, it should reinstate the session via iSCSI session reinstatement > (Section 6.3.5) or continue via session continuation (Section 6.3.6). > > 2) For proposed new text, I have tweaked the version 4 sentence below, ke= eping > the existing implementations in view: > > VERSION 4: > The Time2Wait and Time2Retain > timeout values (see Section 7.55) apply to retention of reserve/rel= ease > reservation state after iSCSI session failure because that state > is part of the I_T Nexus state. > > VERSION 5: > Instead, implementations are strongly encouraged to apply the > Time2Wait and Time2Retain timeout values (see Section 7.5) > to manage retention of reserve/release reservation state after > iSCSI session failure because that state is part of the I_T nexus st= ate. > > > Thanks. > > Mallikarjun > > > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of > Black, David > Sent: Tuesday, September 25, 2012 2:39 PM > To: storm@ietf.org > Subject: [storm] FW: SPC-2 reserve/release: Proposed text - version 4 > > version 4 changes: > > More changes for Mallikarjun's comments, mostly a rewrite of the first > additional caveat bullet. I've chosen not to reference Annex B of > SPC-4 as that seems a bit far afield for a SCSI transport standard like i= SCSI. > > version 3 changes: > > Responding to Mallikarjun's comments (plus some more edits) - > > > 1) IMHO, opening sentence of "I_T nexus state includes reservation stat= e" > > is a little too broad, > > I removed that sentence, removed I_T Nexus from new section title and twe= aked > the persistent reservation text to talk about state in general as opposed= to > I_T Nexus state in particular. > > > 2) Hard reset is not an LU reset - it is either an iSCSI Target Warm > > Reset, or (more likely) an iSCSI Target Cold Reset. In either case, > > any of the three resets should clear the reserve/release state. > > I removed the word "hard" - as you noted, an LU reset is sufficient to re= move > a reserve/release reservation. > > > 3) Should use Time2Retain+Time2Wait, not Default Time2Wait > > Brilliant minds think alike - I got to that one in version 2 ;-). > > > 4) Session recovery does not necessarily preserve the I_T nexus state > > (recovery may not occur within Time2Wait+Time2Retain, and may not use > > the same ISID when it occurs), but session reinstatement and session > > continuation certainly should. > > I changed "session recovery" to "session reinstatement". > > > 5) Should articulate the underlying linkage between Time2Retain (an > > iSCSI concept), and the reserve/release state (a SCSI state). > > I believe I also got to that one in version 2. > > version 2 changes: > > I moved a paragraph break in response to Ralph's comments, and rewrote th= e > first additional consideration to reference the connection timeout values= in > general. I also found some minor problems that need correction in Sectio= n > 4.6.3.3 (see end of this message). > > --- version 4 follows --- > > > > Here's an initial attempt to propose text to deal with the reserve/ relea= se > topic in the consolidated iSCSI draft. Please comment, correct, suggest > edits, etc. > > The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: > > 4.4.3.1. I_T Nexus State > > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages) that may need to be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures). In order for > that state to be restored, the iSCSI initiator should reestablish > its session (re-login) to the same Target Portal Group using the > previous ISID. That is, it should perform session recovery as > described in Chapter 6. This is because the SCSI initiator port > identifier and the SCSI target port identifier (or relative target > port) form the datum that the SCSI logical unit device server uses > to identify the I_T nexus. > > ------------------------------- > > First, for clarity, the following change should be made in the above 4.4.= 3.1 > text to better identify the recovery mechanism: > > OLD > That is, it should perform session recovery as > described in Chapter 6. > NEW > That is, it should recover the session via connection > reinstatement as described in Section 6.3.4. > END > > I believe that a lower-case "should" remains appropriate for this text. > > ----------------------------------- > > NEW TEXT (second draft): > > 4.4.3.2. Reservations > > There are two reservation management methods defined in the SCSI > standards, reserve/release reservations, based on the RESERVE and > RELEASE commands [SPC2], and persistent reservations, based on the > PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. > Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be > used; persistent reservations SHOULD be used instead. > > State for persistent reservations is required to persist > through changes and failures at the iSCSI layer that result in > I_T Nexus failures, see [SPC3] for details and specific requirements. > > In contrast, [SPC2] does not specify detailed persistence > requirements for reserve/release reservation state after an I_T > Nexus failure. Nonetheless, when reserve/release reservations are > supported by an iSCSI target, the preferred implementation approach > is to preserve reserve/release reservation state as part of the > I_T Nexus state for iSCSI session reinstatement (see Section 6.3.5) > or session continuation (see Section 6.3.6). > > Two additional caveats apply to reserve/release reservations: > > - When connection failure causes the iSCSI session to fail, and > the session is not reinstated or continued, target retention > of that session's reserve/release reservation state for an > extended period of time may require the initiator to issue a > reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order > to remove that reservation state. The Time2Wait and Time2Retain > timeout values (see Section 7.55) apply to retention of reserve/rel= ease > reservation state after iSCSI session failure because that state > is part of the I_T Nexus state. The need for resets in this > scenario may be reduced by suitable selection of values for > these two timeouts. > > - Reserve/release reservations may not behave as expected when > persistent reservations are also used on the same logical unit; > see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE > behavior" in [SPC4]. > > ----------------------- > > Reference impacts: both [SPC2] and [SPC3] need to be normative references= , but > [SPC4] can be an informative reference. > > ------------------------- > > I deliberately used "the preferred iSCSI implementation approach" > wording for reserve/release reservation state preservation requirements, = as > SPC-2 is vague on this topic and I'm rather uncomfortable with placing a > requirement as strong as a "SHOULD" on an obsolete mechanism that "SHOULD= NOT" > be used. > > --------------- > > New problem in Section 4.6.3.3 - in this text: > > The Logout response indicates that the connection or session > cleanup is completed and no other responses will arrive on the > connection (if received on the logging out connection). In > addition, the Logout Response indicates how long the target will > continue to hold resources for recovery (e.g., command execution > that continues on a new connection) in the text key Time2Retain > and how long the initiator must wait before proceeding with > recovery in the text key Time2Wait. > > Time2Wait and Time2Retain are not text keys - they're binary fields in th= e > Logout Response PDU. Two changes are in order: > - "text key Time2Retain" -> "Time2Retain field" > - "text key Time2Wait" -> "Time2Wait field" > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > david.black@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From david.brown1@emc.com Tue Sep 25 08:34:37 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D2CF621F87F8 for ; Tue, 25 Sep 2012 08:34:37 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QY+8LG0+KXeY for ; Tue, 25 Sep 2012 08:34:37 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 0ED9521F87D8 for ; Tue, 25 Sep 2012 08:34:36 -0700 (PDT) Received: from hop04-l1d11-si01.isus.emc.com (HOP04-L1D11-SI01.isus.emc.com [10.254.111.54]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PFYXNg004213 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 25 Sep 2012 11:34:34 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.222.129]) by hop04-l1d11-si01.isus.emc.com (RSA Interceptor); Tue, 25 Sep 2012 11:34:19 -0400 Received: from mxhub07.corp.emc.com (mxhub07.corp.emc.com [128.222.70.204]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8PFYI6X006830; Tue, 25 Sep 2012 11:34:18 -0400 Received: from mx32a.corp.emc.com ([169.254.1.141]) by mxhub07.corp.emc.com ([128.222.70.204]) with mapi; Tue, 25 Sep 2012 11:34:17 -0400 From: "brown, david (eng)" To: Milton Scritsmier , "storm@ietf.org" Date: Tue, 25 Sep 2012 11:34:16 -0400 Thread-Topic: [storm] iSCSI drafts - T10 and T11 references Thread-Index: Ac2bK9cmGswOBArNT9++o+M+/H+cRAAAl7Dw Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE71208DD94B1@MX15A.corp.emc.com> <50608249.8080803@alumni.hmc.edu> In-Reply-To: <50608249.8080803@alumni.hmc.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 X-Mailman-Approved-At: Wed, 26 Sep 2012 08:03:54 -0700 Subject: Re: [storm] iSCSI drafts - T10 and T11 references X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 15:50:32 -0000 An I_T nexus in SCSI-2 is different from an I_T nexus in SCSI-3. In parall= el SCSI, as you noted, an I_T nexus is created when a command is received a= nd ends when that command completes. =20 In SCSI-3, the nexus created by an inbound command is called an I_T_L nexus= , not a I_T nexus, since each command must be addressed to a specific LUN. = In SCSI-3, an I_T nexus is created by a port login. That nexus persists e= ven in the absence of commands. In short, a SCSI-2 RESERVE command creates a condition that outlives a SCSI= -2 I_T nexus. (Since the I_T nexus ends when the target sends status for th= e RESERVE command, the reservation had better survive past the end of that = nexus, or else it would be pretty useless.) The question before us is whet= her the reservation should survive after the end of a SCSI-3 I_T nexus. Si= nce SCSI-2 had nothing resembling a login, there is plenty of room to debat= e what happens to a SCSI-2 reservation in a SCSI-3 situation. The closest analogy to a SCSI-3 I_T NEXUS LOSS in SCSI-2 might be a reselec= tion timeout. When that happens, the target discovers that the initiator i= s no longer responding. The target can either retry later or reset the bus= . If he chooses the latter, the reservation ends. DJ Brown -----Original Message----- From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of M= ilton Scritsmier Sent: Monday, September 24, 2012 11:55 AM To: storm@ietf.org Subject: Re: [storm] iSCSI drafts - T10 and T11 references On 9/23/2012 4:38 AM, Knight, Frederick wrote: >=20 > We have references to I_T NEXUS LOSS events in RFC3720. The I_T Nexus lo= ss concept does not exist in SAM-2. SAM-2 does include references to RESER= VE and RELEASE commands. SAM-3 includes I_T Nexus loss, but it has no refe= rence to RESERVE state. There are allusions in RFC3720 about the intersect= ion of RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit. = RFC3720 references other documents (SAM2, SAM3, SBC, SPC3), none of which a= ddress the situation. >=20 The concepts of I_T NEXUS and RESERVE/RELEASE go all the way back to SCSI-2 in the mid-80s, on which I did a little work (see the spec at http://www.mit.edu/afs/sipb.mit.edu/contrib/doc/specs/protocol/scsi-2/s2-r1= 0l.txt ). In SCSI-2, the I_T NEXUS concept was created to deal with disconnect/reconnect on the parallel SCSI bus and the idea that a command could be active on both the host and target even though the bus was disconnected or the target was talking to another initiator. The RESERVE condition was meant to survive anything except a bus reset or a device power cycle (back then it was not assumed that devices had flash memory or any means of storing RESERVE/RELEASE status between power cycles or after a bus reset). So for example, if the initiator aborted a command, or persistent parity errors on the bus prevented communications, any RESERVE condition was still in effect. If a target was reset or power cycled, it would notify the host via CHECK CONDITION of the reset on the next command, which would implicitly tell the host that any RESERVE condition was no longer in effect. Section 9.2.12.1 of the SCSI-2 spec gives the conditions which clear a RESERVE condition, and it's clear that these are rather "catastrophic" conditions. Thus it was always the intent that the RESERVE condition was meant to survive an I_T NEXUS loss, although that concept is not well-defined in SCSI-2. I don't know if these concepts were carried through beyond parallel buses in SCSI-3, but that was the original intent. _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From Frederick.Knight@netapp.com Wed Sep 26 08:39:40 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7598921F851A for ; Wed, 26 Sep 2012 08:39:40 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zRIn5-VtwOoA for ; Wed, 26 Sep 2012 08:39:39 -0700 (PDT) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id F2AC321F84EB for ; Wed, 26 Sep 2012 08:39:38 -0700 (PDT) X-IronPort-AV: E=Sophos;i="4.80,490,1344236400"; d="scan'208";a="694372513" Received: from smtp2.corp.netapp.com ([10.57.159.114]) by mx2-out.netapp.com with ESMTP; 26 Sep 2012 08:39:38 -0700 Received: from vmwexceht01-prd.hq.netapp.com (vmwexceht01-prd.hq.netapp.com [10.106.76.239]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8QFdc93012611; Wed, 26 Sep 2012 08:39:38 -0700 (PDT) Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.102]) by vmwexceht01-prd.hq.netapp.com ([10.106.76.239]) with mapi id 14.02.0309.002; Wed, 26 Sep 2012 08:39:38 -0700 From: "Knight, Frederick" To: "Black, David" , "storm@ietf.org" Thread-Topic: SPC-2 reserve/release: Proposed text - version 4 Thread-Index: Ac2bNKguGPJW+zTbTTC2uDGdWEV2OAAGl/vwAAGNYXAAA3tHQAAIrqHwABYKaSAAA1Dz0AADSX6Q Date: Wed, 26 Sep 2012 15:39:36 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE7120DE79578@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE7969E@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE7969E@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.106.53.53] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2012 15:39:40 -0000 Yes, I think we've got it. Since I can't define "an extended period of tim= e", here's my take: - When connection failure causes the iSCSI session to fail, and the session is not reinstated or continued, target retention of that session's reserve/release reservation state may require an initiator to issue a reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state. The specific Unit Attention condition that results from session re-establishment indicates whether or not nexus state was preserved, see Section 6.3.4 of [SAM4]. Along with: > OLD > That is, it should perform session recovery as > described in Chapter 6. > NEW > That is, it should recover the session via connection > reinstatement as described in Section 6.3.4. > END Fred -----Original Message----- From: Black, David [mailto:david.black@emc.com]=20 Sent: Wednesday, September 26, 2012 10:51 AM To: Knight, Frederick; storm@ietf.org Subject: RE: SPC-2 reserve/release: Proposed text - version 4 Fred, > I don't see anywhere that these timers are associated with any SCSI=20 > state, or with the RESERVE state. Attempting to tie RESERVE state to=20 > these timers is a NEW requirement. Our goal here was specifically to=20 > NOT create any new requirements. I think you have a point that this is too specific. What if we just delete= mention of those timers? The bullet in question could then simply say: - When connection failure causes the iSCSI session to fail, and the session is not reinstated or continued, target retention of that session's reserve/release reservation state for an extended period of time may require the initiator to issue a reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state. That captures the primary concern that I wanted to alert implementers to. > We do not want to impact existing iSCSI implementations. Could we=20 > associate the action that occurred with the observable behavior the=20 > host sees? I don't like putting SCSI layer stuff in a transport layer=20 > spec, but it should just be informative. The text that follows explicitly lists the SCSI ASC/Q values (e.g., 29/07, = in hex). That's something I'd prefer to avoid, e.g., as T10 could easily a= dd additional ASC/Q values that are relevant to this situation. As an alternative, I suggest that we state the principle and point to SAM-4= for the details, e.g., add the following text to the end of 4.4.3.1: The specific Unit Attention condition that results from session re-establishment indicates whether or not nexus state was preserved, see Section 6.3.4 of [SAM4]. Would that be ok? Thanks, --David > -----Original Message----- > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > Sent: Wednesday, September 26, 2012 8:39 AM > To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org > Subject: RE: SPC-2 reserve/release: Proposed text - version 4 > > I'm concerned about this new link between Time2Wait and RESERVE state. =20 > Such a link has not previously existed. I don't think we should=20 > create such a linkage. > > Section 4.6.3.3 includes: > .... In addition, the Logout Response indicates how long the target=20 > will continue to hold resources for recovery (e.g., command execution=20 > that continues on a new connection).... > > This and many other places where Time2Wait and Time2Retain are=20 > discussed, the points all have to do with command continuation; no=20 > mention of I_T state, or RESERVE state, or mode pages, or anything else. > > Section 7.5 says: > .... Time2Wait is the initial "respite time" before attempting an=20 > explicit/implicit Logout for the CID in question or task reassignment=20 > for the affected tasks (if any).... > > Or, task reassignment of affected tasks; with no mention of I_T state. > > Section 7.6 ties the termination of tasks to the Time2Wait /=20 > Time2Retain timers. Section 11.14.5 makes the same statements about=20 > the task termination linkage to those timers. > > Section 11.15.3 says those timers are "...the minimum amount of time,=20 > in seconds, to wait before attempting task reassignment.". > > Section 8.3 defines Session State. And there is one sentence that I=20 > can find in all 345 pages that associates that session state with=20 > these timers. It is in section 11.15.4 where it says: If it is the=20 > last connection of a session, the whole session state is discarded after = Time2Retain. > > I don't see anywhere that these timers are associated with any SCSI=20 > state, or with the RESERVE state. Attempting to tie RESERVE state to=20 > these timers is a NEW requirement. Our goal here was specifically to=20 > NOT create any new requirements. > > We do not want to impact existing iSCSI implementations. Could we=20 > associate the action that occurred with the observable behavior the=20 > host sees? I don't like putting SCSI layer stuff in a transport layer=20 > spec, but it should just be informative. Something like: > > If the initiator receives a UA: 29/00 or 29/01 or 29/04 then the=20 > RESERVE state has been lost. (Maybe 29/04 belongs in a RESERVE state unk= nown list). > > If the initiator receives a UA: 29/07 then the RESERVE state has been=20 > preserved. > > I didn't list 29/02 or 29/03 or 29/05 or 29/06 since I view those as=20 > Parallel SCSI specific. We can debate which items belong in which=20 > list, but would this kind of observable behavior approach be better=20 > than a new requirement? FYI - I didn't invent this; I copied it from=20 > SAM-5r11 (sub-clause 6.3.4) where it states it as a requirement on the=20 > target - that when an I_T NEXUS LOSS event occurs, if the state is=20 > retained, 29/07 is returned, and if the state is not retained, then 29/01= is returned. > > We also don't want to create any new requirement now in this=20 > consolidated draft that conflicts with this SAM-5 text, since we will=20 > eventually have to comply with SAM-5 and its follow-ons. > > Another approach would be to ignore this for the consolidated draft,=20 > and force it into the iSCSI-SAM draft - which references SAM-4 and=20 > SAM-5 where I_T Nexus loss is already defined and the above text already = exists. > > Fred > > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf=20 > Of Mallikarjun Chadalapaka > Sent: Tuesday, September 25, 2012 9:39 PM > To: Black, David; storm@ietf.org > Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 > > Looks good to me overall. Two minor comments: > > 1) For 4.4.3.1, I have replaced the current sentence with the=20 > following one (instead of the proposed): > > That is, it should reinstate the session via iSCSI session=20 > reinstatement (Section 6.3.5) or continue via session continuation (Secti= on 6.3.6). > > 2) For proposed new text, I have tweaked the version 4 sentence below,=20 > keeping the existing implementations in view: > > VERSION 4: > The Time2Wait and Time2Retain > timeout values (see Section 7.55) apply to retention of reserve/rel= ease > reservation state after iSCSI session failure because that state > is part of the I_T Nexus state. > > VERSION 5: > Instead, implementations are strongly encouraged to apply the > Time2Wait and Time2Retain timeout values (see Section 7.5) > to manage retention of reserve/release reservation state after > iSCSI session failure because that state is part of the I_T nexus st= ate. > > > Thanks. > > Mallikarjun > > > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf=20 > Of Black, David > Sent: Tuesday, September 25, 2012 2:39 PM > To: storm@ietf.org > Subject: [storm] FW: SPC-2 reserve/release: Proposed text - version 4 > > version 4 changes: > > More changes for Mallikarjun's comments, mostly a rewrite of the first=20 > additional caveat bullet. I've chosen not to reference Annex B of > SPC-4 as that seems a bit far afield for a SCSI transport standard like i= SCSI. > > version 3 changes: > > Responding to Mallikarjun's comments (plus some more edits) - > > > 1) IMHO, opening sentence of "I_T nexus state includes reservation stat= e" > > is a little too broad, > > I removed that sentence, removed I_T Nexus from new section title and=20 > tweaked the persistent reservation text to talk about state in general=20 > as opposed to I_T Nexus state in particular. > > > 2) Hard reset is not an LU reset - it is either an iSCSI Target Warm=20 > > Reset, or (more likely) an iSCSI Target Cold Reset. In either case,=20 > > any of the three resets should clear the reserve/release state. > > I removed the word "hard" - as you noted, an LU reset is sufficient to=20 > remove a reserve/release reservation. > > > 3) Should use Time2Retain+Time2Wait, not Default Time2Wait > > Brilliant minds think alike - I got to that one in version 2 ;-). > > > 4) Session recovery does not necessarily preserve the I_T nexus=20 > > state (recovery may not occur within Time2Wait+Time2Retain, and may=20 > > not use the same ISID when it occurs), but session reinstatement and=20 > > session continuation certainly should. > > I changed "session recovery" to "session reinstatement". > > > 5) Should articulate the underlying linkage between Time2Retain (an=20 > > iSCSI concept), and the reserve/release state (a SCSI state). > > I believe I also got to that one in version 2. > > version 2 changes: > > I moved a paragraph break in response to Ralph's comments, and rewrote=20 > the first additional consideration to reference the connection timeout=20 > values in general. I also found some minor problems that need=20 > correction in Section > 4.6.3.3 (see end of this message). > > --- version 4 follows --- > > > > Here's an initial attempt to propose text to deal with the reserve/=20 > release topic in the consolidated iSCSI draft. Please comment,=20 > correct, suggest edits, etc. > > The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: > > 4.4.3.1. I_T Nexus State > > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages) that may need to be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures). In order for > that state to be restored, the iSCSI initiator should reestablish > its session (re-login) to the same Target Portal Group using the > previous ISID. That is, it should perform session recovery as > described in Chapter 6. This is because the SCSI initiator port > identifier and the SCSI target port identifier (or relative target > port) form the datum that the SCSI logical unit device server uses > to identify the I_T nexus. > > ------------------------------- > > First, for clarity, the following change should be made in the above=20 > 4.4.3.1 text to better identify the recovery mechanism: > > OLD > That is, it should perform session recovery as > described in Chapter 6. > NEW > That is, it should recover the session via connection > reinstatement as described in Section 6.3.4. > END > > I believe that a lower-case "should" remains appropriate for this text. > > ----------------------------------- > > NEW TEXT (second draft): > > 4.4.3.2. Reservations > > There are two reservation management methods defined in the SCSI > standards, reserve/release reservations, based on the RESERVE and > RELEASE commands [SPC2], and persistent reservations, based on the > PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. > Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be > used; persistent reservations SHOULD be used instead. > > State for persistent reservations is required to persist > through changes and failures at the iSCSI layer that result in > I_T Nexus failures, see [SPC3] for details and specific requirements. > > In contrast, [SPC2] does not specify detailed persistence > requirements for reserve/release reservation state after an I_T > Nexus failure. Nonetheless, when reserve/release reservations are > supported by an iSCSI target, the preferred implementation approach > is to preserve reserve/release reservation state as part of the > I_T Nexus state for iSCSI session reinstatement (see Section 6.3.5) > or session continuation (see Section 6.3.6). > > Two additional caveats apply to reserve/release reservations: > > - When connection failure causes the iSCSI session to fail, and > the session is not reinstated or continued, target retention > of that session's reserve/release reservation state for an > extended period of time may require the initiator to issue a > reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order > to remove that reservation state. The Time2Wait and Time2Retain > timeout values (see Section 7.55) apply to retention of reserve/rel= ease > reservation state after iSCSI session failure because that state > is part of the I_T Nexus state. The need for resets in this > scenario may be reduced by suitable selection of values for > these two timeouts. > > - Reserve/release reservations may not behave as expected when > persistent reservations are also used on the same logical unit; > see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE > behavior" in [SPC4]. > > ----------------------- > > Reference impacts: both [SPC2] and [SPC3] need to be normative=20 > references, but [SPC4] can be an informative reference. > > ------------------------- > > I deliberately used "the preferred iSCSI implementation approach" > wording for reserve/release reservation state preservation=20 > requirements, as > SPC-2 is vague on this topic and I'm rather uncomfortable with placing=20 > a requirement as strong as a "SHOULD" on an obsolete mechanism that "SHOU= LD NOT" > be used. > > --------------- > > New problem in Section 4.6.3.3 - in this text: > > The Logout response indicates that the connection or session > cleanup is completed and no other responses will arrive on the > connection (if received on the logging out connection). In > addition, the Logout Response indicates how long the target will > continue to hold resources for recovery (e.g., command execution > that continues on a new connection) in the text key Time2Retain > and how long the initiator must wait before proceeding with > recovery in the text key Time2Wait. > > Time2Wait and Time2Retain are not text keys - they're binary fields in=20 > the Logout Response PDU. Two changes are in order: > - "text key Time2Retain" -> "Time2Retain field" > - "text key Time2Wait" -> "Time2Wait field" > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer EMC Corporation, 176 South St.,=20 > Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > david.black@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > > > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From david.black@emc.com Wed Sep 26 08:47:28 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3BA8D21F84FA for ; Wed, 26 Sep 2012 08:47:28 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JUwDQkmcI2NB for ; Wed, 26 Sep 2012 08:47:26 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 4A15721F84E4 for ; Wed, 26 Sep 2012 08:47:25 -0700 (PDT) Received: from hop04-l1d11-si01.isus.emc.com (HOP04-L1D11-SI01.isus.emc.com [10.254.111.54]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8QFlMbR003350 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 26 Sep 2012 11:47:25 -0400 Received: from mailhub.lss.emc.com (mailhubhoprd04.lss.emc.com [10.254.222.226]) by hop04-l1d11-si01.isus.emc.com (RSA Interceptor); Wed, 26 Sep 2012 11:46:51 -0400 Received: from mxhub29.corp.emc.com (mxhub29.corp.emc.com [128.222.70.169]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8QFjAPx022087; Wed, 26 Sep 2012 11:46:51 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub29.corp.emc.com ([128.222.70.169]) with mapi; Wed, 26 Sep 2012 11:46:40 -0400 From: "Black, David" To: "Knight, Frederick" , "storm@ietf.org" Date: Wed, 26 Sep 2012 11:46:38 -0400 Thread-Topic: SPC-2 reserve/release: Proposed text - version 4 Thread-Index: Ac2bNKguGPJW+zTbTTC2uDGdWEV2OAAGl/vwAAGNYXAAA3tHQAAIrqHwABYKaSAAA1Dz0AADSX6QAAFXeRA= Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE796DB@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE7120DE79578@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE7969E@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2012 15:47:28 -0000 That text looks good to me, but I have one more minor change ... ... I'd prefer to put the second paragraph at the end of 4.4.3.1, as it applies to all nexus state not just reservations: The specific Unit Attention condition that results from session re-establishment indicates whether or not nexus state was preserved, see Section 6.3.4 of [SAM4]. Thanks, --David > -----Original Message----- > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > Sent: Wednesday, September 26, 2012 11:40 AM > To: Black, David; storm@ietf.org > Subject: RE: SPC-2 reserve/release: Proposed text - version 4 > > Yes, I think we've got it. Since I can't define "an extended period of t= ime", > here's my take: > > - When connection failure causes the iSCSI session to fail, and > the session is not reinstated or continued, target retention > of that session's reserve/release reservation state > may require an initiator to issue a > reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order > to remove that reservation state. > > The specific Unit Attention condition that results from session > re-establishment indicates whether or not nexus state was preserved, > see Section 6.3.4 of [SAM4]. > > Along with: > > OLD > > That is, it should perform session recovery as > > described in Chapter 6. > > NEW > > That is, it should recover the session via connection > > reinstatement as described in Section 6.3.4. > > END > > Fred > > > -----Original Message----- > From: Black, David [mailto:david.black@emc.com] > Sent: Wednesday, September 26, 2012 10:51 AM > To: Knight, Frederick; storm@ietf.org > Subject: RE: SPC-2 reserve/release: Proposed text - version 4 > > Fred, > > > I don't see anywhere that these timers are associated with any SCSI > > state, or with the RESERVE state. Attempting to tie RESERVE state to > > these timers is a NEW requirement. Our goal here was specifically to > > NOT create any new requirements. > > I think you have a point that this is too specific. What if we just dele= te > mention of those timers? The bullet in question could then simply say: > > - When connection failure causes the iSCSI session to fail, and > the session is not reinstated or continued, target retention > of that session's reserve/release reservation state for an > extended period of time may require the initiator to issue a > reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order > to remove that reservation state. > > That captures the primary concern that I wanted to alert implementers to. > > > We do not want to impact existing iSCSI implementations. Could we > > associate the action that occurred with the observable behavior the > > host sees? I don't like putting SCSI layer stuff in a transport layer > > spec, but it should just be informative. > > The text that follows explicitly lists the SCSI ASC/Q values (e.g., 29/07= , in > hex). That's something I'd prefer to avoid, e.g., as T10 could easily ad= d > additional ASC/Q values that are relevant to this situation. > > As an alternative, I suggest that we state the principle and point to SAM= -4 > for the details, e.g., add the following text to the end of 4.4.3.1: > > The specific Unit Attention condition that results from session > re-establishment indicates whether or not nexus state was preserved, > see Section 6.3.4 of [SAM4]. > > Would that be ok? > > Thanks, > --David > > > -----Original Message----- > > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > > Sent: Wednesday, September 26, 2012 8:39 AM > > To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org > > Subject: RE: SPC-2 reserve/release: Proposed text - version 4 > > > > I'm concerned about this new link between Time2Wait and RESERVE state. > > Such a link has not previously existed. I don't think we should > > create such a linkage. > > > > Section 4.6.3.3 includes: > > .... In addition, the Logout Response indicates how long the target > > will continue to hold resources for recovery (e.g., command execution > > that continues on a new connection).... > > > > This and many other places where Time2Wait and Time2Retain are > > discussed, the points all have to do with command continuation; no > > mention of I_T state, or RESERVE state, or mode pages, or anything else= . > > > > Section 7.5 says: > > .... Time2Wait is the initial "respite time" before attempting an > > explicit/implicit Logout for the CID in question or task reassignment > > for the affected tasks (if any).... > > > > Or, task reassignment of affected tasks; with no mention of I_T state. > > > > Section 7.6 ties the termination of tasks to the Time2Wait / > > Time2Retain timers. Section 11.14.5 makes the same statements about > > the task termination linkage to those timers. > > > > Section 11.15.3 says those timers are "...the minimum amount of time, > > in seconds, to wait before attempting task reassignment.". > > > > Section 8.3 defines Session State. And there is one sentence that I > > can find in all 345 pages that associates that session state with > > these timers. It is in section 11.15.4 where it says: If it is the > > last connection of a session, the whole session state is discarded afte= r > Time2Retain. > > > > I don't see anywhere that these timers are associated with any SCSI > > state, or with the RESERVE state. Attempting to tie RESERVE state to > > these timers is a NEW requirement. Our goal here was specifically to > > NOT create any new requirements. > > > > We do not want to impact existing iSCSI implementations. Could we > > associate the action that occurred with the observable behavior the > > host sees? I don't like putting SCSI layer stuff in a transport layer > > spec, but it should just be informative. Something like: > > > > If the initiator receives a UA: 29/00 or 29/01 or 29/04 then the > > RESERVE state has been lost. (Maybe 29/04 belongs in a RESERVE state > unknown list). > > > > If the initiator receives a UA: 29/07 then the RESERVE state has been > > preserved. > > > > I didn't list 29/02 or 29/03 or 29/05 or 29/06 since I view those as > > Parallel SCSI specific. We can debate which items belong in which > > list, but would this kind of observable behavior approach be better > > than a new requirement? FYI - I didn't invent this; I copied it from > > SAM-5r11 (sub-clause 6.3.4) where it states it as a requirement on the > > target - that when an I_T NEXUS LOSS event occurs, if the state is > > retained, 29/07 is returned, and if the state is not retained, then 29/= 01 is > returned. > > > > We also don't want to create any new requirement now in this > > consolidated draft that conflicts with this SAM-5 text, since we will > > eventually have to comply with SAM-5 and its follow-ons. > > > > Another approach would be to ignore this for the consolidated draft, > > and force it into the iSCSI-SAM draft - which references SAM-4 and > > SAM-5 where I_T Nexus loss is already defined and the above text alread= y > exists. > > > > Fred > > > > -----Original Message----- > > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf > > Of Mallikarjun Chadalapaka > > Sent: Tuesday, September 25, 2012 9:39 PM > > To: Black, David; storm@ietf.org > > Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 > > > > Looks good to me overall. Two minor comments: > > > > 1) For 4.4.3.1, I have replaced the current sentence with the > > following one (instead of the proposed): > > > > That is, it should reinstate the session via iSCSI session > > reinstatement (Section 6.3.5) or continue via session continuation (Sec= tion > 6.3.6). > > > > 2) For proposed new text, I have tweaked the version 4 sentence below, > > keeping the existing implementations in view: > > > > VERSION 4: > > The Time2Wait and Time2Retain > > timeout values (see Section 7.55) apply to retention of > reserve/release > > reservation state after iSCSI session failure because that state > > is part of the I_T Nexus state. > > > > VERSION 5: > > Instead, implementations are strongly encouraged to apply the > > Time2Wait and Time2Retain timeout values (see Section 7.5) > > to manage retention of reserve/release reservation state after > > iSCSI session failure because that state is part of the I_T nexus > state. > > > > > > Thanks. > > > > Mallikarjun > > > > > > -----Original Message----- > > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf > > Of Black, David > > Sent: Tuesday, September 25, 2012 2:39 PM > > To: storm@ietf.org > > Subject: [storm] FW: SPC-2 reserve/release: Proposed text - version 4 > > > > version 4 changes: > > > > More changes for Mallikarjun's comments, mostly a rewrite of the first > > additional caveat bullet. I've chosen not to reference Annex B of > > SPC-4 as that seems a bit far afield for a SCSI transport standard like > iSCSI. > > > > version 3 changes: > > > > Responding to Mallikarjun's comments (plus some more edits) - > > > > > 1) IMHO, opening sentence of "I_T nexus state includes reservation st= ate" > > > is a little too broad, > > > > I removed that sentence, removed I_T Nexus from new section title and > > tweaked the persistent reservation text to talk about state in general > > as opposed to I_T Nexus state in particular. > > > > > 2) Hard reset is not an LU reset - it is either an iSCSI Target Warm > > > Reset, or (more likely) an iSCSI Target Cold Reset. In either case, > > > any of the three resets should clear the reserve/release state. > > > > I removed the word "hard" - as you noted, an LU reset is sufficient to > > remove a reserve/release reservation. > > > > > 3) Should use Time2Retain+Time2Wait, not Default Time2Wait > > > > Brilliant minds think alike - I got to that one in version 2 ;-). > > > > > 4) Session recovery does not necessarily preserve the I_T nexus > > > state (recovery may not occur within Time2Wait+Time2Retain, and may > > > not use the same ISID when it occurs), but session reinstatement and > > > session continuation certainly should. > > > > I changed "session recovery" to "session reinstatement". > > > > > 5) Should articulate the underlying linkage between Time2Retain (an > > > iSCSI concept), and the reserve/release state (a SCSI state). > > > > I believe I also got to that one in version 2. > > > > version 2 changes: > > > > I moved a paragraph break in response to Ralph's comments, and rewrote > > the first additional consideration to reference the connection timeout > > values in general. I also found some minor problems that need > > correction in Section > > 4.6.3.3 (see end of this message). > > > > --- version 4 follows --- > > > > > > > > Here's an initial attempt to propose text to deal with the reserve/ > > release topic in the consolidated iSCSI draft. Please comment, > > correct, suggest edits, etc. > > > > The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: > > > > 4.4.3.1. I_T Nexus State > > > > Certain nexus relationships contain an explicit state (e.g., > > initiator-specific mode pages) that may need to be preserved by > > the device server [SAM2] in a logical unit through changes or > > failures in the iSCSI layer (e.g., session failures). In order for > > that state to be restored, the iSCSI initiator should reestablish > > its session (re-login) to the same Target Portal Group using the > > previous ISID. That is, it should perform session recovery as > > described in Chapter 6. This is because the SCSI initiator port > > identifier and the SCSI target port identifier (or relative target > > port) form the datum that the SCSI logical unit device server uses > > to identify the I_T nexus. > > > > ------------------------------- > > > > First, for clarity, the following change should be made in the above > > 4.4.3.1 text to better identify the recovery mechanism: > > > > OLD > > That is, it should perform session recovery as > > described in Chapter 6. > > NEW > > That is, it should recover the session via connection > > reinstatement as described in Section 6.3.4. > > END > > > > I believe that a lower-case "should" remains appropriate for this text. > > > > ----------------------------------- > > > > NEW TEXT (second draft): > > > > 4.4.3.2. Reservations > > > > There are two reservation management methods defined in the SCSI > > standards, reserve/release reservations, based on the RESERVE and > > RELEASE commands [SPC2], and persistent reservations, based on the > > PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. > > Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be > > used; persistent reservations SHOULD be used instead. > > > > State for persistent reservations is required to persist > > through changes and failures at the iSCSI layer that result in > > I_T Nexus failures, see [SPC3] for details and specific requirements. > > > > In contrast, [SPC2] does not specify detailed persistence > > requirements for reserve/release reservation state after an I_T > > Nexus failure. Nonetheless, when reserve/release reservations are > > supported by an iSCSI target, the preferred implementation approach > > is to preserve reserve/release reservation state as part of the > > I_T Nexus state for iSCSI session reinstatement (see Section 6.3.5) > > or session continuation (see Section 6.3.6). > > > > Two additional caveats apply to reserve/release reservations: > > > > - When connection failure causes the iSCSI session to fail, and > > the session is not reinstated or continued, target retention > > of that session's reserve/release reservation state for an > > extended period of time may require the initiator to issue a > > reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order > > to remove that reservation state. The Time2Wait and Time2Retain > > timeout values (see Section 7.55) apply to retention of > reserve/release > > reservation state after iSCSI session failure because that state > > is part of the I_T Nexus state. The need for resets in this > > scenario may be reduced by suitable selection of values for > > these two timeouts. > > > > - Reserve/release reservations may not behave as expected when > > persistent reservations are also used on the same logical unit; > > see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE > > behavior" in [SPC4]. > > > > ----------------------- > > > > Reference impacts: both [SPC2] and [SPC3] need to be normative > > references, but [SPC4] can be an informative reference. > > > > ------------------------- > > > > I deliberately used "the preferred iSCSI implementation approach" > > wording for reserve/release reservation state preservation > > requirements, as > > SPC-2 is vague on this topic and I'm rather uncomfortable with placing > > a requirement as strong as a "SHOULD" on an obsolete mechanism that "SH= OULD > NOT" > > be used. > > > > --------------- > > > > New problem in Section 4.6.3.3 - in this text: > > > > The Logout response indicates that the connection or session > > cleanup is completed and no other responses will arrive on the > > connection (if received on the logging out connection). In > > addition, the Logout Response indicates how long the target will > > continue to hold resources for recovery (e.g., command execution > > that continues on a new connection) in the text key Time2Retain > > and how long the initiator must wait before proceeding with > > recovery in the text key Time2Wait. > > > > Time2Wait and Time2Retain are not text keys - they're binary fields in > > the Logout Response PDU. Two changes are in order: > > - "text key Time2Retain" -> "Time2Retain field" > > - "text key Time2Wait" -> "Time2Wait field" > > > > Thanks, > > --David > > ---------------------------------------------------- > > David L. Black, Distinguished Engineer EMC Corporation, 176 South St., > > Hopkinton, MA 01748 > > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > > david.black@emc.com Mobile: +1 (978) 394-7754 > > ---------------------------------------------------- > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > > > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > From Frederick.Knight@netapp.com Wed Sep 26 09:40:14 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9141E21F8596 for ; Wed, 26 Sep 2012 09:40:14 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Cv6sLQEsWE8h for ; Wed, 26 Sep 2012 09:40:13 -0700 (PDT) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id 31E0621F8595 for ; Wed, 26 Sep 2012 09:40:13 -0700 (PDT) X-IronPort-AV: E=Sophos;i="4.80,491,1344236400"; d="scan'208";a="694397163" Received: from smtp2.corp.netapp.com ([10.57.159.114]) by mx2-out.netapp.com with ESMTP; 26 Sep 2012 09:40:11 -0700 Received: from vmwexceht02-prd.hq.netapp.com (vmwexceht02-prd.hq.netapp.com [10.106.76.240]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8QGeB5W014052; Wed, 26 Sep 2012 09:40:11 -0700 (PDT) Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.102]) by vmwexceht02-prd.hq.netapp.com ([10.106.76.240]) with mapi id 14.02.0309.002; Wed, 26 Sep 2012 09:40:10 -0700 From: "Knight, Frederick" To: "Black, David" , "storm@ietf.org" Thread-Topic: SPC-2 reserve/release: Proposed text - version 4 Thread-Index: Ac2bNKguGPJW+zTbTTC2uDGdWEV2OAAGl/vwAAGNYXAAA3tHQAAIrqHwABYKaSAAA1Dz0AADSX6QAAFXeRAAAOFF0A== Date: Wed, 26 Sep 2012 16:40:09 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE7120DE79578@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE7969E@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE796DB@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE796DB@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.106.53.53] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2012 16:40:14 -0000 Agreed -----Original Message----- From: Black, David [mailto:david.black@emc.com]=20 Sent: Wednesday, September 26, 2012 11:47 AM To: Knight, Frederick; storm@ietf.org Subject: RE: SPC-2 reserve/release: Proposed text - version 4 That text looks good to me, but I have one more minor change ... ... I'd prefer to put the second paragraph at the end of 4.4.3.1, as it app= lies to all nexus state not just reservations: The specific Unit Attention condition that results from session re-establishment indicates whether or not nexus state was preserved, see Section 6.3.4 of [SAM4]. Thanks, --David > -----Original Message----- > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > Sent: Wednesday, September 26, 2012 11:40 AM > To: Black, David; storm@ietf.org > Subject: RE: SPC-2 reserve/release: Proposed text - version 4 > > Yes, I think we've got it. Since I can't define "an extended period=20 > of time", here's my take: > > - When connection failure causes the iSCSI session to fail, and > the session is not reinstated or continued, target retention > of that session's reserve/release reservation state > may require an initiator to issue a > reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order > to remove that reservation state. > > The specific Unit Attention condition that results from session > re-establishment indicates whether or not nexus state was preserved, > see Section 6.3.4 of [SAM4]. > > Along with: > > OLD > > That is, it should perform session recovery as > > described in Chapter 6. > > NEW > > That is, it should recover the session via connection > > reinstatement as described in Section 6.3.4. > > END > > Fred > > > -----Original Message----- > From: Black, David [mailto:david.black@emc.com] > Sent: Wednesday, September 26, 2012 10:51 AM > To: Knight, Frederick; storm@ietf.org > Subject: RE: SPC-2 reserve/release: Proposed text - version 4 > > Fred, > > > I don't see anywhere that these timers are associated with any SCSI=20 > > state, or with the RESERVE state. Attempting to tie RESERVE state=20 > > to these timers is a NEW requirement. Our goal here was=20 > > specifically to NOT create any new requirements. > > I think you have a point that this is too specific. What if we just=20 > delete mention of those timers? The bullet in question could then simply= say: > > - When connection failure causes the iSCSI session to fail, and > the session is not reinstated or continued, target retention > of that session's reserve/release reservation state for an > extended period of time may require the initiator to issue a > reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order > to remove that reservation state. > > That captures the primary concern that I wanted to alert implementers to. > > > We do not want to impact existing iSCSI implementations. Could we=20 > > associate the action that occurred with the observable behavior the=20 > > host sees? I don't like putting SCSI layer stuff in a transport=20 > > layer spec, but it should just be informative. > > The text that follows explicitly lists the SCSI ASC/Q values (e.g.,=20 > 29/07, in hex). That's something I'd prefer to avoid, e.g., as T10=20 > could easily add additional ASC/Q values that are relevant to this situat= ion. > > As an alternative, I suggest that we state the principle and point to=20 > SAM-4 for the details, e.g., add the following text to the end of 4.4.3.1= : > > The specific Unit Attention condition that results from session > re-establishment indicates whether or not nexus state was preserved, > see Section 6.3.4 of [SAM4]. > > Would that be ok? > > Thanks, > --David > > > -----Original Message----- > > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > > Sent: Wednesday, September 26, 2012 8:39 AM > > To: Mallikarjun Chadalapaka; Black, David; storm@ietf.org > > Subject: RE: SPC-2 reserve/release: Proposed text - version 4 > > > > I'm concerned about this new link between Time2Wait and RESERVE state. > > Such a link has not previously existed. I don't think we should=20 > > create such a linkage. > > > > Section 4.6.3.3 includes: > > .... In addition, the Logout Response indicates how long the target=20 > > will continue to hold resources for recovery (e.g., command=20 > > execution that continues on a new connection).... > > > > This and many other places where Time2Wait and Time2Retain are=20 > > discussed, the points all have to do with command continuation; no=20 > > mention of I_T state, or RESERVE state, or mode pages, or anything else= . > > > > Section 7.5 says: > > .... Time2Wait is the initial "respite time" before attempting an=20 > > explicit/implicit Logout for the CID in question or task=20 > > reassignment for the affected tasks (if any).... > > > > Or, task reassignment of affected tasks; with no mention of I_T state. > > > > Section 7.6 ties the termination of tasks to the Time2Wait /=20 > > Time2Retain timers. Section 11.14.5 makes the same statements about=20 > > the task termination linkage to those timers. > > > > Section 11.15.3 says those timers are "...the minimum amount of=20 > > time, in seconds, to wait before attempting task reassignment.". > > > > Section 8.3 defines Session State. And there is one sentence that I=20 > > can find in all 345 pages that associates that session state with=20 > > these timers. It is in section 11.15.4 where it says: If it is the=20 > > last connection of a session, the whole session state is discarded=20 > > after > Time2Retain. > > > > I don't see anywhere that these timers are associated with any SCSI=20 > > state, or with the RESERVE state. Attempting to tie RESERVE state=20 > > to these timers is a NEW requirement. Our goal here was=20 > > specifically to NOT create any new requirements. > > > > We do not want to impact existing iSCSI implementations. Could we=20 > > associate the action that occurred with the observable behavior the=20 > > host sees? I don't like putting SCSI layer stuff in a transport=20 > > layer spec, but it should just be informative. Something like: > > > > If the initiator receives a UA: 29/00 or 29/01 or 29/04 then the=20 > > RESERVE state has been lost. (Maybe 29/04 belongs in a RESERVE=20 > > state > unknown list). > > > > If the initiator receives a UA: 29/07 then the RESERVE state has=20 > > been preserved. > > > > I didn't list 29/02 or 29/03 or 29/05 or 29/06 since I view those as=20 > > Parallel SCSI specific. We can debate which items belong in which=20 > > list, but would this kind of observable behavior approach be better=20 > > than a new requirement? FYI - I didn't invent this; I copied it=20 > > from > > SAM-5r11 (sub-clause 6.3.4) where it states it as a requirement on=20 > > the target - that when an I_T NEXUS LOSS event occurs, if the state=20 > > is retained, 29/07 is returned, and if the state is not retained,=20 > > then 29/01 is > returned. > > > > We also don't want to create any new requirement now in this=20 > > consolidated draft that conflicts with this SAM-5 text, since we=20 > > will eventually have to comply with SAM-5 and its follow-ons. > > > > Another approach would be to ignore this for the consolidated draft,=20 > > and force it into the iSCSI-SAM draft - which references SAM-4 and > > SAM-5 where I_T Nexus loss is already defined and the above text=20 > > already > exists. > > > > Fred > > > > -----Original Message----- > > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On=20 > > Behalf Of Mallikarjun Chadalapaka > > Sent: Tuesday, September 25, 2012 9:39 PM > > To: Black, David; storm@ietf.org > > Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version=20 > > 4 > > > > Looks good to me overall. Two minor comments: > > > > 1) For 4.4.3.1, I have replaced the current sentence with the=20 > > following one (instead of the proposed): > > > > That is, it should reinstate the session via iSCSI session=20 > > reinstatement (Section 6.3.5) or continue via session continuation=20 > > (Section > 6.3.6). > > > > 2) For proposed new text, I have tweaked the version 4 sentence=20 > > below, keeping the existing implementations in view: > > > > VERSION 4: > > The Time2Wait and Time2Retain > > timeout values (see Section 7.55) apply to retention of > reserve/release > > reservation state after iSCSI session failure because that state > > is part of the I_T Nexus state. > > > > VERSION 5: > > Instead, implementations are strongly encouraged to apply the > > Time2Wait and Time2Retain timeout values (see Section 7.5) > > to manage retention of reserve/release reservation state after > > iSCSI session failure because that state is part of the I_T=20 > > nexus > state. > > > > > > Thanks. > > > > Mallikarjun > > > > > > -----Original Message----- > > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On=20 > > Behalf Of Black, David > > Sent: Tuesday, September 25, 2012 2:39 PM > > To: storm@ietf.org > > Subject: [storm] FW: SPC-2 reserve/release: Proposed text - version=20 > > 4 > > > > version 4 changes: > > > > More changes for Mallikarjun's comments, mostly a rewrite of the=20 > > first additional caveat bullet. I've chosen not to reference Annex=20 > > B of > > SPC-4 as that seems a bit far afield for a SCSI transport standard=20 > > like > iSCSI. > > > > version 3 changes: > > > > Responding to Mallikarjun's comments (plus some more edits) - > > > > > 1) IMHO, opening sentence of "I_T nexus state includes reservation st= ate" > > > is a little too broad, > > > > I removed that sentence, removed I_T Nexus from new section title=20 > > and tweaked the persistent reservation text to talk about state in=20 > > general as opposed to I_T Nexus state in particular. > > > > > 2) Hard reset is not an LU reset - it is either an iSCSI Target=20 > > > Warm Reset, or (more likely) an iSCSI Target Cold Reset. In either=20 > > > case, any of the three resets should clear the reserve/release state. > > > > I removed the word "hard" - as you noted, an LU reset is sufficient=20 > > to remove a reserve/release reservation. > > > > > 3) Should use Time2Retain+Time2Wait, not Default Time2Wait > > > > Brilliant minds think alike - I got to that one in version 2 ;-). > > > > > 4) Session recovery does not necessarily preserve the I_T nexus=20 > > > state (recovery may not occur within Time2Wait+Time2Retain, and=20 > > > may not use the same ISID when it occurs), but session=20 > > > reinstatement and session continuation certainly should. > > > > I changed "session recovery" to "session reinstatement". > > > > > 5) Should articulate the underlying linkage between Time2Retain=20 > > > (an iSCSI concept), and the reserve/release state (a SCSI state). > > > > I believe I also got to that one in version 2. > > > > version 2 changes: > > > > I moved a paragraph break in response to Ralph's comments, and=20 > > rewrote the first additional consideration to reference the=20 > > connection timeout values in general. I also found some minor=20 > > problems that need correction in Section > > 4.6.3.3 (see end of this message). > > > > --- version 4 follows --- > > > > > > > > Here's an initial attempt to propose text to deal with the reserve/=20 > > release topic in the consolidated iSCSI draft. Please comment,=20 > > correct, suggest edits, etc. > > > > The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: > > > > 4.4.3.1. I_T Nexus State > > > > Certain nexus relationships contain an explicit state (e.g., > > initiator-specific mode pages) that may need to be preserved by > > the device server [SAM2] in a logical unit through changes or > > failures in the iSCSI layer (e.g., session failures). In order for > > that state to be restored, the iSCSI initiator should reestablish > > its session (re-login) to the same Target Portal Group using the > > previous ISID. That is, it should perform session recovery as > > described in Chapter 6. This is because the SCSI initiator port > > identifier and the SCSI target port identifier (or relative target > > port) form the datum that the SCSI logical unit device server uses > > to identify the I_T nexus. > > > > ------------------------------- > > > > First, for clarity, the following change should be made in the above > > 4.4.3.1 text to better identify the recovery mechanism: > > > > OLD > > That is, it should perform session recovery as > > described in Chapter 6. > > NEW > > That is, it should recover the session via connection > > reinstatement as described in Section 6.3.4. > > END > > > > I believe that a lower-case "should" remains appropriate for this text. > > > > ----------------------------------- > > > > NEW TEXT (second draft): > > > > 4.4.3.2. Reservations > > > > There are two reservation management methods defined in the SCSI > > standards, reserve/release reservations, based on the RESERVE and > > RELEASE commands [SPC2], and persistent reservations, based on the > > PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. > > Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be > > used; persistent reservations SHOULD be used instead. > > > > State for persistent reservations is required to persist > > through changes and failures at the iSCSI layer that result in > > I_T Nexus failures, see [SPC3] for details and specific requirements. > > > > In contrast, [SPC2] does not specify detailed persistence > > requirements for reserve/release reservation state after an I_T > > Nexus failure. Nonetheless, when reserve/release reservations are > > supported by an iSCSI target, the preferred implementation approach > > is to preserve reserve/release reservation state as part of the > > I_T Nexus state for iSCSI session reinstatement (see Section 6.3.5) > > or session continuation (see Section 6.3.6). > > > > Two additional caveats apply to reserve/release reservations: > > > > - When connection failure causes the iSCSI session to fail, and > > the session is not reinstated or continued, target retention > > of that session's reserve/release reservation state for an > > extended period of time may require the initiator to issue a > > reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order > > to remove that reservation state. The Time2Wait and Time2Retain > > timeout values (see Section 7.55) apply to retention of > reserve/release > > reservation state after iSCSI session failure because that state > > is part of the I_T Nexus state. The need for resets in this > > scenario may be reduced by suitable selection of values for > > these two timeouts. > > > > - Reserve/release reservations may not behave as expected when > > persistent reservations are also used on the same logical unit; > > see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE > > behavior" in [SPC4]. > > > > ----------------------- > > > > Reference impacts: both [SPC2] and [SPC3] need to be normative=20 > > references, but [SPC4] can be an informative reference. > > > > ------------------------- > > > > I deliberately used "the preferred iSCSI implementation approach" > > wording for reserve/release reservation state preservation=20 > > requirements, as > > SPC-2 is vague on this topic and I'm rather uncomfortable with=20 > > placing a requirement as strong as a "SHOULD" on an obsolete=20 > > mechanism that "SHOULD > NOT" > > be used. > > > > --------------- > > > > New problem in Section 4.6.3.3 - in this text: > > > > The Logout response indicates that the connection or session > > cleanup is completed and no other responses will arrive on the > > connection (if received on the logging out connection). In > > addition, the Logout Response indicates how long the target will > > continue to hold resources for recovery (e.g., command execution > > that continues on a new connection) in the text key Time2Retain > > and how long the initiator must wait before proceeding with > > recovery in the text key Time2Wait. > > > > Time2Wait and Time2Retain are not text keys - they're binary fields=20 > > in the Logout Response PDU. Two changes are in order: > > - "text key Time2Retain" -> "Time2Retain field" > > - "text key Time2Wait" -> "Time2Wait field" > > > > Thanks, > > --David > > ---------------------------------------------------- > > David L. Black, Distinguished Engineer EMC Corporation, 176 South=20 > > St., Hopkinton, MA 01748 > > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > > david.black@emc.com Mobile: +1 (978) 394-7754 > > ---------------------------------------------------- > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > > > > > > _______________________________________________ > > storm mailing list > > storm@ietf.org > > https://www.ietf.org/mailman/listinfo/storm > From julian@satran.net Wed Sep 26 12:09:21 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DB20521F84FA for ; Wed, 26 Sep 2012 12:09:21 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.203 X-Spam-Level: X-Spam-Status: No, score=-5.203 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_QP_LONG_LINE=1.396, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qWsvqtgpKzxo for ; Wed, 26 Sep 2012 12:09:20 -0700 (PDT) Received: from eu1sys200aog118.obsmtp.com (eu1sys200aog118.obsmtp.com [207.126.144.145]) by ietfa.amsl.com (Postfix) with SMTP id A790121F849C for ; Wed, 26 Sep 2012 12:09:15 -0700 (PDT) Received: from mail-bk0-f44.google.com ([209.85.214.44]) (using TLSv1) by eu1sys200aob118.postini.com ([207.126.147.11]) with SMTP ID DSNKUGNS2AQzz3kAgCwGseoFnSZ6tWRFiA5k@postini.com; Wed, 26 Sep 2012 19:09:15 UTC Received: by bkcjc3 with SMTP id jc3so537967bkc.31 for ; Wed, 26 Sep 2012 12:09:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=references:in-reply-to:mime-version:content-transfer-encoding :content-type:message-id:cc:x-mailer:from:subject:date:to :x-gm-message-state; bh=5T59XMtntjCjNHpNyHhhoj1R9bSap3aLx3DmRHYSn7M=; b=Bh33p6mxEHGD8L2YoRW6Us8uq/oEbnx2WukWK7cvcKgILcDoq0SYduSv+x7AmN0Ag4 wMw9zGaXdQn+4uZs3v4cuSbD58+T2iECagorHOyntZK02Wc3J4vJspuqDnNnKBd4Jig1 SMYvDyp+fZhHMww0q5DzVMZ6j5ic2h95B4/NQrF1aSpBQ0icLiziwVoabBKgyNW+G5K1 zN+/h5VSR0L5rXWfy5gf5O+fHMi8FEoKKc8lq8J8ulXg0gGA67aoL6WEL0frXzBaj7Uu i0qH5MxVaR6NVzyTvs73I8eMYmpLzeE9vhDvITJPaRUfVNFOJ3Jn6fhVY/rnwttXyETF kBew== Received: by 10.204.146.13 with SMTP id f13mr1160262bkv.29.1348686551693; Wed, 26 Sep 2012 12:09:11 -0700 (PDT) Received: from [192.168.0.100] (ip-54-21.sn2.eutelia.it. [83.211.54.21]) by mx.google.com with ESMTPS id hy11sm3151230bkc.5.2012.09.26.12.09.09 (version=SSLv3 cipher=OTHER); Wed, 26 Sep 2012 12:09:10 -0700 (PDT) References: <8D3D17ACE214DC429325B2B98F3AE7120DE79578@MX15A.corp.emc.com> In-Reply-To: Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Message-Id: <9813C654-1D6E-43DE-B21A-2949729E042F@satran.net> X-Mailer: iPad Mail (10A403) From: Julian Satran Date: Wed, 26 Sep 2012 21:09:19 +0200 To: "Knight, Frederick" X-Gm-Message-State: ALoCoQlwmTEaVlLv636lBr/r/TlBqryg4ngqqjgJAX9kEYJuFjhJZbsCD9hlRo5MkyfWSrSr9ZId Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2012 19:09:22 -0000 Fred, I don't think there is any new requirement. Time2retain indicates that a ses= sion (an it nexus) exists. As long as the session exits you can't talk abut n= exus loss - even if you have no connection active. Julo Sent from my iPad On 26 =D7=91=D7=A1=D7=A4=D7=98 2012, at 14:39, "Knight, Frederick" wrote: > I'm concerned about this new link between Time2Wait and RESERVE state. Su= ch a link has not previously existed. I don't think we should create such a= linkage. >=20 > Section 4.6.3.3 includes: > .... In addition, the Logout Response indicates how long the target will c= ontinue to hold resources for recovery (e.g., command execution that continu= es on a new connection).... >=20 > This and many other places where Time2Wait and Time2Retain are discussed, t= he points all have to do with command continuation; no mention of I_T state,= or RESERVE state, or mode pages, or anything else. >=20 > Section 7.5 says: > .... Time2Wait is the initial "respite time" before attempting an explicit= /implicit Logout for the CID in question or task reassignment for the affect= ed tasks (if any).... >=20 > Or, task reassignment of affected tasks; with no mention of I_T state. >=20 > Section 7.6 ties the termination of tasks to the Time2Wait / Time2Retain t= imers. Section 11.14.5 makes the same statements about the task termination= linkage to those timers. >=20 > Section 11.15.3 says those timers are "...the minimum amount of time, in s= econds, to wait before attempting task reassignment.". >=20 > Section 8.3 defines Session State. And there is one sentence that I can f= ind in all 345 pages that associates that session state with these timers. I= t is in section 11.15.4 where it says: If it is the last connection of a ses= sion, the whole session state is discarded after Time2Retain. >=20 > I don't see anywhere that these timers are associated with any SCSI state,= or with the RESERVE state. Attempting to tie RESERVE state to these timers= is a NEW requirement. Our goal here was specifically to NOT create any new= requirements. >=20 > We do not want to impact existing iSCSI implementations. Could we associa= te the action that occurred with the observable behavior the host sees? I d= on't like putting SCSI layer stuff in a transport layer spec, but it should j= ust be informative. Something like: >=20 > If the initiator receives a UA: 29/00 or 29/01 or 29/04 then the RESERVE s= tate has been lost. (Maybe 29/04 belongs in a RESERVE state unknown list). >=20 > If the initiator receives a UA: 29/07 then the RESERVE state has been pres= erved. >=20 > I didn't list 29/02 or 29/03 or 29/05 or 29/06 since I view those as Paral= lel SCSI specific. We can debate which items belong in which list, but woul= d this kind of observable behavior approach be better than a new requirement= ? FYI - I didn't invent this; I copied it from SAM-5r11 (sub-clause 6.3.4) w= here it states it as a requirement on the target - that when an I_T NEXUS L= OSS event occurs, if the state is retained, 29/07 is returned, and if the st= ate is not retained, then 29/01 is returned. >=20 > We also don't want to create any new requirement now in this consolidated d= raft that conflicts with this SAM-5 text, since we will eventually have to c= omply with SAM-5 and its follow-ons. >=20 > Another approach would be to ignore this for the consolidated draft, and f= orce it into the iSCSI-SAM draft - which references SAM-4 and SAM-5 where I_= T Nexus loss is already defined and the above text already exists. >=20 > Fred >=20 > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of M= allikarjun Chadalapaka > Sent: Tuesday, September 25, 2012 9:39 PM > To: Black, David; storm@ietf.org > Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 >=20 > Looks good to me overall. Two minor comments: >=20 > 1) For 4.4.3.1, I have replaced the current sentence with the following on= e (instead of the proposed): >=20 > That is, it should reinstate the session via iSCSI session reinstatement (= Section 6.3.5) or continue via session continuation (Section 6.3.6). >=20 > 2) For proposed new text, I have tweaked the version 4 sentence below, kee= ping the existing implementations in view: >=20 > VERSION 4: > The Time2Wait and Time2Retain > timeout values (see Section 7.55) apply to retention of reserve/relea= se > reservation state after iSCSI session failure because that state > is part of the I_T Nexus state. >=20 > VERSION 5: > Instead, implementations are strongly encouraged to apply the=20 > Time2Wait and Time2Retain timeout values (see Section 7.5)=20 > to manage retention of reserve/release reservation state after=20 > iSCSI session failure because that state is part of the I_T nexus stat= e. >=20 >=20 > Thanks. >=20 > Mallikarjun >=20 >=20 > -----Original Message----- > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of B= lack, David > Sent: Tuesday, September 25, 2012 2:39 PM > To: storm@ietf.org > Subject: [storm] FW: SPC-2 reserve/release: Proposed text - version 4 >=20 > version 4 changes: >=20 > More changes for Mallikarjun's comments, mostly a rewrite of the first add= itional caveat bullet. I've chosen not to reference Annex B of > SPC-4 as that seems a bit far afield for a SCSI transport standard like iS= CSI. >=20 > version 3 changes: >=20 > Responding to Mallikarjun's comments (plus some more edits) - >=20 >> 1) IMHO, opening sentence of "I_T nexus state includes reservation state"= >> is a little too broad, >=20 > I removed that sentence, removed I_T Nexus from new section title and twea= ked the persistent reservation text to talk about state in general as oppose= d to I_T Nexus state in particular. >=20 >> 2) Hard reset is not an LU reset - it is either an iSCSI Target Warm=20 >> Reset, or (more likely) an iSCSI Target Cold Reset. In either case,=20 >> any of the three resets should clear the reserve/release state. >=20 > I removed the word "hard" - as you noted, an LU reset is sufficient to rem= ove a reserve/release reservation. >=20 >> 3) Should use Time2Retain+Time2Wait, not Default Time2Wait >=20 > Brilliant minds think alike - I got to that one in version 2 ;-). >=20 >> 4) Session recovery does not necessarily preserve the I_T nexus state=20 >> (recovery may not occur within Time2Wait+Time2Retain, and may not use=20 >> the same ISID when it occurs), but session reinstatement and session=20 >> continuation certainly should. >=20 > I changed "session recovery" to "session reinstatement". >=20 >> 5) Should articulate the underlying linkage between Time2Retain (an=20 >> iSCSI concept), and the reserve/release state (a SCSI state). >=20 > I believe I also got to that one in version 2. >=20 > version 2 changes: >=20 > I moved a paragraph break in response to Ralph's comments, and rewrote the= first additional consideration to reference the connection timeout values i= n general. I also found some minor problems that need correction in Section= 4.6.3.3 (see end of this message). >=20 > --- version 4 follows --- >=20 > >=20 > Here's an initial attempt to propose text to deal with the reserve/ releas= e topic in the consolidated iSCSI draft. Please comment, correct, suggest e= dits, etc. >=20 > The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: >=20 > 4.4.3.1. I_T Nexus State >=20 > Certain nexus relationships contain an explicit state (e.g., > initiator-specific mode pages) that may need to be preserved by > the device server [SAM2] in a logical unit through changes or > failures in the iSCSI layer (e.g., session failures). In order for > that state to be restored, the iSCSI initiator should reestablish > its session (re-login) to the same Target Portal Group using the > previous ISID. That is, it should perform session recovery as > described in Chapter 6. This is because the SCSI initiator port > identifier and the SCSI target port identifier (or relative target > port) form the datum that the SCSI logical unit device server uses > to identify the I_T nexus. >=20 > ------------------------------- >=20 > First, for clarity, the following change should be made in the above 4.4.3= .1 text to better identify the recovery mechanism: >=20 > OLD > That is, it should perform session recovery as > described in Chapter 6. > NEW > That is, it should recover the session via connection > reinstatement as described in Section 6.3.4. > END >=20 > I believe that a lower-case "should" remains appropriate for this text. >=20 > ----------------------------------- >=20 > NEW TEXT (second draft): >=20 > 4.4.3.2. Reservations >=20 > There are two reservation management methods defined in the SCSI > standards, reserve/release reservations, based on the RESERVE and > RELEASE commands [SPC2], and persistent reservations, based on the > PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. > Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be > used; persistent reservations SHOULD be used instead. >=20 > State for persistent reservations is required to persist > through changes and failures at the iSCSI layer that result in > I_T Nexus failures, see [SPC3] for details and specific requirements. >=20 > In contrast, [SPC2] does not specify detailed persistence > requirements for reserve/release reservation state after an I_T > Nexus failure. Nonetheless, when reserve/release reservations are > supported by an iSCSI target, the preferred implementation approach > is to preserve reserve/release reservation state as part of the > I_T Nexus state for iSCSI session reinstatement (see Section 6.3.5) > or session continuation (see Section 6.3.6). >=20 > Two additional caveats apply to reserve/release reservations: >=20 > - When connection failure causes the iSCSI session to fail, and > the session is not reinstated or continued, target retention > of that session's reserve/release reservation state for an > extended period of time may require the initiator to issue a > reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order > to remove that reservation state. The Time2Wait and Time2Retain > timeout values (see Section 7.55) apply to retention of reserve/relea= se > reservation state after iSCSI session failure because that state > is part of the I_T Nexus state. The need for resets in this > scenario may be reduced by suitable selection of values for > these two timeouts. >=20 > - Reserve/release reservations may not behave as expected when > persistent reservations are also used on the same logical unit; > see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE > behavior" in [SPC4]. >=20 > ----------------------- >=20 > Reference impacts: both [SPC2] and [SPC3] need to be normative references,= but [SPC4] can be an informative reference. >=20 > ------------------------- >=20 > I deliberately used "the preferred iSCSI implementation approach" > wording for reserve/release reservation state preservation requirements, a= s SPC-2 is vague on this topic and I'm rather uncomfortable with placing a r= equirement as strong as a "SHOULD" on an obsolete mechanism that "SHOULD NOT= " be used. >=20 > --------------- >=20 > New problem in Section 4.6.3.3 - in this text: >=20 > The Logout response indicates that the connection or session > cleanup is completed and no other responses will arrive on the > connection (if received on the logging out connection). In > addition, the Logout Response indicates how long the target will > continue to hold resources for recovery (e.g., command execution > that continues on a new connection) in the text key Time2Retain > and how long the initiator must wait before proceeding with > recovery in the text key Time2Wait. >=20 > Time2Wait and Time2Retain are not text keys - they're binary fields in the= Logout Response PDU. Two changes are in order: > - "text key Time2Retain" -> "Time2Retain field" > - "text key Time2Wait" -> "Time2Wait field" >=20 > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > david.black@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm >=20 >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From julian@satran.net Wed Sep 26 12:25:52 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 72C0E21F85A2 for ; Wed, 26 Sep 2012 12:25:52 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.203 X-Spam-Level: X-Spam-Status: No, score=-5.203 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_QP_LONG_LINE=1.396, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GFI7mdINWcVB for ; Wed, 26 Sep 2012 12:25:51 -0700 (PDT) Received: from eu1sys200aog114.obsmtp.com (eu1sys200aog114.obsmtp.com [207.126.144.137]) by ietfa.amsl.com (Postfix) with SMTP id 3E3DE21F84C9 for ; Wed, 26 Sep 2012 12:25:50 -0700 (PDT) Received: from mail-bk0-f44.google.com ([209.85.214.44]) (using TLSv1) by eu1sys200aob114.postini.com ([207.126.147.11]) with SMTP ID DSNKUGNWuBBEua/oAoX0fbXmRxX8TVm3Q4nl@postini.com; Wed, 26 Sep 2012 19:25:50 UTC Received: by bkcjc3 with SMTP id jc3so545946bkc.31 for ; Wed, 26 Sep 2012 12:25:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=references:in-reply-to:mime-version:content-transfer-encoding :content-type:message-id:cc:x-mailer:from:subject:date:to :x-gm-message-state; bh=lH0fvDxhED18dI/qyk4M6aLHl7Wr57WHVbQPgjpJzLQ=; b=V1Ah1wmSi4uYH2P/0OzxMGPiIxrOCJNLhla+U/fLDRKzHoE0rxRBR5xziUoVr2zylF U5rFuh3nulPqbyvT9vN6yipYqbfsOy4wR4n+/IQfPbjox8PWBGfBMFnREEfJrN98zfW9 qcpXvSt9hmdFPu+7cwQ4AIWcf329Idi9ZonZY12q1t7B3QG7FvIyVsJpElfE9ER04kxs CzdeoFCAHKvVpTVgyBdRX1TctqRmJfQMTHUhfJBcACvu1kOIwTCQIGvBwK0DRnbtnx/y Bln35uvuUwEA5ChuV3tFAjG7tk9bWRqnXV4Aun5hfPbGAhGInppoa0LvBt+hAn5w7qmX FjeA== Received: by 10.204.129.27 with SMTP id m27mr1092655bks.115.1348687544427; Wed, 26 Sep 2012 12:25:44 -0700 (PDT) Received: from [192.168.0.100] (ip-54-21.sn2.eutelia.it. [83.211.54.21]) by mx.google.com with ESMTPS id gy18sm3176030bkc.4.2012.09.26.12.25.41 (version=SSLv3 cipher=OTHER); Wed, 26 Sep 2012 12:25:42 -0700 (PDT) References: <8D3D17ACE214DC429325B2B98F3AE7120DE79578@MX15A.corp.emc.com> <9813C654-1D6E-43DE-B21A-2949729E042F@satran.net> In-Reply-To: <9813C654-1D6E-43DE-B21A-2949729E042F@satran.net> Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Message-Id: <2C79B025-BD1D-4E95-9FC1-CB320A5EC8BA@satran.net> X-Mailer: iPad Mail (10A403) From: Julian Satran Date: Wed, 26 Sep 2012 21:25:51 +0200 To: "Knight, Frederick" X-Gm-Message-State: ALoCoQkqA7x4E1gAkrPm1geF0ZL/lm9q2iXnFOyRleO4jDugm+y2j+cebLXeV+qfzg+ZRRik7gKC Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2012 19:25:52 -0000 as for time2wait whether you mention the linkage or not it exists in iSCSI. associting state other than that mentioned in 3270 with the sessiom/it nexus= is a thing that should be relegated to t10 as a target should not behave di= fferently whenaccessed through iscsi or fcp as far as state is concerned. th= e only logic that could allow for a slightly different behavior is a link to= the mechanism by which the trasport perceives the nexus loss. that may requ= ire to mention time2wait for iscsi. Sent from my iPad On 26 =D7=91=D7=A1=D7=A4=D7=98 2012, at 21:09, Julian Satran wrote: > Fred, >=20 > I don't think there is any new requirement. Time2retain indicates that a s= ession (an it nexus) exists. As long as the session exits you can't talk abu= t nexus loss - even if you have no connection active. >=20 > Julo >=20 > Sent from my iPad >=20 > On 26 =D7=91=D7=A1=D7=A4=D7=98 2012, at 14:39, "Knight, Frederick" wrote: >=20 >> I'm concerned about this new link between Time2Wait and RESERVE state. S= uch a link has not previously existed. I don't think we should create such a= linkage. >>=20 >> Section 4.6.3.3 includes: >> .... In addition, the Logout Response indicates how long the target will c= ontinue to hold resources for recovery (e.g., command execution that continu= es on a new connection).... >>=20 >> This and many other places where Time2Wait and Time2Retain are discussed,= the points all have to do with command continuation; no mention of I_T stat= e, or RESERVE state, or mode pages, or anything else. >>=20 >> Section 7.5 says: >> .... Time2Wait is the initial "respite time" before attempting an explici= t/implicit Logout for the CID in question or task reassignment for the affec= ted tasks (if any).... >>=20 >> Or, task reassignment of affected tasks; with no mention of I_T state. >>=20 >> Section 7.6 ties the termination of tasks to the Time2Wait / Time2Retain t= imers. Section 11.14.5 makes the same statements about the task termination= linkage to those timers. >>=20 >> Section 11.15.3 says those timers are "...the minimum amount of time, in s= econds, to wait before attempting task reassignment.". >>=20 >> Section 8.3 defines Session State. And there is one sentence that I can f= ind in all 345 pages that associates that session state with these timers. I= t is in section 11.15.4 where it says: If it is the last connection of a ses= sion, the whole session state is discarded after Time2Retain. >>=20 >> I don't see anywhere that these timers are associated with any SCSI state= , or with the RESERVE state. Attempting to tie RESERVE state to these timer= s is a NEW requirement. Our goal here was specifically to NOT create any ne= w requirements. >>=20 >> We do not want to impact existing iSCSI implementations. Could we associ= ate the action that occurred with the observable behavior the host sees? I d= on't like putting SCSI layer stuff in a transport layer spec, but it should j= ust be informative. Something like: >>=20 >> If the initiator receives a UA: 29/00 or 29/01 or 29/04 then the RESERVE s= tate has been lost. (Maybe 29/04 belongs in a RESERVE state unknown list). >>=20 >> If the initiator receives a UA: 29/07 then the RESERVE state has been pre= served. >>=20 >> I didn't list 29/02 or 29/03 or 29/05 or 29/06 since I view those as Para= llel SCSI specific. We can debate which items belong in which list, but wou= ld this kind of observable behavior approach be better than a new requiremen= t? FYI - I didn't invent this; I copied it from SAM-5r11 (sub-clause 6.3.4)= where it states it as a requirement on the target - that when an I_T NEXUS= LOSS event occurs, if the state is retained, 29/07 is returned, and if the s= tate is not retained, then 29/01 is returned. >>=20 >> We also don't want to create any new requirement now in this consolidated= draft that conflicts with this SAM-5 text, since we will eventually have to= comply with SAM-5 and its follow-ons. >>=20 >> Another approach would be to ignore this for the consolidated draft, and f= orce it into the iSCSI-SAM draft - which references SAM-4 and SAM-5 where I_= T Nexus loss is already defined and the above text already exists. >>=20 >> Fred >>=20 >> -----Original Message----- >> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of= Mallikarjun Chadalapaka >> Sent: Tuesday, September 25, 2012 9:39 PM >> To: Black, David; storm@ietf.org >> Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 >>=20 >> Looks good to me overall. Two minor comments: >>=20 >> 1) For 4.4.3.1, I have replaced the current sentence with the following o= ne (instead of the proposed): >>=20 >> That is, it should reinstate the session via iSCSI session reinstatement (= Section 6.3.5) or continue via session continuation (Section 6.3.6). >>=20 >> 2) For proposed new text, I have tweaked the version 4 sentence below, ke= eping the existing implementations in view: >>=20 >> VERSION 4: >> The Time2Wait and Time2Retain >> timeout values (see Section 7.55) apply to retention of reserve/relea= se >> reservation state after iSCSI session failure because that state >> is part of the I_T Nexus state. >>=20 >> VERSION 5: >> Instead, implementations are strongly encouraged to apply the=20 >> Time2Wait and Time2Retain timeout values (see Section 7.5)=20 >> to manage retention of reserve/release reservation state after=20 >> iSCSI session failure because that state is part of the I_T nexus stat= e. >>=20 >>=20 >> Thanks. >>=20 >> Mallikarjun >>=20 >>=20 >> -----Original Message----- >> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of= Black, David >> Sent: Tuesday, September 25, 2012 2:39 PM >> To: storm@ietf.org >> Subject: [storm] FW: SPC-2 reserve/release: Proposed text - version 4 >>=20 >> version 4 changes: >>=20 >> More changes for Mallikarjun's comments, mostly a rewrite of the first ad= ditional caveat bullet. I've chosen not to reference Annex B of >> SPC-4 as that seems a bit far afield for a SCSI transport standard like i= SCSI. >>=20 >> version 3 changes: >>=20 >> Responding to Mallikarjun's comments (plus some more edits) - >>=20 >>> 1) IMHO, opening sentence of "I_T nexus state includes reservation state= " >>> is a little too broad, >>=20 >> I removed that sentence, removed I_T Nexus from new section title and twe= aked the persistent reservation text to talk about state in general as oppos= ed to I_T Nexus state in particular. >>=20 >>> 2) Hard reset is not an LU reset - it is either an iSCSI Target Warm=20 >>> Reset, or (more likely) an iSCSI Target Cold Reset. In either case,=20 >>> any of the three resets should clear the reserve/release state. >>=20 >> I removed the word "hard" - as you noted, an LU reset is sufficient to re= move a reserve/release reservation. >>=20 >>> 3) Should use Time2Retain+Time2Wait, not Default Time2Wait >>=20 >> Brilliant minds think alike - I got to that one in version 2 ;-). >>=20 >>> 4) Session recovery does not necessarily preserve the I_T nexus state=20= >>> (recovery may not occur within Time2Wait+Time2Retain, and may not use=20= >>> the same ISID when it occurs), but session reinstatement and session=20 >>> continuation certainly should. >>=20 >> I changed "session recovery" to "session reinstatement". >>=20 >>> 5) Should articulate the underlying linkage between Time2Retain (an=20 >>> iSCSI concept), and the reserve/release state (a SCSI state). >>=20 >> I believe I also got to that one in version 2. >>=20 >> version 2 changes: >>=20 >> I moved a paragraph break in response to Ralph's comments, and rewrote th= e first additional consideration to reference the connection timeout values i= n general. I also found some minor problems that need correction in Section= 4.6.3.3 (see end of this message). >>=20 >> --- version 4 follows --- >>=20 >> >>=20 >> Here's an initial attempt to propose text to deal with the reserve/ relea= se topic in the consolidated iSCSI draft. Please comment, correct, suggest e= dits, etc. >>=20 >> The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: >>=20 >> 4.4.3.1. I_T Nexus State >>=20 >> Certain nexus relationships contain an explicit state (e.g., >> initiator-specific mode pages) that may need to be preserved by >> the device server [SAM2] in a logical unit through changes or >> failures in the iSCSI layer (e.g., session failures). In order for >> that state to be restored, the iSCSI initiator should reestablish >> its session (re-login) to the same Target Portal Group using the >> previous ISID. That is, it should perform session recovery as >> described in Chapter 6. This is because the SCSI initiator port >> identifier and the SCSI target port identifier (or relative target >> port) form the datum that the SCSI logical unit device server uses >> to identify the I_T nexus. >>=20 >> ------------------------------- >>=20 >> First, for clarity, the following change should be made in the above 4.4.= 3.1 text to better identify the recovery mechanism: >>=20 >> OLD >> That is, it should perform session recovery as >> described in Chapter 6. >> NEW >> That is, it should recover the session via connection >> reinstatement as described in Section 6.3.4. >> END >>=20 >> I believe that a lower-case "should" remains appropriate for this text. >>=20 >> ----------------------------------- >>=20 >> NEW TEXT (second draft): >>=20 >> 4.4.3.2. Reservations >>=20 >> There are two reservation management methods defined in the SCSI >> standards, reserve/release reservations, based on the RESERVE and >> RELEASE commands [SPC2], and persistent reservations, based on the >> PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. >> Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be >> used; persistent reservations SHOULD be used instead. >>=20 >> State for persistent reservations is required to persist >> through changes and failures at the iSCSI layer that result in >> I_T Nexus failures, see [SPC3] for details and specific requirements. >>=20 >> In contrast, [SPC2] does not specify detailed persistence >> requirements for reserve/release reservation state after an I_T >> Nexus failure. Nonetheless, when reserve/release reservations are >> supported by an iSCSI target, the preferred implementation approach >> is to preserve reserve/release reservation state as part of the >> I_T Nexus state for iSCSI session reinstatement (see Section 6.3.5) >> or session continuation (see Section 6.3.6). >>=20 >> Two additional caveats apply to reserve/release reservations: >>=20 >> - When connection failure causes the iSCSI session to fail, and >> the session is not reinstated or continued, target retention >> of that session's reserve/release reservation state for an >> extended period of time may require the initiator to issue a >> reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order >> to remove that reservation state. The Time2Wait and Time2Retain >> timeout values (see Section 7.55) apply to retention of reserve/relea= se >> reservation state after iSCSI session failure because that state >> is part of the I_T Nexus state. The need for resets in this >> scenario may be reduced by suitable selection of values for >> these two timeouts. >>=20 >> - Reserve/release reservations may not behave as expected when >> persistent reservations are also used on the same logical unit; >> see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE >> behavior" in [SPC4]. >>=20 >> ----------------------- >>=20 >> Reference impacts: both [SPC2] and [SPC3] need to be normative references= , but [SPC4] can be an informative reference. >>=20 >> ------------------------- >>=20 >> I deliberately used "the preferred iSCSI implementation approach" >> wording for reserve/release reservation state preservation requirements, a= s SPC-2 is vague on this topic and I'm rather uncomfortable with placing a r= equirement as strong as a "SHOULD" on an obsolete mechanism that "SHOULD NOT= " be used. >>=20 >> --------------- >>=20 >> New problem in Section 4.6.3.3 - in this text: >>=20 >> The Logout response indicates that the connection or session >> cleanup is completed and no other responses will arrive on the >> connection (if received on the logging out connection). In >> addition, the Logout Response indicates how long the target will >> continue to hold resources for recovery (e.g., command execution >> that continues on a new connection) in the text key Time2Retain >> and how long the initiator must wait before proceeding with >> recovery in the text key Time2Wait. >>=20 >> Time2Wait and Time2Retain are not text keys - they're binary fields in th= e Logout Response PDU. Two changes are in order: >> - "text key Time2Retain" -> "Time2Retain field" >> - "text key Time2Wait" -> "Time2Wait field" >>=20 >> Thanks, >> --David >> ---------------------------------------------------- >> David L. Black, Distinguished Engineer >> EMC Corporation, 176 South St., Hopkinton, MA 01748 >> +1 (508) 293-7953 FAX: +1 (508) 293-7786 >> david.black@emc.com Mobile: +1 (978) 394-7754 >> ---------------------------------------------------- >>=20 >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm >>=20 >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm >>=20 >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm >>=20 >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm >>=20 >>=20 >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm From julian@satran.net Wed Sep 26 12:30:20 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 08D5521F84A7 for ; Wed, 26 Sep 2012 12:30:20 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.203 X-Spam-Level: X-Spam-Status: No, score=-5.203 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_QP_LONG_LINE=1.396, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id s4pdy9ikTbwq for ; Wed, 26 Sep 2012 12:30:18 -0700 (PDT) Received: from eu1sys200aog120.obsmtp.com (eu1sys200aog120.obsmtp.com [207.126.144.149]) by ietfa.amsl.com (Postfix) with SMTP id C4ECB21F8546 for ; Wed, 26 Sep 2012 12:30:17 -0700 (PDT) Received: from mail-bk0-f44.google.com ([209.85.214.44]) (using TLSv1) by eu1sys200aob120.postini.com ([207.126.147.11]) with SMTP ID DSNKUGNXyNFO3NxdGOZjcPZVt+f9MOXGhL3t@postini.com; Wed, 26 Sep 2012 19:30:17 UTC Received: by bkcjc3 with SMTP id jc3so548015bkc.31 for ; Wed, 26 Sep 2012 12:30:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=references:in-reply-to:mime-version:content-transfer-encoding :content-type:message-id:cc:x-mailer:from:subject:date:to :x-gm-message-state; bh=9R1h+fq4mRnJBsCtNsacuJbTU3yEwx4d+L1m5dwgmnE=; b=odbSXAKJUU6DQwtOuPjM6tJTAgquF8Y2rBWedfmmAim71c2Bxrj0RJdRIkEy6roieG jh8ZYexpLt4FQMRXq6AnfkRLpQl1wJ9LugVoebJePGwHuNT/rU02Gkufftz7733XOsNg v45hyWNSafqrHQorg3CF86uUZmpJtLnDiKXhKqQdLPErADozo6VPSoUn/SxHttV0CHr/ fACE6WDDGxNv/cGiqHfk81KISHHZIcsWpDdcehmM0hcUuiNV5rwWhb+WBaaDFm3Pkc1s zOz7VeIAOb5ZkLBRRSzZNNUYD319fgcP+U0jPdNnJSynX1IXD5rd1ewNCaS079ciIDkY 6/bg== Received: by 10.204.145.85 with SMTP id c21mr1201104bkv.46.1348687816477; Wed, 26 Sep 2012 12:30:16 -0700 (PDT) Received: from [192.168.0.100] (ip-54-21.sn2.eutelia.it. [83.211.54.21]) by mx.google.com with ESMTPS id hy11sm3177837bkc.5.2012.09.26.12.30.13 (version=SSLv3 cipher=OTHER); Wed, 26 Sep 2012 12:30:14 -0700 (PDT) References: <8D3D17ACE214DC429325B2B98F3AE7120DE79578@MX15A.corp.emc.com> <9813C654-1D6E-43DE-B21A-2949729E042F@satran.net> <2C79B025-BD1D-4E95-9FC1-CB320A5EC8BA@satran.net> In-Reply-To: <2C79B025-BD1D-4E95-9FC1-CB320A5EC8BA@satran.net> Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Message-Id: X-Mailer: iPad Mail (10A403) From: Julian Satran Date: Wed, 26 Sep 2012 21:30:23 +0200 To: "Knight, Frederick" X-Gm-Message-State: ALoCoQmnTN2I5e08Ulnt4u1jDlONprvaIQJ9US+hXOVWGklR8KMwUwbhLtdmxbmRaBPPze3L1W95 Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2012 19:30:20 -0000 and overall i think that all the new text for reserve/release does more harm= than good. i would mention only that the state that must be maintained while the sessio= n is maintained includes the reseve/release and leave it to that. Sent from my iPad On 26 =D7=91=D7=A1=D7=A4=D7=98 2012, at 21:25, Julian Satran wrote: > as for time2wait whether you mention the linkage or not it exists in iSCSI= . > associting state other than that mentioned in 3270 with the sessiom/it nex= us is a thing that should be relegated to t10 as a target should not behave d= ifferently whenaccessed through iscsi or fcp as far as state is concerned. t= he only logic that could allow for a slightly different behavior is a link t= o the mechanism by which the trasport perceives the nexus loss. that may req= uire to mention time2wait for iscsi. >=20 > Sent from my iPad >=20 > On 26 =D7=91=D7=A1=D7=A4=D7=98 2012, at 21:09, Julian Satran wrote: >=20 >> Fred, >>=20 >> I don't think there is any new requirement. Time2retain indicates that a s= ession (an it nexus) exists. As long as the session exits you can't talk abu= t nexus loss - even if you have no connection active. >>=20 >> Julo >>=20 >> Sent from my iPad >>=20 >> On 26 =D7=91=D7=A1=D7=A4=D7=98 2012, at 14:39, "Knight, Frederick" wrote: >>=20 >>> I'm concerned about this new link between Time2Wait and RESERVE state. S= uch a link has not previously existed. I don't think we should create such a= linkage. >>>=20 >>> Section 4.6.3.3 includes: >>> .... In addition, the Logout Response indicates how long the target will= continue to hold resources for recovery (e.g., command execution that conti= nues on a new connection).... >>>=20 >>> This and many other places where Time2Wait and Time2Retain are discussed= , the points all have to do with command continuation; no mention of I_T sta= te, or RESERVE state, or mode pages, or anything else. >>>=20 >>> Section 7.5 says: >>> .... Time2Wait is the initial "respite time" before attempting an explic= it/implicit Logout for the CID in question or task reassignment for the affe= cted tasks (if any).... >>>=20 >>> Or, task reassignment of affected tasks; with no mention of I_T state. >>>=20 >>> Section 7.6 ties the termination of tasks to the Time2Wait / Time2Retain= timers. Section 11.14.5 makes the same statements about the task terminati= on linkage to those timers. >>>=20 >>> Section 11.15.3 says those timers are "...the minimum amount of time, in= seconds, to wait before attempting task reassignment.". >>>=20 >>> Section 8.3 defines Session State. And there is one sentence that I can= find in all 345 pages that associates that session state with these timers.= It is in section 11.15.4 where it says: If it is the last connection of a s= ession, the whole session state is discarded after Time2Retain. >>>=20 >>> I don't see anywhere that these timers are associated with any SCSI stat= e, or with the RESERVE state. Attempting to tie RESERVE state to these time= rs is a NEW requirement. Our goal here was specifically to NOT create any n= ew requirements. >>>=20 >>> We do not want to impact existing iSCSI implementations. Could we assoc= iate the action that occurred with the observable behavior the host sees? I= don't like putting SCSI layer stuff in a transport layer spec, but it shoul= d just be informative. Something like: >>>=20 >>> If the initiator receives a UA: 29/00 or 29/01 or 29/04 then the RESERVE= state has been lost. (Maybe 29/04 belongs in a RESERVE state unknown list)= . >>>=20 >>> If the initiator receives a UA: 29/07 then the RESERVE state has been pr= eserved. >>>=20 >>> I didn't list 29/02 or 29/03 or 29/05 or 29/06 since I view those as Par= allel SCSI specific. We can debate which items belong in which list, but wo= uld this kind of observable behavior approach be better than a new requireme= nt? FYI - I didn't invent this; I copied it from SAM-5r11 (sub-clause 6.3.4= ) where it states it as a requirement on the target - that when an I_T NEXU= S LOSS event occurs, if the state is retained, 29/07 is returned, and if the= state is not retained, then 29/01 is returned. >>>=20 >>> We also don't want to create any new requirement now in this consolidate= d draft that conflicts with this SAM-5 text, since we will eventually have t= o comply with SAM-5 and its follow-ons. >>>=20 >>> Another approach would be to ignore this for the consolidated draft, and= force it into the iSCSI-SAM draft - which references SAM-4 and SAM-5 where I= _T Nexus loss is already defined and the above text already exists. >>>=20 >>> Fred >>>=20 >>> -----Original Message----- >>> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf O= f Mallikarjun Chadalapaka >>> Sent: Tuesday, September 25, 2012 9:39 PM >>> To: Black, David; storm@ietf.org >>> Subject: Re: [storm] SPC-2 reserve/release: Proposed text - version 4 >>>=20 >>> Looks good to me overall. Two minor comments: >>>=20 >>> 1) For 4.4.3.1, I have replaced the current sentence with the following o= ne (instead of the proposed): >>>=20 >>> That is, it should reinstate the session via iSCSI session reinstatement= (Section 6.3.5) or continue via session continuation (Section 6.3.6). >>>=20 >>> 2) For proposed new text, I have tweaked the version 4 sentence below, k= eeping the existing implementations in view: >>>=20 >>> VERSION 4: >>> The Time2Wait and Time2Retain >>> timeout values (see Section 7.55) apply to retention of reserve/relea= se >>> reservation state after iSCSI session failure because that state >>> is part of the I_T Nexus state. >>>=20 >>> VERSION 5: >>> Instead, implementations are strongly encouraged to apply the=20 >>> Time2Wait and Time2Retain timeout values (see Section 7.5)=20 >>> to manage retention of reserve/release reservation state after=20 >>> iSCSI session failure because that state is part of the I_T nexus stat= e. >>>=20 >>>=20 >>> Thanks. >>>=20 >>> Mallikarjun >>>=20 >>>=20 >>> -----Original Message----- >>> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf O= f Black, David >>> Sent: Tuesday, September 25, 2012 2:39 PM >>> To: storm@ietf.org >>> Subject: [storm] FW: SPC-2 reserve/release: Proposed text - version 4 >>>=20 >>> version 4 changes: >>>=20 >>> More changes for Mallikarjun's comments, mostly a rewrite of the first a= dditional caveat bullet. I've chosen not to reference Annex B of >>> SPC-4 as that seems a bit far afield for a SCSI transport standard like i= SCSI. >>>=20 >>> version 3 changes: >>>=20 >>> Responding to Mallikarjun's comments (plus some more edits) - >>>=20 >>>> 1) IMHO, opening sentence of "I_T nexus state includes reservation stat= e" >>>> is a little too broad, >>>=20 >>> I removed that sentence, removed I_T Nexus from new section title and tw= eaked the persistent reservation text to talk about state in general as oppo= sed to I_T Nexus state in particular. >>>=20 >>>> 2) Hard reset is not an LU reset - it is either an iSCSI Target Warm=20= >>>> Reset, or (more likely) an iSCSI Target Cold Reset. In either case,=20 >>>> any of the three resets should clear the reserve/release state. >>>=20 >>> I removed the word "hard" - as you noted, an LU reset is sufficient to r= emove a reserve/release reservation. >>>=20 >>>> 3) Should use Time2Retain+Time2Wait, not Default Time2Wait >>>=20 >>> Brilliant minds think alike - I got to that one in version 2 ;-). >>>=20 >>>> 4) Session recovery does not necessarily preserve the I_T nexus state=20= >>>> (recovery may not occur within Time2Wait+Time2Retain, and may not use=20= >>>> the same ISID when it occurs), but session reinstatement and session=20= >>>> continuation certainly should. >>>=20 >>> I changed "session recovery" to "session reinstatement". >>>=20 >>>> 5) Should articulate the underlying linkage between Time2Retain (an=20 >>>> iSCSI concept), and the reserve/release state (a SCSI state). >>>=20 >>> I believe I also got to that one in version 2. >>>=20 >>> version 2 changes: >>>=20 >>> I moved a paragraph break in response to Ralph's comments, and rewrote t= he first additional consideration to reference the connection timeout values= in general. I also found some minor problems that need correction in Secti= on 4.6.3.3 (see end of this message). >>>=20 >>> --- version 4 follows --- >>>=20 >>> >>>=20 >>> Here's an initial attempt to propose text to deal with the reserve/ rele= ase topic in the consolidated iSCSI draft. Please comment, correct, suggest= edits, etc. >>>=20 >>> The new text would be a new section 4.4.3.2 that would follow 4.4.3.1: >>>=20 >>> 4.4.3.1. I_T Nexus State >>>=20 >>> Certain nexus relationships contain an explicit state (e.g., >>> initiator-specific mode pages) that may need to be preserved by >>> the device server [SAM2] in a logical unit through changes or >>> failures in the iSCSI layer (e.g., session failures). In order for >>> that state to be restored, the iSCSI initiator should reestablish >>> its session (re-login) to the same Target Portal Group using the >>> previous ISID. That is, it should perform session recovery as >>> described in Chapter 6. This is because the SCSI initiator port >>> identifier and the SCSI target port identifier (or relative target >>> port) form the datum that the SCSI logical unit device server uses >>> to identify the I_T nexus. >>>=20 >>> ------------------------------- >>>=20 >>> First, for clarity, the following change should be made in the above 4.4= .3.1 text to better identify the recovery mechanism: >>>=20 >>> OLD >>> That is, it should perform session recovery as >>> described in Chapter 6. >>> NEW >>> That is, it should recover the session via connection >>> reinstatement as described in Section 6.3.4. >>> END >>>=20 >>> I believe that a lower-case "should" remains appropriate for this text. >>>=20 >>> ----------------------------------- >>>=20 >>> NEW TEXT (second draft): >>>=20 >>> 4.4.3.2. Reservations >>>=20 >>> There are two reservation management methods defined in the SCSI >>> standards, reserve/release reservations, based on the RESERVE and >>> RELEASE commands [SPC2], and persistent reservations, based on the >>> PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. >>> Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be >>> used; persistent reservations SHOULD be used instead. >>>=20 >>> State for persistent reservations is required to persist >>> through changes and failures at the iSCSI layer that result in >>> I_T Nexus failures, see [SPC3] for details and specific requirements. >>>=20 >>> In contrast, [SPC2] does not specify detailed persistence >>> requirements for reserve/release reservation state after an I_T >>> Nexus failure. Nonetheless, when reserve/release reservations are >>> supported by an iSCSI target, the preferred implementation approach >>> is to preserve reserve/release reservation state as part of the >>> I_T Nexus state for iSCSI session reinstatement (see Section 6.3.5) >>> or session continuation (see Section 6.3.6). >>>=20 >>> Two additional caveats apply to reserve/release reservations: >>>=20 >>> - When connection failure causes the iSCSI session to fail, and >>> the session is not reinstated or continued, target retention >>> of that session's reserve/release reservation state for an >>> extended period of time may require the initiator to issue a >>> reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order >>> to remove that reservation state. The Time2Wait and Time2Retain >>> timeout values (see Section 7.55) apply to retention of reserve/relea= se >>> reservation state after iSCSI session failure because that state >>> is part of the I_T Nexus state. The need for resets in this >>> scenario may be reduced by suitable selection of values for >>> these two timeouts. >>>=20 >>> - Reserve/release reservations may not behave as expected when >>> persistent reservations are also used on the same logical unit; >>> see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE >>> behavior" in [SPC4]. >>>=20 >>> ----------------------- >>>=20 >>> Reference impacts: both [SPC2] and [SPC3] need to be normative reference= s, but [SPC4] can be an informative reference. >>>=20 >>> ------------------------- >>>=20 >>> I deliberately used "the preferred iSCSI implementation approach" >>> wording for reserve/release reservation state preservation requirements,= as SPC-2 is vague on this topic and I'm rather uncomfortable with placing a= requirement as strong as a "SHOULD" on an obsolete mechanism that "SHOULD N= OT" be used. >>>=20 >>> --------------- >>>=20 >>> New problem in Section 4.6.3.3 - in this text: >>>=20 >>> The Logout response indicates that the connection or session >>> cleanup is completed and no other responses will arrive on the >>> connection (if received on the logging out connection). In >>> addition, the Logout Response indicates how long the target will >>> continue to hold resources for recovery (e.g., command execution >>> that continues on a new connection) in the text key Time2Retain >>> and how long the initiator must wait before proceeding with >>> recovery in the text key Time2Wait. >>>=20 >>> Time2Wait and Time2Retain are not text keys - they're binary fields in t= he Logout Response PDU. Two changes are in order: >>> - "text key Time2Retain" -> "Time2Retain field" >>> - "text key Time2Wait" -> "Time2Wait field" >>>=20 >>> Thanks, >>> --David >>> ---------------------------------------------------- >>> David L. Black, Distinguished Engineer >>> EMC Corporation, 176 South St., Hopkinton, MA 01748 >>> +1 (508) 293-7953 FAX: +1 (508) 293-7786 >>> david.black@emc.com Mobile: +1 (978) 394-7754 >>> ---------------------------------------------------- >>>=20 >>> _______________________________________________ >>> storm mailing list >>> storm@ietf.org >>> https://www.ietf.org/mailman/listinfo/storm >>>=20 >>> _______________________________________________ >>> storm mailing list >>> storm@ietf.org >>> https://www.ietf.org/mailman/listinfo/storm >>>=20 >>> _______________________________________________ >>> storm mailing list >>> storm@ietf.org >>> https://www.ietf.org/mailman/listinfo/storm >>>=20 >>> _______________________________________________ >>> storm mailing list >>> storm@ietf.org >>> https://www.ietf.org/mailman/listinfo/storm >>>=20 >>>=20 >>> _______________________________________________ >>> storm mailing list >>> storm@ietf.org >>> https://www.ietf.org/mailman/listinfo/storm >>> _______________________________________________ >>> storm mailing list >>> storm@ietf.org >>> https://www.ietf.org/mailman/listinfo/storm From david.black@emc.com Wed Sep 26 15:25:37 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E373221F8551 for ; Wed, 26 Sep 2012 15:25:37 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id R-YEMMSbKEoq for ; Wed, 26 Sep 2012 15:25:37 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 3F17B21F854F for ; Wed, 26 Sep 2012 15:25:36 -0700 (PDT) Received: from hop04-l1d11-si02.isus.emc.com (HOP04-L1D11-SI02.isus.emc.com [10.254.111.55]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8QMPY18023593 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 26 Sep 2012 18:25:35 -0400 Received: from mailhub.lss.emc.com (mailhubhoprd04.lss.emc.com [10.254.222.226]) by hop04-l1d11-si02.isus.emc.com (RSA Interceptor) for ; Wed, 26 Sep 2012 18:25:20 -0400 Received: from mxhub30.corp.emc.com (mxhub30.corp.emc.com [128.222.70.170]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8QMPJmV017800 for ; Wed, 26 Sep 2012 18:25:19 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub30.corp.emc.com ([128.222.70.170]) with mapi; Wed, 26 Sep 2012 18:25:16 -0400 From: "Black, David" To: "storm@ietf.org" Date: Wed, 26 Sep 2012 18:25:14 -0400 Thread-Topic: SPC-2 reserve/release text - version 6 Thread-Index: Ac2cNcys4MzRK8PXQu651fVKNuEIxQ== Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: [storm] SPC-2 reserve/release text - version 6 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2012 22:25:38 -0000 Here's where I think we've arrived (I've made a few minor editorial tweaks to what's been discussed). Despite all the detailed discussion, I think this version 6 comes close to Julian's suggestion that: > i would mention only that the state that must be maintained while > the session is maintained includes the reserve/release and leave it to th= at. Although we can't actually say "must" as that would be a new requirement, and implementations differ in what they do about reserve/release reservation state. The current language characterizes retention of that state as the "preferred implementation approach". Beyond that, I think the two additional caveats about reserve/release reservations (possible need for a reset, potentially unexpected interaction with persistent reservations, see below) are useful to include. ----------------------------------------- [1] For clarity, the following change should be made in 4.4.3.1: OLD That is, it should perform session recovery as described in Chapter 6. NEW That is, it should reinstate the session via iSCSI session reinstatement (Section 6.3.5) or continue via session continuation (Section 6.3.6). END [2] Add the following sentence to the end of 4.4.3.1: The specific Unit Attention condition that results from session reestablishment indicates whether nexus state was preserved, see Section 6.3.4 of [SAM4]. [3] NEW TEXT: 4.4.3.2. Reservations There are two reservation management methods defined in the SCSI standards, reserve/release reservations, based on the RESERVE and RELEASE commands [SPC2], and persistent reservations, based on the PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be used; persistent reservations SHOULD be used instead. State for persistent reservations is required to persist through changes and failures at the iSCSI layer that result in I_T Nexus failures, see [SPC3] for details and specific requirements. In contrast, [SPC2] does not specify detailed persistence requirements for reserve/release reservation state after an I_T Nexus failure. Nonetheless, when reserve/release reservations are supported by an iSCSI target, the preferred implementation approach is to preserve reserve/release reservation state for iSCSI session reinstatement (see Section 6.3.5) or session continuation (see Section 6.3.6). Two additional caveats apply to reserve/release reservations: - When connection failure causes the iSCSI session to fail, and the session is not reinstated or continued, target retention of that session's reserve/release reservation state may require an initiator to issue a reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state. - Reserve/release reservations may not behave as expected when persistent reservations are also used on the same logical unit; see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE behavior" in [SPC4]. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- From julian@satran.net Wed Sep 26 19:25:18 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E766F21F8587 for ; Wed, 26 Sep 2012 19:25:18 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.203 X-Spam-Level: X-Spam-Status: No, score=-5.203 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_QP_LONG_LINE=1.396, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kXkgTgoPQikx for ; Wed, 26 Sep 2012 19:25:18 -0700 (PDT) Received: from eu1sys200aog106.obsmtp.com (eu1sys200aog106.obsmtp.com [207.126.144.121]) by ietfa.amsl.com (Postfix) with SMTP id 5DFF021F861A for ; Wed, 26 Sep 2012 19:25:17 -0700 (PDT) Received: from mail-we0-f198.google.com ([74.125.82.198]) (using TLSv1) by eu1sys200aob106.postini.com ([207.126.147.11]) with SMTP ID DSNKUGO5CcXFgOj74R1bDhfq0CDNlRoXaSxF@postini.com; Thu, 27 Sep 2012 02:25:17 UTC Received: by weyz53 with SMTP id z53so329384wey.1 for ; Wed, 26 Sep 2012 19:25:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=references:in-reply-to:mime-version:content-transfer-encoding :content-type:message-id:cc:x-mailer:from:subject:date:to :x-gm-message-state; bh=r37a9JDmwokyUHYvqws2iqfV4TTf6wy1yU5MyX8AANo=; b=A12e8c7HlKv5K6qBK/zVa0Soa4YZY9y5BV4pz5Hp8e98Lrr5wR7Bl0BHepedQVLyYw tks5fODj7DAfMqjEd0/qOK/ORYaltQLi7gcSG3gqyU7dxxkymhOPxa+PU0rIXGjWLxmy MZkMFMJhVWoAHVykXetExpNR8xFOzMCjwJsSfLAKGQipZ42h65d39FcQshjLmY6otsAA Ph+VF/o/ENvUWLV1T94D/gpIrRrAuJ8OreyD11EfxsicNo4OYb9cmIqqF6nYzIoCzn9r DYkQlTdo0mcUY7J+6Lj+BZLIGT8jNzW33fb53wPziytV4G7hTEN3vqKNdSVSX524eJ8H MURw== Received: by 10.180.82.164 with SMTP id j4mr4983142wiy.18.1348712713887; Wed, 26 Sep 2012 19:25:13 -0700 (PDT) Received: by 10.180.82.164 with SMTP id j4mr4983127wiy.18.1348712713743; Wed, 26 Sep 2012 19:25:13 -0700 (PDT) Received: from [192.168.0.100] (ip-54-21.sn2.eutelia.it. [83.211.54.21]) by mx.google.com with ESMTPS id hv8sm35539509wib.0.2012.09.26.19.25.10 (version=SSLv3 cipher=OTHER); Wed, 26 Sep 2012 19:25:12 -0700 (PDT) References: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Message-Id: <4AE113CD-96D7-414D-8AAB-36ECF7FD89EA@satran.net> X-Mailer: iPad Mail (10A403) From: Julian Satran Date: Thu, 27 Sep 2012 04:25:22 +0200 To: "Black, David" X-Gm-Message-State: ALoCoQmppvwk7tE6n+Lkq1IOy/sB/Y0Kimsjbh165Y/Ifos++LjttuQjpUeM8i+XbdCupuHhZW8+dnO80ROiCEk9cQcdbTGmFVGPtpj7j5qvnnwGaj6dYHxlsETnYKei3c7o88c2VuWWEhsIAzaIpKZ2WFooG3ik/w== Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release text - version 6 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Sep 2012 02:25:19 -0000 David, The text in 4.4.3 states the obvious (there is no other speced method for co= ntinuation or reinstatement). Moreover if the the session is reinstated afte= r time2wait you create a need for a UA (or at least suggest one) and that wa= s not there. Just stating that creation/continuation of a session done accor= ding 6.x includes preservation of reserve/release state -that was already th= ere implied by conformance to spc2 and the relation between it nexus and ses= sion. The new text you suggest to 4.4 is merely a best practices recomendations an= d will need additions once the new "locking" through compare-and-swap get in= widespread use. If instead we refrain from stating anything specific and state that target s= hould maintain whatever state is required by SPC while a session is continue= d and may discard state when a session is terminated and may indicate that i= t discarded state through a ua you are on safer ground and do not create new= requirement. Sent from my iPad On 27 =D7=91=D7=A1=D7=A4=D7=98 2012, at 00:25, "Black, David" wrote: > Here's where I think we've arrived (I've made a few minor editorial > tweaks to what's been discussed). Despite all the detailed discussion, > I think this version 6 comes close to Julian's suggestion that: >=20 >> i would mention only that the state that must be maintained while >> the session is maintained includes the reserve/release and leave it to th= at. >=20 > Although we can't actually say "must" as that would be a new requirement, > and implementations differ in what they do about reserve/release > reservation state. The current language characterizes retention of > that state as the "preferred implementation approach". >=20 > Beyond that, I think the two additional caveats about reserve/release > reservations (possible need for a reset, potentially unexpected > interaction with persistent reservations, see below) are useful to include= . >=20 > ----------------------------------------- >=20 > [1] For clarity, the following change should be made in > 4.4.3.1: >=20 > OLD > That is, it should perform session recovery as > described in Chapter 6. > NEW > That is, it should reinstate the session via iSCSI session > reinstatement (Section 6.3.5) or continue via session > continuation (Section 6.3.6). > END >=20 > [2] Add the following sentence to the end of 4.4.3.1: >=20 > The specific Unit Attention condition that results from session > reestablishment indicates whether nexus state was preserved, > see Section 6.3.4 of [SAM4]. >=20 > [3] NEW TEXT: >=20 > 4.4.3.2. Reservations >=20 > There are two reservation management methods defined in the SCSI > standards, reserve/release reservations, based on the RESERVE and > RELEASE commands [SPC2], and persistent reservations, based on the > PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. > Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be > used; persistent reservations SHOULD be used instead. >=20 > State for persistent reservations is required to persist > through changes and failures at the iSCSI layer that result in > I_T Nexus failures, see [SPC3] for details and specific requirements. >=20 > In contrast, [SPC2] does not specify detailed persistence > requirements for reserve/release reservation state after an I_T > Nexus failure. Nonetheless, when reserve/release reservations are > supported by an iSCSI target, the preferred implementation approach > is to preserve reserve/release reservation state for iSCSI session > reinstatement (see Section 6.3.5) or session continuation > (see Section 6.3.6). >=20 > Two additional caveats apply to reserve/release reservations: >=20 > - When connection failure causes the iSCSI session to fail, and > the session is not reinstated or continued, target retention > of that session's reserve/release reservation state may > require an initiator to issue a reset (e.g., LOGICAL UNIT RESET, > see section 11.5) in order to remove that reservation state. >=20 > - Reserve/release reservations may not behave as expected when > persistent reservations are also used on the same logical unit; > see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE > behavior" in [SPC4]. >=20 > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > david.black@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- >=20 >=20 > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From david.black@emc.com Thu Sep 27 07:47:07 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D82F621F852C for ; Thu, 27 Sep 2012 07:47:07 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Y7dIQJqR-EbW for ; Thu, 27 Sep 2012 07:47:07 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id ECFC721F852E for ; Thu, 27 Sep 2012 07:47:06 -0700 (PDT) Received: from hop04-l1d11-si02.isus.emc.com (HOP04-L1D11-SI02.isus.emc.com [10.254.111.55]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8REl4uJ031141 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 27 Sep 2012 10:47:05 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.222.130]) by hop04-l1d11-si02.isus.emc.com (RSA Interceptor); Thu, 27 Sep 2012 10:46:49 -0400 Received: from mxhub11.corp.emc.com (mxhub11.corp.emc.com [10.254.92.106]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8REkm7T026970; Thu, 27 Sep 2012 10:46:48 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub11.corp.emc.com ([10.254.92.106]) with mapi; Thu, 27 Sep 2012 10:46:47 -0400 From: "Black, David" To: Julian Satran Date: Thu, 27 Sep 2012 10:46:45 -0400 Thread-Topic: [storm] SPC-2 reserve/release text - version 6 Thread-Index: Ac2cV1jTBbQsTowrTQO1GweKNZL/AAAY50HA Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE798F9@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> <4AE113CD-96D7-414D-8AAB-36ECF7FD89EA@satran.net> In-Reply-To: <4AE113CD-96D7-414D-8AAB-36ECF7FD89EA@satran.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-EMM-MHVC: 1 Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release text - version 6 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Sep 2012 14:47:08 -0000 SnVsaWFuLA0KDQo+IFRoZSB0ZXh0IGluIDQuNC4zIHN0YXRlcyB0aGUgb2J2aW91cyAodGhlcmUg aXMgbm8gb3RoZXIgc3BlY2VkIG1ldGhvZCBmb3INCj4gY29udGludWF0aW9uIG9yIHJlaW5zdGF0 ZW1lbnQpLiBNb3Jlb3ZlciBpZiB0aGUgc2Vzc2lvbiBpcyByZWluc3RhdGVkDQo+IGFmdGVyIHRp bWUyd2FpdCB5b3UgY3JlYXRlIGEgbmVlZCBmb3IgYSBVQSAob3IgYXQgbGVhc3Qgc3VnZ2VzdCBv bmUpIGFuZCB0aGF0DQo+IHdhcyBub3QgdGhlcmUuDQoNClRoZSBVQXMgYXJlIGNvdXJ0ZXN5IG9m IFNBTS0zIChhbmQgU0FNLTQpLiAgVG8gZW5jb21wYXNzIHN5c3RlbXMgdGhhdCBtYXkNCm5vdCBk ZWxpdmVyIHRoZXNlIFVBcyBhbmQgc2l0dWF0aW9ucyBpbiB3aGljaCB0aGV5IGRvbid0IG9jY3Vy IChlLmcuLCB0aGUNCnRhcmdldCBkb2Vzbid0IHJlY29nbml6ZSB3aGF0IGhhcHBlbmVkIGFzIG5l eHVzIHJlZXN0YWJsaXNobWVudCwNCmluZGVwZW5kZW50IG9mIHdoYXQgdGhlIGluaXRpYXRvciB0 aGlua3MgaXQncyBkb2luZykgSSB0aGluayB0aGUgdGV4dA0Kc2hvdWxkIGJlIHJlcGhyYXNlZCBh czoNCg0KICAgICBJZiBhIFVuaXQgQXR0ZW50aW9uIGNvbmRpdGlvbiByZXN1bHRzIGZyb20gc2Vz c2lvbiByZWVzdGFibGlzaG1lbnQsDQogICAgIHRoZSBzcGVjaWZpYyBVbml0IEF0dGVudGlvbiBj b25kaXRpb24gaW5kaWNhdGVzIHdoZXRoZXIgbmV4dXMgc3RhdGUNCiAgICAgd2FzIHByZXNlcnZl ZCwgc2VlIFNlY3Rpb24gNi4zLjQgb2YgW1NBTTRdLg0KDQo+IEp1c3Qgc3RhdGluZyB0aGF0IGNy ZWF0aW9uL2NvbnRpbnVhdGlvbiBvZiBhIHNlc3Npb24gZG9uZQ0KPiBhY2NvcmRpbmcgNi54IGlu Y2x1ZGVzIHByZXNlcnZhdGlvbiBvZiByZXNlcnZlL3JlbGVhc2Ugc3RhdGUgLXRoYXQgd2FzIGFs cmVhZHkNCj4gdGhlcmUgaW1wbGllZCBieSBjb25mb3JtYW5jZSB0byBzcGMyIGFuZCB0aGUgcmVs YXRpb24gYmV0d2VlbiBpdCBuZXh1cyBhbmQNCj4gc2Vzc2lvbi4NCg0KVW5mb3J0dW5hdGVseSwg U1BDLTIgaXMgc3VmZmljaWVudGx5IGltcHJlY2lzZSBvbiB0aGlzIHN1YmplY3QgdGhhdCBzdWNo IGFuDQppbXBsaWNhdGlvbiBpcyBhdCBiZXN0IGEgInByZWZlcnJlZCBpbXBsZW1lbnRhdGlvbiBh cHByb2FjaCIuICBJdCdzIG5vdCBhDQpyZXF1aXJlbWVudCB0aGF0IGNhbiBiZSBzYWZlbHkgc3Rh dGVkIGluIGFueSBzdHJvbmdlciBmYXNoaW9uIHdpdGhvdXQgdGhlDQpyaXNrIG9mIHJlbmRlcmlu ZyBleGlzdGluZyBpbXBsZW1lbnRhdGlvbnMgbm9uLWNvbXBsaWFudC4NCg0KPiBUaGUgbmV3IHRl eHQgeW91IHN1Z2dlc3QgdG8gNC40IGlzIG1lcmVseSBhIGJlc3QgcHJhY3RpY2VzIHJlY29tbWVu ZGF0aW9ucyBhbmQNCj4gd2lsbCBuZWVkIGFkZGl0aW9ucyBvbmNlIHRoZSBuZXcgImxvY2tpbmci IHRocm91Z2ggY29tcGFyZS1hbmQtc3dhcCBnZXQgaW4NCj4gd2lkZXNwcmVhZCB1c2UuDQoNCkl0 J3MgZGVmaW5pdGVseSBiZXN0IHByYWN0aWNlcyByZWNvbW1lbmRhdGlvbnMgLSBubyBhcmd1bWVu dCB0aGVyZS4NCg0KRm9yIGNvbnRleHQsIHdoYXQgSnVsaWFuIGlzIHJlZmVycmluZyB0byBpcyB1 c2Ugb2YgdGhlIENPTVBBUkUgQU5EIFdSSVRFIGNvbW1hbmQNCnRoYXQgcGVyZm9ybXMgYW4gYXRv bWljIG9wZXJhdGlvbiBhdCB0aGUgdGFyZ2V0IGluc3RlYWQgb2YgdXNpbmcgUkVTRVJWRSBhbmQg UkVMRUFTRQ0KdG8gY3JlYXRlIGEgY3JpdGljYWwgc2VjdGlvbiBhdCB0aGUgaW5pdGlhdG9yIGJh c2VkIG9uIHJlYWQgYW5kIHdyaXRlIGNvbW1hbmRzLg0KDQpDT01QQVJFIEFORCBXUklURSBpcyBp biB1c2UgLSBhIHNlbnRlbmNlIG1lbnRpb25pbmcgdGhhdCBjb21tYW5kIGNvdWxkIGJlIGFkZGVk LA0KYWx0aG91Z2ggdGhpcyBiZXN0IHByYWN0aWNlcyB0ZXh0IGlzIGFpbWVkIHByaW1hcmlseSBh dCB1c2VzIG9mIHJlc2VydmUvcmVsZWFzZQ0KZm9yIGxvbmctdGVybSByZXNlcnZhdGlvbnMuICBJ IHRoaW5rIHRoZXJlJ3MgYW4gaW1wbGljaXQgIndoZW4gYSByZXNlcnZhdGlvbiBpcw0KdGhlIGRl c2lyZWQgYmVoYXZpb3IiIGltcGxpY2F0aW9uIHRvIHRoZSAicGVyc2lzdGVudCByZXNlcnZhdGlv bnMgU0hPVUxEIGJlIHVzZWQNCmluc3RlYWQiIHRleHQgdGhhdCBJIGRvbid0IHRoaW5rIG5lZWRz IHRvIGJlIG1hZGUgZXhwbGljaXQuICBPVE9ILCB0aGF0ICJTSE9VTEQiDQpjb3VsZCBiZSBjaGFu Z2VkIHRvIGEgInNob3VsZCIsIG9yIGl0IG1pZ2h0IGJlIGV2ZW4gYmV0dGVyIHRvIHJld3JpdGUg dGhhdCB0ZXh0DQphczogDQoNCglwZXJzaXN0ZW50IHJlc2VydmF0aW9ucyBhcmUgc3VnZ2VzdGVk IGFzIGFuIGFsdGVybmF0aXZlLCBzZWUgQW5uZXggQg0KCW9mIFtTUEM0XS4NCg0KSW4gY29udHJh c3QsICJTSE9VTEQgTk9UIGJlIHVzZWQiIGlzIGNsZWFybHkganVzdGlmaWVkIGZvciByZXNlcnZl L3JlbGVhc2UNCnJlc2VydmF0aW9ucyBieSBbU1BDM10uDQoNClRoYW5rcywNCi0tRGF2aWQNCg0K DQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IEp1bGlhbiBTYXRyYW4gW21h aWx0bzpqdWxpYW5Ac2F0cmFuLm5ldF0NCj4gU2VudDogV2VkbmVzZGF5LCBTZXB0ZW1iZXIgMjYs IDIwMTIgMTA6MjUgUE0NCj4gVG86IEJsYWNrLCBEYXZpZA0KPiBDYzogc3Rvcm1AaWV0Zi5vcmcN Cj4gU3ViamVjdDogUmU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJz aW9uIDYNCj4gDQo+IERhdmlkLA0KPiANCj4gVGhlIHRleHQgaW4gNC40LjMgc3RhdGVzIHRoZSBv YnZpb3VzICh0aGVyZSBpcyBubyBvdGhlciBzcGVjZWQgbWV0aG9kIGZvcg0KPiBjb250aW51YXRp b24gb3IgcmVpbnN0YXRlbWVudCkuIE1vcmVvdmVyIGlmIHRoZSB0aGUgc2Vzc2lvbiBpcyByZWlu c3RhdGVkDQo+IGFmdGVyIHRpbWUyd2FpdCB5b3UgY3JlYXRlIGEgbmVlZCBmb3IgYSBVQSAob3Ig YXQgbGVhc3Qgc3VnZ2VzdCBvbmUpIGFuZCB0aGF0DQo+IHdhcyBub3QgdGhlcmUuIEp1c3Qgc3Rh dGluZyB0aGF0IGNyZWF0aW9uL2NvbnRpbnVhdGlvbiBvZiBhIHNlc3Npb24gZG9uZQ0KPiBhY2Nv cmRpbmcgNi54IGluY2x1ZGVzIHByZXNlcnZhdGlvbiBvZiByZXNlcnZlL3JlbGVhc2Ugc3RhdGUg LXRoYXQgd2FzIGFscmVhZHkNCj4gdGhlcmUgaW1wbGllZCBieSBjb25mb3JtYW5jZSB0byBzcGMy IGFuZCB0aGUgcmVsYXRpb24gYmV0d2VlbiBpdCBuZXh1cyBhbmQNCj4gc2Vzc2lvbi4NCj4gDQo+ IFRoZSBuZXcgdGV4dCB5b3Ugc3VnZ2VzdCB0byA0LjQgaXMgbWVyZWx5IGEgYmVzdCBwcmFjdGlj ZXMgcmVjb21lbmRhdGlvbnMgYW5kDQo+IHdpbGwgbmVlZCBhZGRpdGlvbnMgb25jZSB0aGUgbmV3 ICJsb2NraW5nIiB0aHJvdWdoIGNvbXBhcmUtYW5kLXN3YXAgZ2V0IGluDQo+IHdpZGVzcHJlYWQg dXNlLg0KPiBJZiBpbnN0ZWFkIHdlIHJlZnJhaW4gZnJvbSBzdGF0aW5nIGFueXRoaW5nIHNwZWNp ZmljIGFuZCBzdGF0ZSB0aGF0IHRhcmdldA0KPiBzaG91bGQgbWFpbnRhaW4gd2hhdGV2ZXIgc3Rh dGUgaXMgcmVxdWlyZWQgYnkgU1BDIHdoaWxlIGEgc2Vzc2lvbiBpcyBjb250aW51ZWQNCj4gYW5k IG1heSBkaXNjYXJkIHN0YXRlIHdoZW4gYSBzZXNzaW9uIGlzIHRlcm1pbmF0ZWQgYW5kIG1heSBp bmRpY2F0ZSB0aGF0IGl0DQo+IGRpc2NhcmRlZCBzdGF0ZSB0aHJvdWdoIGEgdWEgeW91IGFyZSBv biBzYWZlciBncm91bmQgYW5kIGRvIG5vdCBjcmVhdGUgbmV3DQo+IHJlcXVpcmVtZW50Lg0KPiAN Cj4gU2VudCBmcm9tIG15IGlQYWQNCj4gDQo+IE9uIDI3INeR16HXpNeYIDIwMTIsIGF0IDAwOjI1 LCAiQmxhY2ssIERhdmlkIiA8ZGF2aWQuYmxhY2tAZW1jLmNvbT4gd3JvdGU6DQo+IA0KPiA+IEhl cmUncyB3aGVyZSBJIHRoaW5rIHdlJ3ZlIGFycml2ZWQgKEkndmUgbWFkZSBhIGZldyBtaW5vciBl ZGl0b3JpYWwNCj4gPiB0d2Vha3MgdG8gd2hhdCdzIGJlZW4gZGlzY3Vzc2VkKS4gIERlc3BpdGUg YWxsIHRoZSBkZXRhaWxlZCBkaXNjdXNzaW9uLA0KPiA+IEkgdGhpbmsgdGhpcyB2ZXJzaW9uIDYg Y29tZXMgY2xvc2UgdG8gSnVsaWFuJ3Mgc3VnZ2VzdGlvbiB0aGF0Og0KPiA+DQo+ID4+IGkgd291 bGQgbWVudGlvbiBvbmx5IHRoYXQgdGhlIHN0YXRlIHRoYXQgbXVzdCBiZSBtYWludGFpbmVkIHdo aWxlDQo+ID4+IHRoZSBzZXNzaW9uIGlzIG1haW50YWluZWQgaW5jbHVkZXMgdGhlIHJlc2VydmUv cmVsZWFzZSBhbmQgbGVhdmUgaXQgdG8NCj4gdGhhdC4NCj4gPg0KPiA+IEFsdGhvdWdoIHdlIGNh bid0IGFjdHVhbGx5IHNheSAibXVzdCIgYXMgdGhhdCB3b3VsZCBiZSBhIG5ldyByZXF1aXJlbWVu dCwNCj4gPiBhbmQgaW1wbGVtZW50YXRpb25zIGRpZmZlciBpbiB3aGF0IHRoZXkgZG8gYWJvdXQg cmVzZXJ2ZS9yZWxlYXNlDQo+ID4gcmVzZXJ2YXRpb24gc3RhdGUuICBUaGUgY3VycmVudCBsYW5n dWFnZSBjaGFyYWN0ZXJpemVzIHJldGVudGlvbiBvZg0KPiA+IHRoYXQgc3RhdGUgYXMgdGhlICJw cmVmZXJyZWQgaW1wbGVtZW50YXRpb24gYXBwcm9hY2giLg0KPiA+DQo+ID4gQmV5b25kIHRoYXQs IEkgdGhpbmsgdGhlIHR3byBhZGRpdGlvbmFsIGNhdmVhdHMgYWJvdXQgcmVzZXJ2ZS9yZWxlYXNl DQo+ID4gcmVzZXJ2YXRpb25zIChwb3NzaWJsZSBuZWVkIGZvciBhIHJlc2V0LCBwb3RlbnRpYWxs eSB1bmV4cGVjdGVkDQo+ID4gaW50ZXJhY3Rpb24gd2l0aCBwZXJzaXN0ZW50IHJlc2VydmF0aW9u cywgc2VlIGJlbG93KSBhcmUgdXNlZnVsIHRvIGluY2x1ZGUuDQo+ID4NCj4gPiAtLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiA+DQo+ID4gWzFdIEZvciBjbGFyaXR5 LCB0aGUgZm9sbG93aW5nIGNoYW5nZSBzaG91bGQgYmUgbWFkZSBpbg0KPiA+IDQuNC4zLjE6DQo+ ID4NCj4gPiBPTEQNCj4gPiAgVGhhdCBpcywgaXQgc2hvdWxkIHBlcmZvcm0gc2Vzc2lvbiByZWNv dmVyeSBhcw0KPiA+ICBkZXNjcmliZWQgaW4gQ2hhcHRlciA2Lg0KPiA+IE5FVw0KPiA+ICBUaGF0 IGlzLCBpdCBzaG91bGQgcmVpbnN0YXRlIHRoZSBzZXNzaW9uIHZpYSBpU0NTSSBzZXNzaW9uDQo+ ID4gIHJlaW5zdGF0ZW1lbnQgKFNlY3Rpb24gNi4zLjUpIG9yIGNvbnRpbnVlIHZpYSBzZXNzaW9u DQo+ID4gIGNvbnRpbnVhdGlvbiAoU2VjdGlvbiA2LjMuNikuDQo+ID4gRU5EDQo+ID4NCj4gPiBb Ml0gQWRkIHRoZSBmb2xsb3dpbmcgc2VudGVuY2UgdG8gdGhlIGVuZCBvZiA0LjQuMy4xOg0KPiA+ DQo+ID4gICAgIFRoZSBzcGVjaWZpYyBVbml0IEF0dGVudGlvbiBjb25kaXRpb24gdGhhdCByZXN1 bHRzIGZyb20gc2Vzc2lvbg0KPiA+ICAgICByZWVzdGFibGlzaG1lbnQgaW5kaWNhdGVzIHdoZXRo ZXIgbmV4dXMgc3RhdGUgd2FzIHByZXNlcnZlZCwNCj4gPiAgICAgc2VlIFNlY3Rpb24gNi4zLjQg b2YgW1NBTTRdLg0KPiA+DQo+ID4gWzNdIE5FVyBURVhUOg0KPiA+DQo+ID4gNC40LjMuMi4gUmVz ZXJ2YXRpb25zDQo+ID4NCj4gPiAgVGhlcmUgYXJlIHR3byByZXNlcnZhdGlvbiBtYW5hZ2VtZW50 IG1ldGhvZHMgZGVmaW5lZCBpbiB0aGUgU0NTSQ0KPiA+ICBzdGFuZGFyZHMsIHJlc2VydmUvcmVs ZWFzZSByZXNlcnZhdGlvbnMsIGJhc2VkIG9uIHRoZSBSRVNFUlZFIGFuZA0KPiA+ICBSRUxFQVNF IGNvbW1hbmRzIFtTUEMyXSwgYW5kIHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zLCBiYXNlZCBvbiB0 aGUNCj4gPiAgUEVSU0lTVEVOVCBSRVNFUlZFIElOIGFuZCBQRVJTSVNURU5UIFJFU0VSVkUgT1VU IGNvbW1hbmRzIFtTUEMzXS4NCj4gPiAgUmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9ucyBhcmUg b2Jzb2xldGUgW1NQQzNdIGFuZCBTSE9VTEQgTk9UIGJlDQo+ID4gIHVzZWQ7IHBlcnNpc3RlbnQg cmVzZXJ2YXRpb25zIFNIT1VMRCBiZSB1c2VkIGluc3RlYWQuDQo+ID4NCj4gPiAgU3RhdGUgZm9y IHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIGlzIHJlcXVpcmVkIHRvIHBlcnNpc3QNCj4gPiAgdGhy b3VnaCBjaGFuZ2VzIGFuZCBmYWlsdXJlcyBhdCB0aGUgaVNDU0kgbGF5ZXIgdGhhdCByZXN1bHQg aW4NCj4gPiAgSV9UIE5leHVzIGZhaWx1cmVzLCBzZWUgW1NQQzNdIGZvciBkZXRhaWxzIGFuZCBz cGVjaWZpYyByZXF1aXJlbWVudHMuDQo+ID4NCj4gPiAgSW4gY29udHJhc3QsIFtTUEMyXSBkb2Vz IG5vdCBzcGVjaWZ5IGRldGFpbGVkIHBlcnNpc3RlbmNlDQo+ID4gIHJlcXVpcmVtZW50cyBmb3Ig cmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9uIHN0YXRlIGFmdGVyIGFuIElfVA0KPiA+ICBOZXh1 cyBmYWlsdXJlLiAgTm9uZXRoZWxlc3MsIHdoZW4gcmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9u cyBhcmUNCj4gPiAgc3VwcG9ydGVkIGJ5IGFuIGlTQ1NJIHRhcmdldCwgdGhlIHByZWZlcnJlZCBp bXBsZW1lbnRhdGlvbiBhcHByb2FjaA0KPiA+ICBpcyB0byBwcmVzZXJ2ZSByZXNlcnZlL3JlbGVh c2UgcmVzZXJ2YXRpb24gc3RhdGUgZm9yIGlTQ1NJIHNlc3Npb24NCj4gPiAgcmVpbnN0YXRlbWVu dCAoc2VlIFNlY3Rpb24gNi4zLjUpIG9yIHNlc3Npb24gY29udGludWF0aW9uDQo+ID4gIChzZWUg U2VjdGlvbiA2LjMuNikuDQo+ID4NCj4gPiAgVHdvIGFkZGl0aW9uYWwgY2F2ZWF0cyBhcHBseSB0 byByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zOg0KPiA+DQo+ID4gIC0gV2hlbiBjb25uZWN0 aW9uIGZhaWx1cmUgY2F1c2VzIHRoZSBpU0NTSSBzZXNzaW9uIHRvIGZhaWwsIGFuZA0KPiA+ICAg ICAgdGhlIHNlc3Npb24gaXMgbm90IHJlaW5zdGF0ZWQgb3IgY29udGludWVkLCB0YXJnZXQgcmV0 ZW50aW9uDQo+ID4gICAgICBvZiB0aGF0IHNlc3Npb24ncyByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2 YXRpb24gc3RhdGUgbWF5DQo+ID4gICAgICByZXF1aXJlIGFuIGluaXRpYXRvciB0byBpc3N1ZSBh IHJlc2V0IChlLmcuLCBMT0dJQ0FMIFVOSVQgUkVTRVQsDQo+ID4gICAgICBzZWUgc2VjdGlvbiAx MS41KSBpbiBvcmRlciB0byByZW1vdmUgdGhhdCByZXNlcnZhdGlvbiBzdGF0ZS4NCj4gPg0KPiA+ ICAtIFJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbnMgbWF5IG5vdCBiZWhhdmUgYXMgZXhwZWN0 ZWQgd2hlbg0KPiA+ICAgICAgcGVyc2lzdGVudCByZXNlcnZhdGlvbnMgYXJlIGFsc28gdXNlZCBv biB0aGUgc2FtZSBsb2dpY2FsIHVuaXQ7DQo+ID4gICAgICBzZWUgdGhlIGRpc2N1c3Npb24gb2Yg IkV4Y2VwdGlvbnMgdG8gU1BDLTIgUkVTRVJWRSBhbmQgUkVMRUFTRQ0KPiA+ICAgICAgYmVoYXZp b3IiIGluIFtTUEM0XS4NCj4gPg0KPiA+IFRoYW5rcywNCj4gPiAtLURhdmlkDQo+ID4gLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiA+IERhdmlk IEwuIEJsYWNrLCBEaXN0aW5ndWlzaGVkIEVuZ2luZWVyDQo+ID4gRU1DIENvcnBvcmF0aW9uLCAx NzYgU291dGggU3QuLCBIb3BraW50b24sIE1BICAwMTc0OA0KPiA+ICsxICg1MDgpIDI5My03OTUz ICAgICAgICAgICAgIEZBWDogKzEgKDUwOCkgMjkzLTc3ODYNCj4gPiBkYXZpZC5ibGFja0BlbWMu Y29tICAgICAgICBNb2JpbGU6ICsxICg5NzgpIDM5NC03NzU0DQo+ID4gLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiA+DQo+ID4NCj4gPiBfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiA+IHN0b3JtIG1h aWxpbmcgbGlzdA0KPiA+IHN0b3JtQGlldGYub3JnDQo+ID4gaHR0cHM6Ly93d3cuaWV0Zi5vcmcv bWFpbG1hbi9saXN0aW5mby9zdG9ybQ0KDQo= From Frederick.Knight@netapp.com Thu Sep 27 08:11:03 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B506C21F85B1 for ; Thu, 27 Sep 2012 08:11:03 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 21iatzpaRBqn for ; Thu, 27 Sep 2012 08:11:02 -0700 (PDT) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id D008221F859E for ; Thu, 27 Sep 2012 08:11:02 -0700 (PDT) X-IronPort-AV: E=Sophos;i="4.80,496,1344236400"; d="scan'208";a="694852120" Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx2-out.netapp.com with ESMTP; 27 Sep 2012 08:11:02 -0700 Received: from vmwexceht01-prd.hq.netapp.com (vmwexceht01-prd.hq.netapp.com [10.106.76.239]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8RFB177025331; Thu, 27 Sep 2012 08:11:01 -0700 (PDT) Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.102]) by vmwexceht01-prd.hq.netapp.com ([10.106.76.239]) with mapi id 14.02.0309.002; Thu, 27 Sep 2012 08:11:01 -0700 From: "Knight, Frederick" To: "Black, David" , Julian Satran Thread-Topic: [storm] SPC-2 reserve/release text - version 6 Thread-Index: Ac2cNcys4MzRK8PXQu651fVKNuEIxQAXDfsAABnkfIAADjDW0A== Date: Thu, 27 Sep 2012 15:11:00 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> <4AE113CD-96D7-414D-8AAB-36ECF7FD89EA@satran.net> <8D3D17ACE214DC429325B2B98F3AE7120DE798F9@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE798F9@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.106.53.53] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release text - version 6 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Sep 2012 15:11:03 -0000 SSBhZnJhaWQgSSBoYXZlIHRvIGNvbnRlc3QgYSBsaXR0bGUgb24gdGhpcyAtIGlmIEkgdW5kZXJz dGFuZCB0aGlzIGRpc2N1c3Npb24gY29ycmVjdGx5Lg0KDQpJZiBhIFNDU0kgZGV2aWNlIGxvc2Vz IGFueSBzdGF0ZSBhbmQgZG9lcyBub3QgcmFpc2UgYSBVTklUIEFUVEVOVElPTiwgdGhlbiBpdCBp cyBCUk9LRU4uICBJdCBpcyBub3QgYSBtYXR0ZXIgb2YgYW55dGhpbmcgaW4gaVNDU0kgKDM3MjAp OyBJIHRoaW5rIFNBTSBhbmQgU1BDIGFyZSBwcmV0dHkgY2xlYXIgb24gdGhpcy4gIElmIHRoZXJl IGlzIG5vIFVOSVQgQVRURU5USU9OLCB0aGVuIHN0YXRlIFNIQUxMIGJlIHByZXNlcnZlZDsgb3Ro ZXJ3aXNlLCBhbGwgc29ydHMgb2Ygc3R1ZmYgd2lsbCBuZXZlciB3b3JrIChtb2RlIHBhZ2Ugc2V0 dGluZ3MsIFJFU0VSVkUgY29tbWFuZHMsIGFsbCBzb3J0cyBvZiBzdHVmZikuDQoNCklzIHNvbWVv bmUgcmVhbGx5IHN1Z2dlc3RpbmcgdGhhdCBzb21lIGRldmljZXMgd2lsbCBkcm9wIHN0YXRlIGFu ZCBOT1QgdGVsbCB0aGUgaG9zdCwgYW5kIHNvbWVob3cgdGhhdCBpdCBpcyBPSyB0byBkbyB0aGF0 Pw0KDQpUaGUgc3VnZ2VzdCByZXdvcmRpbmcgc2F5cyB0aGlzIGlzIHBvc3NpYmxlLiANCg0KCUZy ZWQNCg0KLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IHN0b3JtLWJvdW5jZXNAaWV0 Zi5vcmcgW21haWx0bzpzdG9ybS1ib3VuY2VzQGlldGYub3JnXSBPbiBCZWhhbGYgT2YgQmxhY2ss IERhdmlkDQpTZW50OiBUaHVyc2RheSwgU2VwdGVtYmVyIDI3LCAyMDEyIDEwOjQ3IEFNDQpUbzog SnVsaWFuIFNhdHJhbg0KQ2M6IHN0b3JtQGlldGYub3JnDQpTdWJqZWN0OiBSZTogW3N0b3JtXSBT UEMtMiByZXNlcnZlL3JlbGVhc2UgdGV4dCAtIHZlcnNpb24gNg0KDQpKdWxpYW4sDQoNCj4gVGhl IHRleHQgaW4gNC40LjMgc3RhdGVzIHRoZSBvYnZpb3VzICh0aGVyZSBpcyBubyBvdGhlciBzcGVj ZWQgbWV0aG9kIA0KPiBmb3IgY29udGludWF0aW9uIG9yIHJlaW5zdGF0ZW1lbnQpLiBNb3Jlb3Zl ciBpZiB0aGUgc2Vzc2lvbiBpcyANCj4gcmVpbnN0YXRlZCBhZnRlciB0aW1lMndhaXQgeW91IGNy ZWF0ZSBhIG5lZWQgZm9yIGEgVUEgKG9yIGF0IGxlYXN0IA0KPiBzdWdnZXN0IG9uZSkgYW5kIHRo YXQgd2FzIG5vdCB0aGVyZS4NCg0KVGhlIFVBcyBhcmUgY291cnRlc3kgb2YgU0FNLTMgKGFuZCBT QU0tNCkuICBUbyBlbmNvbXBhc3Mgc3lzdGVtcyB0aGF0IG1heSBub3QgZGVsaXZlciB0aGVzZSBV QXMgYW5kIHNpdHVhdGlvbnMgaW4gd2hpY2ggdGhleSBkb24ndCBvY2N1ciAoZS5nLiwgdGhlIHRh cmdldCBkb2Vzbid0IHJlY29nbml6ZSB3aGF0IGhhcHBlbmVkIGFzIG5leHVzIHJlZXN0YWJsaXNo bWVudCwgaW5kZXBlbmRlbnQgb2Ygd2hhdCB0aGUgaW5pdGlhdG9yIHRoaW5rcyBpdCdzIGRvaW5n KSBJIHRoaW5rIHRoZSB0ZXh0IHNob3VsZCBiZSByZXBocmFzZWQgYXM6DQoNCiAgICAgSWYgYSBV bml0IEF0dGVudGlvbiBjb25kaXRpb24gcmVzdWx0cyBmcm9tIHNlc3Npb24gcmVlc3RhYmxpc2ht ZW50LA0KICAgICB0aGUgc3BlY2lmaWMgVW5pdCBBdHRlbnRpb24gY29uZGl0aW9uIGluZGljYXRl cyB3aGV0aGVyIG5leHVzIHN0YXRlDQogICAgIHdhcyBwcmVzZXJ2ZWQsIHNlZSBTZWN0aW9uIDYu My40IG9mIFtTQU00XS4NCg0KPiBKdXN0IHN0YXRpbmcgdGhhdCBjcmVhdGlvbi9jb250aW51YXRp b24gb2YgYSBzZXNzaW9uIGRvbmUgYWNjb3JkaW5nIA0KPiA2LnggaW5jbHVkZXMgcHJlc2VydmF0 aW9uIG9mIHJlc2VydmUvcmVsZWFzZSBzdGF0ZSAtdGhhdCB3YXMgYWxyZWFkeSANCj4gdGhlcmUg aW1wbGllZCBieSBjb25mb3JtYW5jZSB0byBzcGMyIGFuZCB0aGUgcmVsYXRpb24gYmV0d2VlbiBp dCBuZXh1cyANCj4gYW5kIHNlc3Npb24uDQoNClVuZm9ydHVuYXRlbHksIFNQQy0yIGlzIHN1ZmZp Y2llbnRseSBpbXByZWNpc2Ugb24gdGhpcyBzdWJqZWN0IHRoYXQgc3VjaCBhbiBpbXBsaWNhdGlv biBpcyBhdCBiZXN0IGEgInByZWZlcnJlZCBpbXBsZW1lbnRhdGlvbiBhcHByb2FjaCIuICBJdCdz IG5vdCBhIHJlcXVpcmVtZW50IHRoYXQgY2FuIGJlIHNhZmVseSBzdGF0ZWQgaW4gYW55IHN0cm9u Z2VyIGZhc2hpb24gd2l0aG91dCB0aGUgcmlzayBvZiByZW5kZXJpbmcgZXhpc3RpbmcgaW1wbGVt ZW50YXRpb25zIG5vbi1jb21wbGlhbnQuDQoNCj4gVGhlIG5ldyB0ZXh0IHlvdSBzdWdnZXN0IHRv IDQuNCBpcyBtZXJlbHkgYSBiZXN0IHByYWN0aWNlcyANCj4gcmVjb21tZW5kYXRpb25zIGFuZCB3 aWxsIG5lZWQgYWRkaXRpb25zIG9uY2UgdGhlIG5ldyAibG9ja2luZyIgdGhyb3VnaCANCj4gY29t cGFyZS1hbmQtc3dhcCBnZXQgaW4gd2lkZXNwcmVhZCB1c2UuDQoNCkl0J3MgZGVmaW5pdGVseSBi ZXN0IHByYWN0aWNlcyByZWNvbW1lbmRhdGlvbnMgLSBubyBhcmd1bWVudCB0aGVyZS4NCg0KRm9y IGNvbnRleHQsIHdoYXQgSnVsaWFuIGlzIHJlZmVycmluZyB0byBpcyB1c2Ugb2YgdGhlIENPTVBB UkUgQU5EIFdSSVRFIGNvbW1hbmQgdGhhdCBwZXJmb3JtcyBhbiBhdG9taWMgb3BlcmF0aW9uIGF0 IHRoZSB0YXJnZXQgaW5zdGVhZCBvZiB1c2luZyBSRVNFUlZFIGFuZCBSRUxFQVNFIHRvIGNyZWF0 ZSBhIGNyaXRpY2FsIHNlY3Rpb24gYXQgdGhlIGluaXRpYXRvciBiYXNlZCBvbiByZWFkIGFuZCB3 cml0ZSBjb21tYW5kcy4NCg0KQ09NUEFSRSBBTkQgV1JJVEUgaXMgaW4gdXNlIC0gYSBzZW50ZW5j ZSBtZW50aW9uaW5nIHRoYXQgY29tbWFuZCBjb3VsZCBiZSBhZGRlZCwgYWx0aG91Z2ggdGhpcyBi ZXN0IHByYWN0aWNlcyB0ZXh0IGlzIGFpbWVkIHByaW1hcmlseSBhdCB1c2VzIG9mIHJlc2VydmUv cmVsZWFzZSBmb3IgbG9uZy10ZXJtIHJlc2VydmF0aW9ucy4gIEkgdGhpbmsgdGhlcmUncyBhbiBp bXBsaWNpdCAid2hlbiBhIHJlc2VydmF0aW9uIGlzIHRoZSBkZXNpcmVkIGJlaGF2aW9yIiBpbXBs aWNhdGlvbiB0byB0aGUgInBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIFNIT1VMRCBiZSB1c2VkIGlu c3RlYWQiIHRleHQgdGhhdCBJIGRvbid0IHRoaW5rIG5lZWRzIHRvIGJlIG1hZGUgZXhwbGljaXQu ICBPVE9ILCB0aGF0ICJTSE9VTEQiDQpjb3VsZCBiZSBjaGFuZ2VkIHRvIGEgInNob3VsZCIsIG9y IGl0IG1pZ2h0IGJlIGV2ZW4gYmV0dGVyIHRvIHJld3JpdGUgdGhhdCB0ZXh0DQphczogDQoNCglw ZXJzaXN0ZW50IHJlc2VydmF0aW9ucyBhcmUgc3VnZ2VzdGVkIGFzIGFuIGFsdGVybmF0aXZlLCBz ZWUgQW5uZXggQg0KCW9mIFtTUEM0XS4NCg0KSW4gY29udHJhc3QsICJTSE9VTEQgTk9UIGJlIHVz ZWQiIGlzIGNsZWFybHkganVzdGlmaWVkIGZvciByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25z IGJ5IFtTUEMzXS4NCg0KVGhhbmtzLA0KLS1EYXZpZA0KDQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNz YWdlLS0tLS0NCj4gRnJvbTogSnVsaWFuIFNhdHJhbiBbbWFpbHRvOmp1bGlhbkBzYXRyYW4ubmV0 XQ0KPiBTZW50OiBXZWRuZXNkYXksIFNlcHRlbWJlciAyNiwgMjAxMiAxMDoyNSBQTQ0KPiBUbzog QmxhY2ssIERhdmlkDQo+IENjOiBzdG9ybUBpZXRmLm9yZw0KPiBTdWJqZWN0OiBSZTogW3N0b3Jt XSBTUEMtMiByZXNlcnZlL3JlbGVhc2UgdGV4dCAtIHZlcnNpb24gNg0KPiANCj4gRGF2aWQsDQo+ IA0KPiBUaGUgdGV4dCBpbiA0LjQuMyBzdGF0ZXMgdGhlIG9idmlvdXMgKHRoZXJlIGlzIG5vIG90 aGVyIHNwZWNlZCBtZXRob2QgDQo+IGZvciBjb250aW51YXRpb24gb3IgcmVpbnN0YXRlbWVudCku IE1vcmVvdmVyIGlmIHRoZSB0aGUgc2Vzc2lvbiBpcyANCj4gcmVpbnN0YXRlZCBhZnRlciB0aW1l MndhaXQgeW91IGNyZWF0ZSBhIG5lZWQgZm9yIGEgVUEgKG9yIGF0IGxlYXN0IA0KPiBzdWdnZXN0 IG9uZSkgYW5kIHRoYXQgd2FzIG5vdCB0aGVyZS4gSnVzdCBzdGF0aW5nIHRoYXQgDQo+IGNyZWF0 aW9uL2NvbnRpbnVhdGlvbiBvZiBhIHNlc3Npb24gZG9uZSBhY2NvcmRpbmcgNi54IGluY2x1ZGVz IA0KPiBwcmVzZXJ2YXRpb24gb2YgcmVzZXJ2ZS9yZWxlYXNlIHN0YXRlIC10aGF0IHdhcyBhbHJl YWR5IHRoZXJlIGltcGxpZWQgDQo+IGJ5IGNvbmZvcm1hbmNlIHRvIHNwYzIgYW5kIHRoZSByZWxh dGlvbiBiZXR3ZWVuIGl0IG5leHVzIGFuZCBzZXNzaW9uLg0KPiANCj4gVGhlIG5ldyB0ZXh0IHlv dSBzdWdnZXN0IHRvIDQuNCBpcyBtZXJlbHkgYSBiZXN0IHByYWN0aWNlcyANCj4gcmVjb21lbmRh dGlvbnMgYW5kIHdpbGwgbmVlZCBhZGRpdGlvbnMgb25jZSB0aGUgbmV3ICJsb2NraW5nIiB0aHJv dWdoIA0KPiBjb21wYXJlLWFuZC1zd2FwIGdldCBpbiB3aWRlc3ByZWFkIHVzZS4NCj4gSWYgaW5z dGVhZCB3ZSByZWZyYWluIGZyb20gc3RhdGluZyBhbnl0aGluZyBzcGVjaWZpYyBhbmQgc3RhdGUg dGhhdCANCj4gdGFyZ2V0IHNob3VsZCBtYWludGFpbiB3aGF0ZXZlciBzdGF0ZSBpcyByZXF1aXJl ZCBieSBTUEMgd2hpbGUgYSANCj4gc2Vzc2lvbiBpcyBjb250aW51ZWQgYW5kIG1heSBkaXNjYXJk IHN0YXRlIHdoZW4gYSBzZXNzaW9uIGlzIA0KPiB0ZXJtaW5hdGVkIGFuZCBtYXkgaW5kaWNhdGUg dGhhdCBpdCBkaXNjYXJkZWQgc3RhdGUgdGhyb3VnaCBhIHVhIHlvdSANCj4gYXJlIG9uIHNhZmVy IGdyb3VuZCBhbmQgZG8gbm90IGNyZWF0ZSBuZXcgcmVxdWlyZW1lbnQuDQo+IA0KPiBTZW50IGZy b20gbXkgaVBhZA0KPiANCj4gT24gMjcg15HXodek15ggMjAxMiwgYXQgMDA6MjUsICJCbGFjaywg RGF2aWQiIDxkYXZpZC5ibGFja0BlbWMuY29tPiB3cm90ZToNCj4gDQo+ID4gSGVyZSdzIHdoZXJl IEkgdGhpbmsgd2UndmUgYXJyaXZlZCAoSSd2ZSBtYWRlIGEgZmV3IG1pbm9yIGVkaXRvcmlhbCAN Cj4gPiB0d2Vha3MgdG8gd2hhdCdzIGJlZW4gZGlzY3Vzc2VkKS4gIERlc3BpdGUgYWxsIHRoZSBk ZXRhaWxlZCANCj4gPiBkaXNjdXNzaW9uLCBJIHRoaW5rIHRoaXMgdmVyc2lvbiA2IGNvbWVzIGNs b3NlIHRvIEp1bGlhbidzIHN1Z2dlc3Rpb24gdGhhdDoNCj4gPg0KPiA+PiBpIHdvdWxkIG1lbnRp b24gb25seSB0aGF0IHRoZSBzdGF0ZSB0aGF0IG11c3QgYmUgbWFpbnRhaW5lZCB3aGlsZSANCj4g Pj4gdGhlIHNlc3Npb24gaXMgbWFpbnRhaW5lZCBpbmNsdWRlcyB0aGUgcmVzZXJ2ZS9yZWxlYXNl IGFuZCBsZWF2ZSBpdCANCj4gPj4gdG8NCj4gdGhhdC4NCj4gPg0KPiA+IEFsdGhvdWdoIHdlIGNh bid0IGFjdHVhbGx5IHNheSAibXVzdCIgYXMgdGhhdCB3b3VsZCBiZSBhIG5ldyANCj4gPiByZXF1 aXJlbWVudCwgYW5kIGltcGxlbWVudGF0aW9ucyBkaWZmZXIgaW4gd2hhdCB0aGV5IGRvIGFib3V0 IA0KPiA+IHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbiBzdGF0ZS4gIFRoZSBjdXJyZW50IGxh bmd1YWdlIA0KPiA+IGNoYXJhY3Rlcml6ZXMgcmV0ZW50aW9uIG9mIHRoYXQgc3RhdGUgYXMgdGhl ICJwcmVmZXJyZWQgaW1wbGVtZW50YXRpb24gYXBwcm9hY2giLg0KPiA+DQo+ID4gQmV5b25kIHRo YXQsIEkgdGhpbmsgdGhlIHR3byBhZGRpdGlvbmFsIGNhdmVhdHMgYWJvdXQgDQo+ID4gcmVzZXJ2 ZS9yZWxlYXNlIHJlc2VydmF0aW9ucyAocG9zc2libGUgbmVlZCBmb3IgYSByZXNldCwgcG90ZW50 aWFsbHkgDQo+ID4gdW5leHBlY3RlZCBpbnRlcmFjdGlvbiB3aXRoIHBlcnNpc3RlbnQgcmVzZXJ2 YXRpb25zLCBzZWUgYmVsb3cpIGFyZSB1c2VmdWwgdG8gaW5jbHVkZS4NCj4gPg0KPiA+IC0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ID4NCj4gPiBbMV0gRm9yIGNs YXJpdHksIHRoZSBmb2xsb3dpbmcgY2hhbmdlIHNob3VsZCBiZSBtYWRlIGluDQo+ID4gNC40LjMu MToNCj4gPg0KPiA+IE9MRA0KPiA+ICBUaGF0IGlzLCBpdCBzaG91bGQgcGVyZm9ybSBzZXNzaW9u IHJlY292ZXJ5IGFzICBkZXNjcmliZWQgaW4gDQo+ID4gQ2hhcHRlciA2Lg0KPiA+IE5FVw0KPiA+ ICBUaGF0IGlzLCBpdCBzaG91bGQgcmVpbnN0YXRlIHRoZSBzZXNzaW9uIHZpYSBpU0NTSSBzZXNz aW9uICANCj4gPiByZWluc3RhdGVtZW50IChTZWN0aW9uIDYuMy41KSBvciBjb250aW51ZSB2aWEg c2Vzc2lvbiAgY29udGludWF0aW9uIA0KPiA+IChTZWN0aW9uIDYuMy42KS4NCj4gPiBFTkQNCj4g Pg0KPiA+IFsyXSBBZGQgdGhlIGZvbGxvd2luZyBzZW50ZW5jZSB0byB0aGUgZW5kIG9mIDQuNC4z LjE6DQo+ID4NCj4gPiAgICAgVGhlIHNwZWNpZmljIFVuaXQgQXR0ZW50aW9uIGNvbmRpdGlvbiB0 aGF0IHJlc3VsdHMgZnJvbSBzZXNzaW9uDQo+ID4gICAgIHJlZXN0YWJsaXNobWVudCBpbmRpY2F0 ZXMgd2hldGhlciBuZXh1cyBzdGF0ZSB3YXMgcHJlc2VydmVkLA0KPiA+ICAgICBzZWUgU2VjdGlv biA2LjMuNCBvZiBbU0FNNF0uDQo+ID4NCj4gPiBbM10gTkVXIFRFWFQ6DQo+ID4NCj4gPiA0LjQu My4yLiBSZXNlcnZhdGlvbnMNCj4gPg0KPiA+ICBUaGVyZSBhcmUgdHdvIHJlc2VydmF0aW9uIG1h bmFnZW1lbnQgbWV0aG9kcyBkZWZpbmVkIGluIHRoZSBTQ1NJICANCj4gPiBzdGFuZGFyZHMsIHJl c2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbnMsIGJhc2VkIG9uIHRoZSBSRVNFUlZFIGFuZCAgDQo+ ID4gUkVMRUFTRSBjb21tYW5kcyBbU1BDMl0sIGFuZCBwZXJzaXN0ZW50IHJlc2VydmF0aW9ucywg YmFzZWQgb24gdGhlICANCj4gPiBQRVJTSVNURU5UIFJFU0VSVkUgSU4gYW5kIFBFUlNJU1RFTlQg UkVTRVJWRSBPVVQgY29tbWFuZHMgW1NQQzNdLg0KPiA+ICBSZXNlcnZlL3JlbGVhc2UgcmVzZXJ2 YXRpb25zIGFyZSBvYnNvbGV0ZSBbU1BDM10gYW5kIFNIT1VMRCBOT1QgYmUgIA0KPiA+IHVzZWQ7 IHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIFNIT1VMRCBiZSB1c2VkIGluc3RlYWQuDQo+ID4NCj4g PiAgU3RhdGUgZm9yIHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIGlzIHJlcXVpcmVkIHRvIHBlcnNp c3QgIHRocm91Z2ggDQo+ID4gY2hhbmdlcyBhbmQgZmFpbHVyZXMgYXQgdGhlIGlTQ1NJIGxheWVy IHRoYXQgcmVzdWx0IGluICBJX1QgTmV4dXMgDQo+ID4gZmFpbHVyZXMsIHNlZSBbU1BDM10gZm9y IGRldGFpbHMgYW5kIHNwZWNpZmljIHJlcXVpcmVtZW50cy4NCj4gPg0KPiA+ICBJbiBjb250cmFz dCwgW1NQQzJdIGRvZXMgbm90IHNwZWNpZnkgZGV0YWlsZWQgcGVyc2lzdGVuY2UgIA0KPiA+IHJl cXVpcmVtZW50cyBmb3IgcmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9uIHN0YXRlIGFmdGVyIGFu IElfVCAgDQo+ID4gTmV4dXMgZmFpbHVyZS4gIE5vbmV0aGVsZXNzLCB3aGVuIHJlc2VydmUvcmVs ZWFzZSByZXNlcnZhdGlvbnMgYXJlICANCj4gPiBzdXBwb3J0ZWQgYnkgYW4gaVNDU0kgdGFyZ2V0 LCB0aGUgcHJlZmVycmVkIGltcGxlbWVudGF0aW9uIGFwcHJvYWNoICANCj4gPiBpcyB0byBwcmVz ZXJ2ZSByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb24gc3RhdGUgZm9yIGlTQ1NJIHNlc3Npb24g IA0KPiA+IHJlaW5zdGF0ZW1lbnQgKHNlZSBTZWN0aW9uIDYuMy41KSBvciBzZXNzaW9uIGNvbnRp bnVhdGlvbiAgKHNlZSANCj4gPiBTZWN0aW9uIDYuMy42KS4NCj4gPg0KPiA+ICBUd28gYWRkaXRp b25hbCBjYXZlYXRzIGFwcGx5IHRvIHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbnM6DQo+ID4N Cj4gPiAgLSBXaGVuIGNvbm5lY3Rpb24gZmFpbHVyZSBjYXVzZXMgdGhlIGlTQ1NJIHNlc3Npb24g dG8gZmFpbCwgYW5kDQo+ID4gICAgICB0aGUgc2Vzc2lvbiBpcyBub3QgcmVpbnN0YXRlZCBvciBj b250aW51ZWQsIHRhcmdldCByZXRlbnRpb24NCj4gPiAgICAgIG9mIHRoYXQgc2Vzc2lvbidzIHJl c2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbiBzdGF0ZSBtYXkNCj4gPiAgICAgIHJlcXVpcmUgYW4g aW5pdGlhdG9yIHRvIGlzc3VlIGEgcmVzZXQgKGUuZy4sIExPR0lDQUwgVU5JVCBSRVNFVCwNCj4g PiAgICAgIHNlZSBzZWN0aW9uIDExLjUpIGluIG9yZGVyIHRvIHJlbW92ZSB0aGF0IHJlc2VydmF0 aW9uIHN0YXRlLg0KPiA+DQo+ID4gIC0gUmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9ucyBtYXkg bm90IGJlaGF2ZSBhcyBleHBlY3RlZCB3aGVuDQo+ID4gICAgICBwZXJzaXN0ZW50IHJlc2VydmF0 aW9ucyBhcmUgYWxzbyB1c2VkIG9uIHRoZSBzYW1lIGxvZ2ljYWwgdW5pdDsNCj4gPiAgICAgIHNl ZSB0aGUgZGlzY3Vzc2lvbiBvZiAiRXhjZXB0aW9ucyB0byBTUEMtMiBSRVNFUlZFIGFuZCBSRUxF QVNFDQo+ID4gICAgICBiZWhhdmlvciIgaW4gW1NQQzRdLg0KPiA+DQo+ID4gVGhhbmtzLA0KPiA+ IC0tRGF2aWQNCj4gPiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tDQo+ID4gRGF2aWQgTC4gQmxhY2ssIERpc3Rpbmd1aXNoZWQgRW5naW5lZXIgRU1D IENvcnBvcmF0aW9uLCAxNzYgU291dGggDQo+ID4gU3QuLCBIb3BraW50b24sIE1BICAwMTc0OA0K PiA+ICsxICg1MDgpIDI5My03OTUzICAgICAgICAgICAgIEZBWDogKzEgKDUwOCkgMjkzLTc3ODYN Cj4gPiBkYXZpZC5ibGFja0BlbWMuY29tICAgICAgICBNb2JpbGU6ICsxICg5NzgpIDM5NC03NzU0 DQo+ID4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LQ0KPiA+DQo+ID4NCj4gPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fXw0KPiA+IHN0b3JtIG1haWxpbmcgbGlzdA0KPiA+IHN0b3JtQGlldGYub3JnDQo+ID4g aHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9zdG9ybQ0KDQpfX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0Kc3Rvcm0gbWFpbGluZyBsaXN0 DQpzdG9ybUBpZXRmLm9yZw0KaHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9z dG9ybQ0K From david.black@emc.com Thu Sep 27 08:55:55 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 15A6621F84C4 for ; Thu, 27 Sep 2012 08:55:55 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AzK9QtQXH2j2 for ; Thu, 27 Sep 2012 08:55:54 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id D50CB21F847A for ; Thu, 27 Sep 2012 08:55:53 -0700 (PDT) Received: from hop04-l1d11-si02.isus.emc.com (HOP04-L1D11-SI02.isus.emc.com [10.254.111.55]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8RFtqNK011194 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 27 Sep 2012 11:55:53 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.221.253]) by hop04-l1d11-si02.isus.emc.com (RSA Interceptor); Thu, 27 Sep 2012 11:55:43 -0400 Received: from mxhub20.corp.emc.com (mxhub20.corp.emc.com [10.254.93.49]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8RFthh0021503; Thu, 27 Sep 2012 11:55:43 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub20.corp.emc.com ([10.254.93.49]) with mapi; Thu, 27 Sep 2012 11:55:42 -0400 From: "Black, David" To: "Knight, Frederick" Date: Thu, 27 Sep 2012 11:55:40 -0400 Thread-Topic: [storm] SPC-2 reserve/release text - version 6 Thread-Index: Ac2cNcys4MzRK8PXQu651fVKNuEIxQAXDfsAABnkfIAADjDW0AAa4S6g Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE7993C@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> <4AE113CD-96D7-414D-8AAB-36ECF7FD89EA@satran.net> <8D3D17ACE214DC429325B2B98F3AE7120DE798F9@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-EMM-MHVC: 1 Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release text - version 6 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Sep 2012 15:55:55 -0000 RnJlZCwNCg0KVGhlIHByb2JsZW0gaXMgbm90IHRoYXQgdGhlIGRldmljZSBkb2VzIG5vdCByYWlz ZSBhIFVBLCBpdCdzIHRoYXQgdGhlIFVBDQptYXkgbmV2ZXIgYmUgZGVsaXZlcmVkLg0KDQpBIHN1 bW1hcnkgb2YgaG93IHRoaXMgY2FuIGhhcHBlbiBpcyB0aGF0IHRoZSBuZXcgY29ubmVjdGlvbiB3 aXRoIHRoZSBzYW1lDQpJU0lEIG1heSBub3QgYmUgcmVjb2duaXplZCBhcyByZWluc3RhdGluZyBv ciBjb250aW51aW5nIHRoZSBzZXNzaW9uIGF0IHRoZQ0KdGFyZ2V0IGlmIHRoZSB0aW1lb3V0cyBo YXZlIGV4cGlyZWQgLSB0aGUgVUEgZ2V0cyBlc3RhYmxpc2hlZCwgYnV0IG5vYm9keQ0Kc2hvd3Mg dXAgdG8gcGljayBpdCB1cCBpbiB0aW1lLCBzbyB0aGUgVUEgaXMgZGlzY2FyZGVkIGFsb25nIHdp dGggdGhlIHJlc3QNCm9mIHRoZSBzZXNzaW9uIHN0YXRlIGFuZCB0aGUgdGFyZ2V0IGZhaWxzIHRv IHJlY29nbml6ZSB0aGUgaW5pdGlhdG9yJ3MNCnN1YnNlcXVlbnQgSVNJRCByZXVzZSBldmVuIHRo b3VnaCB0aGUgaW5pdGlhdG9yIGJlbGlldmVzIHRoYXQgaXQncyByZS0NCmVzdGFibGlzaGluZyB0 aGUgc2Vzc2lvbi4NCg0KVGhpcyBpcyBmdXJ0aGVyIGNvbXBsaWNhdGVkIGJ5IHRoZSBVQSB0aGF0 IGluZGljYXRlcyBuZXh1cyBzdGF0ZSBwcmVzZXJ2YXRpb24NCm5vdCBiZWluZyBzcGVjaWZpZWQg aW4gU0FNLTIuDQoNClRoYW5rcywNCi0tRGF2aWQNCg0KDQo+IC0tLS0tT3JpZ2luYWwgTWVzc2Fn ZS0tLS0tDQo+IEZyb206IEtuaWdodCwgRnJlZGVyaWNrIFttYWlsdG86RnJlZGVyaWNrLktuaWdo dEBuZXRhcHAuY29tXQ0KPiBTZW50OiBUaHVyc2RheSwgU2VwdGVtYmVyIDI3LCAyMDEyIDExOjEx IEFNDQo+IFRvOiBCbGFjaywgRGF2aWQ7IEp1bGlhbiBTYXRyYW4NCj4gQ2M6IHN0b3JtQGlldGYu b3JnDQo+IFN1YmplY3Q6IFJFOiBbc3Rvcm1dIFNQQy0yIHJlc2VydmUvcmVsZWFzZSB0ZXh0IC0g dmVyc2lvbiA2DQo+IA0KPiBJIGFmcmFpZCBJIGhhdmUgdG8gY29udGVzdCBhIGxpdHRsZSBvbiB0 aGlzIC0gaWYgSSB1bmRlcnN0YW5kIHRoaXMgZGlzY3Vzc2lvbg0KPiBjb3JyZWN0bHkuDQo+IA0K PiBJZiBhIFNDU0kgZGV2aWNlIGxvc2VzIGFueSBzdGF0ZSBhbmQgZG9lcyBub3QgcmFpc2UgYSBV TklUIEFUVEVOVElPTiwgdGhlbiBpdA0KPiBpcyBCUk9LRU4uICBJdCBpcyBub3QgYSBtYXR0ZXIg b2YgYW55dGhpbmcgaW4gaVNDU0kgKDM3MjApOyBJIHRoaW5rIFNBTSBhbmQNCj4gU1BDIGFyZSBw cmV0dHkgY2xlYXIgb24gdGhpcy4gIElmIHRoZXJlIGlzIG5vIFVOSVQgQVRURU5USU9OLCB0aGVu IHN0YXRlIFNIQUxMDQo+IGJlIHByZXNlcnZlZDsgb3RoZXJ3aXNlLCBhbGwgc29ydHMgb2Ygc3R1 ZmYgd2lsbCBuZXZlciB3b3JrIChtb2RlIHBhZ2UNCj4gc2V0dGluZ3MsIFJFU0VSVkUgY29tbWFu ZHMsIGFsbCBzb3J0cyBvZiBzdHVmZikuDQo+IA0KPiBJcyBzb21lb25lIHJlYWxseSBzdWdnZXN0 aW5nIHRoYXQgc29tZSBkZXZpY2VzIHdpbGwgZHJvcCBzdGF0ZSBhbmQgTk9UIHRlbGwNCj4gdGhl IGhvc3QsIGFuZCBzb21laG93IHRoYXQgaXQgaXMgT0sgdG8gZG8gdGhhdD8NCj4gDQo+IFRoZSBz dWdnZXN0IHJld29yZGluZyBzYXlzIHRoaXMgaXMgcG9zc2libGUuDQo+IA0KPiAJRnJlZA0KPiAN Cj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogc3Rvcm0tYm91bmNlc0BpZXRm Lm9yZyBbbWFpbHRvOnN0b3JtLWJvdW5jZXNAaWV0Zi5vcmddIE9uIEJlaGFsZiBPZg0KPiBCbGFj aywgRGF2aWQNCj4gU2VudDogVGh1cnNkYXksIFNlcHRlbWJlciAyNywgMjAxMiAxMDo0NyBBTQ0K PiBUbzogSnVsaWFuIFNhdHJhbg0KPiBDYzogc3Rvcm1AaWV0Zi5vcmcNCj4gU3ViamVjdDogUmU6 IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYNCj4gDQo+IEp1 bGlhbiwNCj4gDQo+ID4gVGhlIHRleHQgaW4gNC40LjMgc3RhdGVzIHRoZSBvYnZpb3VzICh0aGVy ZSBpcyBubyBvdGhlciBzcGVjZWQgbWV0aG9kDQo+ID4gZm9yIGNvbnRpbnVhdGlvbiBvciByZWlu c3RhdGVtZW50KS4gTW9yZW92ZXIgaWYgdGhlIHNlc3Npb24gaXMNCj4gPiByZWluc3RhdGVkIGFm dGVyIHRpbWUyd2FpdCB5b3UgY3JlYXRlIGEgbmVlZCBmb3IgYSBVQSAob3IgYXQgbGVhc3QNCj4g PiBzdWdnZXN0IG9uZSkgYW5kIHRoYXQgd2FzIG5vdCB0aGVyZS4NCj4gDQo+IFRoZSBVQXMgYXJl IGNvdXJ0ZXN5IG9mIFNBTS0zIChhbmQgU0FNLTQpLiAgVG8gZW5jb21wYXNzIHN5c3RlbXMgdGhh dCBtYXkgbm90DQo+IGRlbGl2ZXIgdGhlc2UgVUFzIGFuZCBzaXR1YXRpb25zIGluIHdoaWNoIHRo ZXkgZG9uJ3Qgb2NjdXIgKGUuZy4sIHRoZSB0YXJnZXQNCj4gZG9lc24ndCByZWNvZ25pemUgd2hh dCBoYXBwZW5lZCBhcyBuZXh1cyByZWVzdGFibGlzaG1lbnQsIGluZGVwZW5kZW50IG9mIHdoYXQN Cj4gdGhlIGluaXRpYXRvciB0aGlua3MgaXQncyBkb2luZykgSSB0aGluayB0aGUgdGV4dCBzaG91 bGQgYmUgcmVwaHJhc2VkIGFzOg0KPiANCj4gICAgICBJZiBhIFVuaXQgQXR0ZW50aW9uIGNvbmRp dGlvbiByZXN1bHRzIGZyb20gc2Vzc2lvbiByZWVzdGFibGlzaG1lbnQsDQo+ICAgICAgdGhlIHNw ZWNpZmljIFVuaXQgQXR0ZW50aW9uIGNvbmRpdGlvbiBpbmRpY2F0ZXMgd2hldGhlciBuZXh1cyBz dGF0ZQ0KPiAgICAgIHdhcyBwcmVzZXJ2ZWQsIHNlZSBTZWN0aW9uIDYuMy40IG9mIFtTQU00XS4N Cj4gDQo+ID4gSnVzdCBzdGF0aW5nIHRoYXQgY3JlYXRpb24vY29udGludWF0aW9uIG9mIGEgc2Vz c2lvbiBkb25lIGFjY29yZGluZw0KPiA+IDYueCBpbmNsdWRlcyBwcmVzZXJ2YXRpb24gb2YgcmVz ZXJ2ZS9yZWxlYXNlIHN0YXRlIC10aGF0IHdhcyBhbHJlYWR5DQo+ID4gdGhlcmUgaW1wbGllZCBi eSBjb25mb3JtYW5jZSB0byBzcGMyIGFuZCB0aGUgcmVsYXRpb24gYmV0d2VlbiBpdCBuZXh1cw0K PiA+IGFuZCBzZXNzaW9uLg0KPiANCj4gVW5mb3J0dW5hdGVseSwgU1BDLTIgaXMgc3VmZmljaWVu dGx5IGltcHJlY2lzZSBvbiB0aGlzIHN1YmplY3QgdGhhdCBzdWNoIGFuDQo+IGltcGxpY2F0aW9u IGlzIGF0IGJlc3QgYSAicHJlZmVycmVkIGltcGxlbWVudGF0aW9uIGFwcHJvYWNoIi4gIEl0J3Mg bm90IGENCj4gcmVxdWlyZW1lbnQgdGhhdCBjYW4gYmUgc2FmZWx5IHN0YXRlZCBpbiBhbnkgc3Ry b25nZXIgZmFzaGlvbiB3aXRob3V0IHRoZSByaXNrDQo+IG9mIHJlbmRlcmluZyBleGlzdGluZyBp bXBsZW1lbnRhdGlvbnMgbm9uLWNvbXBsaWFudC4NCj4gDQo+ID4gVGhlIG5ldyB0ZXh0IHlvdSBz dWdnZXN0IHRvIDQuNCBpcyBtZXJlbHkgYSBiZXN0IHByYWN0aWNlcw0KPiA+IHJlY29tbWVuZGF0 aW9ucyBhbmQgd2lsbCBuZWVkIGFkZGl0aW9ucyBvbmNlIHRoZSBuZXcgImxvY2tpbmciIHRocm91 Z2gNCj4gPiBjb21wYXJlLWFuZC1zd2FwIGdldCBpbiB3aWRlc3ByZWFkIHVzZS4NCj4gDQo+IEl0 J3MgZGVmaW5pdGVseSBiZXN0IHByYWN0aWNlcyByZWNvbW1lbmRhdGlvbnMgLSBubyBhcmd1bWVu dCB0aGVyZS4NCj4gDQo+IEZvciBjb250ZXh0LCB3aGF0IEp1bGlhbiBpcyByZWZlcnJpbmcgdG8g aXMgdXNlIG9mIHRoZSBDT01QQVJFIEFORCBXUklURQ0KPiBjb21tYW5kIHRoYXQgcGVyZm9ybXMg YW4gYXRvbWljIG9wZXJhdGlvbiBhdCB0aGUgdGFyZ2V0IGluc3RlYWQgb2YgdXNpbmcNCj4gUkVT RVJWRSBhbmQgUkVMRUFTRSB0byBjcmVhdGUgYSBjcml0aWNhbCBzZWN0aW9uIGF0IHRoZSBpbml0 aWF0b3IgYmFzZWQgb24NCj4gcmVhZCBhbmQgd3JpdGUgY29tbWFuZHMuDQo+IA0KPiBDT01QQVJF IEFORCBXUklURSBpcyBpbiB1c2UgLSBhIHNlbnRlbmNlIG1lbnRpb25pbmcgdGhhdCBjb21tYW5k IGNvdWxkIGJlDQo+IGFkZGVkLCBhbHRob3VnaCB0aGlzIGJlc3QgcHJhY3RpY2VzIHRleHQgaXMg YWltZWQgcHJpbWFyaWx5IGF0IHVzZXMgb2YNCj4gcmVzZXJ2ZS9yZWxlYXNlIGZvciBsb25nLXRl cm0gcmVzZXJ2YXRpb25zLiAgSSB0aGluayB0aGVyZSdzIGFuIGltcGxpY2l0ICJ3aGVuDQo+IGEg cmVzZXJ2YXRpb24gaXMgdGhlIGRlc2lyZWQgYmVoYXZpb3IiIGltcGxpY2F0aW9uIHRvIHRoZSAi cGVyc2lzdGVudA0KPiByZXNlcnZhdGlvbnMgU0hPVUxEIGJlIHVzZWQgaW5zdGVhZCIgdGV4dCB0 aGF0IEkgZG9uJ3QgdGhpbmsgbmVlZHMgdG8gYmUgbWFkZQ0KPiBleHBsaWNpdC4gIE9UT0gsIHRo YXQgIlNIT1VMRCINCj4gY291bGQgYmUgY2hhbmdlZCB0byBhICJzaG91bGQiLCBvciBpdCBtaWdo dCBiZSBldmVuIGJldHRlciB0byByZXdyaXRlIHRoYXQNCj4gdGV4dA0KPiBhczoNCj4gDQo+IAlw ZXJzaXN0ZW50IHJlc2VydmF0aW9ucyBhcmUgc3VnZ2VzdGVkIGFzIGFuIGFsdGVybmF0aXZlLCBz ZWUgQW5uZXggQg0KPiAJb2YgW1NQQzRdLg0KPiANCj4gSW4gY29udHJhc3QsICJTSE9VTEQgTk9U IGJlIHVzZWQiIGlzIGNsZWFybHkganVzdGlmaWVkIGZvciByZXNlcnZlL3JlbGVhc2UNCj4gcmVz ZXJ2YXRpb25zIGJ5IFtTUEMzXS4NCj4gDQo+IFRoYW5rcywNCj4gLS1EYXZpZA0KPiANCj4gDQo+ ID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPiBGcm9tOiBKdWxpYW4gU2F0cmFuIFtt YWlsdG86anVsaWFuQHNhdHJhbi5uZXRdDQo+ID4gU2VudDogV2VkbmVzZGF5LCBTZXB0ZW1iZXIg MjYsIDIwMTIgMTA6MjUgUE0NCj4gPiBUbzogQmxhY2ssIERhdmlkDQo+ID4gQ2M6IHN0b3JtQGll dGYub3JnDQo+ID4gU3ViamVjdDogUmU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRl eHQgLSB2ZXJzaW9uIDYNCj4gPg0KPiA+IERhdmlkLA0KPiA+DQo+ID4gVGhlIHRleHQgaW4gNC40 LjMgc3RhdGVzIHRoZSBvYnZpb3VzICh0aGVyZSBpcyBubyBvdGhlciBzcGVjZWQgbWV0aG9kDQo+ ID4gZm9yIGNvbnRpbnVhdGlvbiBvciByZWluc3RhdGVtZW50KS4gTW9yZW92ZXIgaWYgdGhlIHRo ZSBzZXNzaW9uIGlzDQo+ID4gcmVpbnN0YXRlZCBhZnRlciB0aW1lMndhaXQgeW91IGNyZWF0ZSBh IG5lZWQgZm9yIGEgVUEgKG9yIGF0IGxlYXN0DQo+ID4gc3VnZ2VzdCBvbmUpIGFuZCB0aGF0IHdh cyBub3QgdGhlcmUuIEp1c3Qgc3RhdGluZyB0aGF0DQo+ID4gY3JlYXRpb24vY29udGludWF0aW9u IG9mIGEgc2Vzc2lvbiBkb25lIGFjY29yZGluZyA2LnggaW5jbHVkZXMNCj4gPiBwcmVzZXJ2YXRp b24gb2YgcmVzZXJ2ZS9yZWxlYXNlIHN0YXRlIC10aGF0IHdhcyBhbHJlYWR5IHRoZXJlIGltcGxp ZWQNCj4gPiBieSBjb25mb3JtYW5jZSB0byBzcGMyIGFuZCB0aGUgcmVsYXRpb24gYmV0d2VlbiBp dCBuZXh1cyBhbmQgc2Vzc2lvbi4NCj4gPg0KPiA+IFRoZSBuZXcgdGV4dCB5b3Ugc3VnZ2VzdCB0 byA0LjQgaXMgbWVyZWx5IGEgYmVzdCBwcmFjdGljZXMNCj4gPiByZWNvbWVuZGF0aW9ucyBhbmQg d2lsbCBuZWVkIGFkZGl0aW9ucyBvbmNlIHRoZSBuZXcgImxvY2tpbmciIHRocm91Z2gNCj4gPiBj b21wYXJlLWFuZC1zd2FwIGdldCBpbiB3aWRlc3ByZWFkIHVzZS4NCj4gPiBJZiBpbnN0ZWFkIHdl IHJlZnJhaW4gZnJvbSBzdGF0aW5nIGFueXRoaW5nIHNwZWNpZmljIGFuZCBzdGF0ZSB0aGF0DQo+ ID4gdGFyZ2V0IHNob3VsZCBtYWludGFpbiB3aGF0ZXZlciBzdGF0ZSBpcyByZXF1aXJlZCBieSBT UEMgd2hpbGUgYQ0KPiA+IHNlc3Npb24gaXMgY29udGludWVkIGFuZCBtYXkgZGlzY2FyZCBzdGF0 ZSB3aGVuIGEgc2Vzc2lvbiBpcw0KPiA+IHRlcm1pbmF0ZWQgYW5kIG1heSBpbmRpY2F0ZSB0aGF0 IGl0IGRpc2NhcmRlZCBzdGF0ZSB0aHJvdWdoIGEgdWEgeW91DQo+ID4gYXJlIG9uIHNhZmVyIGdy b3VuZCBhbmQgZG8gbm90IGNyZWF0ZSBuZXcgcmVxdWlyZW1lbnQuDQo+ID4NCj4gPiBTZW50IGZy b20gbXkgaVBhZA0KPiA+DQo+ID4gT24gMjcg15HXodek15ggMjAxMiwgYXQgMDA6MjUsICJCbGFj aywgRGF2aWQiIDxkYXZpZC5ibGFja0BlbWMuY29tPiB3cm90ZToNCj4gPg0KPiA+ID4gSGVyZSdz IHdoZXJlIEkgdGhpbmsgd2UndmUgYXJyaXZlZCAoSSd2ZSBtYWRlIGEgZmV3IG1pbm9yIGVkaXRv cmlhbA0KPiA+ID4gdHdlYWtzIHRvIHdoYXQncyBiZWVuIGRpc2N1c3NlZCkuICBEZXNwaXRlIGFs bCB0aGUgZGV0YWlsZWQNCj4gPiA+IGRpc2N1c3Npb24sIEkgdGhpbmsgdGhpcyB2ZXJzaW9uIDYg Y29tZXMgY2xvc2UgdG8gSnVsaWFuJ3Mgc3VnZ2VzdGlvbg0KPiB0aGF0Og0KPiA+ID4NCj4gPiA+ PiBpIHdvdWxkIG1lbnRpb24gb25seSB0aGF0IHRoZSBzdGF0ZSB0aGF0IG11c3QgYmUgbWFpbnRh aW5lZCB3aGlsZQ0KPiA+ID4+IHRoZSBzZXNzaW9uIGlzIG1haW50YWluZWQgaW5jbHVkZXMgdGhl IHJlc2VydmUvcmVsZWFzZSBhbmQgbGVhdmUgaXQNCj4gPiA+PiB0bw0KPiA+IHRoYXQuDQo+ID4g Pg0KPiA+ID4gQWx0aG91Z2ggd2UgY2FuJ3QgYWN0dWFsbHkgc2F5ICJtdXN0IiBhcyB0aGF0IHdv dWxkIGJlIGEgbmV3DQo+ID4gPiByZXF1aXJlbWVudCwgYW5kIGltcGxlbWVudGF0aW9ucyBkaWZm ZXIgaW4gd2hhdCB0aGV5IGRvIGFib3V0DQo+ID4gPiByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRp b24gc3RhdGUuICBUaGUgY3VycmVudCBsYW5ndWFnZQ0KPiA+ID4gY2hhcmFjdGVyaXplcyByZXRl bnRpb24gb2YgdGhhdCBzdGF0ZSBhcyB0aGUgInByZWZlcnJlZCBpbXBsZW1lbnRhdGlvbg0KPiBh cHByb2FjaCIuDQo+ID4gPg0KPiA+ID4gQmV5b25kIHRoYXQsIEkgdGhpbmsgdGhlIHR3byBhZGRp dGlvbmFsIGNhdmVhdHMgYWJvdXQNCj4gPiA+IHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbnMg KHBvc3NpYmxlIG5lZWQgZm9yIGEgcmVzZXQsIHBvdGVudGlhbGx5DQo+ID4gPiB1bmV4cGVjdGVk IGludGVyYWN0aW9uIHdpdGggcGVyc2lzdGVudCByZXNlcnZhdGlvbnMsIHNlZSBiZWxvdykgYXJl IHVzZWZ1bA0KPiB0byBpbmNsdWRlLg0KPiA+ID4NCj4gPiA+IC0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ID4gPg0KPiA+ID4gWzFdIEZvciBjbGFyaXR5LCB0aGUg Zm9sbG93aW5nIGNoYW5nZSBzaG91bGQgYmUgbWFkZSBpbg0KPiA+ID4gNC40LjMuMToNCj4gPiA+ DQo+ID4gPiBPTEQNCj4gPiA+ICBUaGF0IGlzLCBpdCBzaG91bGQgcGVyZm9ybSBzZXNzaW9uIHJl Y292ZXJ5IGFzICBkZXNjcmliZWQgaW4NCj4gPiA+IENoYXB0ZXIgNi4NCj4gPiA+IE5FVw0KPiA+ ID4gIFRoYXQgaXMsIGl0IHNob3VsZCByZWluc3RhdGUgdGhlIHNlc3Npb24gdmlhIGlTQ1NJIHNl c3Npb24NCj4gPiA+IHJlaW5zdGF0ZW1lbnQgKFNlY3Rpb24gNi4zLjUpIG9yIGNvbnRpbnVlIHZp YSBzZXNzaW9uICBjb250aW51YXRpb24NCj4gPiA+IChTZWN0aW9uIDYuMy42KS4NCj4gPiA+IEVO RA0KPiA+ID4NCj4gPiA+IFsyXSBBZGQgdGhlIGZvbGxvd2luZyBzZW50ZW5jZSB0byB0aGUgZW5k IG9mIDQuNC4zLjE6DQo+ID4gPg0KPiA+ID4gICAgIFRoZSBzcGVjaWZpYyBVbml0IEF0dGVudGlv biBjb25kaXRpb24gdGhhdCByZXN1bHRzIGZyb20gc2Vzc2lvbg0KPiA+ID4gICAgIHJlZXN0YWJs aXNobWVudCBpbmRpY2F0ZXMgd2hldGhlciBuZXh1cyBzdGF0ZSB3YXMgcHJlc2VydmVkLA0KPiA+ ID4gICAgIHNlZSBTZWN0aW9uIDYuMy40IG9mIFtTQU00XS4NCj4gPiA+DQo+ID4gPiBbM10gTkVX IFRFWFQ6DQo+ID4gPg0KPiA+ID4gNC40LjMuMi4gUmVzZXJ2YXRpb25zDQo+ID4gPg0KPiA+ID4g IFRoZXJlIGFyZSB0d28gcmVzZXJ2YXRpb24gbWFuYWdlbWVudCBtZXRob2RzIGRlZmluZWQgaW4g dGhlIFNDU0kNCj4gPiA+IHN0YW5kYXJkcywgcmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9ucywg YmFzZWQgb24gdGhlIFJFU0VSVkUgYW5kDQo+ID4gPiBSRUxFQVNFIGNvbW1hbmRzIFtTUEMyXSwg YW5kIHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zLCBiYXNlZCBvbiB0aGUNCj4gPiA+IFBFUlNJU1RF TlQgUkVTRVJWRSBJTiBhbmQgUEVSU0lTVEVOVCBSRVNFUlZFIE9VVCBjb21tYW5kcyBbU1BDM10u DQo+ID4gPiAgUmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9ucyBhcmUgb2Jzb2xldGUgW1NQQzNd IGFuZCBTSE9VTEQgTk9UIGJlDQo+ID4gPiB1c2VkOyBwZXJzaXN0ZW50IHJlc2VydmF0aW9ucyBT SE9VTEQgYmUgdXNlZCBpbnN0ZWFkLg0KPiA+ID4NCj4gPiA+ICBTdGF0ZSBmb3IgcGVyc2lzdGVu dCByZXNlcnZhdGlvbnMgaXMgcmVxdWlyZWQgdG8gcGVyc2lzdCAgdGhyb3VnaA0KPiA+ID4gY2hh bmdlcyBhbmQgZmFpbHVyZXMgYXQgdGhlIGlTQ1NJIGxheWVyIHRoYXQgcmVzdWx0IGluICBJX1Qg TmV4dXMNCj4gPiA+IGZhaWx1cmVzLCBzZWUgW1NQQzNdIGZvciBkZXRhaWxzIGFuZCBzcGVjaWZp YyByZXF1aXJlbWVudHMuDQo+ID4gPg0KPiA+ID4gIEluIGNvbnRyYXN0LCBbU1BDMl0gZG9lcyBu b3Qgc3BlY2lmeSBkZXRhaWxlZCBwZXJzaXN0ZW5jZQ0KPiA+ID4gcmVxdWlyZW1lbnRzIGZvciBy ZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb24gc3RhdGUgYWZ0ZXIgYW4gSV9UDQo+ID4gPiBOZXh1 cyBmYWlsdXJlLiAgTm9uZXRoZWxlc3MsIHdoZW4gcmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9u cyBhcmUNCj4gPiA+IHN1cHBvcnRlZCBieSBhbiBpU0NTSSB0YXJnZXQsIHRoZSBwcmVmZXJyZWQg aW1wbGVtZW50YXRpb24gYXBwcm9hY2gNCj4gPiA+IGlzIHRvIHByZXNlcnZlIHJlc2VydmUvcmVs ZWFzZSByZXNlcnZhdGlvbiBzdGF0ZSBmb3IgaVNDU0kgc2Vzc2lvbg0KPiA+ID4gcmVpbnN0YXRl bWVudCAoc2VlIFNlY3Rpb24gNi4zLjUpIG9yIHNlc3Npb24gY29udGludWF0aW9uICAoc2VlDQo+ ID4gPiBTZWN0aW9uIDYuMy42KS4NCj4gPiA+DQo+ID4gPiAgVHdvIGFkZGl0aW9uYWwgY2F2ZWF0 cyBhcHBseSB0byByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zOg0KPiA+ID4NCj4gPiA+ICAt IFdoZW4gY29ubmVjdGlvbiBmYWlsdXJlIGNhdXNlcyB0aGUgaVNDU0kgc2Vzc2lvbiB0byBmYWls LCBhbmQNCj4gPiA+ICAgICAgdGhlIHNlc3Npb24gaXMgbm90IHJlaW5zdGF0ZWQgb3IgY29udGlu dWVkLCB0YXJnZXQgcmV0ZW50aW9uDQo+ID4gPiAgICAgIG9mIHRoYXQgc2Vzc2lvbidzIHJlc2Vy dmUvcmVsZWFzZSByZXNlcnZhdGlvbiBzdGF0ZSBtYXkNCj4gPiA+ICAgICAgcmVxdWlyZSBhbiBp bml0aWF0b3IgdG8gaXNzdWUgYSByZXNldCAoZS5nLiwgTE9HSUNBTCBVTklUIFJFU0VULA0KPiA+ ID4gICAgICBzZWUgc2VjdGlvbiAxMS41KSBpbiBvcmRlciB0byByZW1vdmUgdGhhdCByZXNlcnZh dGlvbiBzdGF0ZS4NCj4gPiA+DQo+ID4gPiAgLSBSZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25z IG1heSBub3QgYmVoYXZlIGFzIGV4cGVjdGVkIHdoZW4NCj4gPiA+ICAgICAgcGVyc2lzdGVudCBy ZXNlcnZhdGlvbnMgYXJlIGFsc28gdXNlZCBvbiB0aGUgc2FtZSBsb2dpY2FsIHVuaXQ7DQo+ID4g PiAgICAgIHNlZSB0aGUgZGlzY3Vzc2lvbiBvZiAiRXhjZXB0aW9ucyB0byBTUEMtMiBSRVNFUlZF IGFuZCBSRUxFQVNFDQo+ID4gPiAgICAgIGJlaGF2aW9yIiBpbiBbU1BDNF0uDQo+ID4gPg0KPiA+ ID4gVGhhbmtzLA0KPiA+ID4gLS1EYXZpZA0KPiA+ID4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiA+ID4gRGF2aWQgTC4gQmxhY2ssIERpc3Rp bmd1aXNoZWQgRW5naW5lZXIgRU1DIENvcnBvcmF0aW9uLCAxNzYgU291dGgNCj4gPiA+IFN0Liwg SG9wa2ludG9uLCBNQSAgMDE3NDgNCj4gPiA+ICsxICg1MDgpIDI5My03OTUzICAgICAgICAgICAg IEZBWDogKzEgKDUwOCkgMjkzLTc3ODYNCj4gPiA+IGRhdmlkLmJsYWNrQGVtYy5jb20gICAgICAg IE1vYmlsZTogKzEgKDk3OCkgMzk0LTc3NTQNCj4gPiA+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gPiA+DQo+ID4gPg0KPiA+ID4gX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4gPiA+IHN0b3JtIG1h aWxpbmcgbGlzdA0KPiA+ID4gc3Rvcm1AaWV0Zi5vcmcNCj4gPiA+IGh0dHBzOi8vd3d3LmlldGYu b3JnL21haWxtYW4vbGlzdGluZm8vc3Rvcm0NCj4gDQo+IF9fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fDQo+IHN0b3JtIG1haWxpbmcgbGlzdA0KPiBzdG9ybUBp ZXRmLm9yZw0KPiBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3N0b3JtDQo= From Frederick.Knight@netapp.com Thu Sep 27 15:29:33 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1BA5B21F85B4 for ; Thu, 27 Sep 2012 15:29:33 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id F1v5jKJvMKpX for ; Thu, 27 Sep 2012 15:29:32 -0700 (PDT) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id 2171821F85B1 for ; Thu, 27 Sep 2012 15:29:31 -0700 (PDT) X-IronPort-AV: E=Sophos;i="4.80,498,1344236400"; d="scan'208";a="695023389" Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx2-out.netapp.com with ESMTP; 27 Sep 2012 15:29:31 -0700 Received: from vmwexceht05-prd.hq.netapp.com (vmwexceht05-prd.hq.netapp.com [10.106.77.35]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8RMTVkA017275; Thu, 27 Sep 2012 15:29:31 -0700 (PDT) Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.102]) by vmwexceht05-prd.hq.netapp.com ([10.106.77.35]) with mapi id 14.02.0309.002; Thu, 27 Sep 2012 15:29:30 -0700 From: "Knight, Frederick" To: "Black, David" Thread-Topic: [storm] SPC-2 reserve/release text - version 6 Thread-Index: Ac2cNcys4MzRK8PXQu651fVKNuEIxQAXDfsAABnkfIAADjDW0AAa4S6gACikjGA= Date: Thu, 27 Sep 2012 22:29:30 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> <4AE113CD-96D7-414D-8AAB-36ECF7FD89EA@satran.net> <8D3D17ACE214DC429325B2B98F3AE7120DE798F9@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE7993C@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE7993C@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.106.53.53] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release text - version 6 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Sep 2012 22:29:33 -0000 SSB3b3VsZCB0aGluayBpZiB0aGUgSVNJRCBpcyBub3QgcmVjb2duaXplZCBhcyBhbiBvbGQgb25l LCB0aGVuIGl0IG11c3QgYmUgcmVjb2duaXplZCBhcyBhIG5ldyBvbmUuICBJZiBpdCBpcyBhIG5l dyBvbmUsIHRoZW4gaXQgd2lsbCBnZXQgYSBVQTogMjkvMDAgcmVmbGVjdGluZyB0aGUgZmlyc3Qg YWNjZXNzIGFmdGVyIHBvd2VyIG9uIGZyb20gdGhlICJuZXciIGluaXRpYXRvci4NCg0KSXMgdGhl cmUgc29tZXRoaW5nIG90aGVyIHRoYW4gbmV3IG9yIG9sZD8gIElzIHRoZXJlIHN1Y2ggYSB0aGlu ZyBhcyBhIG1pZGRsZS1hZ2VkIGluaXRpYXRvcj8NCg0KCUZyZWQNCg0KLS0tLS1PcmlnaW5hbCBN ZXNzYWdlLS0tLS0NCkZyb206IEJsYWNrLCBEYXZpZCBbbWFpbHRvOmRhdmlkLmJsYWNrQGVtYy5j b21dIA0KU2VudDogVGh1cnNkYXksIFNlcHRlbWJlciAyNywgMjAxMiAxMTo1NiBBTQ0KVG86IEtu aWdodCwgRnJlZGVyaWNrDQpDYzogc3Rvcm1AaWV0Zi5vcmcNClN1YmplY3Q6IFJFOiBbc3Rvcm1d IFNQQy0yIHJlc2VydmUvcmVsZWFzZSB0ZXh0IC0gdmVyc2lvbiA2DQoNCkZyZWQsDQoNClRoZSBw cm9ibGVtIGlzIG5vdCB0aGF0IHRoZSBkZXZpY2UgZG9lcyBub3QgcmFpc2UgYSBVQSwgaXQncyB0 aGF0IHRoZSBVQSBtYXkgbmV2ZXIgYmUgZGVsaXZlcmVkLg0KDQpBIHN1bW1hcnkgb2YgaG93IHRo aXMgY2FuIGhhcHBlbiBpcyB0aGF0IHRoZSBuZXcgY29ubmVjdGlvbiB3aXRoIHRoZSBzYW1lIElT SUQgbWF5IG5vdCBiZSByZWNvZ25pemVkIGFzIHJlaW5zdGF0aW5nIG9yIGNvbnRpbnVpbmcgdGhl IHNlc3Npb24gYXQgdGhlIHRhcmdldCBpZiB0aGUgdGltZW91dHMgaGF2ZSBleHBpcmVkIC0gdGhl IFVBIGdldHMgZXN0YWJsaXNoZWQsIGJ1dCBub2JvZHkgc2hvd3MgdXAgdG8gcGljayBpdCB1cCBp biB0aW1lLCBzbyB0aGUgVUEgaXMgZGlzY2FyZGVkIGFsb25nIHdpdGggdGhlIHJlc3Qgb2YgdGhl IHNlc3Npb24gc3RhdGUgYW5kIHRoZSB0YXJnZXQgZmFpbHMgdG8gcmVjb2duaXplIHRoZSBpbml0 aWF0b3IncyBzdWJzZXF1ZW50IElTSUQgcmV1c2UgZXZlbiB0aG91Z2ggdGhlIGluaXRpYXRvciBi ZWxpZXZlcyB0aGF0IGl0J3MgcmUtIGVzdGFibGlzaGluZyB0aGUgc2Vzc2lvbi4NCg0KVGhpcyBp cyBmdXJ0aGVyIGNvbXBsaWNhdGVkIGJ5IHRoZSBVQSB0aGF0IGluZGljYXRlcyBuZXh1cyBzdGF0 ZSBwcmVzZXJ2YXRpb24gbm90IGJlaW5nIHNwZWNpZmllZCBpbiBTQU0tMi4NCg0KVGhhbmtzLA0K LS1EYXZpZA0KDQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogS25pZ2h0 LCBGcmVkZXJpY2sgW21haWx0bzpGcmVkZXJpY2suS25pZ2h0QG5ldGFwcC5jb21dDQo+IFNlbnQ6 IFRodXJzZGF5LCBTZXB0ZW1iZXIgMjcsIDIwMTIgMTE6MTEgQU0NCj4gVG86IEJsYWNrLCBEYXZp ZDsgSnVsaWFuIFNhdHJhbg0KPiBDYzogc3Rvcm1AaWV0Zi5vcmcNCj4gU3ViamVjdDogUkU6IFtz dG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYNCj4gDQo+IEkgYWZy YWlkIEkgaGF2ZSB0byBjb250ZXN0IGEgbGl0dGxlIG9uIHRoaXMgLSBpZiBJIHVuZGVyc3RhbmQg dGhpcyANCj4gZGlzY3Vzc2lvbiBjb3JyZWN0bHkuDQo+IA0KPiBJZiBhIFNDU0kgZGV2aWNlIGxv c2VzIGFueSBzdGF0ZSBhbmQgZG9lcyBub3QgcmFpc2UgYSBVTklUIEFUVEVOVElPTiwgDQo+IHRo ZW4gaXQgaXMgQlJPS0VOLiAgSXQgaXMgbm90IGEgbWF0dGVyIG9mIGFueXRoaW5nIGluIGlTQ1NJ ICgzNzIwKTsgSSANCj4gdGhpbmsgU0FNIGFuZCBTUEMgYXJlIHByZXR0eSBjbGVhciBvbiB0aGlz LiAgSWYgdGhlcmUgaXMgbm8gVU5JVCANCj4gQVRURU5USU9OLCB0aGVuIHN0YXRlIFNIQUxMIGJl IHByZXNlcnZlZDsgb3RoZXJ3aXNlLCBhbGwgc29ydHMgb2YgDQo+IHN0dWZmIHdpbGwgbmV2ZXIg d29yayAobW9kZSBwYWdlIHNldHRpbmdzLCBSRVNFUlZFIGNvbW1hbmRzLCBhbGwgc29ydHMgb2Yg c3R1ZmYpLg0KPiANCj4gSXMgc29tZW9uZSByZWFsbHkgc3VnZ2VzdGluZyB0aGF0IHNvbWUgZGV2 aWNlcyB3aWxsIGRyb3Agc3RhdGUgYW5kIE5PVCANCj4gdGVsbCB0aGUgaG9zdCwgYW5kIHNvbWVo b3cgdGhhdCBpdCBpcyBPSyB0byBkbyB0aGF0Pw0KPiANCj4gVGhlIHN1Z2dlc3QgcmV3b3JkaW5n IHNheXMgdGhpcyBpcyBwb3NzaWJsZS4NCj4gDQo+IAlGcmVkDQo+IA0KPiAtLS0tLU9yaWdpbmFs IE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBzdG9ybS1ib3VuY2VzQGlldGYub3JnIFttYWlsdG86c3Rv cm0tYm91bmNlc0BpZXRmLm9yZ10gT24gQmVoYWxmIA0KPiBPZiBCbGFjaywgRGF2aWQNCj4gU2Vu dDogVGh1cnNkYXksIFNlcHRlbWJlciAyNywgMjAxMiAxMDo0NyBBTQ0KPiBUbzogSnVsaWFuIFNh dHJhbg0KPiBDYzogc3Rvcm1AaWV0Zi5vcmcNCj4gU3ViamVjdDogUmU6IFtzdG9ybV0gU1BDLTIg cmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYNCj4gDQo+IEp1bGlhbiwNCj4gDQo+ID4g VGhlIHRleHQgaW4gNC40LjMgc3RhdGVzIHRoZSBvYnZpb3VzICh0aGVyZSBpcyBubyBvdGhlciBz cGVjZWQgDQo+ID4gbWV0aG9kIGZvciBjb250aW51YXRpb24gb3IgcmVpbnN0YXRlbWVudCkuIE1v cmVvdmVyIGlmIHRoZSBzZXNzaW9uIA0KPiA+IGlzIHJlaW5zdGF0ZWQgYWZ0ZXIgdGltZTJ3YWl0 IHlvdSBjcmVhdGUgYSBuZWVkIGZvciBhIFVBIChvciBhdCANCj4gPiBsZWFzdCBzdWdnZXN0IG9u ZSkgYW5kIHRoYXQgd2FzIG5vdCB0aGVyZS4NCj4gDQo+IFRoZSBVQXMgYXJlIGNvdXJ0ZXN5IG9m IFNBTS0zIChhbmQgU0FNLTQpLiAgVG8gZW5jb21wYXNzIHN5c3RlbXMgdGhhdCANCj4gbWF5IG5v dCBkZWxpdmVyIHRoZXNlIFVBcyBhbmQgc2l0dWF0aW9ucyBpbiB3aGljaCB0aGV5IGRvbid0IG9j Y3VyIA0KPiAoZS5nLiwgdGhlIHRhcmdldCBkb2Vzbid0IHJlY29nbml6ZSB3aGF0IGhhcHBlbmVk IGFzIG5leHVzIA0KPiByZWVzdGFibGlzaG1lbnQsIGluZGVwZW5kZW50IG9mIHdoYXQgdGhlIGlu aXRpYXRvciB0aGlua3MgaXQncyBkb2luZykgSSB0aGluayB0aGUgdGV4dCBzaG91bGQgYmUgcmVw aHJhc2VkIGFzOg0KPiANCj4gICAgICBJZiBhIFVuaXQgQXR0ZW50aW9uIGNvbmRpdGlvbiByZXN1 bHRzIGZyb20gc2Vzc2lvbiByZWVzdGFibGlzaG1lbnQsDQo+ICAgICAgdGhlIHNwZWNpZmljIFVu aXQgQXR0ZW50aW9uIGNvbmRpdGlvbiBpbmRpY2F0ZXMgd2hldGhlciBuZXh1cyBzdGF0ZQ0KPiAg ICAgIHdhcyBwcmVzZXJ2ZWQsIHNlZSBTZWN0aW9uIDYuMy40IG9mIFtTQU00XS4NCj4gDQo+ID4g SnVzdCBzdGF0aW5nIHRoYXQgY3JlYXRpb24vY29udGludWF0aW9uIG9mIGEgc2Vzc2lvbiBkb25l IGFjY29yZGluZyANCj4gPiA2LnggaW5jbHVkZXMgcHJlc2VydmF0aW9uIG9mIHJlc2VydmUvcmVs ZWFzZSBzdGF0ZSAtdGhhdCB3YXMgYWxyZWFkeSANCj4gPiB0aGVyZSBpbXBsaWVkIGJ5IGNvbmZv cm1hbmNlIHRvIHNwYzIgYW5kIHRoZSByZWxhdGlvbiBiZXR3ZWVuIGl0IA0KPiA+IG5leHVzIGFu ZCBzZXNzaW9uLg0KPiANCj4gVW5mb3J0dW5hdGVseSwgU1BDLTIgaXMgc3VmZmljaWVudGx5IGlt cHJlY2lzZSBvbiB0aGlzIHN1YmplY3QgdGhhdCANCj4gc3VjaCBhbiBpbXBsaWNhdGlvbiBpcyBh dCBiZXN0IGEgInByZWZlcnJlZCBpbXBsZW1lbnRhdGlvbiBhcHByb2FjaCIuICANCj4gSXQncyBu b3QgYSByZXF1aXJlbWVudCB0aGF0IGNhbiBiZSBzYWZlbHkgc3RhdGVkIGluIGFueSBzdHJvbmdl ciANCj4gZmFzaGlvbiB3aXRob3V0IHRoZSByaXNrIG9mIHJlbmRlcmluZyBleGlzdGluZyBpbXBs ZW1lbnRhdGlvbnMgbm9uLWNvbXBsaWFudC4NCj4gDQo+ID4gVGhlIG5ldyB0ZXh0IHlvdSBzdWdn ZXN0IHRvIDQuNCBpcyBtZXJlbHkgYSBiZXN0IHByYWN0aWNlcyANCj4gPiByZWNvbW1lbmRhdGlv bnMgYW5kIHdpbGwgbmVlZCBhZGRpdGlvbnMgb25jZSB0aGUgbmV3ICJsb2NraW5nIiANCj4gPiB0 aHJvdWdoIGNvbXBhcmUtYW5kLXN3YXAgZ2V0IGluIHdpZGVzcHJlYWQgdXNlLg0KPiANCj4gSXQn cyBkZWZpbml0ZWx5IGJlc3QgcHJhY3RpY2VzIHJlY29tbWVuZGF0aW9ucyAtIG5vIGFyZ3VtZW50 IHRoZXJlLg0KPiANCj4gRm9yIGNvbnRleHQsIHdoYXQgSnVsaWFuIGlzIHJlZmVycmluZyB0byBp cyB1c2Ugb2YgdGhlIENPTVBBUkUgQU5EIA0KPiBXUklURSBjb21tYW5kIHRoYXQgcGVyZm9ybXMg YW4gYXRvbWljIG9wZXJhdGlvbiBhdCB0aGUgdGFyZ2V0IGluc3RlYWQgDQo+IG9mIHVzaW5nIFJF U0VSVkUgYW5kIFJFTEVBU0UgdG8gY3JlYXRlIGEgY3JpdGljYWwgc2VjdGlvbiBhdCB0aGUgDQo+ IGluaXRpYXRvciBiYXNlZCBvbiByZWFkIGFuZCB3cml0ZSBjb21tYW5kcy4NCj4gDQo+IENPTVBB UkUgQU5EIFdSSVRFIGlzIGluIHVzZSAtIGEgc2VudGVuY2UgbWVudGlvbmluZyB0aGF0IGNvbW1h bmQgY291bGQgDQo+IGJlIGFkZGVkLCBhbHRob3VnaCB0aGlzIGJlc3QgcHJhY3RpY2VzIHRleHQg aXMgYWltZWQgcHJpbWFyaWx5IGF0IHVzZXMgDQo+IG9mIHJlc2VydmUvcmVsZWFzZSBmb3IgbG9u Zy10ZXJtIHJlc2VydmF0aW9ucy4gIEkgdGhpbmsgdGhlcmUncyBhbiANCj4gaW1wbGljaXQgIndo ZW4gYSByZXNlcnZhdGlvbiBpcyB0aGUgZGVzaXJlZCBiZWhhdmlvciIgaW1wbGljYXRpb24gdG8g DQo+IHRoZSAicGVyc2lzdGVudCByZXNlcnZhdGlvbnMgU0hPVUxEIGJlIHVzZWQgaW5zdGVhZCIg dGV4dCB0aGF0IEkgZG9uJ3QgDQo+IHRoaW5rIG5lZWRzIHRvIGJlIG1hZGUgZXhwbGljaXQuICBP VE9ILCB0aGF0ICJTSE9VTEQiDQo+IGNvdWxkIGJlIGNoYW5nZWQgdG8gYSAic2hvdWxkIiwgb3Ig aXQgbWlnaHQgYmUgZXZlbiBiZXR0ZXIgdG8gcmV3cml0ZSANCj4gdGhhdCB0ZXh0DQo+IGFzOg0K PiANCj4gCXBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIGFyZSBzdWdnZXN0ZWQgYXMgYW4gYWx0ZXJu YXRpdmUsIHNlZSBBbm5leCBCDQo+IAlvZiBbU1BDNF0uDQo+IA0KPiBJbiBjb250cmFzdCwgIlNI T1VMRCBOT1QgYmUgdXNlZCIgaXMgY2xlYXJseSBqdXN0aWZpZWQgZm9yIA0KPiByZXNlcnZlL3Jl bGVhc2UgcmVzZXJ2YXRpb25zIGJ5IFtTUEMzXS4NCj4gDQo+IFRoYW5rcywNCj4gLS1EYXZpZA0K PiANCj4gDQo+ID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPiBGcm9tOiBKdWxpYW4g U2F0cmFuIFttYWlsdG86anVsaWFuQHNhdHJhbi5uZXRdDQo+ID4gU2VudDogV2VkbmVzZGF5LCBT ZXB0ZW1iZXIgMjYsIDIwMTIgMTA6MjUgUE0NCj4gPiBUbzogQmxhY2ssIERhdmlkDQo+ID4gQ2M6 IHN0b3JtQGlldGYub3JnDQo+ID4gU3ViamVjdDogUmU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9y ZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYNCj4gPg0KPiA+IERhdmlkLA0KPiA+DQo+ID4gVGhlIHRl eHQgaW4gNC40LjMgc3RhdGVzIHRoZSBvYnZpb3VzICh0aGVyZSBpcyBubyBvdGhlciBzcGVjZWQg DQo+ID4gbWV0aG9kIGZvciBjb250aW51YXRpb24gb3IgcmVpbnN0YXRlbWVudCkuIE1vcmVvdmVy IGlmIHRoZSB0aGUgDQo+ID4gc2Vzc2lvbiBpcyByZWluc3RhdGVkIGFmdGVyIHRpbWUyd2FpdCB5 b3UgY3JlYXRlIGEgbmVlZCBmb3IgYSBVQSAob3IgDQo+ID4gYXQgbGVhc3Qgc3VnZ2VzdCBvbmUp IGFuZCB0aGF0IHdhcyBub3QgdGhlcmUuIEp1c3Qgc3RhdGluZyB0aGF0IA0KPiA+IGNyZWF0aW9u L2NvbnRpbnVhdGlvbiBvZiBhIHNlc3Npb24gZG9uZSBhY2NvcmRpbmcgNi54IGluY2x1ZGVzIA0K PiA+IHByZXNlcnZhdGlvbiBvZiByZXNlcnZlL3JlbGVhc2Ugc3RhdGUgLXRoYXQgd2FzIGFscmVh ZHkgdGhlcmUgDQo+ID4gaW1wbGllZCBieSBjb25mb3JtYW5jZSB0byBzcGMyIGFuZCB0aGUgcmVs YXRpb24gYmV0d2VlbiBpdCBuZXh1cyBhbmQgc2Vzc2lvbi4NCj4gPg0KPiA+IFRoZSBuZXcgdGV4 dCB5b3Ugc3VnZ2VzdCB0byA0LjQgaXMgbWVyZWx5IGEgYmVzdCBwcmFjdGljZXMgDQo+ID4gcmVj b21lbmRhdGlvbnMgYW5kIHdpbGwgbmVlZCBhZGRpdGlvbnMgb25jZSB0aGUgbmV3ICJsb2NraW5n IiANCj4gPiB0aHJvdWdoIGNvbXBhcmUtYW5kLXN3YXAgZ2V0IGluIHdpZGVzcHJlYWQgdXNlLg0K PiA+IElmIGluc3RlYWQgd2UgcmVmcmFpbiBmcm9tIHN0YXRpbmcgYW55dGhpbmcgc3BlY2lmaWMg YW5kIHN0YXRlIHRoYXQgDQo+ID4gdGFyZ2V0IHNob3VsZCBtYWludGFpbiB3aGF0ZXZlciBzdGF0 ZSBpcyByZXF1aXJlZCBieSBTUEMgd2hpbGUgYSANCj4gPiBzZXNzaW9uIGlzIGNvbnRpbnVlZCBh bmQgbWF5IGRpc2NhcmQgc3RhdGUgd2hlbiBhIHNlc3Npb24gaXMgDQo+ID4gdGVybWluYXRlZCBh bmQgbWF5IGluZGljYXRlIHRoYXQgaXQgZGlzY2FyZGVkIHN0YXRlIHRocm91Z2ggYSB1YSB5b3Ug DQo+ID4gYXJlIG9uIHNhZmVyIGdyb3VuZCBhbmQgZG8gbm90IGNyZWF0ZSBuZXcgcmVxdWlyZW1l bnQuDQo+ID4NCj4gPiBTZW50IGZyb20gbXkgaVBhZA0KPiA+DQo+ID4gT24gMjcg15HXodek15gg MjAxMiwgYXQgMDA6MjUsICJCbGFjaywgRGF2aWQiIDxkYXZpZC5ibGFja0BlbWMuY29tPiB3cm90 ZToNCj4gPg0KPiA+ID4gSGVyZSdzIHdoZXJlIEkgdGhpbmsgd2UndmUgYXJyaXZlZCAoSSd2ZSBt YWRlIGEgZmV3IG1pbm9yIA0KPiA+ID4gZWRpdG9yaWFsIHR3ZWFrcyB0byB3aGF0J3MgYmVlbiBk aXNjdXNzZWQpLiAgRGVzcGl0ZSBhbGwgdGhlIA0KPiA+ID4gZGV0YWlsZWQgZGlzY3Vzc2lvbiwg SSB0aGluayB0aGlzIHZlcnNpb24gNiBjb21lcyBjbG9zZSB0byANCj4gPiA+IEp1bGlhbidzIHN1 Z2dlc3Rpb24NCj4gdGhhdDoNCj4gPiA+DQo+ID4gPj4gaSB3b3VsZCBtZW50aW9uIG9ubHkgdGhh dCB0aGUgc3RhdGUgdGhhdCBtdXN0IGJlIG1haW50YWluZWQgd2hpbGUgDQo+ID4gPj4gdGhlIHNl c3Npb24gaXMgbWFpbnRhaW5lZCBpbmNsdWRlcyB0aGUgcmVzZXJ2ZS9yZWxlYXNlIGFuZCBsZWF2 ZSANCj4gPiA+PiBpdCB0bw0KPiA+IHRoYXQuDQo+ID4gPg0KPiA+ID4gQWx0aG91Z2ggd2UgY2Fu J3QgYWN0dWFsbHkgc2F5ICJtdXN0IiBhcyB0aGF0IHdvdWxkIGJlIGEgbmV3IA0KPiA+ID4gcmVx dWlyZW1lbnQsIGFuZCBpbXBsZW1lbnRhdGlvbnMgZGlmZmVyIGluIHdoYXQgdGhleSBkbyBhYm91 dCANCj4gPiA+IHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbiBzdGF0ZS4gIFRoZSBjdXJyZW50 IGxhbmd1YWdlIA0KPiA+ID4gY2hhcmFjdGVyaXplcyByZXRlbnRpb24gb2YgdGhhdCBzdGF0ZSBh cyB0aGUgInByZWZlcnJlZCANCj4gPiA+IGltcGxlbWVudGF0aW9uDQo+IGFwcHJvYWNoIi4NCj4g PiA+DQo+ID4gPiBCZXlvbmQgdGhhdCwgSSB0aGluayB0aGUgdHdvIGFkZGl0aW9uYWwgY2F2ZWF0 cyBhYm91dCANCj4gPiA+IHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbnMgKHBvc3NpYmxlIG5l ZWQgZm9yIGEgcmVzZXQsIA0KPiA+ID4gcG90ZW50aWFsbHkgdW5leHBlY3RlZCBpbnRlcmFjdGlv biB3aXRoIHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zLCANCj4gPiA+IHNlZSBiZWxvdykgYXJlIHVz ZWZ1bA0KPiB0byBpbmNsdWRlLg0KPiA+ID4NCj4gPiA+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tDQo+ID4gPg0KPiA+ID4gWzFdIEZvciBjbGFyaXR5LCB0aGUgZm9s bG93aW5nIGNoYW5nZSBzaG91bGQgYmUgbWFkZSBpbg0KPiA+ID4gNC40LjMuMToNCj4gPiA+DQo+ ID4gPiBPTEQNCj4gPiA+ICBUaGF0IGlzLCBpdCBzaG91bGQgcGVyZm9ybSBzZXNzaW9uIHJlY292 ZXJ5IGFzICBkZXNjcmliZWQgaW4gDQo+ID4gPiBDaGFwdGVyIDYuDQo+ID4gPiBORVcNCj4gPiA+ ICBUaGF0IGlzLCBpdCBzaG91bGQgcmVpbnN0YXRlIHRoZSBzZXNzaW9uIHZpYSBpU0NTSSBzZXNz aW9uIA0KPiA+ID4gcmVpbnN0YXRlbWVudCAoU2VjdGlvbiA2LjMuNSkgb3IgY29udGludWUgdmlh IHNlc3Npb24gIA0KPiA+ID4gY29udGludWF0aW9uIChTZWN0aW9uIDYuMy42KS4NCj4gPiA+IEVO RA0KPiA+ID4NCj4gPiA+IFsyXSBBZGQgdGhlIGZvbGxvd2luZyBzZW50ZW5jZSB0byB0aGUgZW5k IG9mIDQuNC4zLjE6DQo+ID4gPg0KPiA+ID4gICAgIFRoZSBzcGVjaWZpYyBVbml0IEF0dGVudGlv biBjb25kaXRpb24gdGhhdCByZXN1bHRzIGZyb20gc2Vzc2lvbg0KPiA+ID4gICAgIHJlZXN0YWJs aXNobWVudCBpbmRpY2F0ZXMgd2hldGhlciBuZXh1cyBzdGF0ZSB3YXMgcHJlc2VydmVkLA0KPiA+ ID4gICAgIHNlZSBTZWN0aW9uIDYuMy40IG9mIFtTQU00XS4NCj4gPiA+DQo+ID4gPiBbM10gTkVX IFRFWFQ6DQo+ID4gPg0KPiA+ID4gNC40LjMuMi4gUmVzZXJ2YXRpb25zDQo+ID4gPg0KPiA+ID4g IFRoZXJlIGFyZSB0d28gcmVzZXJ2YXRpb24gbWFuYWdlbWVudCBtZXRob2RzIGRlZmluZWQgaW4g dGhlIFNDU0kgDQo+ID4gPiBzdGFuZGFyZHMsIHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbnMs IGJhc2VkIG9uIHRoZSBSRVNFUlZFIGFuZCANCj4gPiA+IFJFTEVBU0UgY29tbWFuZHMgW1NQQzJd LCBhbmQgcGVyc2lzdGVudCByZXNlcnZhdGlvbnMsIGJhc2VkIG9uIHRoZSANCj4gPiA+IFBFUlNJ U1RFTlQgUkVTRVJWRSBJTiBhbmQgUEVSU0lTVEVOVCBSRVNFUlZFIE9VVCBjb21tYW5kcyBbU1BD M10uDQo+ID4gPiAgUmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9ucyBhcmUgb2Jzb2xldGUgW1NQ QzNdIGFuZCBTSE9VTEQgTk9UIA0KPiA+ID4gYmUgdXNlZDsgcGVyc2lzdGVudCByZXNlcnZhdGlv bnMgU0hPVUxEIGJlIHVzZWQgaW5zdGVhZC4NCj4gPiA+DQo+ID4gPiAgU3RhdGUgZm9yIHBlcnNp c3RlbnQgcmVzZXJ2YXRpb25zIGlzIHJlcXVpcmVkIHRvIHBlcnNpc3QgIHRocm91Z2ggDQo+ID4g PiBjaGFuZ2VzIGFuZCBmYWlsdXJlcyBhdCB0aGUgaVNDU0kgbGF5ZXIgdGhhdCByZXN1bHQgaW4g IElfVCBOZXh1cyANCj4gPiA+IGZhaWx1cmVzLCBzZWUgW1NQQzNdIGZvciBkZXRhaWxzIGFuZCBz cGVjaWZpYyByZXF1aXJlbWVudHMuDQo+ID4gPg0KPiA+ID4gIEluIGNvbnRyYXN0LCBbU1BDMl0g ZG9lcyBub3Qgc3BlY2lmeSBkZXRhaWxlZCBwZXJzaXN0ZW5jZSANCj4gPiA+IHJlcXVpcmVtZW50 cyBmb3IgcmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9uIHN0YXRlIGFmdGVyIGFuIElfVCANCj4g PiA+IE5leHVzIGZhaWx1cmUuICBOb25ldGhlbGVzcywgd2hlbiByZXNlcnZlL3JlbGVhc2UgcmVz ZXJ2YXRpb25zIGFyZSANCj4gPiA+IHN1cHBvcnRlZCBieSBhbiBpU0NTSSB0YXJnZXQsIHRoZSBw cmVmZXJyZWQgaW1wbGVtZW50YXRpb24gDQo+ID4gPiBhcHByb2FjaCBpcyB0byBwcmVzZXJ2ZSBy ZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb24gc3RhdGUgZm9yIA0KPiA+ID4gaVNDU0kgc2Vzc2lv biByZWluc3RhdGVtZW50IChzZWUgU2VjdGlvbiA2LjMuNSkgb3Igc2Vzc2lvbiANCj4gPiA+IGNv bnRpbnVhdGlvbiAgKHNlZSBTZWN0aW9uIDYuMy42KS4NCj4gPiA+DQo+ID4gPiAgVHdvIGFkZGl0 aW9uYWwgY2F2ZWF0cyBhcHBseSB0byByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zOg0KPiA+ ID4NCj4gPiA+ICAtIFdoZW4gY29ubmVjdGlvbiBmYWlsdXJlIGNhdXNlcyB0aGUgaVNDU0kgc2Vz c2lvbiB0byBmYWlsLCBhbmQNCj4gPiA+ICAgICAgdGhlIHNlc3Npb24gaXMgbm90IHJlaW5zdGF0 ZWQgb3IgY29udGludWVkLCB0YXJnZXQgcmV0ZW50aW9uDQo+ID4gPiAgICAgIG9mIHRoYXQgc2Vz c2lvbidzIHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbiBzdGF0ZSBtYXkNCj4gPiA+ICAgICAg cmVxdWlyZSBhbiBpbml0aWF0b3IgdG8gaXNzdWUgYSByZXNldCAoZS5nLiwgTE9HSUNBTCBVTklU IFJFU0VULA0KPiA+ID4gICAgICBzZWUgc2VjdGlvbiAxMS41KSBpbiBvcmRlciB0byByZW1vdmUg dGhhdCByZXNlcnZhdGlvbiBzdGF0ZS4NCj4gPiA+DQo+ID4gPiAgLSBSZXNlcnZlL3JlbGVhc2Ug cmVzZXJ2YXRpb25zIG1heSBub3QgYmVoYXZlIGFzIGV4cGVjdGVkIHdoZW4NCj4gPiA+ICAgICAg cGVyc2lzdGVudCByZXNlcnZhdGlvbnMgYXJlIGFsc28gdXNlZCBvbiB0aGUgc2FtZSBsb2dpY2Fs IHVuaXQ7DQo+ID4gPiAgICAgIHNlZSB0aGUgZGlzY3Vzc2lvbiBvZiAiRXhjZXB0aW9ucyB0byBT UEMtMiBSRVNFUlZFIGFuZCBSRUxFQVNFDQo+ID4gPiAgICAgIGJlaGF2aW9yIiBpbiBbU1BDNF0u DQo+ID4gPg0KPiA+ID4gVGhhbmtzLA0KPiA+ID4gLS1EYXZpZA0KPiA+ID4gLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiA+ID4gRGF2aWQgTC4g QmxhY2ssIERpc3Rpbmd1aXNoZWQgRW5naW5lZXIgRU1DIENvcnBvcmF0aW9uLCAxNzYgU291dGgg DQo+ID4gPiBTdC4sIEhvcGtpbnRvbiwgTUEgIDAxNzQ4DQo+ID4gPiArMSAoNTA4KSAyOTMtNzk1 MyAgICAgICAgICAgICBGQVg6ICsxICg1MDgpIDI5My03Nzg2DQo+ID4gPiBkYXZpZC5ibGFja0Bl bWMuY29tICAgICAgICBNb2JpbGU6ICsxICg5NzgpIDM5NC03NzU0DQo+ID4gPiAtLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ID4gPg0KPiA+ID4N Cj4gPiA+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+ ID4gPiBzdG9ybSBtYWlsaW5nIGxpc3QNCj4gPiA+IHN0b3JtQGlldGYub3JnDQo+ID4gPiBodHRw czovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3N0b3JtDQo+IA0KPiBfX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiBzdG9ybSBtYWlsaW5nIGxp c3QNCj4gc3Rvcm1AaWV0Zi5vcmcNCj4gaHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0 aW5mby9zdG9ybQ0K From julian@satran.net Thu Sep 27 23:03:23 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6740321F841B for ; Thu, 27 Sep 2012 23:03:23 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.203 X-Spam-Level: X-Spam-Status: No, score=-5.203 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_QP_LONG_LINE=1.396, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RMvr6iQA-r6q for ; Thu, 27 Sep 2012 23:03:18 -0700 (PDT) Received: from eu1sys200aog104.obsmtp.com (eu1sys200aog104.obsmtp.com [207.126.144.117]) by ietfa.amsl.com (Postfix) with SMTP id 1C1F721F845D for ; Thu, 27 Sep 2012 23:03:16 -0700 (PDT) Received: from mail-we0-f172.google.com ([74.125.82.172]) (using TLSv1) by eu1sys200aob104.postini.com ([207.126.147.11]) with SMTP ID DSNKUGU9n0Ta/yIwxZDsdKf01f6dRX5JmTpM@postini.com; Fri, 28 Sep 2012 06:03:17 UTC Received: by weyu46 with SMTP id u46so1189080wey.31 for ; Thu, 27 Sep 2012 23:03:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=references:in-reply-to:mime-version:content-transfer-encoding :content-type:message-id:cc:x-mailer:from:subject:date:to :x-gm-message-state; bh=wiAI+vMNpW98FuTDl3Bj2Ew5NJoUU/AA7HRBcMRvzyg=; b=LpAlucldOIdkWfOLL6oBd8XepDbYxZHNFP2K9BJ6GePebO1w0iWTocYXugG8IY2qaL m4Jg2U9UuSivdwdecOzRkB+gEXtDflac/vsZ6rXzLSbzh4GbqFlFD5bUiMXN5h1BnOe0 czb1NJl6jqWDMlmza5jC6mxR2gxNNy3AVAoAtGQnL3KVJvE4X6A3lPFmQ/ZDpgfvBWyu T4M/sNoor6YKebf1ZfD9AzKllH6DCMtyraJ8BCAGm6YUX9ZcE5N0kH6RYL40CdTS8Srm zS7Q3pLBVD9OHFE3Go9EvFuKpZ5+xV+3UdXm+1BkG2ZlxVfYVj6kX4Rx2S96ouHayHYt DxKw== Received: by 10.216.241.202 with SMTP id g52mr2965952wer.212.1348812191560; Thu, 27 Sep 2012 23:03:11 -0700 (PDT) Received: from [192.168.0.102] (ip-54-21.sn2.eutelia.it. [83.211.54.21]) by mx.google.com with ESMTPS id fb20sm17360181wid.1.2012.09.27.23.03.08 (version=SSLv3 cipher=OTHER); Thu, 27 Sep 2012 23:03:09 -0700 (PDT) References: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> <4AE113CD-96D7-414D-8AAB-36ECF7FD89EA@satran.net> <8D3D17ACE214DC429325B2B98F3AE7120DE798F9@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE7993C@MX15A.corp.emc.com> In-Reply-To: Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Message-Id: X-Mailer: iPad Mail (10A403) From: Julian Satran Date: Fri, 28 Sep 2012 08:03:21 +0200 To: "Knight, Frederick" X-Gm-Message-State: ALoCoQnBQKz1aAQrUyyqcUjNUdyKdqkqWFX2jlXxRuneQr2vC5zQnuKC2rIdQUS3txaYJ4/V/ays Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release text - version 6 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Sep 2012 06:03:23 -0000 There is no such thing. And the reason the timeout is there is to limit the a= mount of state the targets have to keep. A target may choose to accept very l= arge timeouts and then do what you wanted done but beyond that only the powe= r-up ua is required. julo Sent from my iPad On 28 =D7=91=D7=A1=D7=A4=D7=98 2012, at 00:29, "Knight, Frederick" wrote: > I would think if the ISID is not recognized as an old one, then it must be= recognized as a new one. If it is a new one, then it will get a UA: 29/00 r= eflecting the first access after power on from the "new" initiator. >=20 > Is there something other than new or old? Is there such a thing as a midd= le-aged initiator? >=20 > Fred >=20 > -----Original Message----- > From: Black, David [mailto:david.black@emc.com]=20 > Sent: Thursday, September 27, 2012 11:56 AM > To: Knight, Frederick > Cc: storm@ietf.org > Subject: RE: [storm] SPC-2 reserve/release text - version 6 >=20 > Fred, >=20 > The problem is not that the device does not raise a UA, it's that the UA m= ay never be delivered. >=20 > A summary of how this can happen is that the new connection with the same I= SID may not be recognized as reinstating or continuing the session at the ta= rget if the timeouts have expired - the UA gets established, but nobody show= s up to pick it up in time, so the UA is discarded along with the rest of th= e session state and the target fails to recognize the initiator's subsequent= ISID reuse even though the initiator believes that it's re- establishing th= e session. >=20 > This is further complicated by the UA that indicates nexus state preservat= ion not being specified in SAM-2. >=20 > Thanks, > --David >=20 >=20 >> -----Original Message----- >> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] >> Sent: Thursday, September 27, 2012 11:11 AM >> To: Black, David; Julian Satran >> Cc: storm@ietf.org >> Subject: RE: [storm] SPC-2 reserve/release text - version 6 >>=20 >> I afraid I have to contest a little on this - if I understand this=20 >> discussion correctly. >>=20 >> If a SCSI device loses any state and does not raise a UNIT ATTENTION,=20 >> then it is BROKEN. It is not a matter of anything in iSCSI (3720); I=20 >> think SAM and SPC are pretty clear on this. If there is no UNIT=20 >> ATTENTION, then state SHALL be preserved; otherwise, all sorts of=20 >> stuff will never work (mode page settings, RESERVE commands, all sorts of= stuff). >>=20 >> Is someone really suggesting that some devices will drop state and NOT=20= >> tell the host, and somehow that it is OK to do that? >>=20 >> The suggest rewording says this is possible. >>=20 >> Fred >>=20 >> -----Original Message----- >> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf=20= >> Of Black, David >> Sent: Thursday, September 27, 2012 10:47 AM >> To: Julian Satran >> Cc: storm@ietf.org >> Subject: Re: [storm] SPC-2 reserve/release text - version 6 >>=20 >> Julian, >>=20 >>> The text in 4.4.3 states the obvious (there is no other speced=20 >>> method for continuation or reinstatement). Moreover if the session=20 >>> is reinstated after time2wait you create a need for a UA (or at=20 >>> least suggest one) and that was not there. >>=20 >> The UAs are courtesy of SAM-3 (and SAM-4). To encompass systems that=20 >> may not deliver these UAs and situations in which they don't occur=20 >> (e.g., the target doesn't recognize what happened as nexus=20 >> reestablishment, independent of what the initiator thinks it's doing) I t= hink the text should be rephrased as: >>=20 >> If a Unit Attention condition results from session reestablishment, >> the specific Unit Attention condition indicates whether nexus state >> was preserved, see Section 6.3.4 of [SAM4]. >>=20 >>> Just stating that creation/continuation of a session done according=20 >>> 6.x includes preservation of reserve/release state -that was already=20 >>> there implied by conformance to spc2 and the relation between it=20 >>> nexus and session. >>=20 >> Unfortunately, SPC-2 is sufficiently imprecise on this subject that=20 >> such an implication is at best a "preferred implementation approach". =20= >> It's not a requirement that can be safely stated in any stronger=20 >> fashion without the risk of rendering existing implementations non-compli= ant. >>=20 >>> The new text you suggest to 4.4 is merely a best practices=20 >>> recommendations and will need additions once the new "locking"=20 >>> through compare-and-swap get in widespread use. >>=20 >> It's definitely best practices recommendations - no argument there. >>=20 >> For context, what Julian is referring to is use of the COMPARE AND=20 >> WRITE command that performs an atomic operation at the target instead=20 >> of using RESERVE and RELEASE to create a critical section at the=20 >> initiator based on read and write commands. >>=20 >> COMPARE AND WRITE is in use - a sentence mentioning that command could=20= >> be added, although this best practices text is aimed primarily at uses=20= >> of reserve/release for long-term reservations. I think there's an=20 >> implicit "when a reservation is the desired behavior" implication to=20 >> the "persistent reservations SHOULD be used instead" text that I don't=20= >> think needs to be made explicit. OTOH, that "SHOULD" >> could be changed to a "should", or it might be even better to rewrite=20 >> that text >> as: >>=20 >> persistent reservations are suggested as an alternative, see Annex B >> of [SPC4]. >>=20 >> In contrast, "SHOULD NOT be used" is clearly justified for=20 >> reserve/release reservations by [SPC3]. >>=20 >> Thanks, >> --David >>=20 >>=20 >>> -----Original Message----- >>> From: Julian Satran [mailto:julian@satran.net] >>> Sent: Wednesday, September 26, 2012 10:25 PM >>> To: Black, David >>> Cc: storm@ietf.org >>> Subject: Re: [storm] SPC-2 reserve/release text - version 6 >>>=20 >>> David, >>>=20 >>> The text in 4.4.3 states the obvious (there is no other speced=20 >>> method for continuation or reinstatement). Moreover if the the=20 >>> session is reinstated after time2wait you create a need for a UA (or=20 >>> at least suggest one) and that was not there. Just stating that=20 >>> creation/continuation of a session done according 6.x includes=20 >>> preservation of reserve/release state -that was already there=20 >>> implied by conformance to spc2 and the relation between it nexus and ses= sion. >>>=20 >>> The new text you suggest to 4.4 is merely a best practices=20 >>> recomendations and will need additions once the new "locking"=20 >>> through compare-and-swap get in widespread use. >>> If instead we refrain from stating anything specific and state that=20 >>> target should maintain whatever state is required by SPC while a=20 >>> session is continued and may discard state when a session is=20 >>> terminated and may indicate that it discarded state through a ua you=20 >>> are on safer ground and do not create new requirement. >>>=20 >>> Sent from my iPad >>>=20 >>> On 27 =D7=91=D7=A1=D7=A4=D7=98 2012, at 00:25, "Black, David" wrote: >>>=20 >>>> Here's where I think we've arrived (I've made a few minor=20 >>>> editorial tweaks to what's been discussed). Despite all the=20 >>>> detailed discussion, I think this version 6 comes close to=20 >>>> Julian's suggestion >> that: >>>>=20 >>>>> i would mention only that the state that must be maintained while=20 >>>>> the session is maintained includes the reserve/release and leave=20 >>>>> it to >>> that. >>>>=20 >>>> Although we can't actually say "must" as that would be a new=20 >>>> requirement, and implementations differ in what they do about=20 >>>> reserve/release reservation state. The current language=20 >>>> characterizes retention of that state as the "preferred=20 >>>> implementation >> approach". >>>>=20 >>>> Beyond that, I think the two additional caveats about=20 >>>> reserve/release reservations (possible need for a reset,=20 >>>> potentially unexpected interaction with persistent reservations,=20 >>>> see below) are useful >> to include. >>>>=20 >>>> ----------------------------------------- >>>>=20 >>>> [1] For clarity, the following change should be made in >>>> 4.4.3.1: >>>>=20 >>>> OLD >>>> That is, it should perform session recovery as described in=20 >>>> Chapter 6. >>>> NEW >>>> That is, it should reinstate the session via iSCSI session=20 >>>> reinstatement (Section 6.3.5) or continue via session =20 >>>> continuation (Section 6.3.6). >>>> END >>>>=20 >>>> [2] Add the following sentence to the end of 4.4.3.1: >>>>=20 >>>> The specific Unit Attention condition that results from session >>>> reestablishment indicates whether nexus state was preserved, >>>> see Section 6.3.4 of [SAM4]. >>>>=20 >>>> [3] NEW TEXT: >>>>=20 >>>> 4.4.3.2. Reservations >>>>=20 >>>> There are two reservation management methods defined in the SCSI=20 >>>> standards, reserve/release reservations, based on the RESERVE and=20 >>>> RELEASE commands [SPC2], and persistent reservations, based on the=20 >>>> PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. >>>> Reserve/release reservations are obsolete [SPC3] and SHOULD NOT=20 >>>> be used; persistent reservations SHOULD be used instead. >>>>=20 >>>> State for persistent reservations is required to persist through=20 >>>> changes and failures at the iSCSI layer that result in I_T Nexus=20 >>>> failures, see [SPC3] for details and specific requirements. >>>>=20 >>>> In contrast, [SPC2] does not specify detailed persistence=20 >>>> requirements for reserve/release reservation state after an I_T=20 >>>> Nexus failure. Nonetheless, when reserve/release reservations are=20 >>>> supported by an iSCSI target, the preferred implementation=20 >>>> approach is to preserve reserve/release reservation state for=20 >>>> iSCSI session reinstatement (see Section 6.3.5) or session=20 >>>> continuation (see Section 6.3.6). >>>>=20 >>>> Two additional caveats apply to reserve/release reservations: >>>>=20 >>>> - When connection failure causes the iSCSI session to fail, and >>>> the session is not reinstated or continued, target retention >>>> of that session's reserve/release reservation state may >>>> require an initiator to issue a reset (e.g., LOGICAL UNIT RESET, >>>> see section 11.5) in order to remove that reservation state. >>>>=20 >>>> - Reserve/release reservations may not behave as expected when >>>> persistent reservations are also used on the same logical unit; >>>> see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE >>>> behavior" in [SPC4]. >>>>=20 >>>> Thanks, >>>> --David >>>> ---------------------------------------------------- >>>> David L. Black, Distinguished Engineer EMC Corporation, 176 South=20 >>>> St., Hopkinton, MA 01748 >>>> +1 (508) 293-7953 FAX: +1 (508) 293-7786 >>>> david.black@emc.com Mobile: +1 (978) 394-7754 >>>> ---------------------------------------------------- >>>>=20 >>>>=20 >>>> _______________________________________________ >>>> storm mailing list >>>> storm@ietf.org >>>> https://www.ietf.org/mailman/listinfo/storm >>=20 >> _______________________________________________ >> storm mailing list >> storm@ietf.org >> https://www.ietf.org/mailman/listinfo/storm > _______________________________________________ > storm mailing list > storm@ietf.org > https://www.ietf.org/mailman/listinfo/storm From Frederick.Knight@netapp.com Fri Sep 28 05:37:46 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CAA7521F8630 for ; Fri, 28 Sep 2012 05:37:46 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9SKZqXxT7CYi for ; Fri, 28 Sep 2012 05:37:45 -0700 (PDT) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id AA12221F862A for ; Fri, 28 Sep 2012 05:37:45 -0700 (PDT) X-IronPort-AV: E=Sophos;i="4.80,502,1344236400"; d="scan'208";a="695200646" Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx2-out.netapp.com with ESMTP; 28 Sep 2012 05:37:45 -0700 Received: from vmwexceht03-prd.hq.netapp.com (vmwexceht03-prd.hq.netapp.com [10.106.76.241]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8SCbitL021245; Fri, 28 Sep 2012 05:37:44 -0700 (PDT) Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.102]) by vmwexceht03-prd.hq.netapp.com ([10.106.76.241]) with mapi id 14.02.0309.002; Fri, 28 Sep 2012 05:37:44 -0700 From: "Knight, Frederick" To: Julian Satran Thread-Topic: [storm] SPC-2 reserve/release text - version 6 Thread-Index: Ac2cNcys4MzRK8PXQu651fVKNuEIxQAXDfsAABnkfIAADjDW0AAa4S6gACikjGD//nJjgIAAECkg Date: Fri, 28 Sep 2012 12:37:43 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> <4AE113CD-96D7-414D-8AAB-36ECF7FD89EA@satran.net> <8D3D17ACE214DC429325B2B98F3AE7120DE798F9@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE7993C@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.106.53.53] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release text - version 6 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Sep 2012 12:37:46 -0000 T0ssIHRoYXQncyB3aGF0IEkgdGhvdWdodC4NCg0KSWYgc3RhdGUgaXMgcmV0YWluZWQgKGl0J3Mg YW4gb2xkICJJIiByZWNvbm5lY3RpbmcpLCB0aGVuIHdoaWNoIFVBIGlzIGRlZmluZWQgYnkgU0FN LTMuDQoNCklmIHN0YXRlIGlzIG5vdCByZXRhaW5lZCAoaXQncyBhIG5ldyAiSSIpLCB0aGVuIHRo ZSBVQSBpcyBkZWZpbmVkIGJ5IFNBTS0zLg0KDQpTaW5jZSBib3RoIG9sZCAiSSJzIGFuZCBuZXcg IkkicyByZXN1bHQgaW4gdGhlICJJRiIgY29uZGl0aW9uIGJlaW5nIHRydWUsIHdoYXQgaXMgdGhl IGNvbmRpdGlvbiB0aGF0IHdvdWxkIG1ha2UgdGhlICJJRiIgZmFsc2U/DQoNClRoYXQgIklGIiBs YW5ndWFnZSBhbGxvd3MgYSB0YXJnZXQgdG8gbG9vc2Ugc3RhdGUgYW5kIG5vdCBkZWxpdmVyIGFu eSBVQS4NCg0KCUZyZWQNCg0KDQotLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogSnVs aWFuIFNhdHJhbiBbbWFpbHRvOmp1bGlhbkBzYXRyYW4ubmV0XSANClNlbnQ6IEZyaWRheSwgU2Vw dGVtYmVyIDI4LCAyMDEyIDI6MDMgQU0NClRvOiBLbmlnaHQsIEZyZWRlcmljaw0KQ2M6IEJsYWNr LCBEYXZpZDsgc3Rvcm1AaWV0Zi5vcmcNClN1YmplY3Q6IFJlOiBbc3Rvcm1dIFNQQy0yIHJlc2Vy dmUvcmVsZWFzZSB0ZXh0IC0gdmVyc2lvbiA2DQoNClRoZXJlIGlzIG5vIHN1Y2ggdGhpbmcuIEFu ZCB0aGUgcmVhc29uIHRoZSB0aW1lb3V0IGlzIHRoZXJlIGlzIHRvIGxpbWl0IHRoZSBhbW91bnQg b2Ygc3RhdGUgdGhlIHRhcmdldHMgaGF2ZSB0byBrZWVwLiBBIHRhcmdldCBtYXkgY2hvb3NlIHRv IGFjY2VwdCB2ZXJ5IGxhcmdlIHRpbWVvdXRzIGFuZCB0aGVuIGRvIHdoYXQgeW91IHdhbnRlZCBk b25lIGJ1dCBiZXlvbmQgdGhhdCBvbmx5IHRoZSBwb3dlci11cCB1YSBpcyByZXF1aXJlZC4NCg0K anVsbw0KDQpTZW50IGZyb20gbXkgaVBhZA0KDQpPbiAyOCDXkdeh16TXmCAyMDEyLCBhdCAwMDoy OSwgIktuaWdodCwgRnJlZGVyaWNrIiA8RnJlZGVyaWNrLktuaWdodEBuZXRhcHAuY29tPiB3cm90 ZToNCg0KPiBJIHdvdWxkIHRoaW5rIGlmIHRoZSBJU0lEIGlzIG5vdCByZWNvZ25pemVkIGFzIGFu IG9sZCBvbmUsIHRoZW4gaXQgbXVzdCBiZSByZWNvZ25pemVkIGFzIGEgbmV3IG9uZS4gIElmIGl0 IGlzIGEgbmV3IG9uZSwgdGhlbiBpdCB3aWxsIGdldCBhIFVBOiAyOS8wMCByZWZsZWN0aW5nIHRo ZSBmaXJzdCBhY2Nlc3MgYWZ0ZXIgcG93ZXIgb24gZnJvbSB0aGUgIm5ldyIgaW5pdGlhdG9yLg0K PiANCj4gSXMgdGhlcmUgc29tZXRoaW5nIG90aGVyIHRoYW4gbmV3IG9yIG9sZD8gIElzIHRoZXJl IHN1Y2ggYSB0aGluZyBhcyBhIG1pZGRsZS1hZ2VkIGluaXRpYXRvcj8NCj4gDQo+ICAgIEZyZWQN Cj4gDQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IEJsYWNrLCBEYXZpZCBb bWFpbHRvOmRhdmlkLmJsYWNrQGVtYy5jb21dDQo+IFNlbnQ6IFRodXJzZGF5LCBTZXB0ZW1iZXIg MjcsIDIwMTIgMTE6NTYgQU0NCj4gVG86IEtuaWdodCwgRnJlZGVyaWNrDQo+IENjOiBzdG9ybUBp ZXRmLm9yZw0KPiBTdWJqZWN0OiBSRTogW3N0b3JtXSBTUEMtMiByZXNlcnZlL3JlbGVhc2UgdGV4 dCAtIHZlcnNpb24gNg0KPiANCj4gRnJlZCwNCj4gDQo+IFRoZSBwcm9ibGVtIGlzIG5vdCB0aGF0 IHRoZSBkZXZpY2UgZG9lcyBub3QgcmFpc2UgYSBVQSwgaXQncyB0aGF0IHRoZSBVQSBtYXkgbmV2 ZXIgYmUgZGVsaXZlcmVkLg0KPiANCj4gQSBzdW1tYXJ5IG9mIGhvdyB0aGlzIGNhbiBoYXBwZW4g aXMgdGhhdCB0aGUgbmV3IGNvbm5lY3Rpb24gd2l0aCB0aGUgc2FtZSBJU0lEIG1heSBub3QgYmUg cmVjb2duaXplZCBhcyByZWluc3RhdGluZyBvciBjb250aW51aW5nIHRoZSBzZXNzaW9uIGF0IHRo ZSB0YXJnZXQgaWYgdGhlIHRpbWVvdXRzIGhhdmUgZXhwaXJlZCAtIHRoZSBVQSBnZXRzIGVzdGFi bGlzaGVkLCBidXQgbm9ib2R5IHNob3dzIHVwIHRvIHBpY2sgaXQgdXAgaW4gdGltZSwgc28gdGhl IFVBIGlzIGRpc2NhcmRlZCBhbG9uZyB3aXRoIHRoZSByZXN0IG9mIHRoZSBzZXNzaW9uIHN0YXRl IGFuZCB0aGUgdGFyZ2V0IGZhaWxzIHRvIHJlY29nbml6ZSB0aGUgaW5pdGlhdG9yJ3Mgc3Vic2Vx dWVudCBJU0lEIHJldXNlIGV2ZW4gdGhvdWdoIHRoZSBpbml0aWF0b3IgYmVsaWV2ZXMgdGhhdCBp dCdzIHJlLSBlc3RhYmxpc2hpbmcgdGhlIHNlc3Npb24uDQo+IA0KPiBUaGlzIGlzIGZ1cnRoZXIg Y29tcGxpY2F0ZWQgYnkgdGhlIFVBIHRoYXQgaW5kaWNhdGVzIG5leHVzIHN0YXRlIHByZXNlcnZh dGlvbiBub3QgYmVpbmcgc3BlY2lmaWVkIGluIFNBTS0yLg0KPiANCj4gVGhhbmtzLA0KPiAtLURh dmlkDQo+IA0KPiANCj4+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+PiBGcm9tOiBLbmln aHQsIEZyZWRlcmljayBbbWFpbHRvOkZyZWRlcmljay5LbmlnaHRAbmV0YXBwLmNvbV0NCj4+IFNl bnQ6IFRodXJzZGF5LCBTZXB0ZW1iZXIgMjcsIDIwMTIgMTE6MTEgQU0NCj4+IFRvOiBCbGFjaywg RGF2aWQ7IEp1bGlhbiBTYXRyYW4NCj4+IENjOiBzdG9ybUBpZXRmLm9yZw0KPj4gU3ViamVjdDog UkU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYNCj4+IA0K Pj4gSSBhZnJhaWQgSSBoYXZlIHRvIGNvbnRlc3QgYSBsaXR0bGUgb24gdGhpcyAtIGlmIEkgdW5k ZXJzdGFuZCB0aGlzIA0KPj4gZGlzY3Vzc2lvbiBjb3JyZWN0bHkuDQo+PiANCj4+IElmIGEgU0NT SSBkZXZpY2UgbG9zZXMgYW55IHN0YXRlIGFuZCBkb2VzIG5vdCByYWlzZSBhIFVOSVQgQVRURU5U SU9OLCANCj4+IHRoZW4gaXQgaXMgQlJPS0VOLiAgSXQgaXMgbm90IGEgbWF0dGVyIG9mIGFueXRo aW5nIGluIGlTQ1NJICgzNzIwKTsgSSANCj4+IHRoaW5rIFNBTSBhbmQgU1BDIGFyZSBwcmV0dHkg Y2xlYXIgb24gdGhpcy4gIElmIHRoZXJlIGlzIG5vIFVOSVQgDQo+PiBBVFRFTlRJT04sIHRoZW4g c3RhdGUgU0hBTEwgYmUgcHJlc2VydmVkOyBvdGhlcndpc2UsIGFsbCBzb3J0cyBvZiANCj4+IHN0 dWZmIHdpbGwgbmV2ZXIgd29yayAobW9kZSBwYWdlIHNldHRpbmdzLCBSRVNFUlZFIGNvbW1hbmRz LCBhbGwgc29ydHMgb2Ygc3R1ZmYpLg0KPj4gDQo+PiBJcyBzb21lb25lIHJlYWxseSBzdWdnZXN0 aW5nIHRoYXQgc29tZSBkZXZpY2VzIHdpbGwgZHJvcCBzdGF0ZSBhbmQgDQo+PiBOT1QgdGVsbCB0 aGUgaG9zdCwgYW5kIHNvbWVob3cgdGhhdCBpdCBpcyBPSyB0byBkbyB0aGF0Pw0KPj4gDQo+PiBU aGUgc3VnZ2VzdCByZXdvcmRpbmcgc2F5cyB0aGlzIGlzIHBvc3NpYmxlLg0KPj4gDQo+PiAgICBG cmVkDQo+PiANCj4+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+PiBGcm9tOiBzdG9ybS1i b3VuY2VzQGlldGYub3JnIFttYWlsdG86c3Rvcm0tYm91bmNlc0BpZXRmLm9yZ10gT24gDQo+PiBC ZWhhbGYgT2YgQmxhY2ssIERhdmlkDQo+PiBTZW50OiBUaHVyc2RheSwgU2VwdGVtYmVyIDI3LCAy MDEyIDEwOjQ3IEFNDQo+PiBUbzogSnVsaWFuIFNhdHJhbg0KPj4gQ2M6IHN0b3JtQGlldGYub3Jn DQo+PiBTdWJqZWN0OiBSZTogW3N0b3JtXSBTUEMtMiByZXNlcnZlL3JlbGVhc2UgdGV4dCAtIHZl cnNpb24gNg0KPj4gDQo+PiBKdWxpYW4sDQo+PiANCj4+PiBUaGUgdGV4dCBpbiA0LjQuMyBzdGF0 ZXMgdGhlIG9idmlvdXMgKHRoZXJlIGlzIG5vIG90aGVyIHNwZWNlZCANCj4+PiBtZXRob2QgZm9y IGNvbnRpbnVhdGlvbiBvciByZWluc3RhdGVtZW50KS4gTW9yZW92ZXIgaWYgdGhlIHNlc3Npb24g DQo+Pj4gaXMgcmVpbnN0YXRlZCBhZnRlciB0aW1lMndhaXQgeW91IGNyZWF0ZSBhIG5lZWQgZm9y IGEgVUEgKG9yIGF0IA0KPj4+IGxlYXN0IHN1Z2dlc3Qgb25lKSBhbmQgdGhhdCB3YXMgbm90IHRo ZXJlLg0KPj4gDQo+PiBUaGUgVUFzIGFyZSBjb3VydGVzeSBvZiBTQU0tMyAoYW5kIFNBTS00KS4g IFRvIGVuY29tcGFzcyBzeXN0ZW1zIHRoYXQgDQo+PiBtYXkgbm90IGRlbGl2ZXIgdGhlc2UgVUFz IGFuZCBzaXR1YXRpb25zIGluIHdoaWNoIHRoZXkgZG9uJ3Qgb2NjdXIgDQo+PiAoZS5nLiwgdGhl IHRhcmdldCBkb2Vzbid0IHJlY29nbml6ZSB3aGF0IGhhcHBlbmVkIGFzIG5leHVzIA0KPj4gcmVl c3RhYmxpc2htZW50LCBpbmRlcGVuZGVudCBvZiB3aGF0IHRoZSBpbml0aWF0b3IgdGhpbmtzIGl0 J3MgZG9pbmcpIEkgdGhpbmsgdGhlIHRleHQgc2hvdWxkIGJlIHJlcGhyYXNlZCBhczoNCj4+IA0K Pj4gICAgIElmIGEgVW5pdCBBdHRlbnRpb24gY29uZGl0aW9uIHJlc3VsdHMgZnJvbSBzZXNzaW9u IHJlZXN0YWJsaXNobWVudCwNCj4+ICAgICB0aGUgc3BlY2lmaWMgVW5pdCBBdHRlbnRpb24gY29u ZGl0aW9uIGluZGljYXRlcyB3aGV0aGVyIG5leHVzIHN0YXRlDQo+PiAgICAgd2FzIHByZXNlcnZl ZCwgc2VlIFNlY3Rpb24gNi4zLjQgb2YgW1NBTTRdLg0KPj4gDQo+Pj4gSnVzdCBzdGF0aW5nIHRo YXQgY3JlYXRpb24vY29udGludWF0aW9uIG9mIGEgc2Vzc2lvbiBkb25lIGFjY29yZGluZyANCj4+ PiA2LnggaW5jbHVkZXMgcHJlc2VydmF0aW9uIG9mIHJlc2VydmUvcmVsZWFzZSBzdGF0ZSAtdGhh dCB3YXMgYWxyZWFkeSANCj4+PiB0aGVyZSBpbXBsaWVkIGJ5IGNvbmZvcm1hbmNlIHRvIHNwYzIg YW5kIHRoZSByZWxhdGlvbiBiZXR3ZWVuIGl0IA0KPj4+IG5leHVzIGFuZCBzZXNzaW9uLg0KPj4g DQo+PiBVbmZvcnR1bmF0ZWx5LCBTUEMtMiBpcyBzdWZmaWNpZW50bHkgaW1wcmVjaXNlIG9uIHRo aXMgc3ViamVjdCB0aGF0IA0KPj4gc3VjaCBhbiBpbXBsaWNhdGlvbiBpcyBhdCBiZXN0IGEgInBy ZWZlcnJlZCBpbXBsZW1lbnRhdGlvbiBhcHByb2FjaCIuDQo+PiBJdCdzIG5vdCBhIHJlcXVpcmVt ZW50IHRoYXQgY2FuIGJlIHNhZmVseSBzdGF0ZWQgaW4gYW55IHN0cm9uZ2VyIA0KPj4gZmFzaGlv biB3aXRob3V0IHRoZSByaXNrIG9mIHJlbmRlcmluZyBleGlzdGluZyBpbXBsZW1lbnRhdGlvbnMg bm9uLWNvbXBsaWFudC4NCj4+IA0KPj4+IFRoZSBuZXcgdGV4dCB5b3Ugc3VnZ2VzdCB0byA0LjQg aXMgbWVyZWx5IGEgYmVzdCBwcmFjdGljZXMgDQo+Pj4gcmVjb21tZW5kYXRpb25zIGFuZCB3aWxs IG5lZWQgYWRkaXRpb25zIG9uY2UgdGhlIG5ldyAibG9ja2luZyINCj4+PiB0aHJvdWdoIGNvbXBh cmUtYW5kLXN3YXAgZ2V0IGluIHdpZGVzcHJlYWQgdXNlLg0KPj4gDQo+PiBJdCdzIGRlZmluaXRl bHkgYmVzdCBwcmFjdGljZXMgcmVjb21tZW5kYXRpb25zIC0gbm8gYXJndW1lbnQgdGhlcmUuDQo+ PiANCj4+IEZvciBjb250ZXh0LCB3aGF0IEp1bGlhbiBpcyByZWZlcnJpbmcgdG8gaXMgdXNlIG9m IHRoZSBDT01QQVJFIEFORCANCj4+IFdSSVRFIGNvbW1hbmQgdGhhdCBwZXJmb3JtcyBhbiBhdG9t aWMgb3BlcmF0aW9uIGF0IHRoZSB0YXJnZXQgaW5zdGVhZCANCj4+IG9mIHVzaW5nIFJFU0VSVkUg YW5kIFJFTEVBU0UgdG8gY3JlYXRlIGEgY3JpdGljYWwgc2VjdGlvbiBhdCB0aGUgDQo+PiBpbml0 aWF0b3IgYmFzZWQgb24gcmVhZCBhbmQgd3JpdGUgY29tbWFuZHMuDQo+PiANCj4+IENPTVBBUkUg QU5EIFdSSVRFIGlzIGluIHVzZSAtIGEgc2VudGVuY2UgbWVudGlvbmluZyB0aGF0IGNvbW1hbmQg DQo+PiBjb3VsZCBiZSBhZGRlZCwgYWx0aG91Z2ggdGhpcyBiZXN0IHByYWN0aWNlcyB0ZXh0IGlz IGFpbWVkIHByaW1hcmlseSANCj4+IGF0IHVzZXMgb2YgcmVzZXJ2ZS9yZWxlYXNlIGZvciBsb25n LXRlcm0gcmVzZXJ2YXRpb25zLiAgSSB0aGluayANCj4+IHRoZXJlJ3MgYW4gaW1wbGljaXQgIndo ZW4gYSByZXNlcnZhdGlvbiBpcyB0aGUgZGVzaXJlZCBiZWhhdmlvciIgDQo+PiBpbXBsaWNhdGlv biB0byB0aGUgInBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIFNIT1VMRCBiZSB1c2VkIGluc3RlYWQi IA0KPj4gdGV4dCB0aGF0IEkgZG9uJ3QgdGhpbmsgbmVlZHMgdG8gYmUgbWFkZSBleHBsaWNpdC4g IE9UT0gsIHRoYXQgIlNIT1VMRCINCj4+IGNvdWxkIGJlIGNoYW5nZWQgdG8gYSAic2hvdWxkIiwg b3IgaXQgbWlnaHQgYmUgZXZlbiBiZXR0ZXIgdG8gcmV3cml0ZSANCj4+IHRoYXQgdGV4dA0KPj4g YXM6DQo+PiANCj4+ICAgIHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIGFyZSBzdWdnZXN0ZWQgYXMg YW4gYWx0ZXJuYXRpdmUsIHNlZSBBbm5leCBCDQo+PiAgICBvZiBbU1BDNF0uDQo+PiANCj4+IElu IGNvbnRyYXN0LCAiU0hPVUxEIE5PVCBiZSB1c2VkIiBpcyBjbGVhcmx5IGp1c3RpZmllZCBmb3Ig DQo+PiByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zIGJ5IFtTUEMzXS4NCj4+IA0KPj4gVGhh bmtzLA0KPj4gLS1EYXZpZA0KPj4gDQo+PiANCj4+PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0t LQ0KPj4+IEZyb206IEp1bGlhbiBTYXRyYW4gW21haWx0bzpqdWxpYW5Ac2F0cmFuLm5ldF0NCj4+ PiBTZW50OiBXZWRuZXNkYXksIFNlcHRlbWJlciAyNiwgMjAxMiAxMDoyNSBQTQ0KPj4+IFRvOiBC bGFjaywgRGF2aWQNCj4+PiBDYzogc3Rvcm1AaWV0Zi5vcmcNCj4+PiBTdWJqZWN0OiBSZTogW3N0 b3JtXSBTUEMtMiByZXNlcnZlL3JlbGVhc2UgdGV4dCAtIHZlcnNpb24gNg0KPj4+IA0KPj4+IERh dmlkLA0KPj4+IA0KPj4+IFRoZSB0ZXh0IGluIDQuNC4zIHN0YXRlcyB0aGUgb2J2aW91cyAodGhl cmUgaXMgbm8gb3RoZXIgc3BlY2VkIA0KPj4+IG1ldGhvZCBmb3IgY29udGludWF0aW9uIG9yIHJl aW5zdGF0ZW1lbnQpLiBNb3Jlb3ZlciBpZiB0aGUgdGhlIA0KPj4+IHNlc3Npb24gaXMgcmVpbnN0 YXRlZCBhZnRlciB0aW1lMndhaXQgeW91IGNyZWF0ZSBhIG5lZWQgZm9yIGEgVUEgKG9yIA0KPj4+ IGF0IGxlYXN0IHN1Z2dlc3Qgb25lKSBhbmQgdGhhdCB3YXMgbm90IHRoZXJlLiBKdXN0IHN0YXRp bmcgdGhhdCANCj4+PiBjcmVhdGlvbi9jb250aW51YXRpb24gb2YgYSBzZXNzaW9uIGRvbmUgYWNj b3JkaW5nIDYueCBpbmNsdWRlcyANCj4+PiBwcmVzZXJ2YXRpb24gb2YgcmVzZXJ2ZS9yZWxlYXNl IHN0YXRlIC10aGF0IHdhcyBhbHJlYWR5IHRoZXJlIA0KPj4+IGltcGxpZWQgYnkgY29uZm9ybWFu Y2UgdG8gc3BjMiBhbmQgdGhlIHJlbGF0aW9uIGJldHdlZW4gaXQgbmV4dXMgYW5kIHNlc3Npb24u DQo+Pj4gDQo+Pj4gVGhlIG5ldyB0ZXh0IHlvdSBzdWdnZXN0IHRvIDQuNCBpcyBtZXJlbHkgYSBi ZXN0IHByYWN0aWNlcyANCj4+PiByZWNvbWVuZGF0aW9ucyBhbmQgd2lsbCBuZWVkIGFkZGl0aW9u cyBvbmNlIHRoZSBuZXcgImxvY2tpbmciDQo+Pj4gdGhyb3VnaCBjb21wYXJlLWFuZC1zd2FwIGdl dCBpbiB3aWRlc3ByZWFkIHVzZS4NCj4+PiBJZiBpbnN0ZWFkIHdlIHJlZnJhaW4gZnJvbSBzdGF0 aW5nIGFueXRoaW5nIHNwZWNpZmljIGFuZCBzdGF0ZSB0aGF0IA0KPj4+IHRhcmdldCBzaG91bGQg bWFpbnRhaW4gd2hhdGV2ZXIgc3RhdGUgaXMgcmVxdWlyZWQgYnkgU1BDIHdoaWxlIGEgDQo+Pj4g c2Vzc2lvbiBpcyBjb250aW51ZWQgYW5kIG1heSBkaXNjYXJkIHN0YXRlIHdoZW4gYSBzZXNzaW9u IGlzIA0KPj4+IHRlcm1pbmF0ZWQgYW5kIG1heSBpbmRpY2F0ZSB0aGF0IGl0IGRpc2NhcmRlZCBz dGF0ZSB0aHJvdWdoIGEgdWEgeW91IA0KPj4+IGFyZSBvbiBzYWZlciBncm91bmQgYW5kIGRvIG5v dCBjcmVhdGUgbmV3IHJlcXVpcmVtZW50Lg0KPj4+IA0KPj4+IFNlbnQgZnJvbSBteSBpUGFkDQo+ Pj4gDQo+Pj4gT24gMjcg15HXodek15ggMjAxMiwgYXQgMDA6MjUsICJCbGFjaywgRGF2aWQiIDxk YXZpZC5ibGFja0BlbWMuY29tPiB3cm90ZToNCj4+PiANCj4+Pj4gSGVyZSdzIHdoZXJlIEkgdGhp bmsgd2UndmUgYXJyaXZlZCAoSSd2ZSBtYWRlIGEgZmV3IG1pbm9yIGVkaXRvcmlhbCANCj4+Pj4g dHdlYWtzIHRvIHdoYXQncyBiZWVuIGRpc2N1c3NlZCkuICBEZXNwaXRlIGFsbCB0aGUgZGV0YWls ZWQgDQo+Pj4+IGRpc2N1c3Npb24sIEkgdGhpbmsgdGhpcyB2ZXJzaW9uIDYgY29tZXMgY2xvc2Ug dG8gSnVsaWFuJ3MgDQo+Pj4+IHN1Z2dlc3Rpb24NCj4+IHRoYXQ6DQo+Pj4+IA0KPj4+Pj4gaSB3 b3VsZCBtZW50aW9uIG9ubHkgdGhhdCB0aGUgc3RhdGUgdGhhdCBtdXN0IGJlIG1haW50YWluZWQg d2hpbGUgDQo+Pj4+PiB0aGUgc2Vzc2lvbiBpcyBtYWludGFpbmVkIGluY2x1ZGVzIHRoZSByZXNl cnZlL3JlbGVhc2UgYW5kIGxlYXZlIA0KPj4+Pj4gaXQgdG8NCj4+PiB0aGF0Lg0KPj4+PiANCj4+ Pj4gQWx0aG91Z2ggd2UgY2FuJ3QgYWN0dWFsbHkgc2F5ICJtdXN0IiBhcyB0aGF0IHdvdWxkIGJl IGEgbmV3IA0KPj4+PiByZXF1aXJlbWVudCwgYW5kIGltcGxlbWVudGF0aW9ucyBkaWZmZXIgaW4g d2hhdCB0aGV5IGRvIGFib3V0IA0KPj4+PiByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb24gc3Rh dGUuICBUaGUgY3VycmVudCBsYW5ndWFnZSANCj4+Pj4gY2hhcmFjdGVyaXplcyByZXRlbnRpb24g b2YgdGhhdCBzdGF0ZSBhcyB0aGUgInByZWZlcnJlZCANCj4+Pj4gaW1wbGVtZW50YXRpb24NCj4+ IGFwcHJvYWNoIi4NCj4+Pj4gDQo+Pj4+IEJleW9uZCB0aGF0LCBJIHRoaW5rIHRoZSB0d28gYWRk aXRpb25hbCBjYXZlYXRzIGFib3V0IA0KPj4+PiByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25z IChwb3NzaWJsZSBuZWVkIGZvciBhIHJlc2V0LCANCj4+Pj4gcG90ZW50aWFsbHkgdW5leHBlY3Rl ZCBpbnRlcmFjdGlvbiB3aXRoIHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zLCANCj4+Pj4gc2VlIGJl bG93KSBhcmUgdXNlZnVsDQo+PiB0byBpbmNsdWRlLg0KPj4+PiANCj4+Pj4gLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4+Pj4gDQo+Pj4+IFsxXSBGb3IgY2xhcml0 eSwgdGhlIGZvbGxvd2luZyBjaGFuZ2Ugc2hvdWxkIGJlIG1hZGUgaW4NCj4+Pj4gNC40LjMuMToN Cj4+Pj4gDQo+Pj4+IE9MRA0KPj4+PiBUaGF0IGlzLCBpdCBzaG91bGQgcGVyZm9ybSBzZXNzaW9u IHJlY292ZXJ5IGFzICBkZXNjcmliZWQgaW4gDQo+Pj4+IENoYXB0ZXIgNi4NCj4+Pj4gTkVXDQo+ Pj4+IFRoYXQgaXMsIGl0IHNob3VsZCByZWluc3RhdGUgdGhlIHNlc3Npb24gdmlhIGlTQ1NJIHNl c3Npb24gDQo+Pj4+IHJlaW5zdGF0ZW1lbnQgKFNlY3Rpb24gNi4zLjUpIG9yIGNvbnRpbnVlIHZp YSBzZXNzaW9uIGNvbnRpbnVhdGlvbiANCj4+Pj4gKFNlY3Rpb24gNi4zLjYpLg0KPj4+PiBFTkQN Cj4+Pj4gDQo+Pj4+IFsyXSBBZGQgdGhlIGZvbGxvd2luZyBzZW50ZW5jZSB0byB0aGUgZW5kIG9m IDQuNC4zLjE6DQo+Pj4+IA0KPj4+PiAgICBUaGUgc3BlY2lmaWMgVW5pdCBBdHRlbnRpb24gY29u ZGl0aW9uIHRoYXQgcmVzdWx0cyBmcm9tIHNlc3Npb24NCj4+Pj4gICAgcmVlc3RhYmxpc2htZW50 IGluZGljYXRlcyB3aGV0aGVyIG5leHVzIHN0YXRlIHdhcyBwcmVzZXJ2ZWQsDQo+Pj4+ICAgIHNl ZSBTZWN0aW9uIDYuMy40IG9mIFtTQU00XS4NCj4+Pj4gDQo+Pj4+IFszXSBORVcgVEVYVDoNCj4+ Pj4gDQo+Pj4+IDQuNC4zLjIuIFJlc2VydmF0aW9ucw0KPj4+PiANCj4+Pj4gVGhlcmUgYXJlIHR3 byByZXNlcnZhdGlvbiBtYW5hZ2VtZW50IG1ldGhvZHMgZGVmaW5lZCBpbiB0aGUgU0NTSSANCj4+ Pj4gc3RhbmRhcmRzLCByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zLCBiYXNlZCBvbiB0aGUg UkVTRVJWRSBhbmQgDQo+Pj4+IFJFTEVBU0UgY29tbWFuZHMgW1NQQzJdLCBhbmQgcGVyc2lzdGVu dCByZXNlcnZhdGlvbnMsIGJhc2VkIG9uIHRoZSANCj4+Pj4gUEVSU0lTVEVOVCBSRVNFUlZFIElO IGFuZCBQRVJTSVNURU5UIFJFU0VSVkUgT1VUIGNvbW1hbmRzIFtTUEMzXS4NCj4+Pj4gUmVzZXJ2 ZS9yZWxlYXNlIHJlc2VydmF0aW9ucyBhcmUgb2Jzb2xldGUgW1NQQzNdIGFuZCBTSE9VTEQgTk9U IGJlIA0KPj4+PiB1c2VkOyBwZXJzaXN0ZW50IHJlc2VydmF0aW9ucyBTSE9VTEQgYmUgdXNlZCBp bnN0ZWFkLg0KPj4+PiANCj4+Pj4gU3RhdGUgZm9yIHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIGlz IHJlcXVpcmVkIHRvIHBlcnNpc3QgIHRocm91Z2ggDQo+Pj4+IGNoYW5nZXMgYW5kIGZhaWx1cmVz IGF0IHRoZSBpU0NTSSBsYXllciB0aGF0IHJlc3VsdCBpbiAgSV9UIE5leHVzIA0KPj4+PiBmYWls dXJlcywgc2VlIFtTUEMzXSBmb3IgZGV0YWlscyBhbmQgc3BlY2lmaWMgcmVxdWlyZW1lbnRzLg0K Pj4+PiANCj4+Pj4gSW4gY29udHJhc3QsIFtTUEMyXSBkb2VzIG5vdCBzcGVjaWZ5IGRldGFpbGVk IHBlcnNpc3RlbmNlIA0KPj4+PiByZXF1aXJlbWVudHMgZm9yIHJlc2VydmUvcmVsZWFzZSByZXNl cnZhdGlvbiBzdGF0ZSBhZnRlciBhbiBJX1QgDQo+Pj4+IE5leHVzIGZhaWx1cmUuICBOb25ldGhl bGVzcywgd2hlbiByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zIGFyZSANCj4+Pj4gc3VwcG9y dGVkIGJ5IGFuIGlTQ1NJIHRhcmdldCwgdGhlIHByZWZlcnJlZCBpbXBsZW1lbnRhdGlvbiBhcHBy b2FjaCANCj4+Pj4gaXMgdG8gcHJlc2VydmUgcmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9uIHN0 YXRlIGZvciBpU0NTSSBzZXNzaW9uIA0KPj4+PiByZWluc3RhdGVtZW50IChzZWUgU2VjdGlvbiA2 LjMuNSkgb3Igc2Vzc2lvbiBjb250aW51YXRpb24gIChzZWUgDQo+Pj4+IFNlY3Rpb24gNi4zLjYp Lg0KPj4+PiANCj4+Pj4gVHdvIGFkZGl0aW9uYWwgY2F2ZWF0cyBhcHBseSB0byByZXNlcnZlL3Jl bGVhc2UgcmVzZXJ2YXRpb25zOg0KPj4+PiANCj4+Pj4gLSBXaGVuIGNvbm5lY3Rpb24gZmFpbHVy ZSBjYXVzZXMgdGhlIGlTQ1NJIHNlc3Npb24gdG8gZmFpbCwgYW5kDQo+Pj4+ICAgICB0aGUgc2Vz c2lvbiBpcyBub3QgcmVpbnN0YXRlZCBvciBjb250aW51ZWQsIHRhcmdldCByZXRlbnRpb24NCj4+ Pj4gICAgIG9mIHRoYXQgc2Vzc2lvbidzIHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbiBzdGF0 ZSBtYXkNCj4+Pj4gICAgIHJlcXVpcmUgYW4gaW5pdGlhdG9yIHRvIGlzc3VlIGEgcmVzZXQgKGUu Zy4sIExPR0lDQUwgVU5JVCBSRVNFVCwNCj4+Pj4gICAgIHNlZSBzZWN0aW9uIDExLjUpIGluIG9y ZGVyIHRvIHJlbW92ZSB0aGF0IHJlc2VydmF0aW9uIHN0YXRlLg0KPj4+PiANCj4+Pj4gLSBSZXNl cnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zIG1heSBub3QgYmVoYXZlIGFzIGV4cGVjdGVkIHdoZW4N Cj4+Pj4gICAgIHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIGFyZSBhbHNvIHVzZWQgb24gdGhlIHNh bWUgbG9naWNhbCB1bml0Ow0KPj4+PiAgICAgc2VlIHRoZSBkaXNjdXNzaW9uIG9mICJFeGNlcHRp b25zIHRvIFNQQy0yIFJFU0VSVkUgYW5kIFJFTEVBU0UNCj4+Pj4gICAgIGJlaGF2aW9yIiBpbiBb U1BDNF0uDQo+Pj4+IA0KPj4+PiBUaGFua3MsDQo+Pj4+IC0tRGF2aWQNCj4+Pj4gLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPj4+PiBEYXZpZCBM LiBCbGFjaywgRGlzdGluZ3Vpc2hlZCBFbmdpbmVlciBFTUMgQ29ycG9yYXRpb24sIDE3NiBTb3V0 aCANCj4+Pj4gU3QuLCBIb3BraW50b24sIE1BICAwMTc0OA0KPj4+PiArMSAoNTA4KSAyOTMtNzk1 MyAgICAgICAgICAgICBGQVg6ICsxICg1MDgpIDI5My03Nzg2DQo+Pj4+IGRhdmlkLmJsYWNrQGVt Yy5jb20gICAgICAgIE1vYmlsZTogKzEgKDk3OCkgMzk0LTc3NTQNCj4+Pj4gLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPj4+PiANCj4+Pj4gDQo+ Pj4+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+Pj4+ IHN0b3JtIG1haWxpbmcgbGlzdA0KPj4+PiBzdG9ybUBpZXRmLm9yZw0KPj4+PiBodHRwczovL3d3 dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3N0b3JtDQo+PiANCj4+IF9fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+PiBzdG9ybSBtYWlsaW5nIGxpc3QN Cj4+IHN0b3JtQGlldGYub3JnDQo+PiBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3Rp bmZvL3N0b3JtDQo+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fDQo+IHN0b3JtIG1haWxpbmcgbGlzdA0KPiBzdG9ybUBpZXRmLm9yZw0KPiBodHRwczovL3d3 dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3N0b3JtDQo= From david.black@emc.com Fri Sep 28 06:51:31 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BFE8821F859A for ; Fri, 28 Sep 2012 06:51:28 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.584 X-Spam-Level: X-Spam-Status: No, score=-102.584 tagged_above=-999 required=5 tests=[AWL=0.015, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Q5IlX9SjCLvy for ; Fri, 28 Sep 2012 06:51:26 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 50B3E21F85A2 for ; Fri, 28 Sep 2012 06:51:25 -0700 (PDT) Received: from hop04-l1d11-si04.isus.emc.com (HOP04-L1D11-SI04.isus.emc.com [10.254.111.24]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8SDpPZo012489 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 28 Sep 2012 09:51:25 -0400 Received: from mailhub.lss.emc.com (mailhubhoprd02.lss.emc.com [10.254.221.253]) by hop04-l1d11-si04.isus.emc.com (RSA Interceptor); Fri, 28 Sep 2012 09:51:07 -0400 Received: from mxhub02.corp.emc.com (mxhub02.corp.emc.com [10.254.141.104]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8SDp641020145; Fri, 28 Sep 2012 09:51:06 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub02.corp.emc.com ([10.254.141.104]) with mapi; Fri, 28 Sep 2012 09:51:06 -0400 From: "Black, David" To: "Knight, Frederick" Date: Fri, 28 Sep 2012 09:51:04 -0400 Thread-Topic: [storm] SPC-2 reserve/release text - version 6 Thread-Index: Ac2cNcys4MzRK8PXQu651fVKNuEIxQAXDfsAABnkfIAADjDW0AAa4S6gACikjGD//nJjgIAAECkggAAHbKA= Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE79B42@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> <4AE113CD-96D7-414D-8AAB-36ECF7FD89EA@satran.net> <8D3D17ACE214DC429325B2B98F3AE7120DE798F9@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE7993C@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-EMM-MHVC: 1 Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release text - version 6 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Sep 2012 13:51:31 -0000 RnJlZCwNCg0KPiBJZiBzdGF0ZSBpcyByZXRhaW5lZCAoaXQncyBhbiBvbGQgIkkiIHJlY29ubmVj dGluZyksIHRoZW4gd2hpY2ggVUEgaXMgZGVmaW5lZA0KPiBieSBTQU0tMy4NCj4gDQo+IElmIHN0 YXRlIGlzIG5vdCByZXRhaW5lZCAoaXQncyBhIG5ldyAiSSIpLCB0aGVuIHRoZSBVQSBpcyBkZWZp bmVkIGJ5IFNBTS0zLg0KPg0KPiBTaW5jZSBib3RoIG9sZCAiSSJzIGFuZCBuZXcgIkkicyByZXN1 bHQgaW4gdGhlICJJRiIgY29uZGl0aW9uIGJlaW5nIHRydWUsIHdoYXQNCj4gaXMgdGhlIGNvbmRp dGlvbiB0aGF0IHdvdWxkIG1ha2UgdGhlICJJRiIgZmFsc2U/DQoNCllvdSBhbG1vc3QgYW5zd2Vy ZWQgeW91ciBvd24gcXVlc3Rpb24gOy0pLg0KDQpUaGUgYW5zd2VyIGlzIC4uLi4gKGRydW0gcm9s bCkgLi4uIGEgdGFyZ2V0IGltcGxlbWVudGF0aW9uIGJhc2VkIG9uIFJGQyAzNzIwDQpTQU0tMiwg YWx0aG91Z2ggSSB3b3VsZCBob3BlIHRoYXQgc3VjaCBpbXBsZW1lbnRhdGlvbnMgYXQgbGVhc3Qg cGlja2VkDQp1cCBSRkMgNTA0OCdzIGNoYW5nZXMuICANCg0KVGhlIElfVCBOZXh1cyBMb3NzIGNv bmNlcHQgb24gd2hpY2ggdGhpcyBVQSBkaXNjdXNzaW9uIGlzIGJhc2VkIGlzIHBhcnQgb2YNCnRo ZSBTQ1NJIGV2ZW50cyBhbmQgZXZlbnQgbm90aWZpY2F0aW9ucyBtb2RlbCB0aGF0IHdhcyBpbnRy b2R1Y2VkIGluIFNBTS0zLA0KYnV0IG5laXRoZXIgUkZDIDM3MjAgbm9yIFJGQyA1MDQ4IHJlZmVy ZW5jZXMgU0FNLTMuDQoNClRoZXJlIHdlcmUgZGVmaW5pdGVseSBpbXBsZW1lbnRhdGlvbnMgb2Yg aVNDU0kgYmFzZWQgb24gU0FNLTIgLSBJIGFncmVlIHdpdGgNCkp1bGlhbiB0aGF0IHdlIHNob3Vs ZCAqbm90KiBiZSBpbnRyb2R1Y2luZyBhIFVBIHJlcXVpcmVtZW50IGluIHRoaXMgZHJhZnQNCnRo YXQgbWlnaHQgYnJlYWsgc3VjaCBpbXBsZW1lbnRhdGlvbnMuDQoNClRoYW5rcywNCi0tRGF2aWQN Cg0KPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBLbmlnaHQsIEZyZWRlcmlj ayBbbWFpbHRvOkZyZWRlcmljay5LbmlnaHRAbmV0YXBwLmNvbV0NCj4gU2VudDogRnJpZGF5LCBT ZXB0ZW1iZXIgMjgsIDIwMTIgODozOCBBTQ0KPiBUbzogSnVsaWFuIFNhdHJhbg0KPiBDYzogQmxh Y2ssIERhdmlkOyBzdG9ybUBpZXRmLm9yZw0KPiBTdWJqZWN0OiBSRTogW3N0b3JtXSBTUEMtMiBy ZXNlcnZlL3JlbGVhc2UgdGV4dCAtIHZlcnNpb24gNg0KPiANCj4gT0ssIHRoYXQncyB3aGF0IEkg dGhvdWdodC4NCj4gDQo+IElmIHN0YXRlIGlzIHJldGFpbmVkIChpdCdzIGFuIG9sZCAiSSIgcmVj b25uZWN0aW5nKSwgdGhlbiB3aGljaCBVQSBpcyBkZWZpbmVkDQo+IGJ5IFNBTS0zLg0KPiANCj4g SWYgc3RhdGUgaXMgbm90IHJldGFpbmVkIChpdCdzIGEgbmV3ICJJIiksIHRoZW4gdGhlIFVBIGlz IGRlZmluZWQgYnkgU0FNLTMuDQo+IA0KPiBTaW5jZSBib3RoIG9sZCAiSSJzIGFuZCBuZXcgIkki cyByZXN1bHQgaW4gdGhlICJJRiIgY29uZGl0aW9uIGJlaW5nIHRydWUsIHdoYXQNCj4gaXMgdGhl IGNvbmRpdGlvbiB0aGF0IHdvdWxkIG1ha2UgdGhlICJJRiIgZmFsc2U/DQo+IA0KPiBUaGF0ICJJ RiIgbGFuZ3VhZ2UgYWxsb3dzIGEgdGFyZ2V0IHRvIGxvb3NlIHN0YXRlIGFuZCBub3QgZGVsaXZl ciBhbnkgVUEuDQo+IA0KPiAJRnJlZA0KPiANCj4gDQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0t LS0tDQo+IEZyb206IEp1bGlhbiBTYXRyYW4gW21haWx0bzpqdWxpYW5Ac2F0cmFuLm5ldF0NCj4g U2VudDogRnJpZGF5LCBTZXB0ZW1iZXIgMjgsIDIwMTIgMjowMyBBTQ0KPiBUbzogS25pZ2h0LCBG cmVkZXJpY2sNCj4gQ2M6IEJsYWNrLCBEYXZpZDsgc3Rvcm1AaWV0Zi5vcmcNCj4gU3ViamVjdDog UmU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYNCj4gDQo+ IFRoZXJlIGlzIG5vIHN1Y2ggdGhpbmcuIEFuZCB0aGUgcmVhc29uIHRoZSB0aW1lb3V0IGlzIHRo ZXJlIGlzIHRvIGxpbWl0IHRoZQ0KPiBhbW91bnQgb2Ygc3RhdGUgdGhlIHRhcmdldHMgaGF2ZSB0 byBrZWVwLiBBIHRhcmdldCBtYXkgY2hvb3NlIHRvIGFjY2VwdCB2ZXJ5DQo+IGxhcmdlIHRpbWVv dXRzIGFuZCB0aGVuIGRvIHdoYXQgeW91IHdhbnRlZCBkb25lIGJ1dCBiZXlvbmQgdGhhdCBvbmx5 IHRoZQ0KPiBwb3dlci11cCB1YSBpcyByZXF1aXJlZC4NCj4gDQo+IGp1bG8NCj4gDQo+IFNlbnQg ZnJvbSBteSBpUGFkDQo+IA0KPiBPbiAyOCDXkdeh16TXmCAyMDEyLCBhdCAwMDoyOSwgIktuaWdo dCwgRnJlZGVyaWNrIiA8RnJlZGVyaWNrLktuaWdodEBuZXRhcHAuY29tPg0KPiB3cm90ZToNCj4g DQo+ID4gSSB3b3VsZCB0aGluayBpZiB0aGUgSVNJRCBpcyBub3QgcmVjb2duaXplZCBhcyBhbiBv bGQgb25lLCB0aGVuIGl0IG11c3QgYmUNCj4gcmVjb2duaXplZCBhcyBhIG5ldyBvbmUuICBJZiBp dCBpcyBhIG5ldyBvbmUsIHRoZW4gaXQgd2lsbCBnZXQgYSBVQTogMjkvMDANCj4gcmVmbGVjdGlu ZyB0aGUgZmlyc3QgYWNjZXNzIGFmdGVyIHBvd2VyIG9uIGZyb20gdGhlICJuZXciIGluaXRpYXRv ci4NCj4gPg0KPiA+IElzIHRoZXJlIHNvbWV0aGluZyBvdGhlciB0aGFuIG5ldyBvciBvbGQ/ICBJ cyB0aGVyZSBzdWNoIGEgdGhpbmcgYXMgYQ0KPiBtaWRkbGUtYWdlZCBpbml0aWF0b3I/DQo+ID4N Cj4gPiAgICBGcmVkDQo+ID4NCj4gPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiA+IEZy b206IEJsYWNrLCBEYXZpZCBbbWFpbHRvOmRhdmlkLmJsYWNrQGVtYy5jb21dDQo+ID4gU2VudDog VGh1cnNkYXksIFNlcHRlbWJlciAyNywgMjAxMiAxMTo1NiBBTQ0KPiA+IFRvOiBLbmlnaHQsIEZy ZWRlcmljaw0KPiA+IENjOiBzdG9ybUBpZXRmLm9yZw0KPiA+IFN1YmplY3Q6IFJFOiBbc3Rvcm1d IFNQQy0yIHJlc2VydmUvcmVsZWFzZSB0ZXh0IC0gdmVyc2lvbiA2DQo+ID4NCj4gPiBGcmVkLA0K PiA+DQo+ID4gVGhlIHByb2JsZW0gaXMgbm90IHRoYXQgdGhlIGRldmljZSBkb2VzIG5vdCByYWlz ZSBhIFVBLCBpdCdzIHRoYXQgdGhlIFVBIG1heQ0KPiBuZXZlciBiZSBkZWxpdmVyZWQuDQo+ID4N Cj4gPiBBIHN1bW1hcnkgb2YgaG93IHRoaXMgY2FuIGhhcHBlbiBpcyB0aGF0IHRoZSBuZXcgY29u bmVjdGlvbiB3aXRoIHRoZSBzYW1lDQo+IElTSUQgbWF5IG5vdCBiZSByZWNvZ25pemVkIGFzIHJl aW5zdGF0aW5nIG9yIGNvbnRpbnVpbmcgdGhlIHNlc3Npb24gYXQgdGhlDQo+IHRhcmdldCBpZiB0 aGUgdGltZW91dHMgaGF2ZSBleHBpcmVkIC0gdGhlIFVBIGdldHMgZXN0YWJsaXNoZWQsIGJ1dCBu b2JvZHkNCj4gc2hvd3MgdXAgdG8gcGljayBpdCB1cCBpbiB0aW1lLCBzbyB0aGUgVUEgaXMgZGlz Y2FyZGVkIGFsb25nIHdpdGggdGhlIHJlc3Qgb2YNCj4gdGhlIHNlc3Npb24gc3RhdGUgYW5kIHRo ZSB0YXJnZXQgZmFpbHMgdG8gcmVjb2duaXplIHRoZSBpbml0aWF0b3IncyBzdWJzZXF1ZW50DQo+ IElTSUQgcmV1c2UgZXZlbiB0aG91Z2ggdGhlIGluaXRpYXRvciBiZWxpZXZlcyB0aGF0IGl0J3Mg cmUtIGVzdGFibGlzaGluZyB0aGUNCj4gc2Vzc2lvbi4NCj4gPg0KPiA+IFRoaXMgaXMgZnVydGhl ciBjb21wbGljYXRlZCBieSB0aGUgVUEgdGhhdCBpbmRpY2F0ZXMgbmV4dXMgc3RhdGUNCj4gcHJl c2VydmF0aW9uIG5vdCBiZWluZyBzcGVjaWZpZWQgaW4gU0FNLTIuDQo+ID4NCj4gPiBUaGFua3Ms DQo+ID4gLS1EYXZpZA0KPiA+DQo+ID4NCj4gPj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0N Cj4gPj4gRnJvbTogS25pZ2h0LCBGcmVkZXJpY2sgW21haWx0bzpGcmVkZXJpY2suS25pZ2h0QG5l dGFwcC5jb21dDQo+ID4+IFNlbnQ6IFRodXJzZGF5LCBTZXB0ZW1iZXIgMjcsIDIwMTIgMTE6MTEg QU0NCj4gPj4gVG86IEJsYWNrLCBEYXZpZDsgSnVsaWFuIFNhdHJhbg0KPiA+PiBDYzogc3Rvcm1A aWV0Zi5vcmcNCj4gPj4gU3ViamVjdDogUkU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNl IHRleHQgLSB2ZXJzaW9uIDYNCj4gPj4NCj4gPj4gSSBhZnJhaWQgSSBoYXZlIHRvIGNvbnRlc3Qg YSBsaXR0bGUgb24gdGhpcyAtIGlmIEkgdW5kZXJzdGFuZCB0aGlzDQo+ID4+IGRpc2N1c3Npb24g Y29ycmVjdGx5Lg0KPiA+Pg0KPiA+PiBJZiBhIFNDU0kgZGV2aWNlIGxvc2VzIGFueSBzdGF0ZSBh bmQgZG9lcyBub3QgcmFpc2UgYSBVTklUIEFUVEVOVElPTiwNCj4gPj4gdGhlbiBpdCBpcyBCUk9L RU4uICBJdCBpcyBub3QgYSBtYXR0ZXIgb2YgYW55dGhpbmcgaW4gaVNDU0kgKDM3MjApOyBJDQo+ ID4+IHRoaW5rIFNBTSBhbmQgU1BDIGFyZSBwcmV0dHkgY2xlYXIgb24gdGhpcy4gIElmIHRoZXJl IGlzIG5vIFVOSVQNCj4gPj4gQVRURU5USU9OLCB0aGVuIHN0YXRlIFNIQUxMIGJlIHByZXNlcnZl ZDsgb3RoZXJ3aXNlLCBhbGwgc29ydHMgb2YNCj4gPj4gc3R1ZmYgd2lsbCBuZXZlciB3b3JrICht b2RlIHBhZ2Ugc2V0dGluZ3MsIFJFU0VSVkUgY29tbWFuZHMsIGFsbCBzb3J0cyBvZg0KPiBzdHVm ZikuDQo+ID4+DQo+ID4+IElzIHNvbWVvbmUgcmVhbGx5IHN1Z2dlc3RpbmcgdGhhdCBzb21lIGRl dmljZXMgd2lsbCBkcm9wIHN0YXRlIGFuZA0KPiA+PiBOT1QgdGVsbCB0aGUgaG9zdCwgYW5kIHNv bWVob3cgdGhhdCBpdCBpcyBPSyB0byBkbyB0aGF0Pw0KPiA+Pg0KPiA+PiBUaGUgc3VnZ2VzdCBy ZXdvcmRpbmcgc2F5cyB0aGlzIGlzIHBvc3NpYmxlLg0KPiA+Pg0KPiA+PiAgICBGcmVkDQo+ID4+ DQo+ID4+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+ID4+IEZyb206IHN0b3JtLWJvdW5j ZXNAaWV0Zi5vcmcgW21haWx0bzpzdG9ybS1ib3VuY2VzQGlldGYub3JnXSBPbg0KPiA+PiBCZWhh bGYgT2YgQmxhY2ssIERhdmlkDQo+ID4+IFNlbnQ6IFRodXJzZGF5LCBTZXB0ZW1iZXIgMjcsIDIw MTIgMTA6NDcgQU0NCj4gPj4gVG86IEp1bGlhbiBTYXRyYW4NCj4gPj4gQ2M6IHN0b3JtQGlldGYu b3JnDQo+ID4+IFN1YmplY3Q6IFJlOiBbc3Rvcm1dIFNQQy0yIHJlc2VydmUvcmVsZWFzZSB0ZXh0 IC0gdmVyc2lvbiA2DQo+ID4+DQo+ID4+IEp1bGlhbiwNCj4gPj4NCj4gPj4+IFRoZSB0ZXh0IGlu IDQuNC4zIHN0YXRlcyB0aGUgb2J2aW91cyAodGhlcmUgaXMgbm8gb3RoZXIgc3BlY2VkDQo+ID4+ PiBtZXRob2QgZm9yIGNvbnRpbnVhdGlvbiBvciByZWluc3RhdGVtZW50KS4gTW9yZW92ZXIgaWYg dGhlIHNlc3Npb24NCj4gPj4+IGlzIHJlaW5zdGF0ZWQgYWZ0ZXIgdGltZTJ3YWl0IHlvdSBjcmVh dGUgYSBuZWVkIGZvciBhIFVBIChvciBhdA0KPiA+Pj4gbGVhc3Qgc3VnZ2VzdCBvbmUpIGFuZCB0 aGF0IHdhcyBub3QgdGhlcmUuDQo+ID4+DQo+ID4+IFRoZSBVQXMgYXJlIGNvdXJ0ZXN5IG9mIFNB TS0zIChhbmQgU0FNLTQpLiAgVG8gZW5jb21wYXNzIHN5c3RlbXMgdGhhdA0KPiA+PiBtYXkgbm90 IGRlbGl2ZXIgdGhlc2UgVUFzIGFuZCBzaXR1YXRpb25zIGluIHdoaWNoIHRoZXkgZG9uJ3Qgb2Nj dXINCj4gPj4gKGUuZy4sIHRoZSB0YXJnZXQgZG9lc24ndCByZWNvZ25pemUgd2hhdCBoYXBwZW5l ZCBhcyBuZXh1cw0KPiA+PiByZWVzdGFibGlzaG1lbnQsIGluZGVwZW5kZW50IG9mIHdoYXQgdGhl IGluaXRpYXRvciB0aGlua3MgaXQncyBkb2luZykgSQ0KPiB0aGluayB0aGUgdGV4dCBzaG91bGQg YmUgcmVwaHJhc2VkIGFzOg0KPiA+Pg0KPiA+PiAgICAgSWYgYSBVbml0IEF0dGVudGlvbiBjb25k aXRpb24gcmVzdWx0cyBmcm9tIHNlc3Npb24gcmVlc3RhYmxpc2htZW50LA0KPiA+PiAgICAgdGhl IHNwZWNpZmljIFVuaXQgQXR0ZW50aW9uIGNvbmRpdGlvbiBpbmRpY2F0ZXMgd2hldGhlciBuZXh1 cyBzdGF0ZQ0KPiA+PiAgICAgd2FzIHByZXNlcnZlZCwgc2VlIFNlY3Rpb24gNi4zLjQgb2YgW1NB TTRdLg0KPiA+Pg0KPiA+Pj4gSnVzdCBzdGF0aW5nIHRoYXQgY3JlYXRpb24vY29udGludWF0aW9u IG9mIGEgc2Vzc2lvbiBkb25lIGFjY29yZGluZw0KPiA+Pj4gNi54IGluY2x1ZGVzIHByZXNlcnZh dGlvbiBvZiByZXNlcnZlL3JlbGVhc2Ugc3RhdGUgLXRoYXQgd2FzIGFscmVhZHkNCj4gPj4+IHRo ZXJlIGltcGxpZWQgYnkgY29uZm9ybWFuY2UgdG8gc3BjMiBhbmQgdGhlIHJlbGF0aW9uIGJldHdl ZW4gaXQNCj4gPj4+IG5leHVzIGFuZCBzZXNzaW9uLg0KPiA+Pg0KPiA+PiBVbmZvcnR1bmF0ZWx5 LCBTUEMtMiBpcyBzdWZmaWNpZW50bHkgaW1wcmVjaXNlIG9uIHRoaXMgc3ViamVjdCB0aGF0DQo+ ID4+IHN1Y2ggYW4gaW1wbGljYXRpb24gaXMgYXQgYmVzdCBhICJwcmVmZXJyZWQgaW1wbGVtZW50 YXRpb24gYXBwcm9hY2giLg0KPiA+PiBJdCdzIG5vdCBhIHJlcXVpcmVtZW50IHRoYXQgY2FuIGJl IHNhZmVseSBzdGF0ZWQgaW4gYW55IHN0cm9uZ2VyDQo+ID4+IGZhc2hpb24gd2l0aG91dCB0aGUg cmlzayBvZiByZW5kZXJpbmcgZXhpc3RpbmcgaW1wbGVtZW50YXRpb25zIG5vbi0NCj4gY29tcGxp YW50Lg0KPiA+Pg0KPiA+Pj4gVGhlIG5ldyB0ZXh0IHlvdSBzdWdnZXN0IHRvIDQuNCBpcyBtZXJl bHkgYSBiZXN0IHByYWN0aWNlcw0KPiA+Pj4gcmVjb21tZW5kYXRpb25zIGFuZCB3aWxsIG5lZWQg YWRkaXRpb25zIG9uY2UgdGhlIG5ldyAibG9ja2luZyINCj4gPj4+IHRocm91Z2ggY29tcGFyZS1h bmQtc3dhcCBnZXQgaW4gd2lkZXNwcmVhZCB1c2UuDQo+ID4+DQo+ID4+IEl0J3MgZGVmaW5pdGVs eSBiZXN0IHByYWN0aWNlcyByZWNvbW1lbmRhdGlvbnMgLSBubyBhcmd1bWVudCB0aGVyZS4NCj4g Pj4NCj4gPj4gRm9yIGNvbnRleHQsIHdoYXQgSnVsaWFuIGlzIHJlZmVycmluZyB0byBpcyB1c2Ug b2YgdGhlIENPTVBBUkUgQU5EDQo+ID4+IFdSSVRFIGNvbW1hbmQgdGhhdCBwZXJmb3JtcyBhbiBh dG9taWMgb3BlcmF0aW9uIGF0IHRoZSB0YXJnZXQgaW5zdGVhZA0KPiA+PiBvZiB1c2luZyBSRVNF UlZFIGFuZCBSRUxFQVNFIHRvIGNyZWF0ZSBhIGNyaXRpY2FsIHNlY3Rpb24gYXQgdGhlDQo+ID4+ IGluaXRpYXRvciBiYXNlZCBvbiByZWFkIGFuZCB3cml0ZSBjb21tYW5kcy4NCj4gPj4NCj4gPj4g Q09NUEFSRSBBTkQgV1JJVEUgaXMgaW4gdXNlIC0gYSBzZW50ZW5jZSBtZW50aW9uaW5nIHRoYXQg Y29tbWFuZA0KPiA+PiBjb3VsZCBiZSBhZGRlZCwgYWx0aG91Z2ggdGhpcyBiZXN0IHByYWN0aWNl cyB0ZXh0IGlzIGFpbWVkIHByaW1hcmlseQ0KPiA+PiBhdCB1c2VzIG9mIHJlc2VydmUvcmVsZWFz ZSBmb3IgbG9uZy10ZXJtIHJlc2VydmF0aW9ucy4gIEkgdGhpbmsNCj4gPj4gdGhlcmUncyBhbiBp bXBsaWNpdCAid2hlbiBhIHJlc2VydmF0aW9uIGlzIHRoZSBkZXNpcmVkIGJlaGF2aW9yIg0KPiA+ PiBpbXBsaWNhdGlvbiB0byB0aGUgInBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIFNIT1VMRCBiZSB1 c2VkIGluc3RlYWQiDQo+ID4+IHRleHQgdGhhdCBJIGRvbid0IHRoaW5rIG5lZWRzIHRvIGJlIG1h ZGUgZXhwbGljaXQuICBPVE9ILCB0aGF0ICJTSE9VTEQiDQo+ID4+IGNvdWxkIGJlIGNoYW5nZWQg dG8gYSAic2hvdWxkIiwgb3IgaXQgbWlnaHQgYmUgZXZlbiBiZXR0ZXIgdG8gcmV3cml0ZQ0KPiA+ PiB0aGF0IHRleHQNCj4gPj4gYXM6DQo+ID4+DQo+ID4+ICAgIHBlcnNpc3RlbnQgcmVzZXJ2YXRp b25zIGFyZSBzdWdnZXN0ZWQgYXMgYW4gYWx0ZXJuYXRpdmUsIHNlZSBBbm5leCBCDQo+ID4+ICAg IG9mIFtTUEM0XS4NCj4gPj4NCj4gPj4gSW4gY29udHJhc3QsICJTSE9VTEQgTk9UIGJlIHVzZWQi IGlzIGNsZWFybHkganVzdGlmaWVkIGZvcg0KPiA+PiByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRp b25zIGJ5IFtTUEMzXS4NCj4gPj4NCj4gPj4gVGhhbmtzLA0KPiA+PiAtLURhdmlkDQo+ID4+DQo+ ID4+DQo+ID4+PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiA+Pj4gRnJvbTogSnVsaWFu IFNhdHJhbiBbbWFpbHRvOmp1bGlhbkBzYXRyYW4ubmV0XQ0KPiA+Pj4gU2VudDogV2VkbmVzZGF5 LCBTZXB0ZW1iZXIgMjYsIDIwMTIgMTA6MjUgUE0NCj4gPj4+IFRvOiBCbGFjaywgRGF2aWQNCj4g Pj4+IENjOiBzdG9ybUBpZXRmLm9yZw0KPiA+Pj4gU3ViamVjdDogUmU6IFtzdG9ybV0gU1BDLTIg cmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYNCj4gPj4+DQo+ID4+PiBEYXZpZCwNCj4g Pj4+DQo+ID4+PiBUaGUgdGV4dCBpbiA0LjQuMyBzdGF0ZXMgdGhlIG9idmlvdXMgKHRoZXJlIGlz IG5vIG90aGVyIHNwZWNlZA0KPiA+Pj4gbWV0aG9kIGZvciBjb250aW51YXRpb24gb3IgcmVpbnN0 YXRlbWVudCkuIE1vcmVvdmVyIGlmIHRoZSB0aGUNCj4gPj4+IHNlc3Npb24gaXMgcmVpbnN0YXRl ZCBhZnRlciB0aW1lMndhaXQgeW91IGNyZWF0ZSBhIG5lZWQgZm9yIGEgVUEgKG9yDQo+ID4+PiBh dCBsZWFzdCBzdWdnZXN0IG9uZSkgYW5kIHRoYXQgd2FzIG5vdCB0aGVyZS4gSnVzdCBzdGF0aW5n IHRoYXQNCj4gPj4+IGNyZWF0aW9uL2NvbnRpbnVhdGlvbiBvZiBhIHNlc3Npb24gZG9uZSBhY2Nv cmRpbmcgNi54IGluY2x1ZGVzDQo+ID4+PiBwcmVzZXJ2YXRpb24gb2YgcmVzZXJ2ZS9yZWxlYXNl IHN0YXRlIC10aGF0IHdhcyBhbHJlYWR5IHRoZXJlDQo+ID4+PiBpbXBsaWVkIGJ5IGNvbmZvcm1h bmNlIHRvIHNwYzIgYW5kIHRoZSByZWxhdGlvbiBiZXR3ZWVuIGl0IG5leHVzIGFuZA0KPiBzZXNz aW9uLg0KPiA+Pj4NCj4gPj4+IFRoZSBuZXcgdGV4dCB5b3Ugc3VnZ2VzdCB0byA0LjQgaXMgbWVy ZWx5IGEgYmVzdCBwcmFjdGljZXMNCj4gPj4+IHJlY29tZW5kYXRpb25zIGFuZCB3aWxsIG5lZWQg YWRkaXRpb25zIG9uY2UgdGhlIG5ldyAibG9ja2luZyINCj4gPj4+IHRocm91Z2ggY29tcGFyZS1h bmQtc3dhcCBnZXQgaW4gd2lkZXNwcmVhZCB1c2UuDQo+ID4+PiBJZiBpbnN0ZWFkIHdlIHJlZnJh aW4gZnJvbSBzdGF0aW5nIGFueXRoaW5nIHNwZWNpZmljIGFuZCBzdGF0ZSB0aGF0DQo+ID4+PiB0 YXJnZXQgc2hvdWxkIG1haW50YWluIHdoYXRldmVyIHN0YXRlIGlzIHJlcXVpcmVkIGJ5IFNQQyB3 aGlsZSBhDQo+ID4+PiBzZXNzaW9uIGlzIGNvbnRpbnVlZCBhbmQgbWF5IGRpc2NhcmQgc3RhdGUg d2hlbiBhIHNlc3Npb24gaXMNCj4gPj4+IHRlcm1pbmF0ZWQgYW5kIG1heSBpbmRpY2F0ZSB0aGF0 IGl0IGRpc2NhcmRlZCBzdGF0ZSB0aHJvdWdoIGEgdWEgeW91DQo+ID4+PiBhcmUgb24gc2FmZXIg Z3JvdW5kIGFuZCBkbyBub3QgY3JlYXRlIG5ldyByZXF1aXJlbWVudC4NCj4gPj4+DQo+ID4+PiBT ZW50IGZyb20gbXkgaVBhZA0KPiA+Pj4NCj4gPj4+IE9uIDI3INeR16HXpNeYIDIwMTIsIGF0IDAw OjI1LCAiQmxhY2ssIERhdmlkIiA8ZGF2aWQuYmxhY2tAZW1jLmNvbT4gd3JvdGU6DQo+ID4+Pg0K PiA+Pj4+IEhlcmUncyB3aGVyZSBJIHRoaW5rIHdlJ3ZlIGFycml2ZWQgKEkndmUgbWFkZSBhIGZl dyBtaW5vciBlZGl0b3JpYWwNCj4gPj4+PiB0d2Vha3MgdG8gd2hhdCdzIGJlZW4gZGlzY3Vzc2Vk KS4gIERlc3BpdGUgYWxsIHRoZSBkZXRhaWxlZA0KPiA+Pj4+IGRpc2N1c3Npb24sIEkgdGhpbmsg dGhpcyB2ZXJzaW9uIDYgY29tZXMgY2xvc2UgdG8gSnVsaWFuJ3MNCj4gPj4+PiBzdWdnZXN0aW9u DQo+ID4+IHRoYXQ6DQo+ID4+Pj4NCj4gPj4+Pj4gaSB3b3VsZCBtZW50aW9uIG9ubHkgdGhhdCB0 aGUgc3RhdGUgdGhhdCBtdXN0IGJlIG1haW50YWluZWQgd2hpbGUNCj4gPj4+Pj4gdGhlIHNlc3Np b24gaXMgbWFpbnRhaW5lZCBpbmNsdWRlcyB0aGUgcmVzZXJ2ZS9yZWxlYXNlIGFuZCBsZWF2ZQ0K PiA+Pj4+PiBpdCB0bw0KPiA+Pj4gdGhhdC4NCj4gPj4+Pg0KPiA+Pj4+IEFsdGhvdWdoIHdlIGNh bid0IGFjdHVhbGx5IHNheSAibXVzdCIgYXMgdGhhdCB3b3VsZCBiZSBhIG5ldw0KPiA+Pj4+IHJl cXVpcmVtZW50LCBhbmQgaW1wbGVtZW50YXRpb25zIGRpZmZlciBpbiB3aGF0IHRoZXkgZG8gYWJv dXQNCj4gPj4+PiByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb24gc3RhdGUuICBUaGUgY3VycmVu dCBsYW5ndWFnZQ0KPiA+Pj4+IGNoYXJhY3Rlcml6ZXMgcmV0ZW50aW9uIG9mIHRoYXQgc3RhdGUg YXMgdGhlICJwcmVmZXJyZWQNCj4gPj4+PiBpbXBsZW1lbnRhdGlvbg0KPiA+PiBhcHByb2FjaCIu DQo+ID4+Pj4NCj4gPj4+PiBCZXlvbmQgdGhhdCwgSSB0aGluayB0aGUgdHdvIGFkZGl0aW9uYWwg Y2F2ZWF0cyBhYm91dA0KPiA+Pj4+IHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbnMgKHBvc3Np YmxlIG5lZWQgZm9yIGEgcmVzZXQsDQo+ID4+Pj4gcG90ZW50aWFsbHkgdW5leHBlY3RlZCBpbnRl cmFjdGlvbiB3aXRoIHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zLA0KPiA+Pj4+IHNlZSBiZWxvdykg YXJlIHVzZWZ1bA0KPiA+PiB0byBpbmNsdWRlLg0KPiA+Pj4+DQo+ID4+Pj4gLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gPj4+Pg0KPiA+Pj4+IFsxXSBGb3IgY2xh cml0eSwgdGhlIGZvbGxvd2luZyBjaGFuZ2Ugc2hvdWxkIGJlIG1hZGUgaW4NCj4gPj4+PiA0LjQu My4xOg0KPiA+Pj4+DQo+ID4+Pj4gT0xEDQo+ID4+Pj4gVGhhdCBpcywgaXQgc2hvdWxkIHBlcmZv cm0gc2Vzc2lvbiByZWNvdmVyeSBhcyAgZGVzY3JpYmVkIGluDQo+ID4+Pj4gQ2hhcHRlciA2Lg0K PiA+Pj4+IE5FVw0KPiA+Pj4+IFRoYXQgaXMsIGl0IHNob3VsZCByZWluc3RhdGUgdGhlIHNlc3Np b24gdmlhIGlTQ1NJIHNlc3Npb24NCj4gPj4+PiByZWluc3RhdGVtZW50IChTZWN0aW9uIDYuMy41 KSBvciBjb250aW51ZSB2aWEgc2Vzc2lvbiBjb250aW51YXRpb24NCj4gPj4+PiAoU2VjdGlvbiA2 LjMuNikuDQo+ID4+Pj4gRU5EDQo+ID4+Pj4NCj4gPj4+PiBbMl0gQWRkIHRoZSBmb2xsb3dpbmcg c2VudGVuY2UgdG8gdGhlIGVuZCBvZiA0LjQuMy4xOg0KPiA+Pj4+DQo+ID4+Pj4gICAgVGhlIHNw ZWNpZmljIFVuaXQgQXR0ZW50aW9uIGNvbmRpdGlvbiB0aGF0IHJlc3VsdHMgZnJvbSBzZXNzaW9u DQo+ID4+Pj4gICAgcmVlc3RhYmxpc2htZW50IGluZGljYXRlcyB3aGV0aGVyIG5leHVzIHN0YXRl IHdhcyBwcmVzZXJ2ZWQsDQo+ID4+Pj4gICAgc2VlIFNlY3Rpb24gNi4zLjQgb2YgW1NBTTRdLg0K PiA+Pj4+DQo+ID4+Pj4gWzNdIE5FVyBURVhUOg0KPiA+Pj4+DQo+ID4+Pj4gNC40LjMuMi4gUmVz ZXJ2YXRpb25zDQo+ID4+Pj4NCj4gPj4+PiBUaGVyZSBhcmUgdHdvIHJlc2VydmF0aW9uIG1hbmFn ZW1lbnQgbWV0aG9kcyBkZWZpbmVkIGluIHRoZSBTQ1NJDQo+ID4+Pj4gc3RhbmRhcmRzLCByZXNl cnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zLCBiYXNlZCBvbiB0aGUgUkVTRVJWRSBhbmQNCj4gPj4+ PiBSRUxFQVNFIGNvbW1hbmRzIFtTUEMyXSwgYW5kIHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zLCBi YXNlZCBvbiB0aGUNCj4gPj4+PiBQRVJTSVNURU5UIFJFU0VSVkUgSU4gYW5kIFBFUlNJU1RFTlQg UkVTRVJWRSBPVVQgY29tbWFuZHMgW1NQQzNdLg0KPiA+Pj4+IFJlc2VydmUvcmVsZWFzZSByZXNl cnZhdGlvbnMgYXJlIG9ic29sZXRlIFtTUEMzXSBhbmQgU0hPVUxEIE5PVCBiZQ0KPiA+Pj4+IHVz ZWQ7IHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIFNIT1VMRCBiZSB1c2VkIGluc3RlYWQuDQo+ID4+ Pj4NCj4gPj4+PiBTdGF0ZSBmb3IgcGVyc2lzdGVudCByZXNlcnZhdGlvbnMgaXMgcmVxdWlyZWQg dG8gcGVyc2lzdCAgdGhyb3VnaA0KPiA+Pj4+IGNoYW5nZXMgYW5kIGZhaWx1cmVzIGF0IHRoZSBp U0NTSSBsYXllciB0aGF0IHJlc3VsdCBpbiAgSV9UIE5leHVzDQo+ID4+Pj4gZmFpbHVyZXMsIHNl ZSBbU1BDM10gZm9yIGRldGFpbHMgYW5kIHNwZWNpZmljIHJlcXVpcmVtZW50cy4NCj4gPj4+Pg0K PiA+Pj4+IEluIGNvbnRyYXN0LCBbU1BDMl0gZG9lcyBub3Qgc3BlY2lmeSBkZXRhaWxlZCBwZXJz aXN0ZW5jZQ0KPiA+Pj4+IHJlcXVpcmVtZW50cyBmb3IgcmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0 aW9uIHN0YXRlIGFmdGVyIGFuIElfVA0KPiA+Pj4+IE5leHVzIGZhaWx1cmUuICBOb25ldGhlbGVz cywgd2hlbiByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zIGFyZQ0KPiA+Pj4+IHN1cHBvcnRl ZCBieSBhbiBpU0NTSSB0YXJnZXQsIHRoZSBwcmVmZXJyZWQgaW1wbGVtZW50YXRpb24gYXBwcm9h Y2gNCj4gPj4+PiBpcyB0byBwcmVzZXJ2ZSByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb24gc3Rh dGUgZm9yIGlTQ1NJIHNlc3Npb24NCj4gPj4+PiByZWluc3RhdGVtZW50IChzZWUgU2VjdGlvbiA2 LjMuNSkgb3Igc2Vzc2lvbiBjb250aW51YXRpb24gIChzZWUNCj4gPj4+PiBTZWN0aW9uIDYuMy42 KS4NCj4gPj4+Pg0KPiA+Pj4+IFR3byBhZGRpdGlvbmFsIGNhdmVhdHMgYXBwbHkgdG8gcmVzZXJ2 ZS9yZWxlYXNlIHJlc2VydmF0aW9uczoNCj4gPj4+Pg0KPiA+Pj4+IC0gV2hlbiBjb25uZWN0aW9u IGZhaWx1cmUgY2F1c2VzIHRoZSBpU0NTSSBzZXNzaW9uIHRvIGZhaWwsIGFuZA0KPiA+Pj4+ICAg ICB0aGUgc2Vzc2lvbiBpcyBub3QgcmVpbnN0YXRlZCBvciBjb250aW51ZWQsIHRhcmdldCByZXRl bnRpb24NCj4gPj4+PiAgICAgb2YgdGhhdCBzZXNzaW9uJ3MgcmVzZXJ2ZS9yZWxlYXNlIHJlc2Vy dmF0aW9uIHN0YXRlIG1heQ0KPiA+Pj4+ICAgICByZXF1aXJlIGFuIGluaXRpYXRvciB0byBpc3N1 ZSBhIHJlc2V0IChlLmcuLCBMT0dJQ0FMIFVOSVQgUkVTRVQsDQo+ID4+Pj4gICAgIHNlZSBzZWN0 aW9uIDExLjUpIGluIG9yZGVyIHRvIHJlbW92ZSB0aGF0IHJlc2VydmF0aW9uIHN0YXRlLg0KPiA+ Pj4+DQo+ID4+Pj4gLSBSZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zIG1heSBub3QgYmVoYXZl IGFzIGV4cGVjdGVkIHdoZW4NCj4gPj4+PiAgICAgcGVyc2lzdGVudCByZXNlcnZhdGlvbnMgYXJl IGFsc28gdXNlZCBvbiB0aGUgc2FtZSBsb2dpY2FsIHVuaXQ7DQo+ID4+Pj4gICAgIHNlZSB0aGUg ZGlzY3Vzc2lvbiBvZiAiRXhjZXB0aW9ucyB0byBTUEMtMiBSRVNFUlZFIGFuZCBSRUxFQVNFDQo+ ID4+Pj4gICAgIGJlaGF2aW9yIiBpbiBbU1BDNF0uDQo+ID4+Pj4NCj4gPj4+PiBUaGFua3MsDQo+ ID4+Pj4gLS1EYXZpZA0KPiA+Pj4+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0NCj4gPj4+PiBEYXZpZCBMLiBCbGFjaywgRGlzdGluZ3Vpc2hlZCBF bmdpbmVlciBFTUMgQ29ycG9yYXRpb24sIDE3NiBTb3V0aA0KPiA+Pj4+IFN0LiwgSG9wa2ludG9u LCBNQSAgMDE3NDgNCj4gPj4+PiArMSAoNTA4KSAyOTMtNzk1MyAgICAgICAgICAgICBGQVg6ICsx ICg1MDgpIDI5My03Nzg2DQo+ID4+Pj4gZGF2aWQuYmxhY2tAZW1jLmNvbSAgICAgICAgTW9iaWxl OiArMSAoOTc4KSAzOTQtNzc1NA0KPiA+Pj4+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gPj4+Pg0KPiA+Pj4+DQo+ID4+Pj4gX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4gPj4+PiBzdG9ybSBtYWls aW5nIGxpc3QNCj4gPj4+PiBzdG9ybUBpZXRmLm9yZw0KPiA+Pj4+IGh0dHBzOi8vd3d3LmlldGYu b3JnL21haWxtYW4vbGlzdGluZm8vc3Rvcm0NCj4gPj4NCj4gPj4gX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4gPj4gc3Rvcm0gbWFpbGluZyBsaXN0DQo+ ID4+IHN0b3JtQGlldGYub3JnDQo+ID4+IGh0dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlz dGluZm8vc3Rvcm0NCj4gPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fXw0KPiA+IHN0b3JtIG1haWxpbmcgbGlzdA0KPiA+IHN0b3JtQGlldGYub3JnDQo+ID4g aHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9zdG9ybQ0K From Frederick.Knight@netapp.com Fri Sep 28 07:20:55 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7CA0E21F8609 for ; Fri, 28 Sep 2012 07:20:55 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xgvukyUfT5wb for ; Fri, 28 Sep 2012 07:20:54 -0700 (PDT) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id 34BA321F8610 for ; Fri, 28 Sep 2012 07:20:54 -0700 (PDT) X-IronPort-AV: E=Sophos;i="4.80,502,1344236400"; d="scan'208";a="695236707" Received: from smtp2.corp.netapp.com ([10.57.159.114]) by mx2-out.netapp.com with ESMTP; 28 Sep 2012 07:20:53 -0700 Received: from vmwexceht02-prd.hq.netapp.com (vmwexceht02-prd.hq.netapp.com [10.106.76.240]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8SEKqTr026447; Fri, 28 Sep 2012 07:20:53 -0700 (PDT) Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.102]) by vmwexceht02-prd.hq.netapp.com ([10.106.76.240]) with mapi id 14.02.0309.002; Fri, 28 Sep 2012 07:20:52 -0700 From: "Knight, Frederick" To: "Black, David" Thread-Topic: [storm] SPC-2 reserve/release text - version 6 Thread-Index: Ac2cNcys4MzRK8PXQu651fVKNuEIxQAXDfsAABnkfIAADjDW0AAa4S6gACikjGD//nJjgIAAECkggAAHbKCAAAZiEA== Date: Fri, 28 Sep 2012 14:20:50 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> <4AE113CD-96D7-414D-8AAB-36ECF7FD89EA@satran.net> <8D3D17ACE214DC429325B2B98F3AE7120DE798F9@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE7993C@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79B42@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE79B42@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.106.53.53] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release text - version 6 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Sep 2012 14:20:55 -0000 QnV0IHdlIGFsc28gc2hvdWxkbid0IGNvZGlmeSBCUk9LRU4gZGVzaWducyBpbiBhIGJyYW5kIG5l dyBSRkMuICBJdCB3b3VsZCBiZXR0ZXIgKGluIG15IG9waW5pb24pIHRvIGxlYXZlIGl0IGFsb25l IHRoYW4gc2FuY3Rpb24gYW5kIGRlc2NyaWJlIGhvdyB0byBidWlsZCBhIGJyb2tlbiBpbXBsZW1l bnRhdGlvbi4gIEFkZGluZyB0aGUgIklGIiBpcyB3cm9uZy4NCg0KV2UndmUgZml4ZWQgdGhlIGlz c3VlcyByYWlzZWQgYnkgUkZDNTA0OCBieSBpbmNvcnBvcmF0aW5nIGl0IGludG8gdGhlIG5ldyBS RkMuDQoNCglGcmVkDQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBCbGFjaywg RGF2aWQgW21haWx0bzpkYXZpZC5ibGFja0BlbWMuY29tXSANClNlbnQ6IEZyaWRheSwgU2VwdGVt YmVyIDI4LCAyMDEyIDk6NTEgQU0NClRvOiBLbmlnaHQsIEZyZWRlcmljaw0KQ2M6IHN0b3JtQGll dGYub3JnDQpTdWJqZWN0OiBSRTogW3N0b3JtXSBTUEMtMiByZXNlcnZlL3JlbGVhc2UgdGV4dCAt IHZlcnNpb24gNg0KDQpGcmVkLA0KDQo+IElmIHN0YXRlIGlzIHJldGFpbmVkIChpdCdzIGFuIG9s ZCAiSSIgcmVjb25uZWN0aW5nKSwgdGhlbiB3aGljaCBVQSBpcyANCj4gZGVmaW5lZCBieSBTQU0t My4NCj4gDQo+IElmIHN0YXRlIGlzIG5vdCByZXRhaW5lZCAoaXQncyBhIG5ldyAiSSIpLCB0aGVu IHRoZSBVQSBpcyBkZWZpbmVkIGJ5IFNBTS0zLg0KPg0KPiBTaW5jZSBib3RoIG9sZCAiSSJzIGFu ZCBuZXcgIkkicyByZXN1bHQgaW4gdGhlICJJRiIgY29uZGl0aW9uIGJlaW5nIA0KPiB0cnVlLCB3 aGF0IGlzIHRoZSBjb25kaXRpb24gdGhhdCB3b3VsZCBtYWtlIHRoZSAiSUYiIGZhbHNlPw0KDQpZ b3UgYWxtb3N0IGFuc3dlcmVkIHlvdXIgb3duIHF1ZXN0aW9uIDstKS4NCg0KVGhlIGFuc3dlciBp cyAuLi4uIChkcnVtIHJvbGwpIC4uLiBhIHRhcmdldCBpbXBsZW1lbnRhdGlvbiBiYXNlZCBvbiBS RkMgMzcyMCBTQU0tMiwgYWx0aG91Z2ggSSB3b3VsZCBob3BlIHRoYXQgc3VjaCBpbXBsZW1lbnRh dGlvbnMgYXQgbGVhc3QgcGlja2VkIHVwIFJGQyA1MDQ4J3MgY2hhbmdlcy4gIA0KDQpUaGUgSV9U IE5leHVzIExvc3MgY29uY2VwdCBvbiB3aGljaCB0aGlzIFVBIGRpc2N1c3Npb24gaXMgYmFzZWQg aXMgcGFydCBvZiB0aGUgU0NTSSBldmVudHMgYW5kIGV2ZW50IG5vdGlmaWNhdGlvbnMgbW9kZWwg dGhhdCB3YXMgaW50cm9kdWNlZCBpbiBTQU0tMywgYnV0IG5laXRoZXIgUkZDIDM3MjAgbm9yIFJG QyA1MDQ4IHJlZmVyZW5jZXMgU0FNLTMuDQoNClRoZXJlIHdlcmUgZGVmaW5pdGVseSBpbXBsZW1l bnRhdGlvbnMgb2YgaVNDU0kgYmFzZWQgb24gU0FNLTIgLSBJIGFncmVlIHdpdGggSnVsaWFuIHRo YXQgd2Ugc2hvdWxkICpub3QqIGJlIGludHJvZHVjaW5nIGEgVUEgcmVxdWlyZW1lbnQgaW4gdGhp cyBkcmFmdCB0aGF0IG1pZ2h0IGJyZWFrIHN1Y2ggaW1wbGVtZW50YXRpb25zLg0KDQpUaGFua3Ms DQotLURhdmlkDQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogS25pZ2h0 LCBGcmVkZXJpY2sgW21haWx0bzpGcmVkZXJpY2suS25pZ2h0QG5ldGFwcC5jb21dDQo+IFNlbnQ6 IEZyaWRheSwgU2VwdGVtYmVyIDI4LCAyMDEyIDg6MzggQU0NCj4gVG86IEp1bGlhbiBTYXRyYW4N Cj4gQ2M6IEJsYWNrLCBEYXZpZDsgc3Rvcm1AaWV0Zi5vcmcNCj4gU3ViamVjdDogUkU6IFtzdG9y bV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYNCj4gDQo+IE9LLCB0aGF0 J3Mgd2hhdCBJIHRob3VnaHQuDQo+IA0KPiBJZiBzdGF0ZSBpcyByZXRhaW5lZCAoaXQncyBhbiBv bGQgIkkiIHJlY29ubmVjdGluZyksIHRoZW4gd2hpY2ggVUEgaXMgDQo+IGRlZmluZWQgYnkgU0FN LTMuDQo+IA0KPiBJZiBzdGF0ZSBpcyBub3QgcmV0YWluZWQgKGl0J3MgYSBuZXcgIkkiKSwgdGhl biB0aGUgVUEgaXMgZGVmaW5lZCBieSBTQU0tMy4NCj4gDQo+IFNpbmNlIGJvdGggb2xkICJJInMg YW5kIG5ldyAiSSJzIHJlc3VsdCBpbiB0aGUgIklGIiBjb25kaXRpb24gYmVpbmcgDQo+IHRydWUs IHdoYXQgaXMgdGhlIGNvbmRpdGlvbiB0aGF0IHdvdWxkIG1ha2UgdGhlICJJRiIgZmFsc2U/DQo+ IA0KPiBUaGF0ICJJRiIgbGFuZ3VhZ2UgYWxsb3dzIGEgdGFyZ2V0IHRvIGxvb3NlIHN0YXRlIGFu ZCBub3QgZGVsaXZlciBhbnkgVUEuDQo+IA0KPiAJRnJlZA0KPiANCj4gDQo+IC0tLS0tT3JpZ2lu YWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IEp1bGlhbiBTYXRyYW4gW21haWx0bzpqdWxpYW5Ac2F0 cmFuLm5ldF0NCj4gU2VudDogRnJpZGF5LCBTZXB0ZW1iZXIgMjgsIDIwMTIgMjowMyBBTQ0KPiBU bzogS25pZ2h0LCBGcmVkZXJpY2sNCj4gQ2M6IEJsYWNrLCBEYXZpZDsgc3Rvcm1AaWV0Zi5vcmcN Cj4gU3ViamVjdDogUmU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJz aW9uIDYNCj4gDQo+IFRoZXJlIGlzIG5vIHN1Y2ggdGhpbmcuIEFuZCB0aGUgcmVhc29uIHRoZSB0 aW1lb3V0IGlzIHRoZXJlIGlzIHRvIA0KPiBsaW1pdCB0aGUgYW1vdW50IG9mIHN0YXRlIHRoZSB0 YXJnZXRzIGhhdmUgdG8ga2VlcC4gQSB0YXJnZXQgbWF5IA0KPiBjaG9vc2UgdG8gYWNjZXB0IHZl cnkgbGFyZ2UgdGltZW91dHMgYW5kIHRoZW4gZG8gd2hhdCB5b3Ugd2FudGVkIGRvbmUgDQo+IGJ1 dCBiZXlvbmQgdGhhdCBvbmx5IHRoZSBwb3dlci11cCB1YSBpcyByZXF1aXJlZC4NCj4gDQo+IGp1 bG8NCj4gDQo+IFNlbnQgZnJvbSBteSBpUGFkDQo+IA0KPiBPbiAyOCDXkdeh16TXmCAyMDEyLCBh dCAwMDoyOSwgIktuaWdodCwgRnJlZGVyaWNrIiANCj4gPEZyZWRlcmljay5LbmlnaHRAbmV0YXBw LmNvbT4NCj4gd3JvdGU6DQo+IA0KPiA+IEkgd291bGQgdGhpbmsgaWYgdGhlIElTSUQgaXMgbm90 IHJlY29nbml6ZWQgYXMgYW4gb2xkIG9uZSwgdGhlbiBpdCANCj4gPiBtdXN0IGJlDQo+IHJlY29n bml6ZWQgYXMgYSBuZXcgb25lLiAgSWYgaXQgaXMgYSBuZXcgb25lLCB0aGVuIGl0IHdpbGwgZ2V0 IGEgVUE6IA0KPiAyOS8wMCByZWZsZWN0aW5nIHRoZSBmaXJzdCBhY2Nlc3MgYWZ0ZXIgcG93ZXIg b24gZnJvbSB0aGUgIm5ldyIgaW5pdGlhdG9yLg0KPiA+DQo+ID4gSXMgdGhlcmUgc29tZXRoaW5n IG90aGVyIHRoYW4gbmV3IG9yIG9sZD8gIElzIHRoZXJlIHN1Y2ggYSB0aGluZyBhcyANCj4gPiBh DQo+IG1pZGRsZS1hZ2VkIGluaXRpYXRvcj8NCj4gPg0KPiA+ICAgIEZyZWQNCj4gPg0KPiA+IC0t LS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+ID4gRnJvbTogQmxhY2ssIERhdmlkIFttYWlsdG86 ZGF2aWQuYmxhY2tAZW1jLmNvbV0NCj4gPiBTZW50OiBUaHVyc2RheSwgU2VwdGVtYmVyIDI3LCAy MDEyIDExOjU2IEFNDQo+ID4gVG86IEtuaWdodCwgRnJlZGVyaWNrDQo+ID4gQ2M6IHN0b3JtQGll dGYub3JnDQo+ID4gU3ViamVjdDogUkU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRl eHQgLSB2ZXJzaW9uIDYNCj4gPg0KPiA+IEZyZWQsDQo+ID4NCj4gPiBUaGUgcHJvYmxlbSBpcyBu b3QgdGhhdCB0aGUgZGV2aWNlIGRvZXMgbm90IHJhaXNlIGEgVUEsIGl0J3MgdGhhdCANCj4gPiB0 aGUgVUEgbWF5DQo+IG5ldmVyIGJlIGRlbGl2ZXJlZC4NCj4gPg0KPiA+IEEgc3VtbWFyeSBvZiBo b3cgdGhpcyBjYW4gaGFwcGVuIGlzIHRoYXQgdGhlIG5ldyBjb25uZWN0aW9uIHdpdGggdGhlIA0K PiA+IHNhbWUNCj4gSVNJRCBtYXkgbm90IGJlIHJlY29nbml6ZWQgYXMgcmVpbnN0YXRpbmcgb3Ig Y29udGludWluZyB0aGUgc2Vzc2lvbiBhdCANCj4gdGhlIHRhcmdldCBpZiB0aGUgdGltZW91dHMg aGF2ZSBleHBpcmVkIC0gdGhlIFVBIGdldHMgZXN0YWJsaXNoZWQsIGJ1dCANCj4gbm9ib2R5IHNo b3dzIHVwIHRvIHBpY2sgaXQgdXAgaW4gdGltZSwgc28gdGhlIFVBIGlzIGRpc2NhcmRlZCBhbG9u ZyANCj4gd2l0aCB0aGUgcmVzdCBvZiB0aGUgc2Vzc2lvbiBzdGF0ZSBhbmQgdGhlIHRhcmdldCBm YWlscyB0byByZWNvZ25pemUgDQo+IHRoZSBpbml0aWF0b3IncyBzdWJzZXF1ZW50IElTSUQgcmV1 c2UgZXZlbiB0aG91Z2ggdGhlIGluaXRpYXRvciANCj4gYmVsaWV2ZXMgdGhhdCBpdCdzIHJlLSBl c3RhYmxpc2hpbmcgdGhlIHNlc3Npb24uDQo+ID4NCj4gPiBUaGlzIGlzIGZ1cnRoZXIgY29tcGxp Y2F0ZWQgYnkgdGhlIFVBIHRoYXQgaW5kaWNhdGVzIG5leHVzIHN0YXRlDQo+IHByZXNlcnZhdGlv biBub3QgYmVpbmcgc3BlY2lmaWVkIGluIFNBTS0yLg0KPiA+DQo+ID4gVGhhbmtzLA0KPiA+IC0t RGF2aWQNCj4gPg0KPiA+DQo+ID4+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+ID4+IEZy b206IEtuaWdodCwgRnJlZGVyaWNrIFttYWlsdG86RnJlZGVyaWNrLktuaWdodEBuZXRhcHAuY29t XQ0KPiA+PiBTZW50OiBUaHVyc2RheSwgU2VwdGVtYmVyIDI3LCAyMDEyIDExOjExIEFNDQo+ID4+ IFRvOiBCbGFjaywgRGF2aWQ7IEp1bGlhbiBTYXRyYW4NCj4gPj4gQ2M6IHN0b3JtQGlldGYub3Jn DQo+ID4+IFN1YmplY3Q6IFJFOiBbc3Rvcm1dIFNQQy0yIHJlc2VydmUvcmVsZWFzZSB0ZXh0IC0g dmVyc2lvbiA2DQo+ID4+DQo+ID4+IEkgYWZyYWlkIEkgaGF2ZSB0byBjb250ZXN0IGEgbGl0dGxl IG9uIHRoaXMgLSBpZiBJIHVuZGVyc3RhbmQgdGhpcyANCj4gPj4gZGlzY3Vzc2lvbiBjb3JyZWN0 bHkuDQo+ID4+DQo+ID4+IElmIGEgU0NTSSBkZXZpY2UgbG9zZXMgYW55IHN0YXRlIGFuZCBkb2Vz IG5vdCByYWlzZSBhIFVOSVQgDQo+ID4+IEFUVEVOVElPTiwgdGhlbiBpdCBpcyBCUk9LRU4uICBJ dCBpcyBub3QgYSBtYXR0ZXIgb2YgYW55dGhpbmcgaW4gDQo+ID4+IGlTQ1NJICgzNzIwKTsgSSB0 aGluayBTQU0gYW5kIFNQQyBhcmUgcHJldHR5IGNsZWFyIG9uIHRoaXMuICBJZiANCj4gPj4gdGhl cmUgaXMgbm8gVU5JVCBBVFRFTlRJT04sIHRoZW4gc3RhdGUgU0hBTEwgYmUgcHJlc2VydmVkOyAN Cj4gPj4gb3RoZXJ3aXNlLCBhbGwgc29ydHMgb2Ygc3R1ZmYgd2lsbCBuZXZlciB3b3JrIChtb2Rl IHBhZ2Ugc2V0dGluZ3MsIA0KPiA+PiBSRVNFUlZFIGNvbW1hbmRzLCBhbGwgc29ydHMgb2YNCj4g c3R1ZmYpLg0KPiA+Pg0KPiA+PiBJcyBzb21lb25lIHJlYWxseSBzdWdnZXN0aW5nIHRoYXQgc29t ZSBkZXZpY2VzIHdpbGwgZHJvcCBzdGF0ZSBhbmQgDQo+ID4+IE5PVCB0ZWxsIHRoZSBob3N0LCBh bmQgc29tZWhvdyB0aGF0IGl0IGlzIE9LIHRvIGRvIHRoYXQ/DQo+ID4+DQo+ID4+IFRoZSBzdWdn ZXN0IHJld29yZGluZyBzYXlzIHRoaXMgaXMgcG9zc2libGUuDQo+ID4+DQo+ID4+ICAgIEZyZWQN Cj4gPj4NCj4gPj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPj4gRnJvbTogc3Rvcm0t Ym91bmNlc0BpZXRmLm9yZyBbbWFpbHRvOnN0b3JtLWJvdW5jZXNAaWV0Zi5vcmddIE9uIA0KPiA+ PiBCZWhhbGYgT2YgQmxhY2ssIERhdmlkDQo+ID4+IFNlbnQ6IFRodXJzZGF5LCBTZXB0ZW1iZXIg MjcsIDIwMTIgMTA6NDcgQU0NCj4gPj4gVG86IEp1bGlhbiBTYXRyYW4NCj4gPj4gQ2M6IHN0b3Jt QGlldGYub3JnDQo+ID4+IFN1YmplY3Q6IFJlOiBbc3Rvcm1dIFNQQy0yIHJlc2VydmUvcmVsZWFz ZSB0ZXh0IC0gdmVyc2lvbiA2DQo+ID4+DQo+ID4+IEp1bGlhbiwNCj4gPj4NCj4gPj4+IFRoZSB0 ZXh0IGluIDQuNC4zIHN0YXRlcyB0aGUgb2J2aW91cyAodGhlcmUgaXMgbm8gb3RoZXIgc3BlY2Vk IA0KPiA+Pj4gbWV0aG9kIGZvciBjb250aW51YXRpb24gb3IgcmVpbnN0YXRlbWVudCkuIE1vcmVv dmVyIGlmIHRoZSBzZXNzaW9uIA0KPiA+Pj4gaXMgcmVpbnN0YXRlZCBhZnRlciB0aW1lMndhaXQg eW91IGNyZWF0ZSBhIG5lZWQgZm9yIGEgVUEgKG9yIGF0IA0KPiA+Pj4gbGVhc3Qgc3VnZ2VzdCBv bmUpIGFuZCB0aGF0IHdhcyBub3QgdGhlcmUuDQo+ID4+DQo+ID4+IFRoZSBVQXMgYXJlIGNvdXJ0 ZXN5IG9mIFNBTS0zIChhbmQgU0FNLTQpLiAgVG8gZW5jb21wYXNzIHN5c3RlbXMgDQo+ID4+IHRo YXQgbWF5IG5vdCBkZWxpdmVyIHRoZXNlIFVBcyBhbmQgc2l0dWF0aW9ucyBpbiB3aGljaCB0aGV5 IGRvbid0IA0KPiA+PiBvY2N1ciAoZS5nLiwgdGhlIHRhcmdldCBkb2Vzbid0IHJlY29nbml6ZSB3 aGF0IGhhcHBlbmVkIGFzIG5leHVzIA0KPiA+PiByZWVzdGFibGlzaG1lbnQsIGluZGVwZW5kZW50 IG9mIHdoYXQgdGhlIGluaXRpYXRvciB0aGlua3MgaXQncyANCj4gPj4gZG9pbmcpIEkNCj4gdGhp bmsgdGhlIHRleHQgc2hvdWxkIGJlIHJlcGhyYXNlZCBhczoNCj4gPj4NCj4gPj4gICAgIElmIGEg VW5pdCBBdHRlbnRpb24gY29uZGl0aW9uIHJlc3VsdHMgZnJvbSBzZXNzaW9uIHJlZXN0YWJsaXNo bWVudCwNCj4gPj4gICAgIHRoZSBzcGVjaWZpYyBVbml0IEF0dGVudGlvbiBjb25kaXRpb24gaW5k aWNhdGVzIHdoZXRoZXIgbmV4dXMgc3RhdGUNCj4gPj4gICAgIHdhcyBwcmVzZXJ2ZWQsIHNlZSBT ZWN0aW9uIDYuMy40IG9mIFtTQU00XS4NCj4gPj4NCj4gPj4+IEp1c3Qgc3RhdGluZyB0aGF0IGNy ZWF0aW9uL2NvbnRpbnVhdGlvbiBvZiBhIHNlc3Npb24gZG9uZSANCj4gPj4+IGFjY29yZGluZyA2 LnggaW5jbHVkZXMgcHJlc2VydmF0aW9uIG9mIHJlc2VydmUvcmVsZWFzZSBzdGF0ZSAtdGhhdCAN Cj4gPj4+IHdhcyBhbHJlYWR5IHRoZXJlIGltcGxpZWQgYnkgY29uZm9ybWFuY2UgdG8gc3BjMiBh bmQgdGhlIHJlbGF0aW9uIA0KPiA+Pj4gYmV0d2VlbiBpdCBuZXh1cyBhbmQgc2Vzc2lvbi4NCj4g Pj4NCj4gPj4gVW5mb3J0dW5hdGVseSwgU1BDLTIgaXMgc3VmZmljaWVudGx5IGltcHJlY2lzZSBv biB0aGlzIHN1YmplY3QgdGhhdCANCj4gPj4gc3VjaCBhbiBpbXBsaWNhdGlvbiBpcyBhdCBiZXN0 IGEgInByZWZlcnJlZCBpbXBsZW1lbnRhdGlvbiBhcHByb2FjaCIuDQo+ID4+IEl0J3Mgbm90IGEg cmVxdWlyZW1lbnQgdGhhdCBjYW4gYmUgc2FmZWx5IHN0YXRlZCBpbiBhbnkgc3Ryb25nZXIgDQo+ ID4+IGZhc2hpb24gd2l0aG91dCB0aGUgcmlzayBvZiByZW5kZXJpbmcgZXhpc3RpbmcgaW1wbGVt ZW50YXRpb25zIG5vbi0NCj4gY29tcGxpYW50Lg0KPiA+Pg0KPiA+Pj4gVGhlIG5ldyB0ZXh0IHlv dSBzdWdnZXN0IHRvIDQuNCBpcyBtZXJlbHkgYSBiZXN0IHByYWN0aWNlcyANCj4gPj4+IHJlY29t bWVuZGF0aW9ucyBhbmQgd2lsbCBuZWVkIGFkZGl0aW9ucyBvbmNlIHRoZSBuZXcgImxvY2tpbmci DQo+ID4+PiB0aHJvdWdoIGNvbXBhcmUtYW5kLXN3YXAgZ2V0IGluIHdpZGVzcHJlYWQgdXNlLg0K PiA+Pg0KPiA+PiBJdCdzIGRlZmluaXRlbHkgYmVzdCBwcmFjdGljZXMgcmVjb21tZW5kYXRpb25z IC0gbm8gYXJndW1lbnQgdGhlcmUuDQo+ID4+DQo+ID4+IEZvciBjb250ZXh0LCB3aGF0IEp1bGlh biBpcyByZWZlcnJpbmcgdG8gaXMgdXNlIG9mIHRoZSBDT01QQVJFIEFORCANCj4gPj4gV1JJVEUg Y29tbWFuZCB0aGF0IHBlcmZvcm1zIGFuIGF0b21pYyBvcGVyYXRpb24gYXQgdGhlIHRhcmdldCAN Cj4gPj4gaW5zdGVhZCBvZiB1c2luZyBSRVNFUlZFIGFuZCBSRUxFQVNFIHRvIGNyZWF0ZSBhIGNy aXRpY2FsIHNlY3Rpb24gDQo+ID4+IGF0IHRoZSBpbml0aWF0b3IgYmFzZWQgb24gcmVhZCBhbmQg d3JpdGUgY29tbWFuZHMuDQo+ID4+DQo+ID4+IENPTVBBUkUgQU5EIFdSSVRFIGlzIGluIHVzZSAt IGEgc2VudGVuY2UgbWVudGlvbmluZyB0aGF0IGNvbW1hbmQgDQo+ID4+IGNvdWxkIGJlIGFkZGVk LCBhbHRob3VnaCB0aGlzIGJlc3QgcHJhY3RpY2VzIHRleHQgaXMgYWltZWQgDQo+ID4+IHByaW1h cmlseSBhdCB1c2VzIG9mIHJlc2VydmUvcmVsZWFzZSBmb3IgbG9uZy10ZXJtIHJlc2VydmF0aW9u cy4gIEkgDQo+ID4+IHRoaW5rIHRoZXJlJ3MgYW4gaW1wbGljaXQgIndoZW4gYSByZXNlcnZhdGlv biBpcyB0aGUgZGVzaXJlZCBiZWhhdmlvciINCj4gPj4gaW1wbGljYXRpb24gdG8gdGhlICJwZXJz aXN0ZW50IHJlc2VydmF0aW9ucyBTSE9VTEQgYmUgdXNlZCBpbnN0ZWFkIg0KPiA+PiB0ZXh0IHRo YXQgSSBkb24ndCB0aGluayBuZWVkcyB0byBiZSBtYWRlIGV4cGxpY2l0LiAgT1RPSCwgdGhhdCAi U0hPVUxEIg0KPiA+PiBjb3VsZCBiZSBjaGFuZ2VkIHRvIGEgInNob3VsZCIsIG9yIGl0IG1pZ2h0 IGJlIGV2ZW4gYmV0dGVyIHRvIA0KPiA+PiByZXdyaXRlIHRoYXQgdGV4dA0KPiA+PiBhczoNCj4g Pj4NCj4gPj4gICAgcGVyc2lzdGVudCByZXNlcnZhdGlvbnMgYXJlIHN1Z2dlc3RlZCBhcyBhbiBh bHRlcm5hdGl2ZSwgc2VlIEFubmV4IEINCj4gPj4gICAgb2YgW1NQQzRdLg0KPiA+Pg0KPiA+PiBJ biBjb250cmFzdCwgIlNIT1VMRCBOT1QgYmUgdXNlZCIgaXMgY2xlYXJseSBqdXN0aWZpZWQgZm9y IA0KPiA+PiByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zIGJ5IFtTUEMzXS4NCj4gPj4NCj4g Pj4gVGhhbmtzLA0KPiA+PiAtLURhdmlkDQo+ID4+DQo+ID4+DQo+ID4+PiAtLS0tLU9yaWdpbmFs IE1lc3NhZ2UtLS0tLQ0KPiA+Pj4gRnJvbTogSnVsaWFuIFNhdHJhbiBbbWFpbHRvOmp1bGlhbkBz YXRyYW4ubmV0XQ0KPiA+Pj4gU2VudDogV2VkbmVzZGF5LCBTZXB0ZW1iZXIgMjYsIDIwMTIgMTA6 MjUgUE0NCj4gPj4+IFRvOiBCbGFjaywgRGF2aWQNCj4gPj4+IENjOiBzdG9ybUBpZXRmLm9yZw0K PiA+Pj4gU3ViamVjdDogUmU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2 ZXJzaW9uIDYNCj4gPj4+DQo+ID4+PiBEYXZpZCwNCj4gPj4+DQo+ID4+PiBUaGUgdGV4dCBpbiA0 LjQuMyBzdGF0ZXMgdGhlIG9idmlvdXMgKHRoZXJlIGlzIG5vIG90aGVyIHNwZWNlZCANCj4gPj4+ IG1ldGhvZCBmb3IgY29udGludWF0aW9uIG9yIHJlaW5zdGF0ZW1lbnQpLiBNb3Jlb3ZlciBpZiB0 aGUgdGhlIA0KPiA+Pj4gc2Vzc2lvbiBpcyByZWluc3RhdGVkIGFmdGVyIHRpbWUyd2FpdCB5b3Ug Y3JlYXRlIGEgbmVlZCBmb3IgYSBVQSANCj4gPj4+IChvciBhdCBsZWFzdCBzdWdnZXN0IG9uZSkg YW5kIHRoYXQgd2FzIG5vdCB0aGVyZS4gSnVzdCBzdGF0aW5nIA0KPiA+Pj4gdGhhdCBjcmVhdGlv bi9jb250aW51YXRpb24gb2YgYSBzZXNzaW9uIGRvbmUgYWNjb3JkaW5nIDYueCANCj4gPj4+IGlu Y2x1ZGVzIHByZXNlcnZhdGlvbiBvZiByZXNlcnZlL3JlbGVhc2Ugc3RhdGUgLXRoYXQgd2FzIGFs cmVhZHkgDQo+ID4+PiB0aGVyZSBpbXBsaWVkIGJ5IGNvbmZvcm1hbmNlIHRvIHNwYzIgYW5kIHRo ZSByZWxhdGlvbiBiZXR3ZWVuIGl0IA0KPiA+Pj4gbmV4dXMgYW5kDQo+IHNlc3Npb24uDQo+ID4+ Pg0KPiA+Pj4gVGhlIG5ldyB0ZXh0IHlvdSBzdWdnZXN0IHRvIDQuNCBpcyBtZXJlbHkgYSBiZXN0 IHByYWN0aWNlcyANCj4gPj4+IHJlY29tZW5kYXRpb25zIGFuZCB3aWxsIG5lZWQgYWRkaXRpb25z IG9uY2UgdGhlIG5ldyAibG9ja2luZyINCj4gPj4+IHRocm91Z2ggY29tcGFyZS1hbmQtc3dhcCBn ZXQgaW4gd2lkZXNwcmVhZCB1c2UuDQo+ID4+PiBJZiBpbnN0ZWFkIHdlIHJlZnJhaW4gZnJvbSBz dGF0aW5nIGFueXRoaW5nIHNwZWNpZmljIGFuZCBzdGF0ZSANCj4gPj4+IHRoYXQgdGFyZ2V0IHNo b3VsZCBtYWludGFpbiB3aGF0ZXZlciBzdGF0ZSBpcyByZXF1aXJlZCBieSBTUEMgDQo+ID4+PiB3 aGlsZSBhIHNlc3Npb24gaXMgY29udGludWVkIGFuZCBtYXkgZGlzY2FyZCBzdGF0ZSB3aGVuIGEg c2Vzc2lvbiANCj4gPj4+IGlzIHRlcm1pbmF0ZWQgYW5kIG1heSBpbmRpY2F0ZSB0aGF0IGl0IGRp c2NhcmRlZCBzdGF0ZSB0aHJvdWdoIGEgDQo+ID4+PiB1YSB5b3UgYXJlIG9uIHNhZmVyIGdyb3Vu ZCBhbmQgZG8gbm90IGNyZWF0ZSBuZXcgcmVxdWlyZW1lbnQuDQo+ID4+Pg0KPiA+Pj4gU2VudCBm cm9tIG15IGlQYWQNCj4gPj4+DQo+ID4+PiBPbiAyNyDXkdeh16TXmCAyMDEyLCBhdCAwMDoyNSwg IkJsYWNrLCBEYXZpZCIgPGRhdmlkLmJsYWNrQGVtYy5jb20+IHdyb3RlOg0KPiA+Pj4NCj4gPj4+ PiBIZXJlJ3Mgd2hlcmUgSSB0aGluayB3ZSd2ZSBhcnJpdmVkIChJJ3ZlIG1hZGUgYSBmZXcgbWlu b3IgDQo+ID4+Pj4gZWRpdG9yaWFsIHR3ZWFrcyB0byB3aGF0J3MgYmVlbiBkaXNjdXNzZWQpLiAg RGVzcGl0ZSBhbGwgdGhlIA0KPiA+Pj4+IGRldGFpbGVkIGRpc2N1c3Npb24sIEkgdGhpbmsgdGhp cyB2ZXJzaW9uIDYgY29tZXMgY2xvc2UgdG8gDQo+ID4+Pj4gSnVsaWFuJ3Mgc3VnZ2VzdGlvbg0K PiA+PiB0aGF0Og0KPiA+Pj4+DQo+ID4+Pj4+IGkgd291bGQgbWVudGlvbiBvbmx5IHRoYXQgdGhl IHN0YXRlIHRoYXQgbXVzdCBiZSBtYWludGFpbmVkIA0KPiA+Pj4+PiB3aGlsZSB0aGUgc2Vzc2lv biBpcyBtYWludGFpbmVkIGluY2x1ZGVzIHRoZSByZXNlcnZlL3JlbGVhc2UgYW5kIA0KPiA+Pj4+ PiBsZWF2ZSBpdCB0bw0KPiA+Pj4gdGhhdC4NCj4gPj4+Pg0KPiA+Pj4+IEFsdGhvdWdoIHdlIGNh bid0IGFjdHVhbGx5IHNheSAibXVzdCIgYXMgdGhhdCB3b3VsZCBiZSBhIG5ldyANCj4gPj4+PiBy ZXF1aXJlbWVudCwgYW5kIGltcGxlbWVudGF0aW9ucyBkaWZmZXIgaW4gd2hhdCB0aGV5IGRvIGFi b3V0IA0KPiA+Pj4+IHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbiBzdGF0ZS4gIFRoZSBjdXJy ZW50IGxhbmd1YWdlIA0KPiA+Pj4+IGNoYXJhY3Rlcml6ZXMgcmV0ZW50aW9uIG9mIHRoYXQgc3Rh dGUgYXMgdGhlICJwcmVmZXJyZWQgDQo+ID4+Pj4gaW1wbGVtZW50YXRpb24NCj4gPj4gYXBwcm9h Y2giLg0KPiA+Pj4+DQo+ID4+Pj4gQmV5b25kIHRoYXQsIEkgdGhpbmsgdGhlIHR3byBhZGRpdGlv bmFsIGNhdmVhdHMgYWJvdXQgDQo+ID4+Pj4gcmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9ucyAo cG9zc2libGUgbmVlZCBmb3IgYSByZXNldCwgDQo+ID4+Pj4gcG90ZW50aWFsbHkgdW5leHBlY3Rl ZCBpbnRlcmFjdGlvbiB3aXRoIHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zLCANCj4gPj4+PiBzZWUg YmVsb3cpIGFyZSB1c2VmdWwNCj4gPj4gdG8gaW5jbHVkZS4NCj4gPj4+Pg0KPiA+Pj4+IC0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ID4+Pj4NCj4gPj4+PiBbMV0g Rm9yIGNsYXJpdHksIHRoZSBmb2xsb3dpbmcgY2hhbmdlIHNob3VsZCBiZSBtYWRlIGluDQo+ID4+ Pj4gNC40LjMuMToNCj4gPj4+Pg0KPiA+Pj4+IE9MRA0KPiA+Pj4+IFRoYXQgaXMsIGl0IHNob3Vs ZCBwZXJmb3JtIHNlc3Npb24gcmVjb3ZlcnkgYXMgIGRlc2NyaWJlZCBpbiANCj4gPj4+PiBDaGFw dGVyIDYuDQo+ID4+Pj4gTkVXDQo+ID4+Pj4gVGhhdCBpcywgaXQgc2hvdWxkIHJlaW5zdGF0ZSB0 aGUgc2Vzc2lvbiB2aWEgaVNDU0kgc2Vzc2lvbiANCj4gPj4+PiByZWluc3RhdGVtZW50IChTZWN0 aW9uIDYuMy41KSBvciBjb250aW51ZSB2aWEgc2Vzc2lvbiANCj4gPj4+PiBjb250aW51YXRpb24g KFNlY3Rpb24gNi4zLjYpLg0KPiA+Pj4+IEVORA0KPiA+Pj4+DQo+ID4+Pj4gWzJdIEFkZCB0aGUg Zm9sbG93aW5nIHNlbnRlbmNlIHRvIHRoZSBlbmQgb2YgNC40LjMuMToNCj4gPj4+Pg0KPiA+Pj4+ ICAgIFRoZSBzcGVjaWZpYyBVbml0IEF0dGVudGlvbiBjb25kaXRpb24gdGhhdCByZXN1bHRzIGZy b20gc2Vzc2lvbg0KPiA+Pj4+ICAgIHJlZXN0YWJsaXNobWVudCBpbmRpY2F0ZXMgd2hldGhlciBu ZXh1cyBzdGF0ZSB3YXMgcHJlc2VydmVkLA0KPiA+Pj4+ICAgIHNlZSBTZWN0aW9uIDYuMy40IG9m IFtTQU00XS4NCj4gPj4+Pg0KPiA+Pj4+IFszXSBORVcgVEVYVDoNCj4gPj4+Pg0KPiA+Pj4+IDQu NC4zLjIuIFJlc2VydmF0aW9ucw0KPiA+Pj4+DQo+ID4+Pj4gVGhlcmUgYXJlIHR3byByZXNlcnZh dGlvbiBtYW5hZ2VtZW50IG1ldGhvZHMgZGVmaW5lZCBpbiB0aGUgU0NTSSANCj4gPj4+PiBzdGFu ZGFyZHMsIHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbnMsIGJhc2VkIG9uIHRoZSBSRVNFUlZF IGFuZCANCj4gPj4+PiBSRUxFQVNFIGNvbW1hbmRzIFtTUEMyXSwgYW5kIHBlcnNpc3RlbnQgcmVz ZXJ2YXRpb25zLCBiYXNlZCBvbiANCj4gPj4+PiB0aGUgUEVSU0lTVEVOVCBSRVNFUlZFIElOIGFu ZCBQRVJTSVNURU5UIFJFU0VSVkUgT1VUIGNvbW1hbmRzIFtTUEMzXS4NCj4gPj4+PiBSZXNlcnZl L3JlbGVhc2UgcmVzZXJ2YXRpb25zIGFyZSBvYnNvbGV0ZSBbU1BDM10gYW5kIFNIT1VMRCBOT1Qg DQo+ID4+Pj4gYmUgdXNlZDsgcGVyc2lzdGVudCByZXNlcnZhdGlvbnMgU0hPVUxEIGJlIHVzZWQg aW5zdGVhZC4NCj4gPj4+Pg0KPiA+Pj4+IFN0YXRlIGZvciBwZXJzaXN0ZW50IHJlc2VydmF0aW9u cyBpcyByZXF1aXJlZCB0byBwZXJzaXN0ICB0aHJvdWdoIA0KPiA+Pj4+IGNoYW5nZXMgYW5kIGZh aWx1cmVzIGF0IHRoZSBpU0NTSSBsYXllciB0aGF0IHJlc3VsdCBpbiAgSV9UIE5leHVzIA0KPiA+ Pj4+IGZhaWx1cmVzLCBzZWUgW1NQQzNdIGZvciBkZXRhaWxzIGFuZCBzcGVjaWZpYyByZXF1aXJl bWVudHMuDQo+ID4+Pj4NCj4gPj4+PiBJbiBjb250cmFzdCwgW1NQQzJdIGRvZXMgbm90IHNwZWNp ZnkgZGV0YWlsZWQgcGVyc2lzdGVuY2UgDQo+ID4+Pj4gcmVxdWlyZW1lbnRzIGZvciByZXNlcnZl L3JlbGVhc2UgcmVzZXJ2YXRpb24gc3RhdGUgYWZ0ZXIgYW4gSV9UIA0KPiA+Pj4+IE5leHVzIGZh aWx1cmUuICBOb25ldGhlbGVzcywgd2hlbiByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zIA0K PiA+Pj4+IGFyZSBzdXBwb3J0ZWQgYnkgYW4gaVNDU0kgdGFyZ2V0LCB0aGUgcHJlZmVycmVkIGlt cGxlbWVudGF0aW9uIA0KPiA+Pj4+IGFwcHJvYWNoIGlzIHRvIHByZXNlcnZlIHJlc2VydmUvcmVs ZWFzZSByZXNlcnZhdGlvbiBzdGF0ZSBmb3IgDQo+ID4+Pj4gaVNDU0kgc2Vzc2lvbiByZWluc3Rh dGVtZW50IChzZWUgU2VjdGlvbiA2LjMuNSkgb3Igc2Vzc2lvbiANCj4gPj4+PiBjb250aW51YXRp b24gIChzZWUgU2VjdGlvbiA2LjMuNikuDQo+ID4+Pj4NCj4gPj4+PiBUd28gYWRkaXRpb25hbCBj YXZlYXRzIGFwcGx5IHRvIHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbnM6DQo+ID4+Pj4NCj4g Pj4+PiAtIFdoZW4gY29ubmVjdGlvbiBmYWlsdXJlIGNhdXNlcyB0aGUgaVNDU0kgc2Vzc2lvbiB0 byBmYWlsLCBhbmQNCj4gPj4+PiAgICAgdGhlIHNlc3Npb24gaXMgbm90IHJlaW5zdGF0ZWQgb3Ig Y29udGludWVkLCB0YXJnZXQgcmV0ZW50aW9uDQo+ID4+Pj4gICAgIG9mIHRoYXQgc2Vzc2lvbidz IHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbiBzdGF0ZSBtYXkNCj4gPj4+PiAgICAgcmVxdWly ZSBhbiBpbml0aWF0b3IgdG8gaXNzdWUgYSByZXNldCAoZS5nLiwgTE9HSUNBTCBVTklUIFJFU0VU LA0KPiA+Pj4+ICAgICBzZWUgc2VjdGlvbiAxMS41KSBpbiBvcmRlciB0byByZW1vdmUgdGhhdCBy ZXNlcnZhdGlvbiBzdGF0ZS4NCj4gPj4+Pg0KPiA+Pj4+IC0gUmVzZXJ2ZS9yZWxlYXNlIHJlc2Vy dmF0aW9ucyBtYXkgbm90IGJlaGF2ZSBhcyBleHBlY3RlZCB3aGVuDQo+ID4+Pj4gICAgIHBlcnNp c3RlbnQgcmVzZXJ2YXRpb25zIGFyZSBhbHNvIHVzZWQgb24gdGhlIHNhbWUgbG9naWNhbCB1bml0 Ow0KPiA+Pj4+ICAgICBzZWUgdGhlIGRpc2N1c3Npb24gb2YgIkV4Y2VwdGlvbnMgdG8gU1BDLTIg UkVTRVJWRSBhbmQgUkVMRUFTRQ0KPiA+Pj4+ICAgICBiZWhhdmlvciIgaW4gW1NQQzRdLg0KPiA+ Pj4+DQo+ID4+Pj4gVGhhbmtzLA0KPiA+Pj4+IC0tRGF2aWQNCj4gPj4+PiAtLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ID4+Pj4gRGF2aWQgTC4g QmxhY2ssIERpc3Rpbmd1aXNoZWQgRW5naW5lZXIgRU1DIENvcnBvcmF0aW9uLCAxNzYgU291dGgg DQo+ID4+Pj4gU3QuLCBIb3BraW50b24sIE1BICAwMTc0OA0KPiA+Pj4+ICsxICg1MDgpIDI5My03 OTUzICAgICAgICAgICAgIEZBWDogKzEgKDUwOCkgMjkzLTc3ODYNCj4gPj4+PiBkYXZpZC5ibGFj a0BlbWMuY29tICAgICAgICBNb2JpbGU6ICsxICg5NzgpIDM5NC03NzU0DQo+ID4+Pj4gLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiA+Pj4+DQo+ ID4+Pj4NCj4gPj4+PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXw0KPiA+Pj4+IHN0b3JtIG1haWxpbmcgbGlzdA0KPiA+Pj4+IHN0b3JtQGlldGYub3JnDQo+ ID4+Pj4gaHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9zdG9ybQ0KPiA+Pg0K PiA+PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiA+ PiBzdG9ybSBtYWlsaW5nIGxpc3QNCj4gPj4gc3Rvcm1AaWV0Zi5vcmcNCj4gPj4gaHR0cHM6Ly93 d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9zdG9ybQ0KPiA+IF9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+ID4gc3Rvcm0gbWFpbGluZyBsaXN0DQo+ ID4gc3Rvcm1AaWV0Zi5vcmcNCj4gPiBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3Rp bmZvL3N0b3JtDQo= From david.black@emc.com Fri Sep 28 07:34:41 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B267F21F853D for ; Fri, 28 Sep 2012 07:34:41 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UQnFIl8kHGxG for ; Fri, 28 Sep 2012 07:34:40 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id 1E12F21F851C for ; Fri, 28 Sep 2012 07:34:37 -0700 (PDT) Received: from hop04-l1d11-si04.isus.emc.com (HOP04-L1D11-SI04.isus.emc.com [10.254.111.24]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8SEYaJN006935 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 28 Sep 2012 10:34:37 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.221.251]) by hop04-l1d11-si04.isus.emc.com (RSA Interceptor); Fri, 28 Sep 2012 10:34:26 -0400 Received: from mxhub05.corp.emc.com (mxhub05.corp.emc.com [128.222.70.202]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8SEYPY5001520; Fri, 28 Sep 2012 10:34:25 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub05.corp.emc.com ([128.222.70.202]) with mapi; Fri, 28 Sep 2012 10:34:25 -0400 From: "Black, David" To: "Knight, Frederick" Date: Fri, 28 Sep 2012 10:34:24 -0400 Thread-Topic: [storm] SPC-2 reserve/release text - version 6 Thread-Index: Ac2cNcys4MzRK8PXQu651fVKNuEIxQAXDfsAABnkfIAADjDW0AAa4S6gACikjGD//nJjgIAAECkggAAHbKCAAAZiEIAABrHg Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE79B71@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> <4AE113CD-96D7-414D-8AAB-36ECF7FD89EA@satran.net> <8D3D17ACE214DC429325B2B98F3AE7120DE798F9@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE7993C@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79B42@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-EMM-MHVC: 1 Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release text - version 6 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Sep 2012 14:34:41 -0000 SSdtIG5vdCBzdXJlIHdoYXQncyBCUk9LRU4sIGJ1dCBhIHN0YXRlbWVudCByZXF1aXJpbmcgdGhl IFVBcyB3b3VsZA0KYmUgYSBwb3N0LVNBTS0yIHJlcXVpcmVtZW50cyBjaGFuZ2UgdGhhdCB3b3Vs ZCBoYXZlIHRvIGdvIGluIHRoZQ0KLXNhbS0gZHJhZnQgc28gdGhhdCBpdCdzIHN1YmplY3QgdG8g dGhlIG5lZ290aWF0aW9uIG9mIHRoZSB0ZXh0IGtleQ0KaW4gdGhhdCBkcmFmdC4gIEkgaGF2ZSBu byBwcm9ibGVtIHdpdGggYSBibGFua2V0IHN0YXRlbWVudCB0aGF0DQpuZWdvdGlhdGluZyB0aGF0 IHRleHQga2V5IHRvIGEgdmFsdWUgdGhhdCBwcm9taXNlcyB0aGUgZnVuY3Rpb25hbGl0eQ0KaW4g dGhlIC1zYW0tIGRyYWZ0IHJlcXVpcmVzIHRoYXQgdGhlIGltcGxlbWVudGF0aW9uIGFkaGVyZSB0 bw0KZXZlcnl0aGluZyBpbiBTQU0tNCB0aGF0IG1heSBkaWZmZXIgZnJvbSBwcmlvciB2ZXJzaW9u cyBvZiBTQU0sDQppbmNsdWRpbmcgSV9UIE5leHVzIGxvc3MgVUEgYmVoYXZpb3IsIGJ1dCB0aGVy ZSB3aWxsIG5lZWQgdG8gYmUNCnNvbWUgbGFuZ3VhZ2UgaW4gdGhlcmUgYWJvdXQgd2hhdCBoYXBw ZW5zIGFmdGVyIFRpbWUyUmV0YWluIGV4cGlyZXMNCmJlZm9yZSB0aGUgSVNJRCBpcyByZXVzZWQu DQoNCk9UT0gsIEkgYW0gc3Ryb25nbHkgb3Bwb3NlZCB0byByZXZpc2luZyB0aGUgYmFzZWxpbmUg cmVxdWlyZW1lbnRzDQppbiB0aGUgY29uc29saWRhdGVkIGRyYWZ0IGluIGEgZmFzaGlvbiB0aGF0 IG1heSBicmVhayAicnVubmluZyBjb2RlIg0KdGhhdCdzIHdvcmtpbmcgZmluZSwgZXNwZWNpYWxs eSB3aGVuIHRoZSBjaGFuZ2UgaXMgYmFzZWQgb24gYSBwb2ludA0Kb2YgcHJpbmNpcGxlIHRoYXQg ZG9lcyBub3QgZXhpc3QgaW4gdGhlIHNwZWNpZmljYXRpb25zIGFnYWluc3QNCndoaWNoIHRoYXQg InJ1bm5pbmcgY29kZSIgd2FzIGRldmVsb3BlZCAoaS5lLiwgU0FNLTIpLg0KDQpUaGFua3MsDQot LURhdmlkDQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogS25pZ2h0LCBG cmVkZXJpY2sgW21haWx0bzpGcmVkZXJpY2suS25pZ2h0QG5ldGFwcC5jb21dDQo+IFNlbnQ6IEZy aWRheSwgU2VwdGVtYmVyIDI4LCAyMDEyIDEwOjIxIEFNDQo+IFRvOiBCbGFjaywgRGF2aWQNCj4g Q2M6IHN0b3JtQGlldGYub3JnDQo+IFN1YmplY3Q6IFJFOiBbc3Rvcm1dIFNQQy0yIHJlc2VydmUv cmVsZWFzZSB0ZXh0IC0gdmVyc2lvbiA2DQo+DQo+IEJ1dCB3ZSBhbHNvIHNob3VsZG4ndCBjb2Rp ZnkgQlJPS0VOIGRlc2lnbnMgaW4gYSBicmFuZCBuZXcgUkZDLiAgSXQgd291bGQNCj4gYmV0dGVy IChpbiBteSBvcGluaW9uKSB0byBsZWF2ZSBpdCBhbG9uZSB0aGFuIHNhbmN0aW9uIGFuZCBkZXNj cmliZSBob3cgdG8NCj4gYnVpbGQgYSBicm9rZW4gaW1wbGVtZW50YXRpb24uICBBZGRpbmcgdGhl ICJJRiIgaXMgd3JvbmcuDQo+DQo+IFdlJ3ZlIGZpeGVkIHRoZSBpc3N1ZXMgcmFpc2VkIGJ5IFJG QzUwNDggYnkgaW5jb3Jwb3JhdGluZyBpdCBpbnRvIHRoZSBuZXcgUkZDLg0KPg0KPiAgICAgICBG cmVkDQo+DQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IEJsYWNrLCBEYXZp ZCBbbWFpbHRvOmRhdmlkLmJsYWNrQGVtYy5jb21dDQo+IFNlbnQ6IEZyaWRheSwgU2VwdGVtYmVy IDI4LCAyMDEyIDk6NTEgQU0NCj4gVG86IEtuaWdodCwgRnJlZGVyaWNrDQo+IENjOiBzdG9ybUBp ZXRmLm9yZw0KPiBTdWJqZWN0OiBSRTogW3N0b3JtXSBTUEMtMiByZXNlcnZlL3JlbGVhc2UgdGV4 dCAtIHZlcnNpb24gNg0KPg0KPiBGcmVkLA0KPg0KPiA+IElmIHN0YXRlIGlzIHJldGFpbmVkIChp dCdzIGFuIG9sZCAiSSIgcmVjb25uZWN0aW5nKSwgdGhlbiB3aGljaCBVQSBpcw0KPiA+IGRlZmlu ZWQgYnkgU0FNLTMuDQo+ID4NCj4gPiBJZiBzdGF0ZSBpcyBub3QgcmV0YWluZWQgKGl0J3MgYSBu ZXcgIkkiKSwgdGhlbiB0aGUgVUEgaXMgZGVmaW5lZCBieSBTQU0tMy4NCj4gPg0KPiA+IFNpbmNl IGJvdGggb2xkICJJInMgYW5kIG5ldyAiSSJzIHJlc3VsdCBpbiB0aGUgIklGIiBjb25kaXRpb24g YmVpbmcNCj4gPiB0cnVlLCB3aGF0IGlzIHRoZSBjb25kaXRpb24gdGhhdCB3b3VsZCBtYWtlIHRo ZSAiSUYiIGZhbHNlPw0KPg0KPiBZb3UgYWxtb3N0IGFuc3dlcmVkIHlvdXIgb3duIHF1ZXN0aW9u IDstKS4NCj4NCj4gVGhlIGFuc3dlciBpcyAuLi4uIChkcnVtIHJvbGwpIC4uLiBhIHRhcmdldCBp bXBsZW1lbnRhdGlvbiBiYXNlZCBvbiBSRkMgMzcyMA0KPiBTQU0tMiwgYWx0aG91Z2ggSSB3b3Vs ZCBob3BlIHRoYXQgc3VjaCBpbXBsZW1lbnRhdGlvbnMgYXQgbGVhc3QgcGlja2VkIHVwIFJGQw0K PiA1MDQ4J3MgY2hhbmdlcy4NCj4NCj4gVGhlIElfVCBOZXh1cyBMb3NzIGNvbmNlcHQgb24gd2hp Y2ggdGhpcyBVQSBkaXNjdXNzaW9uIGlzIGJhc2VkIGlzIHBhcnQgb2YgdGhlDQo+IFNDU0kgZXZl bnRzIGFuZCBldmVudCBub3RpZmljYXRpb25zIG1vZGVsIHRoYXQgd2FzIGludHJvZHVjZWQgaW4g U0FNLTMsIGJ1dA0KPiBuZWl0aGVyIFJGQyAzNzIwIG5vciBSRkMgNTA0OCByZWZlcmVuY2VzIFNB TS0zLg0KPg0KPiBUaGVyZSB3ZXJlIGRlZmluaXRlbHkgaW1wbGVtZW50YXRpb25zIG9mIGlTQ1NJ IGJhc2VkIG9uIFNBTS0yIC0gSSBhZ3JlZSB3aXRoDQo+IEp1bGlhbiB0aGF0IHdlIHNob3VsZCAq bm90KiBiZSBpbnRyb2R1Y2luZyBhIFVBIHJlcXVpcmVtZW50IGluIHRoaXMgZHJhZnQgdGhhdA0K PiBtaWdodCBicmVhayBzdWNoIGltcGxlbWVudGF0aW9ucy4NCj4NCj4gVGhhbmtzLA0KPiAtLURh dmlkDQo+DQo+ID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPiBGcm9tOiBLbmlnaHQs IEZyZWRlcmljayBbbWFpbHRvOkZyZWRlcmljay5LbmlnaHRAbmV0YXBwLmNvbV0NCj4gPiBTZW50 OiBGcmlkYXksIFNlcHRlbWJlciAyOCwgMjAxMiA4OjM4IEFNDQo+ID4gVG86IEp1bGlhbiBTYXRy YW4NCj4gPiBDYzogQmxhY2ssIERhdmlkOyBzdG9ybUBpZXRmLm9yZw0KPiA+IFN1YmplY3Q6IFJF OiBbc3Rvcm1dIFNQQy0yIHJlc2VydmUvcmVsZWFzZSB0ZXh0IC0gdmVyc2lvbiA2DQo+ID4NCj4g PiBPSywgdGhhdCdzIHdoYXQgSSB0aG91Z2h0Lg0KPiA+DQo+ID4gSWYgc3RhdGUgaXMgcmV0YWlu ZWQgKGl0J3MgYW4gb2xkICJJIiByZWNvbm5lY3RpbmcpLCB0aGVuIHdoaWNoIFVBIGlzDQo+ID4g ZGVmaW5lZCBieSBTQU0tMy4NCj4gPg0KPiA+IElmIHN0YXRlIGlzIG5vdCByZXRhaW5lZCAoaXQn cyBhIG5ldyAiSSIpLCB0aGVuIHRoZSBVQSBpcyBkZWZpbmVkIGJ5IFNBTS0zLg0KPiA+DQo+ID4g U2luY2UgYm90aCBvbGQgIkkicyBhbmQgbmV3ICJJInMgcmVzdWx0IGluIHRoZSAiSUYiIGNvbmRp dGlvbiBiZWluZw0KPiA+IHRydWUsIHdoYXQgaXMgdGhlIGNvbmRpdGlvbiB0aGF0IHdvdWxkIG1h a2UgdGhlICJJRiIgZmFsc2U/DQo+ID4NCj4gPiBUaGF0ICJJRiIgbGFuZ3VhZ2UgYWxsb3dzIGEg dGFyZ2V0IHRvIGxvb3NlIHN0YXRlIGFuZCBub3QgZGVsaXZlciBhbnkgVUEuDQo+ID4NCj4gPiAg ICAgRnJlZA0KPiA+DQo+ID4NCj4gPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiA+IEZy b206IEp1bGlhbiBTYXRyYW4gW21haWx0bzpqdWxpYW5Ac2F0cmFuLm5ldF0NCj4gPiBTZW50OiBG cmlkYXksIFNlcHRlbWJlciAyOCwgMjAxMiAyOjAzIEFNDQo+ID4gVG86IEtuaWdodCwgRnJlZGVy aWNrDQo+ID4gQ2M6IEJsYWNrLCBEYXZpZDsgc3Rvcm1AaWV0Zi5vcmcNCj4gPiBTdWJqZWN0OiBS ZTogW3N0b3JtXSBTUEMtMiByZXNlcnZlL3JlbGVhc2UgdGV4dCAtIHZlcnNpb24gNg0KPiA+DQo+ ID4gVGhlcmUgaXMgbm8gc3VjaCB0aGluZy4gQW5kIHRoZSByZWFzb24gdGhlIHRpbWVvdXQgaXMg dGhlcmUgaXMgdG8NCj4gPiBsaW1pdCB0aGUgYW1vdW50IG9mIHN0YXRlIHRoZSB0YXJnZXRzIGhh dmUgdG8ga2VlcC4gQSB0YXJnZXQgbWF5DQo+ID4gY2hvb3NlIHRvIGFjY2VwdCB2ZXJ5IGxhcmdl IHRpbWVvdXRzIGFuZCB0aGVuIGRvIHdoYXQgeW91IHdhbnRlZCBkb25lDQo+ID4gYnV0IGJleW9u ZCB0aGF0IG9ubHkgdGhlIHBvd2VyLXVwIHVhIGlzIHJlcXVpcmVkLg0KPiA+DQo+ID4ganVsbw0K PiA+DQo+ID4gU2VudCBmcm9tIG15IGlQYWQNCj4gPg0KPiA+IE9uIDI4INeR16HXpNeYIDIwMTIs IGF0IDAwOjI5LCAiS25pZ2h0LCBGcmVkZXJpY2siDQo+ID4gPEZyZWRlcmljay5LbmlnaHRAbmV0 YXBwLmNvbT4NCj4gPiB3cm90ZToNCj4gPg0KPiA+ID4gSSB3b3VsZCB0aGluayBpZiB0aGUgSVNJ RCBpcyBub3QgcmVjb2duaXplZCBhcyBhbiBvbGQgb25lLCB0aGVuIGl0DQo+ID4gPiBtdXN0IGJl DQo+ID4gcmVjb2duaXplZCBhcyBhIG5ldyBvbmUuICBJZiBpdCBpcyBhIG5ldyBvbmUsIHRoZW4g aXQgd2lsbCBnZXQgYSBVQToNCj4gPiAyOS8wMCByZWZsZWN0aW5nIHRoZSBmaXJzdCBhY2Nlc3Mg YWZ0ZXIgcG93ZXIgb24gZnJvbSB0aGUgIm5ldyIgaW5pdGlhdG9yLg0KPiA+ID4NCj4gPiA+IElz IHRoZXJlIHNvbWV0aGluZyBvdGhlciB0aGFuIG5ldyBvciBvbGQ/ICBJcyB0aGVyZSBzdWNoIGEg dGhpbmcgYXMNCj4gPiA+IGENCj4gPiBtaWRkbGUtYWdlZCBpbml0aWF0b3I/DQo+ID4gPg0KPiA+ ID4gICAgRnJlZA0KPiA+ID4NCj4gPiA+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+ID4g PiBGcm9tOiBCbGFjaywgRGF2aWQgW21haWx0bzpkYXZpZC5ibGFja0BlbWMuY29tXQ0KPiA+ID4g U2VudDogVGh1cnNkYXksIFNlcHRlbWJlciAyNywgMjAxMiAxMTo1NiBBTQ0KPiA+ID4gVG86IEtu aWdodCwgRnJlZGVyaWNrDQo+ID4gPiBDYzogc3Rvcm1AaWV0Zi5vcmcNCj4gPiA+IFN1YmplY3Q6 IFJFOiBbc3Rvcm1dIFNQQy0yIHJlc2VydmUvcmVsZWFzZSB0ZXh0IC0gdmVyc2lvbiA2DQo+ID4g Pg0KPiA+ID4gRnJlZCwNCj4gPiA+DQo+ID4gPiBUaGUgcHJvYmxlbSBpcyBub3QgdGhhdCB0aGUg ZGV2aWNlIGRvZXMgbm90IHJhaXNlIGEgVUEsIGl0J3MgdGhhdA0KPiA+ID4gdGhlIFVBIG1heQ0K PiA+IG5ldmVyIGJlIGRlbGl2ZXJlZC4NCj4gPiA+DQo+ID4gPiBBIHN1bW1hcnkgb2YgaG93IHRo aXMgY2FuIGhhcHBlbiBpcyB0aGF0IHRoZSBuZXcgY29ubmVjdGlvbiB3aXRoIHRoZQ0KPiA+ID4g c2FtZQ0KPiA+IElTSUQgbWF5IG5vdCBiZSByZWNvZ25pemVkIGFzIHJlaW5zdGF0aW5nIG9yIGNv bnRpbnVpbmcgdGhlIHNlc3Npb24gYXQNCj4gPiB0aGUgdGFyZ2V0IGlmIHRoZSB0aW1lb3V0cyBo YXZlIGV4cGlyZWQgLSB0aGUgVUEgZ2V0cyBlc3RhYmxpc2hlZCwgYnV0DQo+ID4gbm9ib2R5IHNo b3dzIHVwIHRvIHBpY2sgaXQgdXAgaW4gdGltZSwgc28gdGhlIFVBIGlzIGRpc2NhcmRlZCBhbG9u Zw0KPiA+IHdpdGggdGhlIHJlc3Qgb2YgdGhlIHNlc3Npb24gc3RhdGUgYW5kIHRoZSB0YXJnZXQg ZmFpbHMgdG8gcmVjb2duaXplDQo+ID4gdGhlIGluaXRpYXRvcidzIHN1YnNlcXVlbnQgSVNJRCBy ZXVzZSBldmVuIHRob3VnaCB0aGUgaW5pdGlhdG9yDQo+ID4gYmVsaWV2ZXMgdGhhdCBpdCdzIHJl LSBlc3RhYmxpc2hpbmcgdGhlIHNlc3Npb24uDQo+ID4gPg0KPiA+ID4gVGhpcyBpcyBmdXJ0aGVy IGNvbXBsaWNhdGVkIGJ5IHRoZSBVQSB0aGF0IGluZGljYXRlcyBuZXh1cyBzdGF0ZQ0KPiA+IHBy ZXNlcnZhdGlvbiBub3QgYmVpbmcgc3BlY2lmaWVkIGluIFNBTS0yLg0KPiA+ID4NCj4gPiA+IFRo YW5rcywNCj4gPiA+IC0tRGF2aWQNCj4gPiA+DQo+ID4gPg0KPiA+ID4+IC0tLS0tT3JpZ2luYWwg TWVzc2FnZS0tLS0tDQo+ID4gPj4gRnJvbTogS25pZ2h0LCBGcmVkZXJpY2sgW21haWx0bzpGcmVk ZXJpY2suS25pZ2h0QG5ldGFwcC5jb21dDQo+ID4gPj4gU2VudDogVGh1cnNkYXksIFNlcHRlbWJl ciAyNywgMjAxMiAxMToxMSBBTQ0KPiA+ID4+IFRvOiBCbGFjaywgRGF2aWQ7IEp1bGlhbiBTYXRy YW4NCj4gPiA+PiBDYzogc3Rvcm1AaWV0Zi5vcmcNCj4gPiA+PiBTdWJqZWN0OiBSRTogW3N0b3Jt XSBTUEMtMiByZXNlcnZlL3JlbGVhc2UgdGV4dCAtIHZlcnNpb24gNg0KPiA+ID4+DQo+ID4gPj4g SSBhZnJhaWQgSSBoYXZlIHRvIGNvbnRlc3QgYSBsaXR0bGUgb24gdGhpcyAtIGlmIEkgdW5kZXJz dGFuZCB0aGlzDQo+ID4gPj4gZGlzY3Vzc2lvbiBjb3JyZWN0bHkuDQo+ID4gPj4NCj4gPiA+PiBJ ZiBhIFNDU0kgZGV2aWNlIGxvc2VzIGFueSBzdGF0ZSBhbmQgZG9lcyBub3QgcmFpc2UgYSBVTklU DQo+ID4gPj4gQVRURU5USU9OLCB0aGVuIGl0IGlzIEJST0tFTi4gIEl0IGlzIG5vdCBhIG1hdHRl ciBvZiBhbnl0aGluZyBpbg0KPiA+ID4+IGlTQ1NJICgzNzIwKTsgSSB0aGluayBTQU0gYW5kIFNQ QyBhcmUgcHJldHR5IGNsZWFyIG9uIHRoaXMuICBJZg0KPiA+ID4+IHRoZXJlIGlzIG5vIFVOSVQg QVRURU5USU9OLCB0aGVuIHN0YXRlIFNIQUxMIGJlIHByZXNlcnZlZDsNCj4gPiA+PiBvdGhlcndp c2UsIGFsbCBzb3J0cyBvZiBzdHVmZiB3aWxsIG5ldmVyIHdvcmsgKG1vZGUgcGFnZSBzZXR0aW5n cywNCj4gPiA+PiBSRVNFUlZFIGNvbW1hbmRzLCBhbGwgc29ydHMgb2YNCj4gPiBzdHVmZikuDQo+ ID4gPj4NCj4gPiA+PiBJcyBzb21lb25lIHJlYWxseSBzdWdnZXN0aW5nIHRoYXQgc29tZSBkZXZp Y2VzIHdpbGwgZHJvcCBzdGF0ZSBhbmQNCj4gPiA+PiBOT1QgdGVsbCB0aGUgaG9zdCwgYW5kIHNv bWVob3cgdGhhdCBpdCBpcyBPSyB0byBkbyB0aGF0Pw0KPiA+ID4+DQo+ID4gPj4gVGhlIHN1Z2dl c3QgcmV3b3JkaW5nIHNheXMgdGhpcyBpcyBwb3NzaWJsZS4NCj4gPiA+Pg0KPiA+ID4+ICAgIEZy ZWQNCj4gPiA+Pg0KPiA+ID4+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+ID4gPj4gRnJv bTogc3Rvcm0tYm91bmNlc0BpZXRmLm9yZyBbbWFpbHRvOnN0b3JtLWJvdW5jZXNAaWV0Zi5vcmdd IE9uDQo+ID4gPj4gQmVoYWxmIE9mIEJsYWNrLCBEYXZpZA0KPiA+ID4+IFNlbnQ6IFRodXJzZGF5 LCBTZXB0ZW1iZXIgMjcsIDIwMTIgMTA6NDcgQU0NCj4gPiA+PiBUbzogSnVsaWFuIFNhdHJhbg0K PiA+ID4+IENjOiBzdG9ybUBpZXRmLm9yZw0KPiA+ID4+IFN1YmplY3Q6IFJlOiBbc3Rvcm1dIFNQ Qy0yIHJlc2VydmUvcmVsZWFzZSB0ZXh0IC0gdmVyc2lvbiA2DQo+ID4gPj4NCj4gPiA+PiBKdWxp YW4sDQo+ID4gPj4NCj4gPiA+Pj4gVGhlIHRleHQgaW4gNC40LjMgc3RhdGVzIHRoZSBvYnZpb3Vz ICh0aGVyZSBpcyBubyBvdGhlciBzcGVjZWQNCj4gPiA+Pj4gbWV0aG9kIGZvciBjb250aW51YXRp b24gb3IgcmVpbnN0YXRlbWVudCkuIE1vcmVvdmVyIGlmIHRoZSBzZXNzaW9uDQo+ID4gPj4+IGlz IHJlaW5zdGF0ZWQgYWZ0ZXIgdGltZTJ3YWl0IHlvdSBjcmVhdGUgYSBuZWVkIGZvciBhIFVBIChv ciBhdA0KPiA+ID4+PiBsZWFzdCBzdWdnZXN0IG9uZSkgYW5kIHRoYXQgd2FzIG5vdCB0aGVyZS4N Cj4gPiA+Pg0KPiA+ID4+IFRoZSBVQXMgYXJlIGNvdXJ0ZXN5IG9mIFNBTS0zIChhbmQgU0FNLTQp LiAgVG8gZW5jb21wYXNzIHN5c3RlbXMNCj4gPiA+PiB0aGF0IG1heSBub3QgZGVsaXZlciB0aGVz ZSBVQXMgYW5kIHNpdHVhdGlvbnMgaW4gd2hpY2ggdGhleSBkb24ndA0KPiA+ID4+IG9jY3VyIChl LmcuLCB0aGUgdGFyZ2V0IGRvZXNuJ3QgcmVjb2duaXplIHdoYXQgaGFwcGVuZWQgYXMgbmV4dXMN Cj4gPiA+PiByZWVzdGFibGlzaG1lbnQsIGluZGVwZW5kZW50IG9mIHdoYXQgdGhlIGluaXRpYXRv ciB0aGlua3MgaXQncw0KPiA+ID4+IGRvaW5nKSBJDQo+ID4gdGhpbmsgdGhlIHRleHQgc2hvdWxk IGJlIHJlcGhyYXNlZCBhczoNCj4gPiA+Pg0KPiA+ID4+ICAgICBJZiBhIFVuaXQgQXR0ZW50aW9u IGNvbmRpdGlvbiByZXN1bHRzIGZyb20gc2Vzc2lvbiByZWVzdGFibGlzaG1lbnQsDQo+ID4gPj4g ICAgIHRoZSBzcGVjaWZpYyBVbml0IEF0dGVudGlvbiBjb25kaXRpb24gaW5kaWNhdGVzIHdoZXRo ZXIgbmV4dXMgc3RhdGUNCj4gPiA+PiAgICAgd2FzIHByZXNlcnZlZCwgc2VlIFNlY3Rpb24gNi4z LjQgb2YgW1NBTTRdLg0KPiA+ID4+DQo+ID4gPj4+IEp1c3Qgc3RhdGluZyB0aGF0IGNyZWF0aW9u L2NvbnRpbnVhdGlvbiBvZiBhIHNlc3Npb24gZG9uZQ0KPiA+ID4+PiBhY2NvcmRpbmcgNi54IGlu Y2x1ZGVzIHByZXNlcnZhdGlvbiBvZiByZXNlcnZlL3JlbGVhc2Ugc3RhdGUgLXRoYXQNCj4gPiA+ Pj4gd2FzIGFscmVhZHkgdGhlcmUgaW1wbGllZCBieSBjb25mb3JtYW5jZSB0byBzcGMyIGFuZCB0 aGUgcmVsYXRpb24NCj4gPiA+Pj4gYmV0d2VlbiBpdCBuZXh1cyBhbmQgc2Vzc2lvbi4NCj4gPiA+ Pg0KPiA+ID4+IFVuZm9ydHVuYXRlbHksIFNQQy0yIGlzIHN1ZmZpY2llbnRseSBpbXByZWNpc2Ug b24gdGhpcyBzdWJqZWN0IHRoYXQNCj4gPiA+PiBzdWNoIGFuIGltcGxpY2F0aW9uIGlzIGF0IGJl c3QgYSAicHJlZmVycmVkIGltcGxlbWVudGF0aW9uIGFwcHJvYWNoIi4NCj4gPiA+PiBJdCdzIG5v dCBhIHJlcXVpcmVtZW50IHRoYXQgY2FuIGJlIHNhZmVseSBzdGF0ZWQgaW4gYW55IHN0cm9uZ2Vy DQo+ID4gPj4gZmFzaGlvbiB3aXRob3V0IHRoZSByaXNrIG9mIHJlbmRlcmluZyBleGlzdGluZyBp bXBsZW1lbnRhdGlvbnMgbm9uLQ0KPiA+IGNvbXBsaWFudC4NCj4gPiA+Pg0KPiA+ID4+PiBUaGUg bmV3IHRleHQgeW91IHN1Z2dlc3QgdG8gNC40IGlzIG1lcmVseSBhIGJlc3QgcHJhY3RpY2VzDQo+ ID4gPj4+IHJlY29tbWVuZGF0aW9ucyBhbmQgd2lsbCBuZWVkIGFkZGl0aW9ucyBvbmNlIHRoZSBu ZXcgImxvY2tpbmciDQo+ID4gPj4+IHRocm91Z2ggY29tcGFyZS1hbmQtc3dhcCBnZXQgaW4gd2lk ZXNwcmVhZCB1c2UuDQo+ID4gPj4NCj4gPiA+PiBJdCdzIGRlZmluaXRlbHkgYmVzdCBwcmFjdGlj ZXMgcmVjb21tZW5kYXRpb25zIC0gbm8gYXJndW1lbnQgdGhlcmUuDQo+ID4gPj4NCj4gPiA+PiBG b3IgY29udGV4dCwgd2hhdCBKdWxpYW4gaXMgcmVmZXJyaW5nIHRvIGlzIHVzZSBvZiB0aGUgQ09N UEFSRSBBTkQNCj4gPiA+PiBXUklURSBjb21tYW5kIHRoYXQgcGVyZm9ybXMgYW4gYXRvbWljIG9w ZXJhdGlvbiBhdCB0aGUgdGFyZ2V0DQo+ID4gPj4gaW5zdGVhZCBvZiB1c2luZyBSRVNFUlZFIGFu ZCBSRUxFQVNFIHRvIGNyZWF0ZSBhIGNyaXRpY2FsIHNlY3Rpb24NCj4gPiA+PiBhdCB0aGUgaW5p dGlhdG9yIGJhc2VkIG9uIHJlYWQgYW5kIHdyaXRlIGNvbW1hbmRzLg0KPiA+ID4+DQo+ID4gPj4g Q09NUEFSRSBBTkQgV1JJVEUgaXMgaW4gdXNlIC0gYSBzZW50ZW5jZSBtZW50aW9uaW5nIHRoYXQg Y29tbWFuZA0KPiA+ID4+IGNvdWxkIGJlIGFkZGVkLCBhbHRob3VnaCB0aGlzIGJlc3QgcHJhY3Rp Y2VzIHRleHQgaXMgYWltZWQNCj4gPiA+PiBwcmltYXJpbHkgYXQgdXNlcyBvZiByZXNlcnZlL3Jl bGVhc2UgZm9yIGxvbmctdGVybSByZXNlcnZhdGlvbnMuICBJDQo+ID4gPj4gdGhpbmsgdGhlcmUn cyBhbiBpbXBsaWNpdCAid2hlbiBhIHJlc2VydmF0aW9uIGlzIHRoZSBkZXNpcmVkIGJlaGF2aW9y Ig0KPiA+ID4+IGltcGxpY2F0aW9uIHRvIHRoZSAicGVyc2lzdGVudCByZXNlcnZhdGlvbnMgU0hP VUxEIGJlIHVzZWQgaW5zdGVhZCINCj4gPiA+PiB0ZXh0IHRoYXQgSSBkb24ndCB0aGluayBuZWVk cyB0byBiZSBtYWRlIGV4cGxpY2l0LiAgT1RPSCwgdGhhdCAiU0hPVUxEIg0KPiA+ID4+IGNvdWxk IGJlIGNoYW5nZWQgdG8gYSAic2hvdWxkIiwgb3IgaXQgbWlnaHQgYmUgZXZlbiBiZXR0ZXIgdG8N Cj4gPiA+PiByZXdyaXRlIHRoYXQgdGV4dA0KPiA+ID4+IGFzOg0KPiA+ID4+DQo+ID4gPj4gICAg cGVyc2lzdGVudCByZXNlcnZhdGlvbnMgYXJlIHN1Z2dlc3RlZCBhcyBhbiBhbHRlcm5hdGl2ZSwg c2VlIEFubmV4IEINCj4gPiA+PiAgICBvZiBbU1BDNF0uDQo+ID4gPj4NCj4gPiA+PiBJbiBjb250 cmFzdCwgIlNIT1VMRCBOT1QgYmUgdXNlZCIgaXMgY2xlYXJseSBqdXN0aWZpZWQgZm9yDQo+ID4g Pj4gcmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9ucyBieSBbU1BDM10uDQo+ID4gPj4NCj4gPiA+ PiBUaGFua3MsDQo+ID4gPj4gLS1EYXZpZA0KPiA+ID4+DQo+ID4gPj4NCj4gPiA+Pj4gLS0tLS1P cmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPiA+Pj4gRnJvbTogSnVsaWFuIFNhdHJhbiBbbWFpbHRv Omp1bGlhbkBzYXRyYW4ubmV0XQ0KPiA+ID4+PiBTZW50OiBXZWRuZXNkYXksIFNlcHRlbWJlciAy NiwgMjAxMiAxMDoyNSBQTQ0KPiA+ID4+PiBUbzogQmxhY2ssIERhdmlkDQo+ID4gPj4+IENjOiBz dG9ybUBpZXRmLm9yZw0KPiA+ID4+PiBTdWJqZWN0OiBSZTogW3N0b3JtXSBTUEMtMiByZXNlcnZl L3JlbGVhc2UgdGV4dCAtIHZlcnNpb24gNg0KPiA+ID4+Pg0KPiA+ID4+PiBEYXZpZCwNCj4gPiA+ Pj4NCj4gPiA+Pj4gVGhlIHRleHQgaW4gNC40LjMgc3RhdGVzIHRoZSBvYnZpb3VzICh0aGVyZSBp cyBubyBvdGhlciBzcGVjZWQNCj4gPiA+Pj4gbWV0aG9kIGZvciBjb250aW51YXRpb24gb3IgcmVp bnN0YXRlbWVudCkuIE1vcmVvdmVyIGlmIHRoZSB0aGUNCj4gPiA+Pj4gc2Vzc2lvbiBpcyByZWlu c3RhdGVkIGFmdGVyIHRpbWUyd2FpdCB5b3UgY3JlYXRlIGEgbmVlZCBmb3IgYSBVQQ0KPiA+ID4+ PiAob3IgYXQgbGVhc3Qgc3VnZ2VzdCBvbmUpIGFuZCB0aGF0IHdhcyBub3QgdGhlcmUuIEp1c3Qg c3RhdGluZw0KPiA+ID4+PiB0aGF0IGNyZWF0aW9uL2NvbnRpbnVhdGlvbiBvZiBhIHNlc3Npb24g ZG9uZSBhY2NvcmRpbmcgNi54DQo+ID4gPj4+IGluY2x1ZGVzIHByZXNlcnZhdGlvbiBvZiByZXNl cnZlL3JlbGVhc2Ugc3RhdGUgLXRoYXQgd2FzIGFscmVhZHkNCj4gPiA+Pj4gdGhlcmUgaW1wbGll ZCBieSBjb25mb3JtYW5jZSB0byBzcGMyIGFuZCB0aGUgcmVsYXRpb24gYmV0d2VlbiBpdA0KPiA+ ID4+PiBuZXh1cyBhbmQNCj4gPiBzZXNzaW9uLg0KPiA+ID4+Pg0KPiA+ID4+PiBUaGUgbmV3IHRl eHQgeW91IHN1Z2dlc3QgdG8gNC40IGlzIG1lcmVseSBhIGJlc3QgcHJhY3RpY2VzDQo+ID4gPj4+ IHJlY29tZW5kYXRpb25zIGFuZCB3aWxsIG5lZWQgYWRkaXRpb25zIG9uY2UgdGhlIG5ldyAibG9j a2luZyINCj4gPiA+Pj4gdGhyb3VnaCBjb21wYXJlLWFuZC1zd2FwIGdldCBpbiB3aWRlc3ByZWFk IHVzZS4NCj4gPiA+Pj4gSWYgaW5zdGVhZCB3ZSByZWZyYWluIGZyb20gc3RhdGluZyBhbnl0aGlu ZyBzcGVjaWZpYyBhbmQgc3RhdGUNCj4gPiA+Pj4gdGhhdCB0YXJnZXQgc2hvdWxkIG1haW50YWlu IHdoYXRldmVyIHN0YXRlIGlzIHJlcXVpcmVkIGJ5IFNQQw0KPiA+ID4+PiB3aGlsZSBhIHNlc3Np b24gaXMgY29udGludWVkIGFuZCBtYXkgZGlzY2FyZCBzdGF0ZSB3aGVuIGEgc2Vzc2lvbg0KPiA+ ID4+PiBpcyB0ZXJtaW5hdGVkIGFuZCBtYXkgaW5kaWNhdGUgdGhhdCBpdCBkaXNjYXJkZWQgc3Rh dGUgdGhyb3VnaCBhDQo+ID4gPj4+IHVhIHlvdSBhcmUgb24gc2FmZXIgZ3JvdW5kIGFuZCBkbyBu b3QgY3JlYXRlIG5ldyByZXF1aXJlbWVudC4NCj4gPiA+Pj4NCj4gPiA+Pj4gU2VudCBmcm9tIG15 IGlQYWQNCj4gPiA+Pj4NCj4gPiA+Pj4gT24gMjcg15HXodek15ggMjAxMiwgYXQgMDA6MjUsICJC bGFjaywgRGF2aWQiIDxkYXZpZC5ibGFja0BlbWMuY29tPiB3cm90ZToNCj4gPiA+Pj4NCj4gPiA+ Pj4+IEhlcmUncyB3aGVyZSBJIHRoaW5rIHdlJ3ZlIGFycml2ZWQgKEkndmUgbWFkZSBhIGZldyBt aW5vcg0KPiA+ID4+Pj4gZWRpdG9yaWFsIHR3ZWFrcyB0byB3aGF0J3MgYmVlbiBkaXNjdXNzZWQp LiAgRGVzcGl0ZSBhbGwgdGhlDQo+ID4gPj4+PiBkZXRhaWxlZCBkaXNjdXNzaW9uLCBJIHRoaW5r IHRoaXMgdmVyc2lvbiA2IGNvbWVzIGNsb3NlIHRvDQo+ID4gPj4+PiBKdWxpYW4ncyBzdWdnZXN0 aW9uDQo+ID4gPj4gdGhhdDoNCj4gPiA+Pj4+DQo+ID4gPj4+Pj4gaSB3b3VsZCBtZW50aW9uIG9u bHkgdGhhdCB0aGUgc3RhdGUgdGhhdCBtdXN0IGJlIG1haW50YWluZWQNCj4gPiA+Pj4+PiB3aGls ZSB0aGUgc2Vzc2lvbiBpcyBtYWludGFpbmVkIGluY2x1ZGVzIHRoZSByZXNlcnZlL3JlbGVhc2Ug YW5kDQo+ID4gPj4+Pj4gbGVhdmUgaXQgdG8NCj4gPiA+Pj4gdGhhdC4NCj4gPiA+Pj4+DQo+ID4g Pj4+PiBBbHRob3VnaCB3ZSBjYW4ndCBhY3R1YWxseSBzYXkgIm11c3QiIGFzIHRoYXQgd291bGQg YmUgYSBuZXcNCj4gPiA+Pj4+IHJlcXVpcmVtZW50LCBhbmQgaW1wbGVtZW50YXRpb25zIGRpZmZl ciBpbiB3aGF0IHRoZXkgZG8gYWJvdXQNCj4gPiA+Pj4+IHJlc2VydmUvcmVsZWFzZSByZXNlcnZh dGlvbiBzdGF0ZS4gIFRoZSBjdXJyZW50IGxhbmd1YWdlDQo+ID4gPj4+PiBjaGFyYWN0ZXJpemVz IHJldGVudGlvbiBvZiB0aGF0IHN0YXRlIGFzIHRoZSAicHJlZmVycmVkDQo+ID4gPj4+PiBpbXBs ZW1lbnRhdGlvbg0KPiA+ID4+IGFwcHJvYWNoIi4NCj4gPiA+Pj4+DQo+ID4gPj4+PiBCZXlvbmQg dGhhdCwgSSB0aGluayB0aGUgdHdvIGFkZGl0aW9uYWwgY2F2ZWF0cyBhYm91dA0KPiA+ID4+Pj4g cmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9ucyAocG9zc2libGUgbmVlZCBmb3IgYSByZXNldCwN Cj4gPiA+Pj4+IHBvdGVudGlhbGx5IHVuZXhwZWN0ZWQgaW50ZXJhY3Rpb24gd2l0aCBwZXJzaXN0 ZW50IHJlc2VydmF0aW9ucywNCj4gPiA+Pj4+IHNlZSBiZWxvdykgYXJlIHVzZWZ1bA0KPiA+ID4+ IHRvIGluY2x1ZGUuDQo+ID4gPj4+Pg0KPiA+ID4+Pj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0NCj4gPiA+Pj4+DQo+ID4gPj4+PiBbMV0gRm9yIGNsYXJpdHksIHRo ZSBmb2xsb3dpbmcgY2hhbmdlIHNob3VsZCBiZSBtYWRlIGluDQo+ID4gPj4+PiA0LjQuMy4xOg0K PiA+ID4+Pj4NCj4gPiA+Pj4+IE9MRA0KPiA+ID4+Pj4gVGhhdCBpcywgaXQgc2hvdWxkIHBlcmZv cm0gc2Vzc2lvbiByZWNvdmVyeSBhcyAgZGVzY3JpYmVkIGluDQo+ID4gPj4+PiBDaGFwdGVyIDYu DQo+ID4gPj4+PiBORVcNCj4gPiA+Pj4+IFRoYXQgaXMsIGl0IHNob3VsZCByZWluc3RhdGUgdGhl IHNlc3Npb24gdmlhIGlTQ1NJIHNlc3Npb24NCj4gPiA+Pj4+IHJlaW5zdGF0ZW1lbnQgKFNlY3Rp b24gNi4zLjUpIG9yIGNvbnRpbnVlIHZpYSBzZXNzaW9uDQo+ID4gPj4+PiBjb250aW51YXRpb24g KFNlY3Rpb24gNi4zLjYpLg0KPiA+ID4+Pj4gRU5EDQo+ID4gPj4+Pg0KPiA+ID4+Pj4gWzJdIEFk ZCB0aGUgZm9sbG93aW5nIHNlbnRlbmNlIHRvIHRoZSBlbmQgb2YgNC40LjMuMToNCj4gPiA+Pj4+ DQo+ID4gPj4+PiAgICBUaGUgc3BlY2lmaWMgVW5pdCBBdHRlbnRpb24gY29uZGl0aW9uIHRoYXQg cmVzdWx0cyBmcm9tIHNlc3Npb24NCj4gPiA+Pj4+ICAgIHJlZXN0YWJsaXNobWVudCBpbmRpY2F0 ZXMgd2hldGhlciBuZXh1cyBzdGF0ZSB3YXMgcHJlc2VydmVkLA0KPiA+ID4+Pj4gICAgc2VlIFNl Y3Rpb24gNi4zLjQgb2YgW1NBTTRdLg0KPiA+ID4+Pj4NCj4gPiA+Pj4+IFszXSBORVcgVEVYVDoN Cj4gPiA+Pj4+DQo+ID4gPj4+PiA0LjQuMy4yLiBSZXNlcnZhdGlvbnMNCj4gPiA+Pj4+DQo+ID4g Pj4+PiBUaGVyZSBhcmUgdHdvIHJlc2VydmF0aW9uIG1hbmFnZW1lbnQgbWV0aG9kcyBkZWZpbmVk IGluIHRoZSBTQ1NJDQo+ID4gPj4+PiBzdGFuZGFyZHMsIHJlc2VydmUvcmVsZWFzZSByZXNlcnZh dGlvbnMsIGJhc2VkIG9uIHRoZSBSRVNFUlZFIGFuZA0KPiA+ID4+Pj4gUkVMRUFTRSBjb21tYW5k cyBbU1BDMl0sIGFuZCBwZXJzaXN0ZW50IHJlc2VydmF0aW9ucywgYmFzZWQgb24NCj4gPiA+Pj4+ IHRoZSBQRVJTSVNURU5UIFJFU0VSVkUgSU4gYW5kIFBFUlNJU1RFTlQgUkVTRVJWRSBPVVQgY29t bWFuZHMgW1NQQzNdLg0KPiA+ID4+Pj4gUmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9ucyBhcmUg b2Jzb2xldGUgW1NQQzNdIGFuZCBTSE9VTEQgTk9UDQo+ID4gPj4+PiBiZSB1c2VkOyBwZXJzaXN0 ZW50IHJlc2VydmF0aW9ucyBTSE9VTEQgYmUgdXNlZCBpbnN0ZWFkLg0KPiA+ID4+Pj4NCj4gPiA+ Pj4+IFN0YXRlIGZvciBwZXJzaXN0ZW50IHJlc2VydmF0aW9ucyBpcyByZXF1aXJlZCB0byBwZXJz aXN0ICB0aHJvdWdoDQo+ID4gPj4+PiBjaGFuZ2VzIGFuZCBmYWlsdXJlcyBhdCB0aGUgaVNDU0kg bGF5ZXIgdGhhdCByZXN1bHQgaW4gIElfVCBOZXh1cw0KPiA+ID4+Pj4gZmFpbHVyZXMsIHNlZSBb U1BDM10gZm9yIGRldGFpbHMgYW5kIHNwZWNpZmljIHJlcXVpcmVtZW50cy4NCj4gPiA+Pj4+DQo+ ID4gPj4+PiBJbiBjb250cmFzdCwgW1NQQzJdIGRvZXMgbm90IHNwZWNpZnkgZGV0YWlsZWQgcGVy c2lzdGVuY2UNCj4gPiA+Pj4+IHJlcXVpcmVtZW50cyBmb3IgcmVzZXJ2ZS9yZWxlYXNlIHJlc2Vy dmF0aW9uIHN0YXRlIGFmdGVyIGFuIElfVA0KPiA+ID4+Pj4gTmV4dXMgZmFpbHVyZS4gIE5vbmV0 aGVsZXNzLCB3aGVuIHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbnMNCj4gPiA+Pj4+IGFyZSBz dXBwb3J0ZWQgYnkgYW4gaVNDU0kgdGFyZ2V0LCB0aGUgcHJlZmVycmVkIGltcGxlbWVudGF0aW9u DQo+ID4gPj4+PiBhcHByb2FjaCBpcyB0byBwcmVzZXJ2ZSByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2 YXRpb24gc3RhdGUgZm9yDQo+ID4gPj4+PiBpU0NTSSBzZXNzaW9uIHJlaW5zdGF0ZW1lbnQgKHNl ZSBTZWN0aW9uIDYuMy41KSBvciBzZXNzaW9uDQo+ID4gPj4+PiBjb250aW51YXRpb24gIChzZWUg U2VjdGlvbiA2LjMuNikuDQo+ID4gPj4+Pg0KPiA+ID4+Pj4gVHdvIGFkZGl0aW9uYWwgY2F2ZWF0 cyBhcHBseSB0byByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zOg0KPiA+ID4+Pj4NCj4gPiA+ Pj4+IC0gV2hlbiBjb25uZWN0aW9uIGZhaWx1cmUgY2F1c2VzIHRoZSBpU0NTSSBzZXNzaW9uIHRv IGZhaWwsIGFuZA0KPiA+ID4+Pj4gICAgIHRoZSBzZXNzaW9uIGlzIG5vdCByZWluc3RhdGVkIG9y IGNvbnRpbnVlZCwgdGFyZ2V0IHJldGVudGlvbg0KPiA+ID4+Pj4gICAgIG9mIHRoYXQgc2Vzc2lv bidzIHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbiBzdGF0ZSBtYXkNCj4gPiA+Pj4+ICAgICBy ZXF1aXJlIGFuIGluaXRpYXRvciB0byBpc3N1ZSBhIHJlc2V0IChlLmcuLCBMT0dJQ0FMIFVOSVQg UkVTRVQsDQo+ID4gPj4+PiAgICAgc2VlIHNlY3Rpb24gMTEuNSkgaW4gb3JkZXIgdG8gcmVtb3Zl IHRoYXQgcmVzZXJ2YXRpb24gc3RhdGUuDQo+ID4gPj4+Pg0KPiA+ID4+Pj4gLSBSZXNlcnZlL3Jl bGVhc2UgcmVzZXJ2YXRpb25zIG1heSBub3QgYmVoYXZlIGFzIGV4cGVjdGVkIHdoZW4NCj4gPiA+ Pj4+ICAgICBwZXJzaXN0ZW50IHJlc2VydmF0aW9ucyBhcmUgYWxzbyB1c2VkIG9uIHRoZSBzYW1l IGxvZ2ljYWwgdW5pdDsNCj4gPiA+Pj4+ICAgICBzZWUgdGhlIGRpc2N1c3Npb24gb2YgIkV4Y2Vw dGlvbnMgdG8gU1BDLTIgUkVTRVJWRSBhbmQgUkVMRUFTRQ0KPiA+ID4+Pj4gICAgIGJlaGF2aW9y IiBpbiBbU1BDNF0uDQo+ID4gPj4+Pg0KPiA+ID4+Pj4gVGhhbmtzLA0KPiA+ID4+Pj4gLS1EYXZp ZA0KPiA+ID4+Pj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLQ0KPiA+ID4+Pj4gRGF2aWQgTC4gQmxhY2ssIERpc3Rpbmd1aXNoZWQgRW5naW5lZXIg RU1DIENvcnBvcmF0aW9uLCAxNzYgU291dGgNCj4gPiA+Pj4+IFN0LiwgSG9wa2ludG9uLCBNQSAg MDE3NDgNCj4gPiA+Pj4+ICsxICg1MDgpIDI5My03OTUzICAgICAgICAgICAgIEZBWDogKzEgKDUw OCkgMjkzLTc3ODYNCj4gPiA+Pj4+IGRhdmlkLmJsYWNrQGVtYy5jb20gICAgICAgIE1vYmlsZTog KzEgKDk3OCkgMzk0LTc3NTQNCj4gPiA+Pj4+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gPiA+Pj4+DQo+ID4gPj4+Pg0KPiA+ID4+Pj4gX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4gPiA+Pj4+IHN0 b3JtIG1haWxpbmcgbGlzdA0KPiA+ID4+Pj4gc3Rvcm1AaWV0Zi5vcmcNCj4gPiA+Pj4+IGh0dHBz Oi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlzdGluZm8vc3Rvcm0NCj4gPiA+Pg0KPiA+ID4+IF9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+ID4gPj4gc3Rv cm0gbWFpbGluZyBsaXN0DQo+ID4gPj4gc3Rvcm1AaWV0Zi5vcmcNCj4gPiA+PiBodHRwczovL3d3 dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3N0b3JtDQo+ID4gPiBfX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiA+ID4gc3Rvcm0gbWFpbGluZyBsaXN0 DQo+ID4gPiBzdG9ybUBpZXRmLm9yZw0KPiA+ID4gaHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1h bi9saXN0aW5mby9zdG9ybQ0K From Frederick.Knight@netapp.com Fri Sep 28 08:08:59 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2184A21F84F1 for ; Fri, 28 Sep 2012 08:08:59 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id T1H4QzxTGdUk for ; Fri, 28 Sep 2012 08:08:57 -0700 (PDT) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id A683A21F867B for ; Fri, 28 Sep 2012 08:08:57 -0700 (PDT) X-IronPort-AV: E=Sophos;i="4.80,502,1344236400"; d="scan'208";a="695255121" Received: from smtp2.corp.netapp.com ([10.57.159.114]) by mx2-out.netapp.com with ESMTP; 28 Sep 2012 08:08:57 -0700 Received: from vmwexceht05-prd.hq.netapp.com (vmwexceht05-prd.hq.netapp.com [10.106.77.35]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8SF8u1Y006093; Fri, 28 Sep 2012 08:08:56 -0700 (PDT) Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.102]) by vmwexceht05-prd.hq.netapp.com ([10.106.77.35]) with mapi id 14.02.0309.002; Fri, 28 Sep 2012 08:08:56 -0700 From: "Knight, Frederick" To: "Black, David" Thread-Topic: [storm] SPC-2 reserve/release text - version 6 Thread-Index: Ac2cNcys4MzRK8PXQu651fVKNuEIxQAXDfsAABnkfIAADjDW0AAa4S6gACikjGD//nJjgIAAECkggAAHbKCAAAZiEIAABrHggAAIv6A= Date: Fri, 28 Sep 2012 15:08:56 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> <4AE113CD-96D7-414D-8AAB-36ECF7FD89EA@satran.net> <8D3D17ACE214DC429325B2B98F3AE7120DE798F9@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE7993C@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79B42@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79B71@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE79B71@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.106.53.53] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release text - version 6 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Sep 2012 15:08:59 -0000 VGhlIElGIHBhcnQgb2YgdGhlIHN0YXRlbWVudDoNCg0KPiA+ID4+ICAgICBJZiBhIFVuaXQgQXR0 ZW50aW9uIGNvbmRpdGlvbiByZXN1bHRzIGZyb20gc2Vzc2lvbiByZWVzdGFibGlzaG1lbnQsDQo+ ID4gPj4gICAgIHRoZSBzcGVjaWZpYyBVbml0IEF0dGVudGlvbiBjb25kaXRpb24gaW5kaWNhdGVz IHdoZXRoZXIgbmV4dXMgc3RhdGUNCj4gPiA+PiAgICAgd2FzIHByZXNlcnZlZCwgc2VlIFNlY3Rp b24gNi4zLjQgb2YgW1NBTTRdLg0KDQpJcyB3aGF0IGlzIGJyb2tlbi4NCg0KSUYgdGhlIHRhcmdl dCBkZWNpZGVzIHRvIGVzdGFibGlzaCBhIFVBLCB0aGVuIGl0IG11c3QgZm9sbG93IFNBTTQgb3Ro ZXJ3aXNlLCBpZiB0aGUgdGFyZ2V0IGRlY2lkZXMgbm90IHRvIGVzdGFibGlzaCBhIFVBLCB0aGVu IGV2ZXJ5dGhpbmcgaXMgZmluZS4gIENvbnNpZGVyIHRoZSBmb2xsb3dpbmcgc2VxdWVuY2U6DQoN CkluaXRpYXRvciBBIC0gUkVTRVJWRSBjb21tYW5kIChHT09EIHN0YXR1cykNCkluaXRpYXRvciBC IC0gV1JJVEUgKFJFU0VSVkFUSU9OIENPTkZMSUNUIHN0YXR1cykNCkluaXRpYXRvciBBIGdldHMg ZGlzY29ubmVjdGVkL3JlY29ubmVjdGVkIGFuZCBSRVNFUlZFIHN0YXRlIGlzIGxvc3QNCkluaXRp YXRvciBCIC0gV1JJVEUgKEdPT0Qgc3RhdHVzKQ0KDQpUaGUgdGFyZ2V0IGRlY2lkZWQgbm90IHRv IGVzdGFibGlzaCBhIFVBIGFzIGEgcmVzdWx0IG9mIHRoZSBzZXNzaW9uIHJlZXN0YWJsaXNobWVu dCwgc28gZXZlcnl0aGluZyBqdXN0IHByb2NlZWRzIG9uIGl0cyBtZXJyeSB3YXk/ICBIZXJlJ3Mg YSBzdWdnZXN0aW9uIHRoYXQgcmVtb3ZlcyB0aGUgIklGIiBhbmQgZG9lc24ndCBhZGQgYW55IG5l dyBtdXN0L3Nob3VsZCByZXF1aXJlbWVudHM6DQoNClVuaXQgQXR0ZW50aW9uIGNvbmRpdGlvbnMg dGhhdCByZXN1bHQgZnJvbSBzZXNzaW9uIHJlZXN0YWJsaXNobWVudA0KaW5kaWNhdGUgd2hldGhl ciBuZXh1cyBzdGF0ZSB3YXMgcHJlc2VydmVkLCBzZWUgU2VjdGlvbiA2LjMuNCBvZiBbU0FNNF0u DQoNCglGcmVkDQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBCbGFjaywgRGF2 aWQgW21haWx0bzpkYXZpZC5ibGFja0BlbWMuY29tXSANClNlbnQ6IEZyaWRheSwgU2VwdGVtYmVy IDI4LCAyMDEyIDEwOjM0IEFNDQpUbzogS25pZ2h0LCBGcmVkZXJpY2sNCkNjOiBzdG9ybUBpZXRm Lm9yZw0KU3ViamVjdDogUkU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2 ZXJzaW9uIDYNCg0KSSdtIG5vdCBzdXJlIHdoYXQncyBCUk9LRU4sIGJ1dCBhIHN0YXRlbWVudCBy ZXF1aXJpbmcgdGhlIFVBcyB3b3VsZCBiZSBhIHBvc3QtU0FNLTIgcmVxdWlyZW1lbnRzIGNoYW5n ZSB0aGF0IHdvdWxkIGhhdmUgdG8gZ28gaW4gdGhlDQotc2FtLSBkcmFmdCBzbyB0aGF0IGl0J3Mg c3ViamVjdCB0byB0aGUgbmVnb3RpYXRpb24gb2YgdGhlIHRleHQga2V5IGluIHRoYXQgZHJhZnQu ICBJIGhhdmUgbm8gcHJvYmxlbSB3aXRoIGEgYmxhbmtldCBzdGF0ZW1lbnQgdGhhdCBuZWdvdGlh dGluZyB0aGF0IHRleHQga2V5IHRvIGEgdmFsdWUgdGhhdCBwcm9taXNlcyB0aGUgZnVuY3Rpb25h bGl0eSBpbiB0aGUgLXNhbS0gZHJhZnQgcmVxdWlyZXMgdGhhdCB0aGUgaW1wbGVtZW50YXRpb24g YWRoZXJlIHRvIGV2ZXJ5dGhpbmcgaW4gU0FNLTQgdGhhdCBtYXkgZGlmZmVyIGZyb20gcHJpb3Ig dmVyc2lvbnMgb2YgU0FNLCBpbmNsdWRpbmcgSV9UIE5leHVzIGxvc3MgVUEgYmVoYXZpb3IsIGJ1 dCB0aGVyZSB3aWxsIG5lZWQgdG8gYmUgc29tZSBsYW5ndWFnZSBpbiB0aGVyZSBhYm91dCB3aGF0 IGhhcHBlbnMgYWZ0ZXIgVGltZTJSZXRhaW4gZXhwaXJlcyBiZWZvcmUgdGhlIElTSUQgaXMgcmV1 c2VkLg0KDQpPVE9ILCBJIGFtIHN0cm9uZ2x5IG9wcG9zZWQgdG8gcmV2aXNpbmcgdGhlIGJhc2Vs aW5lIHJlcXVpcmVtZW50cyBpbiB0aGUgY29uc29saWRhdGVkIGRyYWZ0IGluIGEgZmFzaGlvbiB0 aGF0IG1heSBicmVhayAicnVubmluZyBjb2RlIg0KdGhhdCdzIHdvcmtpbmcgZmluZSwgZXNwZWNp YWxseSB3aGVuIHRoZSBjaGFuZ2UgaXMgYmFzZWQgb24gYSBwb2ludCBvZiBwcmluY2lwbGUgdGhh dCBkb2VzIG5vdCBleGlzdCBpbiB0aGUgc3BlY2lmaWNhdGlvbnMgYWdhaW5zdCB3aGljaCB0aGF0 ICJydW5uaW5nIGNvZGUiIHdhcyBkZXZlbG9wZWQgKGkuZS4sIFNBTS0yKS4NCg0KVGhhbmtzLA0K LS1EYXZpZA0KDQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IEtuaWdodCwg RnJlZGVyaWNrIFttYWlsdG86RnJlZGVyaWNrLktuaWdodEBuZXRhcHAuY29tXQ0KPiBTZW50OiBG cmlkYXksIFNlcHRlbWJlciAyOCwgMjAxMiAxMDoyMSBBTQ0KPiBUbzogQmxhY2ssIERhdmlkDQo+ IENjOiBzdG9ybUBpZXRmLm9yZw0KPiBTdWJqZWN0OiBSRTogW3N0b3JtXSBTUEMtMiByZXNlcnZl L3JlbGVhc2UgdGV4dCAtIHZlcnNpb24gNg0KPg0KPiBCdXQgd2UgYWxzbyBzaG91bGRuJ3QgY29k aWZ5IEJST0tFTiBkZXNpZ25zIGluIGEgYnJhbmQgbmV3IFJGQy4gIEl0IA0KPiB3b3VsZCBiZXR0 ZXIgKGluIG15IG9waW5pb24pIHRvIGxlYXZlIGl0IGFsb25lIHRoYW4gc2FuY3Rpb24gYW5kIA0K PiBkZXNjcmliZSBob3cgdG8gYnVpbGQgYSBicm9rZW4gaW1wbGVtZW50YXRpb24uICBBZGRpbmcg dGhlICJJRiIgaXMgd3JvbmcuDQo+DQo+IFdlJ3ZlIGZpeGVkIHRoZSBpc3N1ZXMgcmFpc2VkIGJ5 IFJGQzUwNDggYnkgaW5jb3Jwb3JhdGluZyBpdCBpbnRvIHRoZSBuZXcgUkZDLg0KPg0KPiAgICAg ICBGcmVkDQo+DQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IEJsYWNrLCBE YXZpZCBbbWFpbHRvOmRhdmlkLmJsYWNrQGVtYy5jb21dDQo+IFNlbnQ6IEZyaWRheSwgU2VwdGVt YmVyIDI4LCAyMDEyIDk6NTEgQU0NCj4gVG86IEtuaWdodCwgRnJlZGVyaWNrDQo+IENjOiBzdG9y bUBpZXRmLm9yZw0KPiBTdWJqZWN0OiBSRTogW3N0b3JtXSBTUEMtMiByZXNlcnZlL3JlbGVhc2Ug dGV4dCAtIHZlcnNpb24gNg0KPg0KPiBGcmVkLA0KPg0KPiA+IElmIHN0YXRlIGlzIHJldGFpbmVk IChpdCdzIGFuIG9sZCAiSSIgcmVjb25uZWN0aW5nKSwgdGhlbiB3aGljaCBVQSANCj4gPiBpcyBk ZWZpbmVkIGJ5IFNBTS0zLg0KPiA+DQo+ID4gSWYgc3RhdGUgaXMgbm90IHJldGFpbmVkIChpdCdz IGEgbmV3ICJJIiksIHRoZW4gdGhlIFVBIGlzIGRlZmluZWQgYnkgU0FNLTMuDQo+ID4NCj4gPiBT aW5jZSBib3RoIG9sZCAiSSJzIGFuZCBuZXcgIkkicyByZXN1bHQgaW4gdGhlICJJRiIgY29uZGl0 aW9uIGJlaW5nIA0KPiA+IHRydWUsIHdoYXQgaXMgdGhlIGNvbmRpdGlvbiB0aGF0IHdvdWxkIG1h a2UgdGhlICJJRiIgZmFsc2U/DQo+DQo+IFlvdSBhbG1vc3QgYW5zd2VyZWQgeW91ciBvd24gcXVl c3Rpb24gOy0pLg0KPg0KPiBUaGUgYW5zd2VyIGlzIC4uLi4gKGRydW0gcm9sbCkgLi4uIGEgdGFy Z2V0IGltcGxlbWVudGF0aW9uIGJhc2VkIG9uIA0KPiBSRkMgMzcyMCBTQU0tMiwgYWx0aG91Z2gg SSB3b3VsZCBob3BlIHRoYXQgc3VjaCBpbXBsZW1lbnRhdGlvbnMgYXQgDQo+IGxlYXN0IHBpY2tl ZCB1cCBSRkMgNTA0OCdzIGNoYW5nZXMuDQo+DQo+IFRoZSBJX1QgTmV4dXMgTG9zcyBjb25jZXB0 IG9uIHdoaWNoIHRoaXMgVUEgZGlzY3Vzc2lvbiBpcyBiYXNlZCBpcyANCj4gcGFydCBvZiB0aGUg U0NTSSBldmVudHMgYW5kIGV2ZW50IG5vdGlmaWNhdGlvbnMgbW9kZWwgdGhhdCB3YXMgDQo+IGlu dHJvZHVjZWQgaW4gU0FNLTMsIGJ1dCBuZWl0aGVyIFJGQyAzNzIwIG5vciBSRkMgNTA0OCByZWZl cmVuY2VzIFNBTS0zLg0KPg0KPiBUaGVyZSB3ZXJlIGRlZmluaXRlbHkgaW1wbGVtZW50YXRpb25z IG9mIGlTQ1NJIGJhc2VkIG9uIFNBTS0yIC0gSSANCj4gYWdyZWUgd2l0aCBKdWxpYW4gdGhhdCB3 ZSBzaG91bGQgKm5vdCogYmUgaW50cm9kdWNpbmcgYSBVQSByZXF1aXJlbWVudCANCj4gaW4gdGhp cyBkcmFmdCB0aGF0IG1pZ2h0IGJyZWFrIHN1Y2ggaW1wbGVtZW50YXRpb25zLg0KPg0KPiBUaGFu a3MsDQo+IC0tRGF2aWQNCj4NCj4gPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiA+IEZy b206IEtuaWdodCwgRnJlZGVyaWNrIFttYWlsdG86RnJlZGVyaWNrLktuaWdodEBuZXRhcHAuY29t XQ0KPiA+IFNlbnQ6IEZyaWRheSwgU2VwdGVtYmVyIDI4LCAyMDEyIDg6MzggQU0NCj4gPiBUbzog SnVsaWFuIFNhdHJhbg0KPiA+IENjOiBCbGFjaywgRGF2aWQ7IHN0b3JtQGlldGYub3JnDQo+ID4g U3ViamVjdDogUkU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9u IDYNCj4gPg0KPiA+IE9LLCB0aGF0J3Mgd2hhdCBJIHRob3VnaHQuDQo+ID4NCj4gPiBJZiBzdGF0 ZSBpcyByZXRhaW5lZCAoaXQncyBhbiBvbGQgIkkiIHJlY29ubmVjdGluZyksIHRoZW4gd2hpY2gg VUEgDQo+ID4gaXMgZGVmaW5lZCBieSBTQU0tMy4NCj4gPg0KPiA+IElmIHN0YXRlIGlzIG5vdCBy ZXRhaW5lZCAoaXQncyBhIG5ldyAiSSIpLCB0aGVuIHRoZSBVQSBpcyBkZWZpbmVkIGJ5IFNBTS0z Lg0KPiA+DQo+ID4gU2luY2UgYm90aCBvbGQgIkkicyBhbmQgbmV3ICJJInMgcmVzdWx0IGluIHRo ZSAiSUYiIGNvbmRpdGlvbiBiZWluZyANCj4gPiB0cnVlLCB3aGF0IGlzIHRoZSBjb25kaXRpb24g dGhhdCB3b3VsZCBtYWtlIHRoZSAiSUYiIGZhbHNlPw0KPiA+DQo+ID4gVGhhdCAiSUYiIGxhbmd1 YWdlIGFsbG93cyBhIHRhcmdldCB0byBsb29zZSBzdGF0ZSBhbmQgbm90IGRlbGl2ZXIgYW55IFVB Lg0KPiA+DQo+ID4gICAgIEZyZWQNCj4gPg0KPiA+DQo+ID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdl LS0tLS0NCj4gPiBGcm9tOiBKdWxpYW4gU2F0cmFuIFttYWlsdG86anVsaWFuQHNhdHJhbi5uZXRd DQo+ID4gU2VudDogRnJpZGF5LCBTZXB0ZW1iZXIgMjgsIDIwMTIgMjowMyBBTQ0KPiA+IFRvOiBL bmlnaHQsIEZyZWRlcmljaw0KPiA+IENjOiBCbGFjaywgRGF2aWQ7IHN0b3JtQGlldGYub3JnDQo+ ID4gU3ViamVjdDogUmU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJz aW9uIDYNCj4gPg0KPiA+IFRoZXJlIGlzIG5vIHN1Y2ggdGhpbmcuIEFuZCB0aGUgcmVhc29uIHRo ZSB0aW1lb3V0IGlzIHRoZXJlIGlzIHRvIA0KPiA+IGxpbWl0IHRoZSBhbW91bnQgb2Ygc3RhdGUg dGhlIHRhcmdldHMgaGF2ZSB0byBrZWVwLiBBIHRhcmdldCBtYXkgDQo+ID4gY2hvb3NlIHRvIGFj Y2VwdCB2ZXJ5IGxhcmdlIHRpbWVvdXRzIGFuZCB0aGVuIGRvIHdoYXQgeW91IHdhbnRlZCANCj4g PiBkb25lIGJ1dCBiZXlvbmQgdGhhdCBvbmx5IHRoZSBwb3dlci11cCB1YSBpcyByZXF1aXJlZC4N Cj4gPg0KPiA+IGp1bG8NCj4gPg0KPiA+IFNlbnQgZnJvbSBteSBpUGFkDQo+ID4NCj4gPiBPbiAy OCDXkdeh16TXmCAyMDEyLCBhdCAwMDoyOSwgIktuaWdodCwgRnJlZGVyaWNrIg0KPiA+IDxGcmVk ZXJpY2suS25pZ2h0QG5ldGFwcC5jb20+DQo+ID4gd3JvdGU6DQo+ID4NCj4gPiA+IEkgd291bGQg dGhpbmsgaWYgdGhlIElTSUQgaXMgbm90IHJlY29nbml6ZWQgYXMgYW4gb2xkIG9uZSwgdGhlbiBp dCANCj4gPiA+IG11c3QgYmUNCj4gPiByZWNvZ25pemVkIGFzIGEgbmV3IG9uZS4gIElmIGl0IGlz IGEgbmV3IG9uZSwgdGhlbiBpdCB3aWxsIGdldCBhIFVBOg0KPiA+IDI5LzAwIHJlZmxlY3Rpbmcg dGhlIGZpcnN0IGFjY2VzcyBhZnRlciBwb3dlciBvbiBmcm9tIHRoZSAibmV3IiBpbml0aWF0b3Iu DQo+ID4gPg0KPiA+ID4gSXMgdGhlcmUgc29tZXRoaW5nIG90aGVyIHRoYW4gbmV3IG9yIG9sZD8g IElzIHRoZXJlIHN1Y2ggYSB0aGluZyANCj4gPiA+IGFzIGENCj4gPiBtaWRkbGUtYWdlZCBpbml0 aWF0b3I/DQo+ID4gPg0KPiA+ID4gICAgRnJlZA0KPiA+ID4NCj4gPiA+IC0tLS0tT3JpZ2luYWwg TWVzc2FnZS0tLS0tDQo+ID4gPiBGcm9tOiBCbGFjaywgRGF2aWQgW21haWx0bzpkYXZpZC5ibGFj a0BlbWMuY29tXQ0KPiA+ID4gU2VudDogVGh1cnNkYXksIFNlcHRlbWJlciAyNywgMjAxMiAxMTo1 NiBBTQ0KPiA+ID4gVG86IEtuaWdodCwgRnJlZGVyaWNrDQo+ID4gPiBDYzogc3Rvcm1AaWV0Zi5v cmcNCj4gPiA+IFN1YmplY3Q6IFJFOiBbc3Rvcm1dIFNQQy0yIHJlc2VydmUvcmVsZWFzZSB0ZXh0 IC0gdmVyc2lvbiA2DQo+ID4gPg0KPiA+ID4gRnJlZCwNCj4gPiA+DQo+ID4gPiBUaGUgcHJvYmxl bSBpcyBub3QgdGhhdCB0aGUgZGV2aWNlIGRvZXMgbm90IHJhaXNlIGEgVUEsIGl0J3MgdGhhdCAN Cj4gPiA+IHRoZSBVQSBtYXkNCj4gPiBuZXZlciBiZSBkZWxpdmVyZWQuDQo+ID4gPg0KPiA+ID4g QSBzdW1tYXJ5IG9mIGhvdyB0aGlzIGNhbiBoYXBwZW4gaXMgdGhhdCB0aGUgbmV3IGNvbm5lY3Rp b24gd2l0aCANCj4gPiA+IHRoZSBzYW1lDQo+ID4gSVNJRCBtYXkgbm90IGJlIHJlY29nbml6ZWQg YXMgcmVpbnN0YXRpbmcgb3IgY29udGludWluZyB0aGUgc2Vzc2lvbiANCj4gPiBhdCB0aGUgdGFy Z2V0IGlmIHRoZSB0aW1lb3V0cyBoYXZlIGV4cGlyZWQgLSB0aGUgVUEgZ2V0cyANCj4gPiBlc3Rh Ymxpc2hlZCwgYnV0IG5vYm9keSBzaG93cyB1cCB0byBwaWNrIGl0IHVwIGluIHRpbWUsIHNvIHRo ZSBVQSBpcyANCj4gPiBkaXNjYXJkZWQgYWxvbmcgd2l0aCB0aGUgcmVzdCBvZiB0aGUgc2Vzc2lv biBzdGF0ZSBhbmQgdGhlIHRhcmdldCANCj4gPiBmYWlscyB0byByZWNvZ25pemUgdGhlIGluaXRp YXRvcidzIHN1YnNlcXVlbnQgSVNJRCByZXVzZSBldmVuIHRob3VnaCANCj4gPiB0aGUgaW5pdGlh dG9yIGJlbGlldmVzIHRoYXQgaXQncyByZS0gZXN0YWJsaXNoaW5nIHRoZSBzZXNzaW9uLg0KPiA+ ID4NCj4gPiA+IFRoaXMgaXMgZnVydGhlciBjb21wbGljYXRlZCBieSB0aGUgVUEgdGhhdCBpbmRp Y2F0ZXMgbmV4dXMgc3RhdGUNCj4gPiBwcmVzZXJ2YXRpb24gbm90IGJlaW5nIHNwZWNpZmllZCBp biBTQU0tMi4NCj4gPiA+DQo+ID4gPiBUaGFua3MsDQo+ID4gPiAtLURhdmlkDQo+ID4gPg0KPiA+ ID4NCj4gPiA+PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiA+ID4+IEZyb206IEtuaWdo dCwgRnJlZGVyaWNrIFttYWlsdG86RnJlZGVyaWNrLktuaWdodEBuZXRhcHAuY29tXQ0KPiA+ID4+ IFNlbnQ6IFRodXJzZGF5LCBTZXB0ZW1iZXIgMjcsIDIwMTIgMTE6MTEgQU0NCj4gPiA+PiBUbzog QmxhY2ssIERhdmlkOyBKdWxpYW4gU2F0cmFuDQo+ID4gPj4gQ2M6IHN0b3JtQGlldGYub3JnDQo+ ID4gPj4gU3ViamVjdDogUkU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2 ZXJzaW9uIDYNCj4gPiA+Pg0KPiA+ID4+IEkgYWZyYWlkIEkgaGF2ZSB0byBjb250ZXN0IGEgbGl0 dGxlIG9uIHRoaXMgLSBpZiBJIHVuZGVyc3RhbmQgDQo+ID4gPj4gdGhpcyBkaXNjdXNzaW9uIGNv cnJlY3RseS4NCj4gPiA+Pg0KPiA+ID4+IElmIGEgU0NTSSBkZXZpY2UgbG9zZXMgYW55IHN0YXRl IGFuZCBkb2VzIG5vdCByYWlzZSBhIFVOSVQgDQo+ID4gPj4gQVRURU5USU9OLCB0aGVuIGl0IGlz IEJST0tFTi4gIEl0IGlzIG5vdCBhIG1hdHRlciBvZiBhbnl0aGluZyBpbiANCj4gPiA+PiBpU0NT SSAoMzcyMCk7IEkgdGhpbmsgU0FNIGFuZCBTUEMgYXJlIHByZXR0eSBjbGVhciBvbiB0aGlzLiAg SWYgDQo+ID4gPj4gdGhlcmUgaXMgbm8gVU5JVCBBVFRFTlRJT04sIHRoZW4gc3RhdGUgU0hBTEwg YmUgcHJlc2VydmVkOyANCj4gPiA+PiBvdGhlcndpc2UsIGFsbCBzb3J0cyBvZiBzdHVmZiB3aWxs IG5ldmVyIHdvcmsgKG1vZGUgcGFnZSANCj4gPiA+PiBzZXR0aW5ncywgUkVTRVJWRSBjb21tYW5k cywgYWxsIHNvcnRzIG9mDQo+ID4gc3R1ZmYpLg0KPiA+ID4+DQo+ID4gPj4gSXMgc29tZW9uZSBy ZWFsbHkgc3VnZ2VzdGluZyB0aGF0IHNvbWUgZGV2aWNlcyB3aWxsIGRyb3Agc3RhdGUgDQo+ID4g Pj4gYW5kIE5PVCB0ZWxsIHRoZSBob3N0LCBhbmQgc29tZWhvdyB0aGF0IGl0IGlzIE9LIHRvIGRv IHRoYXQ/DQo+ID4gPj4NCj4gPiA+PiBUaGUgc3VnZ2VzdCByZXdvcmRpbmcgc2F5cyB0aGlzIGlz IHBvc3NpYmxlLg0KPiA+ID4+DQo+ID4gPj4gICAgRnJlZA0KPiA+ID4+DQo+ID4gPj4gLS0tLS1P cmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPiA+PiBGcm9tOiBzdG9ybS1ib3VuY2VzQGlldGYub3Jn IFttYWlsdG86c3Rvcm0tYm91bmNlc0BpZXRmLm9yZ10gT24gDQo+ID4gPj4gQmVoYWxmIE9mIEJs YWNrLCBEYXZpZA0KPiA+ID4+IFNlbnQ6IFRodXJzZGF5LCBTZXB0ZW1iZXIgMjcsIDIwMTIgMTA6 NDcgQU0NCj4gPiA+PiBUbzogSnVsaWFuIFNhdHJhbg0KPiA+ID4+IENjOiBzdG9ybUBpZXRmLm9y Zw0KPiA+ID4+IFN1YmplY3Q6IFJlOiBbc3Rvcm1dIFNQQy0yIHJlc2VydmUvcmVsZWFzZSB0ZXh0 IC0gdmVyc2lvbiA2DQo+ID4gPj4NCj4gPiA+PiBKdWxpYW4sDQo+ID4gPj4NCj4gPiA+Pj4gVGhl IHRleHQgaW4gNC40LjMgc3RhdGVzIHRoZSBvYnZpb3VzICh0aGVyZSBpcyBubyBvdGhlciBzcGVj ZWQgDQo+ID4gPj4+IG1ldGhvZCBmb3IgY29udGludWF0aW9uIG9yIHJlaW5zdGF0ZW1lbnQpLiBN b3Jlb3ZlciBpZiB0aGUgDQo+ID4gPj4+IHNlc3Npb24gaXMgcmVpbnN0YXRlZCBhZnRlciB0aW1l MndhaXQgeW91IGNyZWF0ZSBhIG5lZWQgZm9yIGEgVUEgDQo+ID4gPj4+IChvciBhdCBsZWFzdCBz dWdnZXN0IG9uZSkgYW5kIHRoYXQgd2FzIG5vdCB0aGVyZS4NCj4gPiA+Pg0KPiA+ID4+IFRoZSBV QXMgYXJlIGNvdXJ0ZXN5IG9mIFNBTS0zIChhbmQgU0FNLTQpLiAgVG8gZW5jb21wYXNzIHN5c3Rl bXMgDQo+ID4gPj4gdGhhdCBtYXkgbm90IGRlbGl2ZXIgdGhlc2UgVUFzIGFuZCBzaXR1YXRpb25z IGluIHdoaWNoIHRoZXkgZG9uJ3QgDQo+ID4gPj4gb2NjdXIgKGUuZy4sIHRoZSB0YXJnZXQgZG9l c24ndCByZWNvZ25pemUgd2hhdCBoYXBwZW5lZCBhcyBuZXh1cyANCj4gPiA+PiByZWVzdGFibGlz aG1lbnQsIGluZGVwZW5kZW50IG9mIHdoYXQgdGhlIGluaXRpYXRvciB0aGlua3MgaXQncw0KPiA+ ID4+IGRvaW5nKSBJDQo+ID4gdGhpbmsgdGhlIHRleHQgc2hvdWxkIGJlIHJlcGhyYXNlZCBhczoN Cj4gPiA+Pg0KPiA+ID4+ICAgICBJZiBhIFVuaXQgQXR0ZW50aW9uIGNvbmRpdGlvbiByZXN1bHRz IGZyb20gc2Vzc2lvbiByZWVzdGFibGlzaG1lbnQsDQo+ID4gPj4gICAgIHRoZSBzcGVjaWZpYyBV bml0IEF0dGVudGlvbiBjb25kaXRpb24gaW5kaWNhdGVzIHdoZXRoZXIgbmV4dXMgc3RhdGUNCj4g PiA+PiAgICAgd2FzIHByZXNlcnZlZCwgc2VlIFNlY3Rpb24gNi4zLjQgb2YgW1NBTTRdLg0KPiA+ ID4+DQo+ID4gPj4+IEp1c3Qgc3RhdGluZyB0aGF0IGNyZWF0aW9uL2NvbnRpbnVhdGlvbiBvZiBh IHNlc3Npb24gZG9uZSANCj4gPiA+Pj4gYWNjb3JkaW5nIDYueCBpbmNsdWRlcyBwcmVzZXJ2YXRp b24gb2YgcmVzZXJ2ZS9yZWxlYXNlIHN0YXRlIA0KPiA+ID4+PiAtdGhhdCB3YXMgYWxyZWFkeSB0 aGVyZSBpbXBsaWVkIGJ5IGNvbmZvcm1hbmNlIHRvIHNwYzIgYW5kIHRoZSANCj4gPiA+Pj4gcmVs YXRpb24gYmV0d2VlbiBpdCBuZXh1cyBhbmQgc2Vzc2lvbi4NCj4gPiA+Pg0KPiA+ID4+IFVuZm9y dHVuYXRlbHksIFNQQy0yIGlzIHN1ZmZpY2llbnRseSBpbXByZWNpc2Ugb24gdGhpcyBzdWJqZWN0 IA0KPiA+ID4+IHRoYXQgc3VjaCBhbiBpbXBsaWNhdGlvbiBpcyBhdCBiZXN0IGEgInByZWZlcnJl ZCBpbXBsZW1lbnRhdGlvbiBhcHByb2FjaCIuDQo+ID4gPj4gSXQncyBub3QgYSByZXF1aXJlbWVu dCB0aGF0IGNhbiBiZSBzYWZlbHkgc3RhdGVkIGluIGFueSBzdHJvbmdlciANCj4gPiA+PiBmYXNo aW9uIHdpdGhvdXQgdGhlIHJpc2sgb2YgcmVuZGVyaW5nIGV4aXN0aW5nIGltcGxlbWVudGF0aW9u cyANCj4gPiA+PiBub24tDQo+ID4gY29tcGxpYW50Lg0KPiA+ID4+DQo+ID4gPj4+IFRoZSBuZXcg dGV4dCB5b3Ugc3VnZ2VzdCB0byA0LjQgaXMgbWVyZWx5IGEgYmVzdCBwcmFjdGljZXMgDQo+ID4g Pj4+IHJlY29tbWVuZGF0aW9ucyBhbmQgd2lsbCBuZWVkIGFkZGl0aW9ucyBvbmNlIHRoZSBuZXcg ImxvY2tpbmciDQo+ID4gPj4+IHRocm91Z2ggY29tcGFyZS1hbmQtc3dhcCBnZXQgaW4gd2lkZXNw cmVhZCB1c2UuDQo+ID4gPj4NCj4gPiA+PiBJdCdzIGRlZmluaXRlbHkgYmVzdCBwcmFjdGljZXMg cmVjb21tZW5kYXRpb25zIC0gbm8gYXJndW1lbnQgdGhlcmUuDQo+ID4gPj4NCj4gPiA+PiBGb3Ig Y29udGV4dCwgd2hhdCBKdWxpYW4gaXMgcmVmZXJyaW5nIHRvIGlzIHVzZSBvZiB0aGUgQ09NUEFS RSANCj4gPiA+PiBBTkQgV1JJVEUgY29tbWFuZCB0aGF0IHBlcmZvcm1zIGFuIGF0b21pYyBvcGVy YXRpb24gYXQgdGhlIHRhcmdldCANCj4gPiA+PiBpbnN0ZWFkIG9mIHVzaW5nIFJFU0VSVkUgYW5k IFJFTEVBU0UgdG8gY3JlYXRlIGEgY3JpdGljYWwgc2VjdGlvbiANCj4gPiA+PiBhdCB0aGUgaW5p dGlhdG9yIGJhc2VkIG9uIHJlYWQgYW5kIHdyaXRlIGNvbW1hbmRzLg0KPiA+ID4+DQo+ID4gPj4g Q09NUEFSRSBBTkQgV1JJVEUgaXMgaW4gdXNlIC0gYSBzZW50ZW5jZSBtZW50aW9uaW5nIHRoYXQg Y29tbWFuZCANCj4gPiA+PiBjb3VsZCBiZSBhZGRlZCwgYWx0aG91Z2ggdGhpcyBiZXN0IHByYWN0 aWNlcyB0ZXh0IGlzIGFpbWVkIA0KPiA+ID4+IHByaW1hcmlseSBhdCB1c2VzIG9mIHJlc2VydmUv cmVsZWFzZSBmb3IgbG9uZy10ZXJtIHJlc2VydmF0aW9ucy4gIA0KPiA+ID4+IEkgdGhpbmsgdGhl cmUncyBhbiBpbXBsaWNpdCAid2hlbiBhIHJlc2VydmF0aW9uIGlzIHRoZSBkZXNpcmVkIGJlaGF2 aW9yIg0KPiA+ID4+IGltcGxpY2F0aW9uIHRvIHRoZSAicGVyc2lzdGVudCByZXNlcnZhdGlvbnMg U0hPVUxEIGJlIHVzZWQgaW5zdGVhZCINCj4gPiA+PiB0ZXh0IHRoYXQgSSBkb24ndCB0aGluayBu ZWVkcyB0byBiZSBtYWRlIGV4cGxpY2l0LiAgT1RPSCwgdGhhdCAiU0hPVUxEIg0KPiA+ID4+IGNv dWxkIGJlIGNoYW5nZWQgdG8gYSAic2hvdWxkIiwgb3IgaXQgbWlnaHQgYmUgZXZlbiBiZXR0ZXIg dG8gDQo+ID4gPj4gcmV3cml0ZSB0aGF0IHRleHQNCj4gPiA+PiBhczoNCj4gPiA+Pg0KPiA+ID4+ ICAgIHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIGFyZSBzdWdnZXN0ZWQgYXMgYW4gYWx0ZXJuYXRp dmUsIHNlZSBBbm5leCBCDQo+ID4gPj4gICAgb2YgW1NQQzRdLg0KPiA+ID4+DQo+ID4gPj4gSW4g Y29udHJhc3QsICJTSE9VTEQgTk9UIGJlIHVzZWQiIGlzIGNsZWFybHkganVzdGlmaWVkIGZvciAN Cj4gPiA+PiByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zIGJ5IFtTUEMzXS4NCj4gPiA+Pg0K PiA+ID4+IFRoYW5rcywNCj4gPiA+PiAtLURhdmlkDQo+ID4gPj4NCj4gPiA+Pg0KPiA+ID4+PiAt LS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiA+ID4+PiBGcm9tOiBKdWxpYW4gU2F0cmFuIFtt YWlsdG86anVsaWFuQHNhdHJhbi5uZXRdDQo+ID4gPj4+IFNlbnQ6IFdlZG5lc2RheSwgU2VwdGVt YmVyIDI2LCAyMDEyIDEwOjI1IFBNDQo+ID4gPj4+IFRvOiBCbGFjaywgRGF2aWQNCj4gPiA+Pj4g Q2M6IHN0b3JtQGlldGYub3JnDQo+ID4gPj4+IFN1YmplY3Q6IFJlOiBbc3Rvcm1dIFNQQy0yIHJl c2VydmUvcmVsZWFzZSB0ZXh0IC0gdmVyc2lvbiA2DQo+ID4gPj4+DQo+ID4gPj4+IERhdmlkLA0K PiA+ID4+Pg0KPiA+ID4+PiBUaGUgdGV4dCBpbiA0LjQuMyBzdGF0ZXMgdGhlIG9idmlvdXMgKHRo ZXJlIGlzIG5vIG90aGVyIHNwZWNlZCANCj4gPiA+Pj4gbWV0aG9kIGZvciBjb250aW51YXRpb24g b3IgcmVpbnN0YXRlbWVudCkuIE1vcmVvdmVyIGlmIHRoZSB0aGUgDQo+ID4gPj4+IHNlc3Npb24g aXMgcmVpbnN0YXRlZCBhZnRlciB0aW1lMndhaXQgeW91IGNyZWF0ZSBhIG5lZWQgZm9yIGEgVUEg DQo+ID4gPj4+IChvciBhdCBsZWFzdCBzdWdnZXN0IG9uZSkgYW5kIHRoYXQgd2FzIG5vdCB0aGVy ZS4gSnVzdCBzdGF0aW5nIA0KPiA+ID4+PiB0aGF0IGNyZWF0aW9uL2NvbnRpbnVhdGlvbiBvZiBh IHNlc3Npb24gZG9uZSBhY2NvcmRpbmcgNi54IA0KPiA+ID4+PiBpbmNsdWRlcyBwcmVzZXJ2YXRp b24gb2YgcmVzZXJ2ZS9yZWxlYXNlIHN0YXRlIC10aGF0IHdhcyBhbHJlYWR5IA0KPiA+ID4+PiB0 aGVyZSBpbXBsaWVkIGJ5IGNvbmZvcm1hbmNlIHRvIHNwYzIgYW5kIHRoZSByZWxhdGlvbiBiZXR3 ZWVuIGl0IA0KPiA+ID4+PiBuZXh1cyBhbmQNCj4gPiBzZXNzaW9uLg0KPiA+ID4+Pg0KPiA+ID4+ PiBUaGUgbmV3IHRleHQgeW91IHN1Z2dlc3QgdG8gNC40IGlzIG1lcmVseSBhIGJlc3QgcHJhY3Rp Y2VzIA0KPiA+ID4+PiByZWNvbWVuZGF0aW9ucyBhbmQgd2lsbCBuZWVkIGFkZGl0aW9ucyBvbmNl IHRoZSBuZXcgImxvY2tpbmciDQo+ID4gPj4+IHRocm91Z2ggY29tcGFyZS1hbmQtc3dhcCBnZXQg aW4gd2lkZXNwcmVhZCB1c2UuDQo+ID4gPj4+IElmIGluc3RlYWQgd2UgcmVmcmFpbiBmcm9tIHN0 YXRpbmcgYW55dGhpbmcgc3BlY2lmaWMgYW5kIHN0YXRlIA0KPiA+ID4+PiB0aGF0IHRhcmdldCBz aG91bGQgbWFpbnRhaW4gd2hhdGV2ZXIgc3RhdGUgaXMgcmVxdWlyZWQgYnkgU1BDIA0KPiA+ID4+ PiB3aGlsZSBhIHNlc3Npb24gaXMgY29udGludWVkIGFuZCBtYXkgZGlzY2FyZCBzdGF0ZSB3aGVu IGEgDQo+ID4gPj4+IHNlc3Npb24gaXMgdGVybWluYXRlZCBhbmQgbWF5IGluZGljYXRlIHRoYXQg aXQgZGlzY2FyZGVkIHN0YXRlIA0KPiA+ID4+PiB0aHJvdWdoIGEgdWEgeW91IGFyZSBvbiBzYWZl ciBncm91bmQgYW5kIGRvIG5vdCBjcmVhdGUgbmV3IHJlcXVpcmVtZW50Lg0KPiA+ID4+Pg0KPiA+ ID4+PiBTZW50IGZyb20gbXkgaVBhZA0KPiA+ID4+Pg0KPiA+ID4+PiBPbiAyNyDXkdeh16TXmCAy MDEyLCBhdCAwMDoyNSwgIkJsYWNrLCBEYXZpZCIgPGRhdmlkLmJsYWNrQGVtYy5jb20+IHdyb3Rl Og0KPiA+ID4+Pg0KPiA+ID4+Pj4gSGVyZSdzIHdoZXJlIEkgdGhpbmsgd2UndmUgYXJyaXZlZCAo SSd2ZSBtYWRlIGEgZmV3IG1pbm9yIA0KPiA+ID4+Pj4gZWRpdG9yaWFsIHR3ZWFrcyB0byB3aGF0 J3MgYmVlbiBkaXNjdXNzZWQpLiAgRGVzcGl0ZSBhbGwgdGhlIA0KPiA+ID4+Pj4gZGV0YWlsZWQg ZGlzY3Vzc2lvbiwgSSB0aGluayB0aGlzIHZlcnNpb24gNiBjb21lcyBjbG9zZSB0byANCj4gPiA+ Pj4+IEp1bGlhbidzIHN1Z2dlc3Rpb24NCj4gPiA+PiB0aGF0Og0KPiA+ID4+Pj4NCj4gPiA+Pj4+ PiBpIHdvdWxkIG1lbnRpb24gb25seSB0aGF0IHRoZSBzdGF0ZSB0aGF0IG11c3QgYmUgbWFpbnRh aW5lZCANCj4gPiA+Pj4+PiB3aGlsZSB0aGUgc2Vzc2lvbiBpcyBtYWludGFpbmVkIGluY2x1ZGVz IHRoZSByZXNlcnZlL3JlbGVhc2UgDQo+ID4gPj4+Pj4gYW5kIGxlYXZlIGl0IHRvDQo+ID4gPj4+ IHRoYXQuDQo+ID4gPj4+Pg0KPiA+ID4+Pj4gQWx0aG91Z2ggd2UgY2FuJ3QgYWN0dWFsbHkgc2F5 ICJtdXN0IiBhcyB0aGF0IHdvdWxkIGJlIGEgbmV3IA0KPiA+ID4+Pj4gcmVxdWlyZW1lbnQsIGFu ZCBpbXBsZW1lbnRhdGlvbnMgZGlmZmVyIGluIHdoYXQgdGhleSBkbyBhYm91dCANCj4gPiA+Pj4+ IHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbiBzdGF0ZS4gIFRoZSBjdXJyZW50IGxhbmd1YWdl IA0KPiA+ID4+Pj4gY2hhcmFjdGVyaXplcyByZXRlbnRpb24gb2YgdGhhdCBzdGF0ZSBhcyB0aGUg InByZWZlcnJlZCANCj4gPiA+Pj4+IGltcGxlbWVudGF0aW9uDQo+ID4gPj4gYXBwcm9hY2giLg0K PiA+ID4+Pj4NCj4gPiA+Pj4+IEJleW9uZCB0aGF0LCBJIHRoaW5rIHRoZSB0d28gYWRkaXRpb25h bCBjYXZlYXRzIGFib3V0IA0KPiA+ID4+Pj4gcmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9ucyAo cG9zc2libGUgbmVlZCBmb3IgYSByZXNldCwgDQo+ID4gPj4+PiBwb3RlbnRpYWxseSB1bmV4cGVj dGVkIGludGVyYWN0aW9uIHdpdGggcGVyc2lzdGVudCANCj4gPiA+Pj4+IHJlc2VydmF0aW9ucywg c2VlIGJlbG93KSBhcmUgdXNlZnVsDQo+ID4gPj4gdG8gaW5jbHVkZS4NCj4gPiA+Pj4+DQo+ID4g Pj4+PiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiA+ID4+Pj4N Cj4gPiA+Pj4+IFsxXSBGb3IgY2xhcml0eSwgdGhlIGZvbGxvd2luZyBjaGFuZ2Ugc2hvdWxkIGJl IG1hZGUgaW4NCj4gPiA+Pj4+IDQuNC4zLjE6DQo+ID4gPj4+Pg0KPiA+ID4+Pj4gT0xEDQo+ID4g Pj4+PiBUaGF0IGlzLCBpdCBzaG91bGQgcGVyZm9ybSBzZXNzaW9uIHJlY292ZXJ5IGFzICBkZXNj cmliZWQgaW4gDQo+ID4gPj4+PiBDaGFwdGVyIDYuDQo+ID4gPj4+PiBORVcNCj4gPiA+Pj4+IFRo YXQgaXMsIGl0IHNob3VsZCByZWluc3RhdGUgdGhlIHNlc3Npb24gdmlhIGlTQ1NJIHNlc3Npb24g DQo+ID4gPj4+PiByZWluc3RhdGVtZW50IChTZWN0aW9uIDYuMy41KSBvciBjb250aW51ZSB2aWEg c2Vzc2lvbiANCj4gPiA+Pj4+IGNvbnRpbnVhdGlvbiAoU2VjdGlvbiA2LjMuNikuDQo+ID4gPj4+ PiBFTkQNCj4gPiA+Pj4+DQo+ID4gPj4+PiBbMl0gQWRkIHRoZSBmb2xsb3dpbmcgc2VudGVuY2Ug dG8gdGhlIGVuZCBvZiA0LjQuMy4xOg0KPiA+ID4+Pj4NCj4gPiA+Pj4+ICAgIFRoZSBzcGVjaWZp YyBVbml0IEF0dGVudGlvbiBjb25kaXRpb24gdGhhdCByZXN1bHRzIGZyb20gc2Vzc2lvbg0KPiA+ ID4+Pj4gICAgcmVlc3RhYmxpc2htZW50IGluZGljYXRlcyB3aGV0aGVyIG5leHVzIHN0YXRlIHdh cyBwcmVzZXJ2ZWQsDQo+ID4gPj4+PiAgICBzZWUgU2VjdGlvbiA2LjMuNCBvZiBbU0FNNF0uDQo+ ID4gPj4+Pg0KPiA+ID4+Pj4gWzNdIE5FVyBURVhUOg0KPiA+ID4+Pj4NCj4gPiA+Pj4+IDQuNC4z LjIuIFJlc2VydmF0aW9ucw0KPiA+ID4+Pj4NCj4gPiA+Pj4+IFRoZXJlIGFyZSB0d28gcmVzZXJ2 YXRpb24gbWFuYWdlbWVudCBtZXRob2RzIGRlZmluZWQgaW4gdGhlIA0KPiA+ID4+Pj4gU0NTSSBz dGFuZGFyZHMsIHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbnMsIGJhc2VkIG9uIHRoZSANCj4g PiA+Pj4+IFJFU0VSVkUgYW5kIFJFTEVBU0UgY29tbWFuZHMgW1NQQzJdLCBhbmQgcGVyc2lzdGVu dCANCj4gPiA+Pj4+IHJlc2VydmF0aW9ucywgYmFzZWQgb24gdGhlIFBFUlNJU1RFTlQgUkVTRVJW RSBJTiBhbmQgUEVSU0lTVEVOVCBSRVNFUlZFIE9VVCBjb21tYW5kcyBbU1BDM10uDQo+ID4gPj4+ PiBSZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zIGFyZSBvYnNvbGV0ZSBbU1BDM10gYW5kIFNI T1VMRCBOT1QgDQo+ID4gPj4+PiBiZSB1c2VkOyBwZXJzaXN0ZW50IHJlc2VydmF0aW9ucyBTSE9V TEQgYmUgdXNlZCBpbnN0ZWFkLg0KPiA+ID4+Pj4NCj4gPiA+Pj4+IFN0YXRlIGZvciBwZXJzaXN0 ZW50IHJlc2VydmF0aW9ucyBpcyByZXF1aXJlZCB0byBwZXJzaXN0ICANCj4gPiA+Pj4+IHRocm91 Z2ggY2hhbmdlcyBhbmQgZmFpbHVyZXMgYXQgdGhlIGlTQ1NJIGxheWVyIHRoYXQgcmVzdWx0IGlu ICANCj4gPiA+Pj4+IElfVCBOZXh1cyBmYWlsdXJlcywgc2VlIFtTUEMzXSBmb3IgZGV0YWlscyBh bmQgc3BlY2lmaWMgcmVxdWlyZW1lbnRzLg0KPiA+ID4+Pj4NCj4gPiA+Pj4+IEluIGNvbnRyYXN0 LCBbU1BDMl0gZG9lcyBub3Qgc3BlY2lmeSBkZXRhaWxlZCBwZXJzaXN0ZW5jZSANCj4gPiA+Pj4+ IHJlcXVpcmVtZW50cyBmb3IgcmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9uIHN0YXRlIGFmdGVy IGFuIElfVCANCj4gPiA+Pj4+IE5leHVzIGZhaWx1cmUuICBOb25ldGhlbGVzcywgd2hlbiByZXNl cnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zIA0KPiA+ID4+Pj4gYXJlIHN1cHBvcnRlZCBieSBhbiBp U0NTSSB0YXJnZXQsIHRoZSBwcmVmZXJyZWQgaW1wbGVtZW50YXRpb24gDQo+ID4gPj4+PiBhcHBy b2FjaCBpcyB0byBwcmVzZXJ2ZSByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb24gc3RhdGUgZm9y IA0KPiA+ID4+Pj4gaVNDU0kgc2Vzc2lvbiByZWluc3RhdGVtZW50IChzZWUgU2VjdGlvbiA2LjMu NSkgb3Igc2Vzc2lvbiANCj4gPiA+Pj4+IGNvbnRpbnVhdGlvbiAgKHNlZSBTZWN0aW9uIDYuMy42 KS4NCj4gPiA+Pj4+DQo+ID4gPj4+PiBUd28gYWRkaXRpb25hbCBjYXZlYXRzIGFwcGx5IHRvIHJl c2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbnM6DQo+ID4gPj4+Pg0KPiA+ID4+Pj4gLSBXaGVuIGNv bm5lY3Rpb24gZmFpbHVyZSBjYXVzZXMgdGhlIGlTQ1NJIHNlc3Npb24gdG8gZmFpbCwgYW5kDQo+ ID4gPj4+PiAgICAgdGhlIHNlc3Npb24gaXMgbm90IHJlaW5zdGF0ZWQgb3IgY29udGludWVkLCB0 YXJnZXQgcmV0ZW50aW9uDQo+ID4gPj4+PiAgICAgb2YgdGhhdCBzZXNzaW9uJ3MgcmVzZXJ2ZS9y ZWxlYXNlIHJlc2VydmF0aW9uIHN0YXRlIG1heQ0KPiA+ID4+Pj4gICAgIHJlcXVpcmUgYW4gaW5p dGlhdG9yIHRvIGlzc3VlIGEgcmVzZXQgKGUuZy4sIExPR0lDQUwgVU5JVCBSRVNFVCwNCj4gPiA+ Pj4+ICAgICBzZWUgc2VjdGlvbiAxMS41KSBpbiBvcmRlciB0byByZW1vdmUgdGhhdCByZXNlcnZh dGlvbiBzdGF0ZS4NCj4gPiA+Pj4+DQo+ID4gPj4+PiAtIFJlc2VydmUvcmVsZWFzZSByZXNlcnZh dGlvbnMgbWF5IG5vdCBiZWhhdmUgYXMgZXhwZWN0ZWQgd2hlbg0KPiA+ID4+Pj4gICAgIHBlcnNp c3RlbnQgcmVzZXJ2YXRpb25zIGFyZSBhbHNvIHVzZWQgb24gdGhlIHNhbWUgbG9naWNhbCB1bml0 Ow0KPiA+ID4+Pj4gICAgIHNlZSB0aGUgZGlzY3Vzc2lvbiBvZiAiRXhjZXB0aW9ucyB0byBTUEMt MiBSRVNFUlZFIGFuZCBSRUxFQVNFDQo+ID4gPj4+PiAgICAgYmVoYXZpb3IiIGluIFtTUEM0XS4N Cj4gPiA+Pj4+DQo+ID4gPj4+PiBUaGFua3MsDQo+ID4gPj4+PiAtLURhdmlkDQo+ID4gPj4+PiAt LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ID4g Pj4+PiBEYXZpZCBMLiBCbGFjaywgRGlzdGluZ3Vpc2hlZCBFbmdpbmVlciBFTUMgQ29ycG9yYXRp b24sIDE3NiANCj4gPiA+Pj4+IFNvdXRoIFN0LiwgSG9wa2ludG9uLCBNQSAgMDE3NDgNCj4gPiA+ Pj4+ICsxICg1MDgpIDI5My03OTUzICAgICAgICAgICAgIEZBWDogKzEgKDUwOCkgMjkzLTc3ODYN Cj4gPiA+Pj4+IGRhdmlkLmJsYWNrQGVtYy5jb20gICAgICAgIE1vYmlsZTogKzEgKDk3OCkgMzk0 LTc3NTQNCj4gPiA+Pj4+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0NCj4gPiA+Pj4+DQo+ID4gPj4+Pg0KPiA+ID4+Pj4gX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4gPiA+Pj4+IHN0b3JtIG1haWxpbmcg bGlzdA0KPiA+ID4+Pj4gc3Rvcm1AaWV0Zi5vcmcNCj4gPiA+Pj4+IGh0dHBzOi8vd3d3LmlldGYu b3JnL21haWxtYW4vbGlzdGluZm8vc3Rvcm0NCj4gPiA+Pg0KPiA+ID4+IF9fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+ID4gPj4gc3Rvcm0gbWFpbGluZyBs aXN0DQo+ID4gPj4gc3Rvcm1AaWV0Zi5vcmcNCj4gPiA+PiBodHRwczovL3d3dy5pZXRmLm9yZy9t YWlsbWFuL2xpc3RpbmZvL3N0b3JtDQo+ID4gPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fXw0KPiA+ID4gc3Rvcm0gbWFpbGluZyBsaXN0DQo+ID4gPiBzdG9y bUBpZXRmLm9yZw0KPiA+ID4gaHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9z dG9ybQ0K From david.black@emc.com Fri Sep 28 08:44:13 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A04A521F863C for ; Fri, 28 Sep 2012 08:44:13 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.585 X-Spam-Level: X-Spam-Status: No, score=-102.585 tagged_above=-999 required=5 tests=[AWL=0.014, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6802wDw29vTD for ; Fri, 28 Sep 2012 08:44:12 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id CAB1021F863F for ; Fri, 28 Sep 2012 08:44:11 -0700 (PDT) Received: from hop04-l1d11-si04.isus.emc.com (HOP04-L1D11-SI04.isus.emc.com [10.254.111.24]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8SFi916020910 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 28 Sep 2012 11:44:10 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.222.130]) by hop04-l1d11-si04.isus.emc.com (RSA Interceptor); Fri, 28 Sep 2012 11:43:51 -0400 Received: from mxhub03.corp.emc.com (mxhub03.corp.emc.com [10.254.141.105]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8SFhpKe023206; Fri, 28 Sep 2012 11:43:51 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub03.corp.emc.com ([10.254.141.105]) with mapi; Fri, 28 Sep 2012 11:43:50 -0400 From: "Black, David" To: "Knight, Frederick" Date: Fri, 28 Sep 2012 11:43:48 -0400 Thread-Topic: [storm] SPC-2 reserve/release text - version 6 Thread-Index: Ac2cNcys4MzRK8PXQu651fVKNuEIxQAXDfsAABnkfIAADjDW0AAa4S6gACikjGD//nJjgIAAECkggAAHbKCAAAZiEIAABrHggAAIv6CAAAR6sA== Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DE79BB8@MX15A.corp.emc.com> References: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> <4AE113CD-96D7-414D-8AAB-36ECF7FD89EA@satran.net> <8D3D17ACE214DC429325B2B98F3AE7120DE798F9@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE7993C@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79B42@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79B71@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-EMM-MHVC: 1 Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release text - version 6 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Sep 2012 15:44:13 -0000 RnJlZCwNCg0KPiBIZXJlJ3MgYSBzdWdnZXN0aW9uIHRoYXQgcmVtb3ZlcyB0aGUgIklGIiBhbmQg ZG9lc24ndCBhZGQgYW55IG5ldyBtdXN0L3Nob3VsZA0KPiByZXF1aXJlbWVudHM6DQo+DQo+IFVu aXQgQXR0ZW50aW9uIGNvbmRpdGlvbnMgdGhhdCByZXN1bHQgZnJvbSBzZXNzaW9uIHJlZXN0YWJs aXNobWVudA0KPiBpbmRpY2F0ZSB3aGV0aGVyIG5leHVzIHN0YXRlIHdhcyBwcmVzZXJ2ZWQsIHNl ZSBTZWN0aW9uIDYuMy40IG9mIFtTQU00XS4NCg0KVGhhdCB0ZXh0IGlzIGFjY2VwdGFibGUgdG8g bWUuDQoNCkkgZG9uJ3QgdGhpbmsgd2UgbmVlZCBhbnl0aGluZyBiZXlvbmQgdGhhdCB0ZXh0IGlu IHRoZSBkcmFmdCwgYnV0IGhlcmUncw0KdGhlIHJlc3Qgb2YgdGhlIGV4cGxhbmF0aW9uIC4uLg0K DQpBbiBpbXBvcnRhbnQgYXNwZWN0IG9mIHdoYXQncyBnb2luZyBvbiBoZXJlIGlzIHRoYXQgd2l0 aCByZXNwZWN0IHRvIHRoZQ0KdXNlIG9mIHRoZSB0ZXJtICJyZS1lc3RhYmxpc2htZW50IiBmb3Ig YW4gSV9UIE5leHVzIGluIFNlY3Rpb24gNi4zLjQgb2YNCltTQU00XSwgYWZ0ZXIgVGltZTJSZXRh aW4gZXhwaXJlcywgYW4gaVNDU0kgdGFyZ2V0IGlzIGVudGl0bGVkIHRvIGNvbnNpZGVyDQpJU0lE IHJlLXVzZSB0byBub3QgYmUgYSAicmUtZXN0YWJsaXNobWVudCIgb2YgdGhlIElfVCBOZXh1cy4g IFRoaXMgaXMNCmJlY2F1c2Ugd2hlbiBUaW1lMlJldGFpbiBleHBpcmVzLCB0aGUgdGFyZ2V0IGlz IGFsbG93ZWQgdG8gZGlzY2FyZCBhbGwNCm9mIGl0cyBzdGF0ZSBmb3IgdGhhdCBuZXh1cywgaW5j bHVkaW5nIHRoZSB0YXJnZXQncyByZWNvcmQgb2YgdGhlIElTSUQNCnZhbHVlLg0KDQpBdm9pZGFu Y2Ugb2YgdGhlIHByb2JsZW1hdGljIHNjZW5hcmlvIGludm9sdmluZyBhIHJlc2VydmF0aW9uIGxv c3MgaXMgYQ0KZmFjdG9yIHRvIGNvbnNpZGVyIGluIHNlbGVjdGluZyBUaW1lMlJldGFpbiB2YWx1 ZXM6DQoNCj4gSW5pdGlhdG9yIEEgLSBSRVNFUlZFIGNvbW1hbmQgKEdPT0Qgc3RhdHVzKQ0KPiBJ bml0aWF0b3IgQiAtIFdSSVRFIChSRVNFUlZBVElPTiBDT05GTElDVCBzdGF0dXMpDQo+IEluaXRp YXRvciBBIGdldHMgZGlzY29ubmVjdGVkL3JlY29ubmVjdGVkIGFuZCBSRVNFUlZFIHN0YXRlIGlz IGxvc3QNCj4gSW5pdGlhdG9yIEIgLSBXUklURSAoR09PRCBzdGF0dXMpDQoNClRoYW5rcywNCi0t RGF2aWQNCg0KPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBLbmlnaHQsIEZy ZWRlcmljayBbbWFpbHRvOkZyZWRlcmljay5LbmlnaHRAbmV0YXBwLmNvbV0NCj4gU2VudDogRnJp ZGF5LCBTZXB0ZW1iZXIgMjgsIDIwMTIgMTE6MDkgQU0NCj4gVG86IEJsYWNrLCBEYXZpZA0KPiBD Yzogc3Rvcm1AaWV0Zi5vcmcNCj4gU3ViamVjdDogUkU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9y ZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYNCj4NCj4gVGhlIElGIHBhcnQgb2YgdGhlIHN0YXRlbWVu dDoNCj4NCj4gPiA+ID4+ICAgICBJZiBhIFVuaXQgQXR0ZW50aW9uIGNvbmRpdGlvbiByZXN1bHRz IGZyb20gc2Vzc2lvbiByZWVzdGFibGlzaG1lbnQsDQo+ID4gPiA+PiAgICAgdGhlIHNwZWNpZmlj IFVuaXQgQXR0ZW50aW9uIGNvbmRpdGlvbiBpbmRpY2F0ZXMgd2hldGhlciBuZXh1cyBzdGF0ZQ0K PiA+ID4gPj4gICAgIHdhcyBwcmVzZXJ2ZWQsIHNlZSBTZWN0aW9uIDYuMy40IG9mIFtTQU00XS4N Cj4NCj4gSXMgd2hhdCBpcyBicm9rZW4uDQo+DQo+IElGIHRoZSB0YXJnZXQgZGVjaWRlcyB0byBl c3RhYmxpc2ggYSBVQSwgdGhlbiBpdCBtdXN0IGZvbGxvdyBTQU00IG90aGVyd2lzZSwNCj4gaWYg dGhlIHRhcmdldCBkZWNpZGVzIG5vdCB0byBlc3RhYmxpc2ggYSBVQSwgdGhlbiBldmVyeXRoaW5n IGlzIGZpbmUuDQo+IENvbnNpZGVyIHRoZSBmb2xsb3dpbmcgc2VxdWVuY2U6DQo+DQo+IEluaXRp YXRvciBBIC0gUkVTRVJWRSBjb21tYW5kIChHT09EIHN0YXR1cykNCj4gSW5pdGlhdG9yIEIgLSBX UklURSAoUkVTRVJWQVRJT04gQ09ORkxJQ1Qgc3RhdHVzKQ0KPiBJbml0aWF0b3IgQSBnZXRzIGRp c2Nvbm5lY3RlZC9yZWNvbm5lY3RlZCBhbmQgUkVTRVJWRSBzdGF0ZSBpcyBsb3N0DQo+IEluaXRp YXRvciBCIC0gV1JJVEUgKEdPT0Qgc3RhdHVzKQ0KPg0KPiBUaGUgdGFyZ2V0IGRlY2lkZWQgbm90 IHRvIGVzdGFibGlzaCBhIFVBIGFzIGEgcmVzdWx0IG9mIHRoZSBzZXNzaW9uDQo+IHJlZXN0YWJs aXNobWVudCwgc28gZXZlcnl0aGluZyBqdXN0IHByb2NlZWRzIG9uIGl0cyBtZXJyeSB3YXk/ICBI ZXJlJ3MgYQ0KPiBzdWdnZXN0aW9uIHRoYXQgcmVtb3ZlcyB0aGUgIklGIiBhbmQgZG9lc24ndCBh ZGQgYW55IG5ldyBtdXN0L3Nob3VsZA0KPiByZXF1aXJlbWVudHM6DQo+DQo+IFVuaXQgQXR0ZW50 aW9uIGNvbmRpdGlvbnMgdGhhdCByZXN1bHQgZnJvbSBzZXNzaW9uIHJlZXN0YWJsaXNobWVudA0K PiBpbmRpY2F0ZSB3aGV0aGVyIG5leHVzIHN0YXRlIHdhcyBwcmVzZXJ2ZWQsIHNlZSBTZWN0aW9u IDYuMy40IG9mIFtTQU00XS4NCj4NCj4gICAgICAgRnJlZA0KPg0KPiAtLS0tLU9yaWdpbmFsIE1l c3NhZ2UtLS0tLQ0KPiBGcm9tOiBCbGFjaywgRGF2aWQgW21haWx0bzpkYXZpZC5ibGFja0BlbWMu Y29tXQ0KPiBTZW50OiBGcmlkYXksIFNlcHRlbWJlciAyOCwgMjAxMiAxMDozNCBBTQ0KPiBUbzog S25pZ2h0LCBGcmVkZXJpY2sNCj4gQ2M6IHN0b3JtQGlldGYub3JnDQo+IFN1YmplY3Q6IFJFOiBb c3Rvcm1dIFNQQy0yIHJlc2VydmUvcmVsZWFzZSB0ZXh0IC0gdmVyc2lvbiA2DQo+DQo+IEknbSBu b3Qgc3VyZSB3aGF0J3MgQlJPS0VOLCBidXQgYSBzdGF0ZW1lbnQgcmVxdWlyaW5nIHRoZSBVQXMg d291bGQgYmUgYSBwb3N0LQ0KPiBTQU0tMiByZXF1aXJlbWVudHMgY2hhbmdlIHRoYXQgd291bGQg aGF2ZSB0byBnbyBpbiB0aGUNCj4gLXNhbS0gZHJhZnQgc28gdGhhdCBpdCdzIHN1YmplY3QgdG8g dGhlIG5lZ290aWF0aW9uIG9mIHRoZSB0ZXh0IGtleSBpbiB0aGF0DQo+IGRyYWZ0LiAgSSBoYXZl IG5vIHByb2JsZW0gd2l0aCBhIGJsYW5rZXQgc3RhdGVtZW50IHRoYXQgbmVnb3RpYXRpbmcgdGhh dCB0ZXh0DQo+IGtleSB0byBhIHZhbHVlIHRoYXQgcHJvbWlzZXMgdGhlIGZ1bmN0aW9uYWxpdHkg aW4gdGhlIC1zYW0tIGRyYWZ0IHJlcXVpcmVzDQo+IHRoYXQgdGhlIGltcGxlbWVudGF0aW9uIGFk aGVyZSB0byBldmVyeXRoaW5nIGluIFNBTS00IHRoYXQgbWF5IGRpZmZlciBmcm9tDQo+IHByaW9y IHZlcnNpb25zIG9mIFNBTSwgaW5jbHVkaW5nIElfVCBOZXh1cyBsb3NzIFVBIGJlaGF2aW9yLCBi dXQgdGhlcmUgd2lsbA0KPiBuZWVkIHRvIGJlIHNvbWUgbGFuZ3VhZ2UgaW4gdGhlcmUgYWJvdXQg d2hhdCBoYXBwZW5zIGFmdGVyIFRpbWUyUmV0YWluIGV4cGlyZXMNCj4gYmVmb3JlIHRoZSBJU0lE IGlzIHJldXNlZC4NCj4NCj4gT1RPSCwgSSBhbSBzdHJvbmdseSBvcHBvc2VkIHRvIHJldmlzaW5n IHRoZSBiYXNlbGluZSByZXF1aXJlbWVudHMgaW4gdGhlDQo+IGNvbnNvbGlkYXRlZCBkcmFmdCBp biBhIGZhc2hpb24gdGhhdCBtYXkgYnJlYWsgInJ1bm5pbmcgY29kZSINCj4gdGhhdCdzIHdvcmtp bmcgZmluZSwgZXNwZWNpYWxseSB3aGVuIHRoZSBjaGFuZ2UgaXMgYmFzZWQgb24gYSBwb2ludCBv Zg0KPiBwcmluY2lwbGUgdGhhdCBkb2VzIG5vdCBleGlzdCBpbiB0aGUgc3BlY2lmaWNhdGlvbnMg YWdhaW5zdCB3aGljaCB0aGF0DQo+ICJydW5uaW5nIGNvZGUiIHdhcyBkZXZlbG9wZWQgKGkuZS4s IFNBTS0yKS4NCj4NCj4gVGhhbmtzLA0KPiAtLURhdmlkDQo+DQo+ID4gLS0tLS1PcmlnaW5hbCBN ZXNzYWdlLS0tLS0NCj4gPiBGcm9tOiBLbmlnaHQsIEZyZWRlcmljayBbbWFpbHRvOkZyZWRlcmlj ay5LbmlnaHRAbmV0YXBwLmNvbV0NCj4gPiBTZW50OiBGcmlkYXksIFNlcHRlbWJlciAyOCwgMjAx MiAxMDoyMSBBTQ0KPiA+IFRvOiBCbGFjaywgRGF2aWQNCj4gPiBDYzogc3Rvcm1AaWV0Zi5vcmcN Cj4gPiBTdWJqZWN0OiBSRTogW3N0b3JtXSBTUEMtMiByZXNlcnZlL3JlbGVhc2UgdGV4dCAtIHZl cnNpb24gNg0KPiA+DQo+ID4gQnV0IHdlIGFsc28gc2hvdWxkbid0IGNvZGlmeSBCUk9LRU4gZGVz aWducyBpbiBhIGJyYW5kIG5ldyBSRkMuICBJdA0KPiA+IHdvdWxkIGJldHRlciAoaW4gbXkgb3Bp bmlvbikgdG8gbGVhdmUgaXQgYWxvbmUgdGhhbiBzYW5jdGlvbiBhbmQNCj4gPiBkZXNjcmliZSBo b3cgdG8gYnVpbGQgYSBicm9rZW4gaW1wbGVtZW50YXRpb24uICBBZGRpbmcgdGhlICJJRiIgaXMg d3JvbmcuDQo+ID4NCj4gPiBXZSd2ZSBmaXhlZCB0aGUgaXNzdWVzIHJhaXNlZCBieSBSRkM1MDQ4 IGJ5IGluY29ycG9yYXRpbmcgaXQgaW50byB0aGUgbmV3DQo+IFJGQy4NCj4gPg0KPiA+ICAgICAg IEZyZWQNCj4gPg0KPiA+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+ID4gRnJvbTogQmxh Y2ssIERhdmlkIFttYWlsdG86ZGF2aWQuYmxhY2tAZW1jLmNvbV0NCj4gPiBTZW50OiBGcmlkYXks IFNlcHRlbWJlciAyOCwgMjAxMiA5OjUxIEFNDQo+ID4gVG86IEtuaWdodCwgRnJlZGVyaWNrDQo+ ID4gQ2M6IHN0b3JtQGlldGYub3JnDQo+ID4gU3ViamVjdDogUkU6IFtzdG9ybV0gU1BDLTIgcmVz ZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYNCj4gPg0KPiA+IEZyZWQsDQo+ID4NCj4gPiA+ IElmIHN0YXRlIGlzIHJldGFpbmVkIChpdCdzIGFuIG9sZCAiSSIgcmVjb25uZWN0aW5nKSwgdGhl biB3aGljaCBVQQ0KPiA+ID4gaXMgZGVmaW5lZCBieSBTQU0tMy4NCj4gPiA+DQo+ID4gPiBJZiBz dGF0ZSBpcyBub3QgcmV0YWluZWQgKGl0J3MgYSBuZXcgIkkiKSwgdGhlbiB0aGUgVUEgaXMgZGVm aW5lZCBieSBTQU0tDQo+IDMuDQo+ID4gPg0KPiA+ID4gU2luY2UgYm90aCBvbGQgIkkicyBhbmQg bmV3ICJJInMgcmVzdWx0IGluIHRoZSAiSUYiIGNvbmRpdGlvbiBiZWluZw0KPiA+ID4gdHJ1ZSwg d2hhdCBpcyB0aGUgY29uZGl0aW9uIHRoYXQgd291bGQgbWFrZSB0aGUgIklGIiBmYWxzZT8NCj4g Pg0KPiA+IFlvdSBhbG1vc3QgYW5zd2VyZWQgeW91ciBvd24gcXVlc3Rpb24gOy0pLg0KPiA+DQo+ ID4gVGhlIGFuc3dlciBpcyAuLi4uIChkcnVtIHJvbGwpIC4uLiBhIHRhcmdldCBpbXBsZW1lbnRh dGlvbiBiYXNlZCBvbg0KPiA+IFJGQyAzNzIwIFNBTS0yLCBhbHRob3VnaCBJIHdvdWxkIGhvcGUg dGhhdCBzdWNoIGltcGxlbWVudGF0aW9ucyBhdA0KPiA+IGxlYXN0IHBpY2tlZCB1cCBSRkMgNTA0 OCdzIGNoYW5nZXMuDQo+ID4NCj4gPiBUaGUgSV9UIE5leHVzIExvc3MgY29uY2VwdCBvbiB3aGlj aCB0aGlzIFVBIGRpc2N1c3Npb24gaXMgYmFzZWQgaXMNCj4gPiBwYXJ0IG9mIHRoZSBTQ1NJIGV2 ZW50cyBhbmQgZXZlbnQgbm90aWZpY2F0aW9ucyBtb2RlbCB0aGF0IHdhcw0KPiA+IGludHJvZHVj ZWQgaW4gU0FNLTMsIGJ1dCBuZWl0aGVyIFJGQyAzNzIwIG5vciBSRkMgNTA0OCByZWZlcmVuY2Vz IFNBTS0zLg0KPiA+DQo+ID4gVGhlcmUgd2VyZSBkZWZpbml0ZWx5IGltcGxlbWVudGF0aW9ucyBv ZiBpU0NTSSBiYXNlZCBvbiBTQU0tMiAtIEkNCj4gPiBhZ3JlZSB3aXRoIEp1bGlhbiB0aGF0IHdl IHNob3VsZCAqbm90KiBiZSBpbnRyb2R1Y2luZyBhIFVBIHJlcXVpcmVtZW50DQo+ID4gaW4gdGhp cyBkcmFmdCB0aGF0IG1pZ2h0IGJyZWFrIHN1Y2ggaW1wbGVtZW50YXRpb25zLg0KPiA+DQo+ID4g VGhhbmtzLA0KPiA+IC0tRGF2aWQNCj4gPg0KPiA+ID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0t LS0NCj4gPiA+IEZyb206IEtuaWdodCwgRnJlZGVyaWNrIFttYWlsdG86RnJlZGVyaWNrLktuaWdo dEBuZXRhcHAuY29tXQ0KPiA+ID4gU2VudDogRnJpZGF5LCBTZXB0ZW1iZXIgMjgsIDIwMTIgODoz OCBBTQ0KPiA+ID4gVG86IEp1bGlhbiBTYXRyYW4NCj4gPiA+IENjOiBCbGFjaywgRGF2aWQ7IHN0 b3JtQGlldGYub3JnDQo+ID4gPiBTdWJqZWN0OiBSRTogW3N0b3JtXSBTUEMtMiByZXNlcnZlL3Jl bGVhc2UgdGV4dCAtIHZlcnNpb24gNg0KPiA+ID4NCj4gPiA+IE9LLCB0aGF0J3Mgd2hhdCBJIHRo b3VnaHQuDQo+ID4gPg0KPiA+ID4gSWYgc3RhdGUgaXMgcmV0YWluZWQgKGl0J3MgYW4gb2xkICJJ IiByZWNvbm5lY3RpbmcpLCB0aGVuIHdoaWNoIFVBDQo+ID4gPiBpcyBkZWZpbmVkIGJ5IFNBTS0z Lg0KPiA+ID4NCj4gPiA+IElmIHN0YXRlIGlzIG5vdCByZXRhaW5lZCAoaXQncyBhIG5ldyAiSSIp LCB0aGVuIHRoZSBVQSBpcyBkZWZpbmVkIGJ5IFNBTS0NCj4gMy4NCj4gPiA+DQo+ID4gPiBTaW5j ZSBib3RoIG9sZCAiSSJzIGFuZCBuZXcgIkkicyByZXN1bHQgaW4gdGhlICJJRiIgY29uZGl0aW9u IGJlaW5nDQo+ID4gPiB0cnVlLCB3aGF0IGlzIHRoZSBjb25kaXRpb24gdGhhdCB3b3VsZCBtYWtl IHRoZSAiSUYiIGZhbHNlPw0KPiA+ID4NCj4gPiA+IFRoYXQgIklGIiBsYW5ndWFnZSBhbGxvd3Mg YSB0YXJnZXQgdG8gbG9vc2Ugc3RhdGUgYW5kIG5vdCBkZWxpdmVyIGFueSBVQS4NCj4gPiA+DQo+ ID4gPiAgICAgRnJlZA0KPiA+ID4NCj4gPiA+DQo+ID4gPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2Ut LS0tLQ0KPiA+ID4gRnJvbTogSnVsaWFuIFNhdHJhbiBbbWFpbHRvOmp1bGlhbkBzYXRyYW4ubmV0 XQ0KPiA+ID4gU2VudDogRnJpZGF5LCBTZXB0ZW1iZXIgMjgsIDIwMTIgMjowMyBBTQ0KPiA+ID4g VG86IEtuaWdodCwgRnJlZGVyaWNrDQo+ID4gPiBDYzogQmxhY2ssIERhdmlkOyBzdG9ybUBpZXRm Lm9yZw0KPiA+ID4gU3ViamVjdDogUmU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRl eHQgLSB2ZXJzaW9uIDYNCj4gPiA+DQo+ID4gPiBUaGVyZSBpcyBubyBzdWNoIHRoaW5nLiBBbmQg dGhlIHJlYXNvbiB0aGUgdGltZW91dCBpcyB0aGVyZSBpcyB0bw0KPiA+ID4gbGltaXQgdGhlIGFt b3VudCBvZiBzdGF0ZSB0aGUgdGFyZ2V0cyBoYXZlIHRvIGtlZXAuIEEgdGFyZ2V0IG1heQ0KPiA+ ID4gY2hvb3NlIHRvIGFjY2VwdCB2ZXJ5IGxhcmdlIHRpbWVvdXRzIGFuZCB0aGVuIGRvIHdoYXQg eW91IHdhbnRlZA0KPiA+ID4gZG9uZSBidXQgYmV5b25kIHRoYXQgb25seSB0aGUgcG93ZXItdXAg dWEgaXMgcmVxdWlyZWQuDQo+ID4gPg0KPiA+ID4ganVsbw0KPiA+ID4NCj4gPiA+IFNlbnQgZnJv bSBteSBpUGFkDQo+ID4gPg0KPiA+ID4gT24gMjgg15HXodek15ggMjAxMiwgYXQgMDA6MjksICJL bmlnaHQsIEZyZWRlcmljayINCj4gPiA+IDxGcmVkZXJpY2suS25pZ2h0QG5ldGFwcC5jb20+DQo+ ID4gPiB3cm90ZToNCj4gPiA+DQo+ID4gPiA+IEkgd291bGQgdGhpbmsgaWYgdGhlIElTSUQgaXMg bm90IHJlY29nbml6ZWQgYXMgYW4gb2xkIG9uZSwgdGhlbiBpdA0KPiA+ID4gPiBtdXN0IGJlDQo+ ID4gPiByZWNvZ25pemVkIGFzIGEgbmV3IG9uZS4gIElmIGl0IGlzIGEgbmV3IG9uZSwgdGhlbiBp dCB3aWxsIGdldCBhIFVBOg0KPiA+ID4gMjkvMDAgcmVmbGVjdGluZyB0aGUgZmlyc3QgYWNjZXNz IGFmdGVyIHBvd2VyIG9uIGZyb20gdGhlICJuZXciIGluaXRpYXRvci4NCj4gPiA+ID4NCj4gPiA+ ID4gSXMgdGhlcmUgc29tZXRoaW5nIG90aGVyIHRoYW4gbmV3IG9yIG9sZD8gIElzIHRoZXJlIHN1 Y2ggYSB0aGluZw0KPiA+ID4gPiBhcyBhDQo+ID4gPiBtaWRkbGUtYWdlZCBpbml0aWF0b3I/DQo+ ID4gPiA+DQo+ID4gPiA+ICAgIEZyZWQNCj4gPiA+ID4NCj4gPiA+ID4gLS0tLS1PcmlnaW5hbCBN ZXNzYWdlLS0tLS0NCj4gPiA+ID4gRnJvbTogQmxhY2ssIERhdmlkIFttYWlsdG86ZGF2aWQuYmxh Y2tAZW1jLmNvbV0NCj4gPiA+ID4gU2VudDogVGh1cnNkYXksIFNlcHRlbWJlciAyNywgMjAxMiAx MTo1NiBBTQ0KPiA+ID4gPiBUbzogS25pZ2h0LCBGcmVkZXJpY2sNCj4gPiA+ID4gQ2M6IHN0b3Jt QGlldGYub3JnDQo+ID4gPiA+IFN1YmplY3Q6IFJFOiBbc3Rvcm1dIFNQQy0yIHJlc2VydmUvcmVs ZWFzZSB0ZXh0IC0gdmVyc2lvbiA2DQo+ID4gPiA+DQo+ID4gPiA+IEZyZWQsDQo+ID4gPiA+DQo+ ID4gPiA+IFRoZSBwcm9ibGVtIGlzIG5vdCB0aGF0IHRoZSBkZXZpY2UgZG9lcyBub3QgcmFpc2Ug YSBVQSwgaXQncyB0aGF0DQo+ID4gPiA+IHRoZSBVQSBtYXkNCj4gPiA+IG5ldmVyIGJlIGRlbGl2 ZXJlZC4NCj4gPiA+ID4NCj4gPiA+ID4gQSBzdW1tYXJ5IG9mIGhvdyB0aGlzIGNhbiBoYXBwZW4g aXMgdGhhdCB0aGUgbmV3IGNvbm5lY3Rpb24gd2l0aA0KPiA+ID4gPiB0aGUgc2FtZQ0KPiA+ID4g SVNJRCBtYXkgbm90IGJlIHJlY29nbml6ZWQgYXMgcmVpbnN0YXRpbmcgb3IgY29udGludWluZyB0 aGUgc2Vzc2lvbg0KPiA+ID4gYXQgdGhlIHRhcmdldCBpZiB0aGUgdGltZW91dHMgaGF2ZSBleHBp cmVkIC0gdGhlIFVBIGdldHMNCj4gPiA+IGVzdGFibGlzaGVkLCBidXQgbm9ib2R5IHNob3dzIHVw IHRvIHBpY2sgaXQgdXAgaW4gdGltZSwgc28gdGhlIFVBIGlzDQo+ID4gPiBkaXNjYXJkZWQgYWxv bmcgd2l0aCB0aGUgcmVzdCBvZiB0aGUgc2Vzc2lvbiBzdGF0ZSBhbmQgdGhlIHRhcmdldA0KPiA+ ID4gZmFpbHMgdG8gcmVjb2duaXplIHRoZSBpbml0aWF0b3IncyBzdWJzZXF1ZW50IElTSUQgcmV1 c2UgZXZlbiB0aG91Z2gNCj4gPiA+IHRoZSBpbml0aWF0b3IgYmVsaWV2ZXMgdGhhdCBpdCdzIHJl LSBlc3RhYmxpc2hpbmcgdGhlIHNlc3Npb24uDQo+ID4gPiA+DQo+ID4gPiA+IFRoaXMgaXMgZnVy dGhlciBjb21wbGljYXRlZCBieSB0aGUgVUEgdGhhdCBpbmRpY2F0ZXMgbmV4dXMgc3RhdGUNCj4g PiA+IHByZXNlcnZhdGlvbiBub3QgYmVpbmcgc3BlY2lmaWVkIGluIFNBTS0yLg0KPiA+ID4gPg0K PiA+ID4gPiBUaGFua3MsDQo+ID4gPiA+IC0tRGF2aWQNCj4gPiA+ID4NCj4gPiA+ID4NCj4gPiA+ ID4+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+ID4gPiA+PiBGcm9tOiBLbmlnaHQsIEZy ZWRlcmljayBbbWFpbHRvOkZyZWRlcmljay5LbmlnaHRAbmV0YXBwLmNvbV0NCj4gPiA+ID4+IFNl bnQ6IFRodXJzZGF5LCBTZXB0ZW1iZXIgMjcsIDIwMTIgMTE6MTEgQU0NCj4gPiA+ID4+IFRvOiBC bGFjaywgRGF2aWQ7IEp1bGlhbiBTYXRyYW4NCj4gPiA+ID4+IENjOiBzdG9ybUBpZXRmLm9yZw0K PiA+ID4gPj4gU3ViamVjdDogUkU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQg LSB2ZXJzaW9uIDYNCj4gPiA+ID4+DQo+ID4gPiA+PiBJIGFmcmFpZCBJIGhhdmUgdG8gY29udGVz dCBhIGxpdHRsZSBvbiB0aGlzIC0gaWYgSSB1bmRlcnN0YW5kDQo+ID4gPiA+PiB0aGlzIGRpc2N1 c3Npb24gY29ycmVjdGx5Lg0KPiA+ID4gPj4NCj4gPiA+ID4+IElmIGEgU0NTSSBkZXZpY2UgbG9z ZXMgYW55IHN0YXRlIGFuZCBkb2VzIG5vdCByYWlzZSBhIFVOSVQNCj4gPiA+ID4+IEFUVEVOVElP TiwgdGhlbiBpdCBpcyBCUk9LRU4uICBJdCBpcyBub3QgYSBtYXR0ZXIgb2YgYW55dGhpbmcgaW4N Cj4gPiA+ID4+IGlTQ1NJICgzNzIwKTsgSSB0aGluayBTQU0gYW5kIFNQQyBhcmUgcHJldHR5IGNs ZWFyIG9uIHRoaXMuICBJZg0KPiA+ID4gPj4gdGhlcmUgaXMgbm8gVU5JVCBBVFRFTlRJT04sIHRo ZW4gc3RhdGUgU0hBTEwgYmUgcHJlc2VydmVkOw0KPiA+ID4gPj4gb3RoZXJ3aXNlLCBhbGwgc29y dHMgb2Ygc3R1ZmYgd2lsbCBuZXZlciB3b3JrIChtb2RlIHBhZ2UNCj4gPiA+ID4+IHNldHRpbmdz LCBSRVNFUlZFIGNvbW1hbmRzLCBhbGwgc29ydHMgb2YNCj4gPiA+IHN0dWZmKS4NCj4gPiA+ID4+ DQo+ID4gPiA+PiBJcyBzb21lb25lIHJlYWxseSBzdWdnZXN0aW5nIHRoYXQgc29tZSBkZXZpY2Vz IHdpbGwgZHJvcCBzdGF0ZQ0KPiA+ID4gPj4gYW5kIE5PVCB0ZWxsIHRoZSBob3N0LCBhbmQgc29t ZWhvdyB0aGF0IGl0IGlzIE9LIHRvIGRvIHRoYXQ/DQo+ID4gPiA+Pg0KPiA+ID4gPj4gVGhlIHN1 Z2dlc3QgcmV3b3JkaW5nIHNheXMgdGhpcyBpcyBwb3NzaWJsZS4NCj4gPiA+ID4+DQo+ID4gPiA+ PiAgICBGcmVkDQo+ID4gPiA+Pg0KPiA+ID4gPj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0N Cj4gPiA+ID4+IEZyb206IHN0b3JtLWJvdW5jZXNAaWV0Zi5vcmcgW21haWx0bzpzdG9ybS1ib3Vu Y2VzQGlldGYub3JnXSBPbg0KPiA+ID4gPj4gQmVoYWxmIE9mIEJsYWNrLCBEYXZpZA0KPiA+ID4g Pj4gU2VudDogVGh1cnNkYXksIFNlcHRlbWJlciAyNywgMjAxMiAxMDo0NyBBTQ0KPiA+ID4gPj4g VG86IEp1bGlhbiBTYXRyYW4NCj4gPiA+ID4+IENjOiBzdG9ybUBpZXRmLm9yZw0KPiA+ID4gPj4g U3ViamVjdDogUmU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9u IDYNCj4gPiA+ID4+DQo+ID4gPiA+PiBKdWxpYW4sDQo+ID4gPiA+Pg0KPiA+ID4gPj4+IFRoZSB0 ZXh0IGluIDQuNC4zIHN0YXRlcyB0aGUgb2J2aW91cyAodGhlcmUgaXMgbm8gb3RoZXIgc3BlY2Vk DQo+ID4gPiA+Pj4gbWV0aG9kIGZvciBjb250aW51YXRpb24gb3IgcmVpbnN0YXRlbWVudCkuIE1v cmVvdmVyIGlmIHRoZQ0KPiA+ID4gPj4+IHNlc3Npb24gaXMgcmVpbnN0YXRlZCBhZnRlciB0aW1l MndhaXQgeW91IGNyZWF0ZSBhIG5lZWQgZm9yIGEgVUENCj4gPiA+ID4+PiAob3IgYXQgbGVhc3Qg c3VnZ2VzdCBvbmUpIGFuZCB0aGF0IHdhcyBub3QgdGhlcmUuDQo+ID4gPiA+Pg0KPiA+ID4gPj4g VGhlIFVBcyBhcmUgY291cnRlc3kgb2YgU0FNLTMgKGFuZCBTQU0tNCkuICBUbyBlbmNvbXBhc3Mg c3lzdGVtcw0KPiA+ID4gPj4gdGhhdCBtYXkgbm90IGRlbGl2ZXIgdGhlc2UgVUFzIGFuZCBzaXR1 YXRpb25zIGluIHdoaWNoIHRoZXkgZG9uJ3QNCj4gPiA+ID4+IG9jY3VyIChlLmcuLCB0aGUgdGFy Z2V0IGRvZXNuJ3QgcmVjb2duaXplIHdoYXQgaGFwcGVuZWQgYXMgbmV4dXMNCj4gPiA+ID4+IHJl ZXN0YWJsaXNobWVudCwgaW5kZXBlbmRlbnQgb2Ygd2hhdCB0aGUgaW5pdGlhdG9yIHRoaW5rcyBp dCdzDQo+ID4gPiA+PiBkb2luZykgSQ0KPiA+ID4gdGhpbmsgdGhlIHRleHQgc2hvdWxkIGJlIHJl cGhyYXNlZCBhczoNCj4gPiA+ID4+DQo+ID4gPiA+PiAgICAgSWYgYSBVbml0IEF0dGVudGlvbiBj b25kaXRpb24gcmVzdWx0cyBmcm9tIHNlc3Npb24gcmVlc3RhYmxpc2htZW50LA0KPiA+ID4gPj4g ICAgIHRoZSBzcGVjaWZpYyBVbml0IEF0dGVudGlvbiBjb25kaXRpb24gaW5kaWNhdGVzIHdoZXRo ZXIgbmV4dXMgc3RhdGUNCj4gPiA+ID4+ICAgICB3YXMgcHJlc2VydmVkLCBzZWUgU2VjdGlvbiA2 LjMuNCBvZiBbU0FNNF0uDQo+ID4gPiA+Pg0KPiA+ID4gPj4+IEp1c3Qgc3RhdGluZyB0aGF0IGNy ZWF0aW9uL2NvbnRpbnVhdGlvbiBvZiBhIHNlc3Npb24gZG9uZQ0KPiA+ID4gPj4+IGFjY29yZGlu ZyA2LnggaW5jbHVkZXMgcHJlc2VydmF0aW9uIG9mIHJlc2VydmUvcmVsZWFzZSBzdGF0ZQ0KPiA+ ID4gPj4+IC10aGF0IHdhcyBhbHJlYWR5IHRoZXJlIGltcGxpZWQgYnkgY29uZm9ybWFuY2UgdG8g c3BjMiBhbmQgdGhlDQo+ID4gPiA+Pj4gcmVsYXRpb24gYmV0d2VlbiBpdCBuZXh1cyBhbmQgc2Vz c2lvbi4NCj4gPiA+ID4+DQo+ID4gPiA+PiBVbmZvcnR1bmF0ZWx5LCBTUEMtMiBpcyBzdWZmaWNp ZW50bHkgaW1wcmVjaXNlIG9uIHRoaXMgc3ViamVjdA0KPiA+ID4gPj4gdGhhdCBzdWNoIGFuIGlt cGxpY2F0aW9uIGlzIGF0IGJlc3QgYSAicHJlZmVycmVkIGltcGxlbWVudGF0aW9uDQo+IGFwcHJv YWNoIi4NCj4gPiA+ID4+IEl0J3Mgbm90IGEgcmVxdWlyZW1lbnQgdGhhdCBjYW4gYmUgc2FmZWx5 IHN0YXRlZCBpbiBhbnkgc3Ryb25nZXINCj4gPiA+ID4+IGZhc2hpb24gd2l0aG91dCB0aGUgcmlz ayBvZiByZW5kZXJpbmcgZXhpc3RpbmcgaW1wbGVtZW50YXRpb25zDQo+ID4gPiA+PiBub24tDQo+ ID4gPiBjb21wbGlhbnQuDQo+ID4gPiA+Pg0KPiA+ID4gPj4+IFRoZSBuZXcgdGV4dCB5b3Ugc3Vn Z2VzdCB0byA0LjQgaXMgbWVyZWx5IGEgYmVzdCBwcmFjdGljZXMNCj4gPiA+ID4+PiByZWNvbW1l bmRhdGlvbnMgYW5kIHdpbGwgbmVlZCBhZGRpdGlvbnMgb25jZSB0aGUgbmV3ICJsb2NraW5nIg0K PiA+ID4gPj4+IHRocm91Z2ggY29tcGFyZS1hbmQtc3dhcCBnZXQgaW4gd2lkZXNwcmVhZCB1c2Uu DQo+ID4gPiA+Pg0KPiA+ID4gPj4gSXQncyBkZWZpbml0ZWx5IGJlc3QgcHJhY3RpY2VzIHJlY29t bWVuZGF0aW9ucyAtIG5vIGFyZ3VtZW50IHRoZXJlLg0KPiA+ID4gPj4NCj4gPiA+ID4+IEZvciBj b250ZXh0LCB3aGF0IEp1bGlhbiBpcyByZWZlcnJpbmcgdG8gaXMgdXNlIG9mIHRoZSBDT01QQVJF DQo+ID4gPiA+PiBBTkQgV1JJVEUgY29tbWFuZCB0aGF0IHBlcmZvcm1zIGFuIGF0b21pYyBvcGVy YXRpb24gYXQgdGhlIHRhcmdldA0KPiA+ID4gPj4gaW5zdGVhZCBvZiB1c2luZyBSRVNFUlZFIGFu ZCBSRUxFQVNFIHRvIGNyZWF0ZSBhIGNyaXRpY2FsIHNlY3Rpb24NCj4gPiA+ID4+IGF0IHRoZSBp bml0aWF0b3IgYmFzZWQgb24gcmVhZCBhbmQgd3JpdGUgY29tbWFuZHMuDQo+ID4gPiA+Pg0KPiA+ ID4gPj4gQ09NUEFSRSBBTkQgV1JJVEUgaXMgaW4gdXNlIC0gYSBzZW50ZW5jZSBtZW50aW9uaW5n IHRoYXQgY29tbWFuZA0KPiA+ID4gPj4gY291bGQgYmUgYWRkZWQsIGFsdGhvdWdoIHRoaXMgYmVz dCBwcmFjdGljZXMgdGV4dCBpcyBhaW1lZA0KPiA+ID4gPj4gcHJpbWFyaWx5IGF0IHVzZXMgb2Yg cmVzZXJ2ZS9yZWxlYXNlIGZvciBsb25nLXRlcm0gcmVzZXJ2YXRpb25zLg0KPiA+ID4gPj4gSSB0 aGluayB0aGVyZSdzIGFuIGltcGxpY2l0ICJ3aGVuIGEgcmVzZXJ2YXRpb24gaXMgdGhlIGRlc2ly ZWQNCj4gYmVoYXZpb3IiDQo+ID4gPiA+PiBpbXBsaWNhdGlvbiB0byB0aGUgInBlcnNpc3RlbnQg cmVzZXJ2YXRpb25zIFNIT1VMRCBiZSB1c2VkIGluc3RlYWQiDQo+ID4gPiA+PiB0ZXh0IHRoYXQg SSBkb24ndCB0aGluayBuZWVkcyB0byBiZSBtYWRlIGV4cGxpY2l0LiAgT1RPSCwgdGhhdCAiU0hP VUxEIg0KPiA+ID4gPj4gY291bGQgYmUgY2hhbmdlZCB0byBhICJzaG91bGQiLCBvciBpdCBtaWdo dCBiZSBldmVuIGJldHRlciB0bw0KPiA+ID4gPj4gcmV3cml0ZSB0aGF0IHRleHQNCj4gPiA+ID4+ IGFzOg0KPiA+ID4gPj4NCj4gPiA+ID4+ICAgIHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIGFyZSBz dWdnZXN0ZWQgYXMgYW4gYWx0ZXJuYXRpdmUsIHNlZSBBbm5leCBCDQo+ID4gPiA+PiAgICBvZiBb U1BDNF0uDQo+ID4gPiA+Pg0KPiA+ID4gPj4gSW4gY29udHJhc3QsICJTSE9VTEQgTk9UIGJlIHVz ZWQiIGlzIGNsZWFybHkganVzdGlmaWVkIGZvcg0KPiA+ID4gPj4gcmVzZXJ2ZS9yZWxlYXNlIHJl c2VydmF0aW9ucyBieSBbU1BDM10uDQo+ID4gPiA+Pg0KPiA+ID4gPj4gVGhhbmtzLA0KPiA+ID4g Pj4gLS1EYXZpZA0KPiA+ID4gPj4NCj4gPiA+ID4+DQo+ID4gPiA+Pj4gLS0tLS1PcmlnaW5hbCBN ZXNzYWdlLS0tLS0NCj4gPiA+ID4+PiBGcm9tOiBKdWxpYW4gU2F0cmFuIFttYWlsdG86anVsaWFu QHNhdHJhbi5uZXRdDQo+ID4gPiA+Pj4gU2VudDogV2VkbmVzZGF5LCBTZXB0ZW1iZXIgMjYsIDIw MTIgMTA6MjUgUE0NCj4gPiA+ID4+PiBUbzogQmxhY2ssIERhdmlkDQo+ID4gPiA+Pj4gQ2M6IHN0 b3JtQGlldGYub3JnDQo+ID4gPiA+Pj4gU3ViamVjdDogUmU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2 ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYNCj4gPiA+ID4+Pg0KPiA+ID4gPj4+IERhdmlkLA0K PiA+ID4gPj4+DQo+ID4gPiA+Pj4gVGhlIHRleHQgaW4gNC40LjMgc3RhdGVzIHRoZSBvYnZpb3Vz ICh0aGVyZSBpcyBubyBvdGhlciBzcGVjZWQNCj4gPiA+ID4+PiBtZXRob2QgZm9yIGNvbnRpbnVh dGlvbiBvciByZWluc3RhdGVtZW50KS4gTW9yZW92ZXIgaWYgdGhlIHRoZQ0KPiA+ID4gPj4+IHNl c3Npb24gaXMgcmVpbnN0YXRlZCBhZnRlciB0aW1lMndhaXQgeW91IGNyZWF0ZSBhIG5lZWQgZm9y IGEgVUENCj4gPiA+ID4+PiAob3IgYXQgbGVhc3Qgc3VnZ2VzdCBvbmUpIGFuZCB0aGF0IHdhcyBu b3QgdGhlcmUuIEp1c3Qgc3RhdGluZw0KPiA+ID4gPj4+IHRoYXQgY3JlYXRpb24vY29udGludWF0 aW9uIG9mIGEgc2Vzc2lvbiBkb25lIGFjY29yZGluZyA2LngNCj4gPiA+ID4+PiBpbmNsdWRlcyBw cmVzZXJ2YXRpb24gb2YgcmVzZXJ2ZS9yZWxlYXNlIHN0YXRlIC10aGF0IHdhcyBhbHJlYWR5DQo+ ID4gPiA+Pj4gdGhlcmUgaW1wbGllZCBieSBjb25mb3JtYW5jZSB0byBzcGMyIGFuZCB0aGUgcmVs YXRpb24gYmV0d2VlbiBpdA0KPiA+ID4gPj4+IG5leHVzIGFuZA0KPiA+ID4gc2Vzc2lvbi4NCj4g PiA+ID4+Pg0KPiA+ID4gPj4+IFRoZSBuZXcgdGV4dCB5b3Ugc3VnZ2VzdCB0byA0LjQgaXMgbWVy ZWx5IGEgYmVzdCBwcmFjdGljZXMNCj4gPiA+ID4+PiByZWNvbWVuZGF0aW9ucyBhbmQgd2lsbCBu ZWVkIGFkZGl0aW9ucyBvbmNlIHRoZSBuZXcgImxvY2tpbmciDQo+ID4gPiA+Pj4gdGhyb3VnaCBj b21wYXJlLWFuZC1zd2FwIGdldCBpbiB3aWRlc3ByZWFkIHVzZS4NCj4gPiA+ID4+PiBJZiBpbnN0 ZWFkIHdlIHJlZnJhaW4gZnJvbSBzdGF0aW5nIGFueXRoaW5nIHNwZWNpZmljIGFuZCBzdGF0ZQ0K PiA+ID4gPj4+IHRoYXQgdGFyZ2V0IHNob3VsZCBtYWludGFpbiB3aGF0ZXZlciBzdGF0ZSBpcyBy ZXF1aXJlZCBieSBTUEMNCj4gPiA+ID4+PiB3aGlsZSBhIHNlc3Npb24gaXMgY29udGludWVkIGFu ZCBtYXkgZGlzY2FyZCBzdGF0ZSB3aGVuIGENCj4gPiA+ID4+PiBzZXNzaW9uIGlzIHRlcm1pbmF0 ZWQgYW5kIG1heSBpbmRpY2F0ZSB0aGF0IGl0IGRpc2NhcmRlZCBzdGF0ZQ0KPiA+ID4gPj4+IHRo cm91Z2ggYSB1YSB5b3UgYXJlIG9uIHNhZmVyIGdyb3VuZCBhbmQgZG8gbm90IGNyZWF0ZSBuZXcN Cj4gcmVxdWlyZW1lbnQuDQo+ID4gPiA+Pj4NCj4gPiA+ID4+PiBTZW50IGZyb20gbXkgaVBhZA0K PiA+ID4gPj4+DQo+ID4gPiA+Pj4gT24gMjcg15HXodek15ggMjAxMiwgYXQgMDA6MjUsICJCbGFj aywgRGF2aWQiIDxkYXZpZC5ibGFja0BlbWMuY29tPiB3cm90ZToNCj4gPiA+ID4+Pg0KPiA+ID4g Pj4+PiBIZXJlJ3Mgd2hlcmUgSSB0aGluayB3ZSd2ZSBhcnJpdmVkIChJJ3ZlIG1hZGUgYSBmZXcg bWlub3INCj4gPiA+ID4+Pj4gZWRpdG9yaWFsIHR3ZWFrcyB0byB3aGF0J3MgYmVlbiBkaXNjdXNz ZWQpLiAgRGVzcGl0ZSBhbGwgdGhlDQo+ID4gPiA+Pj4+IGRldGFpbGVkIGRpc2N1c3Npb24sIEkg dGhpbmsgdGhpcyB2ZXJzaW9uIDYgY29tZXMgY2xvc2UgdG8NCj4gPiA+ID4+Pj4gSnVsaWFuJ3Mg c3VnZ2VzdGlvbg0KPiA+ID4gPj4gdGhhdDoNCj4gPiA+ID4+Pj4NCj4gPiA+ID4+Pj4+IGkgd291 bGQgbWVudGlvbiBvbmx5IHRoYXQgdGhlIHN0YXRlIHRoYXQgbXVzdCBiZSBtYWludGFpbmVkDQo+ ID4gPiA+Pj4+PiB3aGlsZSB0aGUgc2Vzc2lvbiBpcyBtYWludGFpbmVkIGluY2x1ZGVzIHRoZSBy ZXNlcnZlL3JlbGVhc2UNCj4gPiA+ID4+Pj4+IGFuZCBsZWF2ZSBpdCB0bw0KPiA+ID4gPj4+IHRo YXQuDQo+ID4gPiA+Pj4+DQo+ID4gPiA+Pj4+IEFsdGhvdWdoIHdlIGNhbid0IGFjdHVhbGx5IHNh eSAibXVzdCIgYXMgdGhhdCB3b3VsZCBiZSBhIG5ldw0KPiA+ID4gPj4+PiByZXF1aXJlbWVudCwg YW5kIGltcGxlbWVudGF0aW9ucyBkaWZmZXIgaW4gd2hhdCB0aGV5IGRvIGFib3V0DQo+ID4gPiA+ Pj4+IHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbiBzdGF0ZS4gIFRoZSBjdXJyZW50IGxhbmd1 YWdlDQo+ID4gPiA+Pj4+IGNoYXJhY3Rlcml6ZXMgcmV0ZW50aW9uIG9mIHRoYXQgc3RhdGUgYXMg dGhlICJwcmVmZXJyZWQNCj4gPiA+ID4+Pj4gaW1wbGVtZW50YXRpb24NCj4gPiA+ID4+IGFwcHJv YWNoIi4NCj4gPiA+ID4+Pj4NCj4gPiA+ID4+Pj4gQmV5b25kIHRoYXQsIEkgdGhpbmsgdGhlIHR3 byBhZGRpdGlvbmFsIGNhdmVhdHMgYWJvdXQNCj4gPiA+ID4+Pj4gcmVzZXJ2ZS9yZWxlYXNlIHJl c2VydmF0aW9ucyAocG9zc2libGUgbmVlZCBmb3IgYSByZXNldCwNCj4gPiA+ID4+Pj4gcG90ZW50 aWFsbHkgdW5leHBlY3RlZCBpbnRlcmFjdGlvbiB3aXRoIHBlcnNpc3RlbnQNCj4gPiA+ID4+Pj4g cmVzZXJ2YXRpb25zLCBzZWUgYmVsb3cpIGFyZSB1c2VmdWwNCj4gPiA+ID4+IHRvIGluY2x1ZGUu DQo+ID4gPiA+Pj4+DQo+ID4gPiA+Pj4+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tDQo+ID4gPiA+Pj4+DQo+ID4gPiA+Pj4+IFsxXSBGb3IgY2xhcml0eSwgdGhlIGZv bGxvd2luZyBjaGFuZ2Ugc2hvdWxkIGJlIG1hZGUgaW4NCj4gPiA+ID4+Pj4gNC40LjMuMToNCj4g PiA+ID4+Pj4NCj4gPiA+ID4+Pj4gT0xEDQo+ID4gPiA+Pj4+IFRoYXQgaXMsIGl0IHNob3VsZCBw ZXJmb3JtIHNlc3Npb24gcmVjb3ZlcnkgYXMgIGRlc2NyaWJlZCBpbg0KPiA+ID4gPj4+PiBDaGFw dGVyIDYuDQo+ID4gPiA+Pj4+IE5FVw0KPiA+ID4gPj4+PiBUaGF0IGlzLCBpdCBzaG91bGQgcmVp bnN0YXRlIHRoZSBzZXNzaW9uIHZpYSBpU0NTSSBzZXNzaW9uDQo+ID4gPiA+Pj4+IHJlaW5zdGF0 ZW1lbnQgKFNlY3Rpb24gNi4zLjUpIG9yIGNvbnRpbnVlIHZpYSBzZXNzaW9uDQo+ID4gPiA+Pj4+ IGNvbnRpbnVhdGlvbiAoU2VjdGlvbiA2LjMuNikuDQo+ID4gPiA+Pj4+IEVORA0KPiA+ID4gPj4+ Pg0KPiA+ID4gPj4+PiBbMl0gQWRkIHRoZSBmb2xsb3dpbmcgc2VudGVuY2UgdG8gdGhlIGVuZCBv ZiA0LjQuMy4xOg0KPiA+ID4gPj4+Pg0KPiA+ID4gPj4+PiAgICBUaGUgc3BlY2lmaWMgVW5pdCBB dHRlbnRpb24gY29uZGl0aW9uIHRoYXQgcmVzdWx0cyBmcm9tIHNlc3Npb24NCj4gPiA+ID4+Pj4g ICAgcmVlc3RhYmxpc2htZW50IGluZGljYXRlcyB3aGV0aGVyIG5leHVzIHN0YXRlIHdhcyBwcmVz ZXJ2ZWQsDQo+ID4gPiA+Pj4+ICAgIHNlZSBTZWN0aW9uIDYuMy40IG9mIFtTQU00XS4NCj4gPiA+ ID4+Pj4NCj4gPiA+ID4+Pj4gWzNdIE5FVyBURVhUOg0KPiA+ID4gPj4+Pg0KPiA+ID4gPj4+PiA0 LjQuMy4yLiBSZXNlcnZhdGlvbnMNCj4gPiA+ID4+Pj4NCj4gPiA+ID4+Pj4gVGhlcmUgYXJlIHR3 byByZXNlcnZhdGlvbiBtYW5hZ2VtZW50IG1ldGhvZHMgZGVmaW5lZCBpbiB0aGUNCj4gPiA+ID4+ Pj4gU0NTSSBzdGFuZGFyZHMsIHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbnMsIGJhc2VkIG9u IHRoZQ0KPiA+ID4gPj4+PiBSRVNFUlZFIGFuZCBSRUxFQVNFIGNvbW1hbmRzIFtTUEMyXSwgYW5k IHBlcnNpc3RlbnQNCj4gPiA+ID4+Pj4gcmVzZXJ2YXRpb25zLCBiYXNlZCBvbiB0aGUgUEVSU0lT VEVOVCBSRVNFUlZFIElOIGFuZCBQRVJTSVNURU5UDQo+IFJFU0VSVkUgT1VUIGNvbW1hbmRzIFtT UEMzXS4NCj4gPiA+ID4+Pj4gUmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9ucyBhcmUgb2Jzb2xl dGUgW1NQQzNdIGFuZCBTSE9VTEQgTk9UDQo+ID4gPiA+Pj4+IGJlIHVzZWQ7IHBlcnNpc3RlbnQg cmVzZXJ2YXRpb25zIFNIT1VMRCBiZSB1c2VkIGluc3RlYWQuDQo+ID4gPiA+Pj4+DQo+ID4gPiA+ Pj4+IFN0YXRlIGZvciBwZXJzaXN0ZW50IHJlc2VydmF0aW9ucyBpcyByZXF1aXJlZCB0byBwZXJz aXN0DQo+ID4gPiA+Pj4+IHRocm91Z2ggY2hhbmdlcyBhbmQgZmFpbHVyZXMgYXQgdGhlIGlTQ1NJ IGxheWVyIHRoYXQgcmVzdWx0IGluDQo+ID4gPiA+Pj4+IElfVCBOZXh1cyBmYWlsdXJlcywgc2Vl IFtTUEMzXSBmb3IgZGV0YWlscyBhbmQgc3BlY2lmaWMgcmVxdWlyZW1lbnRzLg0KPiA+ID4gPj4+ Pg0KPiA+ID4gPj4+PiBJbiBjb250cmFzdCwgW1NQQzJdIGRvZXMgbm90IHNwZWNpZnkgZGV0YWls ZWQgcGVyc2lzdGVuY2UNCj4gPiA+ID4+Pj4gcmVxdWlyZW1lbnRzIGZvciByZXNlcnZlL3JlbGVh c2UgcmVzZXJ2YXRpb24gc3RhdGUgYWZ0ZXIgYW4gSV9UDQo+ID4gPiA+Pj4+IE5leHVzIGZhaWx1 cmUuICBOb25ldGhlbGVzcywgd2hlbiByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zDQo+ID4g PiA+Pj4+IGFyZSBzdXBwb3J0ZWQgYnkgYW4gaVNDU0kgdGFyZ2V0LCB0aGUgcHJlZmVycmVkIGlt cGxlbWVudGF0aW9uDQo+ID4gPiA+Pj4+IGFwcHJvYWNoIGlzIHRvIHByZXNlcnZlIHJlc2VydmUv cmVsZWFzZSByZXNlcnZhdGlvbiBzdGF0ZSBmb3INCj4gPiA+ID4+Pj4gaVNDU0kgc2Vzc2lvbiBy ZWluc3RhdGVtZW50IChzZWUgU2VjdGlvbiA2LjMuNSkgb3Igc2Vzc2lvbg0KPiA+ID4gPj4+PiBj b250aW51YXRpb24gIChzZWUgU2VjdGlvbiA2LjMuNikuDQo+ID4gPiA+Pj4+DQo+ID4gPiA+Pj4+ IFR3byBhZGRpdGlvbmFsIGNhdmVhdHMgYXBwbHkgdG8gcmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0 aW9uczoNCj4gPiA+ID4+Pj4NCj4gPiA+ID4+Pj4gLSBXaGVuIGNvbm5lY3Rpb24gZmFpbHVyZSBj YXVzZXMgdGhlIGlTQ1NJIHNlc3Npb24gdG8gZmFpbCwgYW5kDQo+ID4gPiA+Pj4+ICAgICB0aGUg c2Vzc2lvbiBpcyBub3QgcmVpbnN0YXRlZCBvciBjb250aW51ZWQsIHRhcmdldCByZXRlbnRpb24N Cj4gPiA+ID4+Pj4gICAgIG9mIHRoYXQgc2Vzc2lvbidzIHJlc2VydmUvcmVsZWFzZSByZXNlcnZh dGlvbiBzdGF0ZSBtYXkNCj4gPiA+ID4+Pj4gICAgIHJlcXVpcmUgYW4gaW5pdGlhdG9yIHRvIGlz c3VlIGEgcmVzZXQgKGUuZy4sIExPR0lDQUwgVU5JVCBSRVNFVCwNCj4gPiA+ID4+Pj4gICAgIHNl ZSBzZWN0aW9uIDExLjUpIGluIG9yZGVyIHRvIHJlbW92ZSB0aGF0IHJlc2VydmF0aW9uIHN0YXRl Lg0KPiA+ID4gPj4+Pg0KPiA+ID4gPj4+PiAtIFJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbnMg bWF5IG5vdCBiZWhhdmUgYXMgZXhwZWN0ZWQgd2hlbg0KPiA+ID4gPj4+PiAgICAgcGVyc2lzdGVu dCByZXNlcnZhdGlvbnMgYXJlIGFsc28gdXNlZCBvbiB0aGUgc2FtZSBsb2dpY2FsIHVuaXQ7DQo+ ID4gPiA+Pj4+ICAgICBzZWUgdGhlIGRpc2N1c3Npb24gb2YgIkV4Y2VwdGlvbnMgdG8gU1BDLTIg UkVTRVJWRSBhbmQgUkVMRUFTRQ0KPiA+ID4gPj4+PiAgICAgYmVoYXZpb3IiIGluIFtTUEM0XS4N Cj4gPiA+ID4+Pj4NCj4gPiA+ID4+Pj4gVGhhbmtzLA0KPiA+ID4gPj4+PiAtLURhdmlkDQo+ID4g PiA+Pj4+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0NCj4gPiA+ID4+Pj4gRGF2aWQgTC4gQmxhY2ssIERpc3Rpbmd1aXNoZWQgRW5naW5lZXIgRU1D IENvcnBvcmF0aW9uLCAxNzYNCj4gPiA+ID4+Pj4gU291dGggU3QuLCBIb3BraW50b24sIE1BICAw MTc0OA0KPiA+ID4gPj4+PiArMSAoNTA4KSAyOTMtNzk1MyAgICAgICAgICAgICBGQVg6ICsxICg1 MDgpIDI5My03Nzg2DQo+ID4gPiA+Pj4+IGRhdmlkLmJsYWNrQGVtYy5jb20gICAgICAgIE1vYmls ZTogKzEgKDk3OCkgMzk0LTc3NTQNCj4gPiA+ID4+Pj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiA+ID4gPj4+Pg0KPiA+ID4gPj4+Pg0KPiA+ ID4gPj4+PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0K PiA+ID4gPj4+PiBzdG9ybSBtYWlsaW5nIGxpc3QNCj4gPiA+ID4+Pj4gc3Rvcm1AaWV0Zi5vcmcN Cj4gPiA+ID4+Pj4gaHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9zdG9ybQ0K PiA+ID4gPj4NCj4gPiA+ID4+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fDQo+ID4gPiA+PiBzdG9ybSBtYWlsaW5nIGxpc3QNCj4gPiA+ID4+IHN0b3JtQGll dGYub3JnDQo+ID4gPiA+PiBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3N0 b3JtDQo+ID4gPiA+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fDQo+ID4gPiA+IHN0b3JtIG1haWxpbmcgbGlzdA0KPiA+ID4gPiBzdG9ybUBpZXRmLm9yZw0K PiA+ID4gPiBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3N0b3JtDQo= From Frederick.Knight@netapp.com Fri Sep 28 09:40:15 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0229521F851E for ; Fri, 28 Sep 2012 09:40:15 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1QMLLzE5yazC for ; Fri, 28 Sep 2012 09:40:13 -0700 (PDT) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id 65B1E21F851C for ; Fri, 28 Sep 2012 09:40:13 -0700 (PDT) X-IronPort-AV: E=Sophos;i="4.80,502,1344236400"; d="scan'208";a="695287335" Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx2-out.netapp.com with ESMTP; 28 Sep 2012 09:40:12 -0700 Received: from vmwexceht05-prd.hq.netapp.com (vmwexceht05-prd.hq.netapp.com [10.106.77.35]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8SGeBve026428; Fri, 28 Sep 2012 09:40:12 -0700 (PDT) Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.102]) by vmwexceht05-prd.hq.netapp.com ([10.106.77.35]) with mapi id 14.02.0309.002; Fri, 28 Sep 2012 09:40:11 -0700 From: "Knight, Frederick" To: "Black, David" Thread-Topic: [storm] SPC-2 reserve/release text - version 6 Thread-Index: Ac2cNcys4MzRK8PXQu651fVKNuEIxQAXDfsAABnkfIAADjDW0AAa4S6gACikjGD//nJjgIAAECkggAAHbKCAAAZiEIAABrHggAAIv6CAAAR6sP///AFA Date: Fri, 28 Sep 2012 16:40:10 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> <4AE113CD-96D7-414D-8AAB-36ECF7FD89EA@satran.net> <8D3D17ACE214DC429325B2B98F3AE7120DE798F9@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE7993C@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79B42@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79B71@MX15A.corp.emc.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79BB8@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE79BB8@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.106.53.53] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Cc: "storm@ietf.org" Subject: Re: [storm] SPC-2 reserve/release text - version 6 X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Sep 2012 16:40:15 -0000 VGltZXJzIGRvbid0IHJlc29sdmUgdGhlIHNpdHVhdGlvbiAodGhleSBhcmUganVzdCBhIHRyYWRl b2ZmKS4gIEluIG15IG9waW5pb24sIGEgZGV2aWNlIHRoYXQgYWxsb3dzIHRoYXQgc2VxdWVuY2U6 DQoNCj4gSW5pdGlhdG9yIEEgLSBSRVNFUlZFIGNvbW1hbmQgKEdPT0Qgc3RhdHVzKQ0KPiBJbml0 aWF0b3IgQiAtIFdSSVRFIChSRVNFUlZBVElPTiBDT05GTElDVCBzdGF0dXMpDQo+IEluaXRpYXRv ciBBIGdldHMgZGlzY29ubmVjdGVkL3JlY29ubmVjdGVkIGFuZCBSRVNFUlZFIHN0YXRlIGlzIGxv c3QNCj4gSW5pdGlhdG9yIEIgLSBXUklURSAoR09PRCBzdGF0dXMpDQoNCklzIGluIHZpb2xhdGlv biBvZiB0aGUgdGV4dCBpbiBTUEMtMiB0aGF0IHNheXM6DQoNCiJSZXNlcnZlL3JlbGVhc2UgbWFu YWdlZCByZXNlcnZhdGlvbnMgYXJlIHJldGFpbmVkIGJ5DQp0aGUgZGV2aWNlIHNlcnZlciB1bnRp bCByZWxlYXNlZCBvciB1bnRpbCByZXNldCBieSBtZWNoYW5pc21zIHNwZWNpZmllZCBpbiB0aGlz IHN0YW5kYXJkLiINCg0KU2luY2UgLSBkaXNjb25uZWN0L3JlY29ubmVjdCBpcyBOT1Qgc3BlY2lm aWVkIGluIFNQQy0yLCBzb21lIGZvcm0gb2YgcmVzZXQgaXMgdGhlIG9ubHkgbW9kZWwgbGVmdCAo YW5kIHJlc2V0cyBlc3RhYmxpc2ggVUFzKS4NCg0KQW55d2F5LCBJJ20gZ2xhZCB3ZSBjb3VsZCBz ZXR0bGUgb24gc29tZSB0ZXh0Lg0KDQoJRnJlZA0KDQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0t LS0tDQpGcm9tOiBCbGFjaywgRGF2aWQgW21haWx0bzpkYXZpZC5ibGFja0BlbWMuY29tXSANClNl bnQ6IEZyaWRheSwgU2VwdGVtYmVyIDI4LCAyMDEyIDExOjQ0IEFNDQpUbzogS25pZ2h0LCBGcmVk ZXJpY2sNCkNjOiBzdG9ybUBpZXRmLm9yZw0KU3ViamVjdDogUkU6IFtzdG9ybV0gU1BDLTIgcmVz ZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYNCg0KRnJlZCwNCg0KPiBIZXJlJ3MgYSBzdWdn ZXN0aW9uIHRoYXQgcmVtb3ZlcyB0aGUgIklGIiBhbmQgZG9lc24ndCBhZGQgYW55IG5ldyBtdXN0 L3Nob3VsZA0KPiByZXF1aXJlbWVudHM6DQo+DQo+IFVuaXQgQXR0ZW50aW9uIGNvbmRpdGlvbnMg dGhhdCByZXN1bHQgZnJvbSBzZXNzaW9uIHJlZXN0YWJsaXNobWVudA0KPiBpbmRpY2F0ZSB3aGV0 aGVyIG5leHVzIHN0YXRlIHdhcyBwcmVzZXJ2ZWQsIHNlZSBTZWN0aW9uIDYuMy40IG9mIFtTQU00 XS4NCg0KVGhhdCB0ZXh0IGlzIGFjY2VwdGFibGUgdG8gbWUuDQoNCkkgZG9uJ3QgdGhpbmsgd2Ug bmVlZCBhbnl0aGluZyBiZXlvbmQgdGhhdCB0ZXh0IGluIHRoZSBkcmFmdCwgYnV0IGhlcmUncw0K dGhlIHJlc3Qgb2YgdGhlIGV4cGxhbmF0aW9uIC4uLg0KDQpBbiBpbXBvcnRhbnQgYXNwZWN0IG9m IHdoYXQncyBnb2luZyBvbiBoZXJlIGlzIHRoYXQgd2l0aCByZXNwZWN0IHRvIHRoZQ0KdXNlIG9m IHRoZSB0ZXJtICJyZS1lc3RhYmxpc2htZW50IiBmb3IgYW4gSV9UIE5leHVzIGluIFNlY3Rpb24g Ni4zLjQgb2YNCltTQU00XSwgYWZ0ZXIgVGltZTJSZXRhaW4gZXhwaXJlcywgYW4gaVNDU0kgdGFy Z2V0IGlzIGVudGl0bGVkIHRvIGNvbnNpZGVyDQpJU0lEIHJlLXVzZSB0byBub3QgYmUgYSAicmUt ZXN0YWJsaXNobWVudCIgb2YgdGhlIElfVCBOZXh1cy4gIFRoaXMgaXMNCmJlY2F1c2Ugd2hlbiBU aW1lMlJldGFpbiBleHBpcmVzLCB0aGUgdGFyZ2V0IGlzIGFsbG93ZWQgdG8gZGlzY2FyZCBhbGwN Cm9mIGl0cyBzdGF0ZSBmb3IgdGhhdCBuZXh1cywgaW5jbHVkaW5nIHRoZSB0YXJnZXQncyByZWNv cmQgb2YgdGhlIElTSUQNCnZhbHVlLg0KDQpBdm9pZGFuY2Ugb2YgdGhlIHByb2JsZW1hdGljIHNj ZW5hcmlvIGludm9sdmluZyBhIHJlc2VydmF0aW9uIGxvc3MgaXMgYQ0KZmFjdG9yIHRvIGNvbnNp ZGVyIGluIHNlbGVjdGluZyBUaW1lMlJldGFpbiB2YWx1ZXM6DQoNCj4gSW5pdGlhdG9yIEEgLSBS RVNFUlZFIGNvbW1hbmQgKEdPT0Qgc3RhdHVzKQ0KPiBJbml0aWF0b3IgQiAtIFdSSVRFIChSRVNF UlZBVElPTiBDT05GTElDVCBzdGF0dXMpDQo+IEluaXRpYXRvciBBIGdldHMgZGlzY29ubmVjdGVk L3JlY29ubmVjdGVkIGFuZCBSRVNFUlZFIHN0YXRlIGlzIGxvc3QNCj4gSW5pdGlhdG9yIEIgLSBX UklURSAoR09PRCBzdGF0dXMpDQoNClRoYW5rcywNCi0tRGF2aWQNCg0KPiAtLS0tLU9yaWdpbmFs IE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBLbmlnaHQsIEZyZWRlcmljayBbbWFpbHRvOkZyZWRlcmlj ay5LbmlnaHRAbmV0YXBwLmNvbV0NCj4gU2VudDogRnJpZGF5LCBTZXB0ZW1iZXIgMjgsIDIwMTIg MTE6MDkgQU0NCj4gVG86IEJsYWNrLCBEYXZpZA0KPiBDYzogc3Rvcm1AaWV0Zi5vcmcNCj4gU3Vi amVjdDogUkU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYN Cj4NCj4gVGhlIElGIHBhcnQgb2YgdGhlIHN0YXRlbWVudDoNCj4NCj4gPiA+ID4+ICAgICBJZiBh IFVuaXQgQXR0ZW50aW9uIGNvbmRpdGlvbiByZXN1bHRzIGZyb20gc2Vzc2lvbiByZWVzdGFibGlz aG1lbnQsDQo+ID4gPiA+PiAgICAgdGhlIHNwZWNpZmljIFVuaXQgQXR0ZW50aW9uIGNvbmRpdGlv biBpbmRpY2F0ZXMgd2hldGhlciBuZXh1cyBzdGF0ZQ0KPiA+ID4gPj4gICAgIHdhcyBwcmVzZXJ2 ZWQsIHNlZSBTZWN0aW9uIDYuMy40IG9mIFtTQU00XS4NCj4NCj4gSXMgd2hhdCBpcyBicm9rZW4u DQo+DQo+IElGIHRoZSB0YXJnZXQgZGVjaWRlcyB0byBlc3RhYmxpc2ggYSBVQSwgdGhlbiBpdCBt dXN0IGZvbGxvdyBTQU00IG90aGVyd2lzZSwNCj4gaWYgdGhlIHRhcmdldCBkZWNpZGVzIG5vdCB0 byBlc3RhYmxpc2ggYSBVQSwgdGhlbiBldmVyeXRoaW5nIGlzIGZpbmUuDQo+IENvbnNpZGVyIHRo ZSBmb2xsb3dpbmcgc2VxdWVuY2U6DQo+DQo+IEluaXRpYXRvciBBIC0gUkVTRVJWRSBjb21tYW5k IChHT09EIHN0YXR1cykNCj4gSW5pdGlhdG9yIEIgLSBXUklURSAoUkVTRVJWQVRJT04gQ09ORkxJ Q1Qgc3RhdHVzKQ0KPiBJbml0aWF0b3IgQSBnZXRzIGRpc2Nvbm5lY3RlZC9yZWNvbm5lY3RlZCBh bmQgUkVTRVJWRSBzdGF0ZSBpcyBsb3N0DQo+IEluaXRpYXRvciBCIC0gV1JJVEUgKEdPT0Qgc3Rh dHVzKQ0KPg0KPiBUaGUgdGFyZ2V0IGRlY2lkZWQgbm90IHRvIGVzdGFibGlzaCBhIFVBIGFzIGEg cmVzdWx0IG9mIHRoZSBzZXNzaW9uDQo+IHJlZXN0YWJsaXNobWVudCwgc28gZXZlcnl0aGluZyBq dXN0IHByb2NlZWRzIG9uIGl0cyBtZXJyeSB3YXk/ICBIZXJlJ3MgYQ0KPiBzdWdnZXN0aW9uIHRo YXQgcmVtb3ZlcyB0aGUgIklGIiBhbmQgZG9lc24ndCBhZGQgYW55IG5ldyBtdXN0L3Nob3VsZA0K PiByZXF1aXJlbWVudHM6DQo+DQo+IFVuaXQgQXR0ZW50aW9uIGNvbmRpdGlvbnMgdGhhdCByZXN1 bHQgZnJvbSBzZXNzaW9uIHJlZXN0YWJsaXNobWVudA0KPiBpbmRpY2F0ZSB3aGV0aGVyIG5leHVz IHN0YXRlIHdhcyBwcmVzZXJ2ZWQsIHNlZSBTZWN0aW9uIDYuMy40IG9mIFtTQU00XS4NCj4NCj4g ICAgICAgRnJlZA0KPg0KPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBCbGFj aywgRGF2aWQgW21haWx0bzpkYXZpZC5ibGFja0BlbWMuY29tXQ0KPiBTZW50OiBGcmlkYXksIFNl cHRlbWJlciAyOCwgMjAxMiAxMDozNCBBTQ0KPiBUbzogS25pZ2h0LCBGcmVkZXJpY2sNCj4gQ2M6 IHN0b3JtQGlldGYub3JnDQo+IFN1YmplY3Q6IFJFOiBbc3Rvcm1dIFNQQy0yIHJlc2VydmUvcmVs ZWFzZSB0ZXh0IC0gdmVyc2lvbiA2DQo+DQo+IEknbSBub3Qgc3VyZSB3aGF0J3MgQlJPS0VOLCBi dXQgYSBzdGF0ZW1lbnQgcmVxdWlyaW5nIHRoZSBVQXMgd291bGQgYmUgYSBwb3N0LQ0KPiBTQU0t MiByZXF1aXJlbWVudHMgY2hhbmdlIHRoYXQgd291bGQgaGF2ZSB0byBnbyBpbiB0aGUNCj4gLXNh bS0gZHJhZnQgc28gdGhhdCBpdCdzIHN1YmplY3QgdG8gdGhlIG5lZ290aWF0aW9uIG9mIHRoZSB0 ZXh0IGtleSBpbiB0aGF0DQo+IGRyYWZ0LiAgSSBoYXZlIG5vIHByb2JsZW0gd2l0aCBhIGJsYW5r ZXQgc3RhdGVtZW50IHRoYXQgbmVnb3RpYXRpbmcgdGhhdCB0ZXh0DQo+IGtleSB0byBhIHZhbHVl IHRoYXQgcHJvbWlzZXMgdGhlIGZ1bmN0aW9uYWxpdHkgaW4gdGhlIC1zYW0tIGRyYWZ0IHJlcXVp cmVzDQo+IHRoYXQgdGhlIGltcGxlbWVudGF0aW9uIGFkaGVyZSB0byBldmVyeXRoaW5nIGluIFNB TS00IHRoYXQgbWF5IGRpZmZlciBmcm9tDQo+IHByaW9yIHZlcnNpb25zIG9mIFNBTSwgaW5jbHVk aW5nIElfVCBOZXh1cyBsb3NzIFVBIGJlaGF2aW9yLCBidXQgdGhlcmUgd2lsbA0KPiBuZWVkIHRv IGJlIHNvbWUgbGFuZ3VhZ2UgaW4gdGhlcmUgYWJvdXQgd2hhdCBoYXBwZW5zIGFmdGVyIFRpbWUy UmV0YWluIGV4cGlyZXMNCj4gYmVmb3JlIHRoZSBJU0lEIGlzIHJldXNlZC4NCj4NCj4gT1RPSCwg SSBhbSBzdHJvbmdseSBvcHBvc2VkIHRvIHJldmlzaW5nIHRoZSBiYXNlbGluZSByZXF1aXJlbWVu dHMgaW4gdGhlDQo+IGNvbnNvbGlkYXRlZCBkcmFmdCBpbiBhIGZhc2hpb24gdGhhdCBtYXkgYnJl YWsgInJ1bm5pbmcgY29kZSINCj4gdGhhdCdzIHdvcmtpbmcgZmluZSwgZXNwZWNpYWxseSB3aGVu IHRoZSBjaGFuZ2UgaXMgYmFzZWQgb24gYSBwb2ludCBvZg0KPiBwcmluY2lwbGUgdGhhdCBkb2Vz IG5vdCBleGlzdCBpbiB0aGUgc3BlY2lmaWNhdGlvbnMgYWdhaW5zdCB3aGljaCB0aGF0DQo+ICJy dW5uaW5nIGNvZGUiIHdhcyBkZXZlbG9wZWQgKGkuZS4sIFNBTS0yKS4NCj4NCj4gVGhhbmtzLA0K PiAtLURhdmlkDQo+DQo+ID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPiBGcm9tOiBL bmlnaHQsIEZyZWRlcmljayBbbWFpbHRvOkZyZWRlcmljay5LbmlnaHRAbmV0YXBwLmNvbV0NCj4g PiBTZW50OiBGcmlkYXksIFNlcHRlbWJlciAyOCwgMjAxMiAxMDoyMSBBTQ0KPiA+IFRvOiBCbGFj aywgRGF2aWQNCj4gPiBDYzogc3Rvcm1AaWV0Zi5vcmcNCj4gPiBTdWJqZWN0OiBSRTogW3N0b3Jt XSBTUEMtMiByZXNlcnZlL3JlbGVhc2UgdGV4dCAtIHZlcnNpb24gNg0KPiA+DQo+ID4gQnV0IHdl IGFsc28gc2hvdWxkbid0IGNvZGlmeSBCUk9LRU4gZGVzaWducyBpbiBhIGJyYW5kIG5ldyBSRkMu ICBJdA0KPiA+IHdvdWxkIGJldHRlciAoaW4gbXkgb3BpbmlvbikgdG8gbGVhdmUgaXQgYWxvbmUg dGhhbiBzYW5jdGlvbiBhbmQNCj4gPiBkZXNjcmliZSBob3cgdG8gYnVpbGQgYSBicm9rZW4gaW1w bGVtZW50YXRpb24uICBBZGRpbmcgdGhlICJJRiIgaXMgd3JvbmcuDQo+ID4NCj4gPiBXZSd2ZSBm aXhlZCB0aGUgaXNzdWVzIHJhaXNlZCBieSBSRkM1MDQ4IGJ5IGluY29ycG9yYXRpbmcgaXQgaW50 byB0aGUgbmV3DQo+IFJGQy4NCj4gPg0KPiA+ICAgICAgIEZyZWQNCj4gPg0KPiA+IC0tLS0tT3Jp Z2luYWwgTWVzc2FnZS0tLS0tDQo+ID4gRnJvbTogQmxhY2ssIERhdmlkIFttYWlsdG86ZGF2aWQu YmxhY2tAZW1jLmNvbV0NCj4gPiBTZW50OiBGcmlkYXksIFNlcHRlbWJlciAyOCwgMjAxMiA5OjUx IEFNDQo+ID4gVG86IEtuaWdodCwgRnJlZGVyaWNrDQo+ID4gQ2M6IHN0b3JtQGlldGYub3JnDQo+ ID4gU3ViamVjdDogUkU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJz aW9uIDYNCj4gPg0KPiA+IEZyZWQsDQo+ID4NCj4gPiA+IElmIHN0YXRlIGlzIHJldGFpbmVkIChp dCdzIGFuIG9sZCAiSSIgcmVjb25uZWN0aW5nKSwgdGhlbiB3aGljaCBVQQ0KPiA+ID4gaXMgZGVm aW5lZCBieSBTQU0tMy4NCj4gPiA+DQo+ID4gPiBJZiBzdGF0ZSBpcyBub3QgcmV0YWluZWQgKGl0 J3MgYSBuZXcgIkkiKSwgdGhlbiB0aGUgVUEgaXMgZGVmaW5lZCBieSBTQU0tDQo+IDMuDQo+ID4g Pg0KPiA+ID4gU2luY2UgYm90aCBvbGQgIkkicyBhbmQgbmV3ICJJInMgcmVzdWx0IGluIHRoZSAi SUYiIGNvbmRpdGlvbiBiZWluZw0KPiA+ID4gdHJ1ZSwgd2hhdCBpcyB0aGUgY29uZGl0aW9uIHRo YXQgd291bGQgbWFrZSB0aGUgIklGIiBmYWxzZT8NCj4gPg0KPiA+IFlvdSBhbG1vc3QgYW5zd2Vy ZWQgeW91ciBvd24gcXVlc3Rpb24gOy0pLg0KPiA+DQo+ID4gVGhlIGFuc3dlciBpcyAuLi4uIChk cnVtIHJvbGwpIC4uLiBhIHRhcmdldCBpbXBsZW1lbnRhdGlvbiBiYXNlZCBvbg0KPiA+IFJGQyAz NzIwIFNBTS0yLCBhbHRob3VnaCBJIHdvdWxkIGhvcGUgdGhhdCBzdWNoIGltcGxlbWVudGF0aW9u cyBhdA0KPiA+IGxlYXN0IHBpY2tlZCB1cCBSRkMgNTA0OCdzIGNoYW5nZXMuDQo+ID4NCj4gPiBU aGUgSV9UIE5leHVzIExvc3MgY29uY2VwdCBvbiB3aGljaCB0aGlzIFVBIGRpc2N1c3Npb24gaXMg YmFzZWQgaXMNCj4gPiBwYXJ0IG9mIHRoZSBTQ1NJIGV2ZW50cyBhbmQgZXZlbnQgbm90aWZpY2F0 aW9ucyBtb2RlbCB0aGF0IHdhcw0KPiA+IGludHJvZHVjZWQgaW4gU0FNLTMsIGJ1dCBuZWl0aGVy IFJGQyAzNzIwIG5vciBSRkMgNTA0OCByZWZlcmVuY2VzIFNBTS0zLg0KPiA+DQo+ID4gVGhlcmUg d2VyZSBkZWZpbml0ZWx5IGltcGxlbWVudGF0aW9ucyBvZiBpU0NTSSBiYXNlZCBvbiBTQU0tMiAt IEkNCj4gPiBhZ3JlZSB3aXRoIEp1bGlhbiB0aGF0IHdlIHNob3VsZCAqbm90KiBiZSBpbnRyb2R1 Y2luZyBhIFVBIHJlcXVpcmVtZW50DQo+ID4gaW4gdGhpcyBkcmFmdCB0aGF0IG1pZ2h0IGJyZWFr IHN1Y2ggaW1wbGVtZW50YXRpb25zLg0KPiA+DQo+ID4gVGhhbmtzLA0KPiA+IC0tRGF2aWQNCj4g Pg0KPiA+ID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPiA+IEZyb206IEtuaWdodCwg RnJlZGVyaWNrIFttYWlsdG86RnJlZGVyaWNrLktuaWdodEBuZXRhcHAuY29tXQ0KPiA+ID4gU2Vu dDogRnJpZGF5LCBTZXB0ZW1iZXIgMjgsIDIwMTIgODozOCBBTQ0KPiA+ID4gVG86IEp1bGlhbiBT YXRyYW4NCj4gPiA+IENjOiBCbGFjaywgRGF2aWQ7IHN0b3JtQGlldGYub3JnDQo+ID4gPiBTdWJq ZWN0OiBSRTogW3N0b3JtXSBTUEMtMiByZXNlcnZlL3JlbGVhc2UgdGV4dCAtIHZlcnNpb24gNg0K PiA+ID4NCj4gPiA+IE9LLCB0aGF0J3Mgd2hhdCBJIHRob3VnaHQuDQo+ID4gPg0KPiA+ID4gSWYg c3RhdGUgaXMgcmV0YWluZWQgKGl0J3MgYW4gb2xkICJJIiByZWNvbm5lY3RpbmcpLCB0aGVuIHdo aWNoIFVBDQo+ID4gPiBpcyBkZWZpbmVkIGJ5IFNBTS0zLg0KPiA+ID4NCj4gPiA+IElmIHN0YXRl IGlzIG5vdCByZXRhaW5lZCAoaXQncyBhIG5ldyAiSSIpLCB0aGVuIHRoZSBVQSBpcyBkZWZpbmVk IGJ5IFNBTS0NCj4gMy4NCj4gPiA+DQo+ID4gPiBTaW5jZSBib3RoIG9sZCAiSSJzIGFuZCBuZXcg IkkicyByZXN1bHQgaW4gdGhlICJJRiIgY29uZGl0aW9uIGJlaW5nDQo+ID4gPiB0cnVlLCB3aGF0 IGlzIHRoZSBjb25kaXRpb24gdGhhdCB3b3VsZCBtYWtlIHRoZSAiSUYiIGZhbHNlPw0KPiA+ID4N Cj4gPiA+IFRoYXQgIklGIiBsYW5ndWFnZSBhbGxvd3MgYSB0YXJnZXQgdG8gbG9vc2Ugc3RhdGUg YW5kIG5vdCBkZWxpdmVyIGFueSBVQS4NCj4gPiA+DQo+ID4gPiAgICAgRnJlZA0KPiA+ID4NCj4g PiA+DQo+ID4gPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiA+ID4gRnJvbTogSnVsaWFu IFNhdHJhbiBbbWFpbHRvOmp1bGlhbkBzYXRyYW4ubmV0XQ0KPiA+ID4gU2VudDogRnJpZGF5LCBT ZXB0ZW1iZXIgMjgsIDIwMTIgMjowMyBBTQ0KPiA+ID4gVG86IEtuaWdodCwgRnJlZGVyaWNrDQo+ ID4gPiBDYzogQmxhY2ssIERhdmlkOyBzdG9ybUBpZXRmLm9yZw0KPiA+ID4gU3ViamVjdDogUmU6 IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYNCj4gPiA+DQo+ ID4gPiBUaGVyZSBpcyBubyBzdWNoIHRoaW5nLiBBbmQgdGhlIHJlYXNvbiB0aGUgdGltZW91dCBp cyB0aGVyZSBpcyB0bw0KPiA+ID4gbGltaXQgdGhlIGFtb3VudCBvZiBzdGF0ZSB0aGUgdGFyZ2V0 cyBoYXZlIHRvIGtlZXAuIEEgdGFyZ2V0IG1heQ0KPiA+ID4gY2hvb3NlIHRvIGFjY2VwdCB2ZXJ5 IGxhcmdlIHRpbWVvdXRzIGFuZCB0aGVuIGRvIHdoYXQgeW91IHdhbnRlZA0KPiA+ID4gZG9uZSBi dXQgYmV5b25kIHRoYXQgb25seSB0aGUgcG93ZXItdXAgdWEgaXMgcmVxdWlyZWQuDQo+ID4gPg0K PiA+ID4ganVsbw0KPiA+ID4NCj4gPiA+IFNlbnQgZnJvbSBteSBpUGFkDQo+ID4gPg0KPiA+ID4g T24gMjgg15HXodek15ggMjAxMiwgYXQgMDA6MjksICJLbmlnaHQsIEZyZWRlcmljayINCj4gPiA+ IDxGcmVkZXJpY2suS25pZ2h0QG5ldGFwcC5jb20+DQo+ID4gPiB3cm90ZToNCj4gPiA+DQo+ID4g PiA+IEkgd291bGQgdGhpbmsgaWYgdGhlIElTSUQgaXMgbm90IHJlY29nbml6ZWQgYXMgYW4gb2xk IG9uZSwgdGhlbiBpdA0KPiA+ID4gPiBtdXN0IGJlDQo+ID4gPiByZWNvZ25pemVkIGFzIGEgbmV3 IG9uZS4gIElmIGl0IGlzIGEgbmV3IG9uZSwgdGhlbiBpdCB3aWxsIGdldCBhIFVBOg0KPiA+ID4g MjkvMDAgcmVmbGVjdGluZyB0aGUgZmlyc3QgYWNjZXNzIGFmdGVyIHBvd2VyIG9uIGZyb20gdGhl ICJuZXciIGluaXRpYXRvci4NCj4gPiA+ID4NCj4gPiA+ID4gSXMgdGhlcmUgc29tZXRoaW5nIG90 aGVyIHRoYW4gbmV3IG9yIG9sZD8gIElzIHRoZXJlIHN1Y2ggYSB0aGluZw0KPiA+ID4gPiBhcyBh DQo+ID4gPiBtaWRkbGUtYWdlZCBpbml0aWF0b3I/DQo+ID4gPiA+DQo+ID4gPiA+ICAgIEZyZWQN Cj4gPiA+ID4NCj4gPiA+ID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPiA+ID4gRnJv bTogQmxhY2ssIERhdmlkIFttYWlsdG86ZGF2aWQuYmxhY2tAZW1jLmNvbV0NCj4gPiA+ID4gU2Vu dDogVGh1cnNkYXksIFNlcHRlbWJlciAyNywgMjAxMiAxMTo1NiBBTQ0KPiA+ID4gPiBUbzogS25p Z2h0LCBGcmVkZXJpY2sNCj4gPiA+ID4gQ2M6IHN0b3JtQGlldGYub3JnDQo+ID4gPiA+IFN1Ympl Y3Q6IFJFOiBbc3Rvcm1dIFNQQy0yIHJlc2VydmUvcmVsZWFzZSB0ZXh0IC0gdmVyc2lvbiA2DQo+ ID4gPiA+DQo+ID4gPiA+IEZyZWQsDQo+ID4gPiA+DQo+ID4gPiA+IFRoZSBwcm9ibGVtIGlzIG5v dCB0aGF0IHRoZSBkZXZpY2UgZG9lcyBub3QgcmFpc2UgYSBVQSwgaXQncyB0aGF0DQo+ID4gPiA+ IHRoZSBVQSBtYXkNCj4gPiA+IG5ldmVyIGJlIGRlbGl2ZXJlZC4NCj4gPiA+ID4NCj4gPiA+ID4g QSBzdW1tYXJ5IG9mIGhvdyB0aGlzIGNhbiBoYXBwZW4gaXMgdGhhdCB0aGUgbmV3IGNvbm5lY3Rp b24gd2l0aA0KPiA+ID4gPiB0aGUgc2FtZQ0KPiA+ID4gSVNJRCBtYXkgbm90IGJlIHJlY29nbml6 ZWQgYXMgcmVpbnN0YXRpbmcgb3IgY29udGludWluZyB0aGUgc2Vzc2lvbg0KPiA+ID4gYXQgdGhl IHRhcmdldCBpZiB0aGUgdGltZW91dHMgaGF2ZSBleHBpcmVkIC0gdGhlIFVBIGdldHMNCj4gPiA+ IGVzdGFibGlzaGVkLCBidXQgbm9ib2R5IHNob3dzIHVwIHRvIHBpY2sgaXQgdXAgaW4gdGltZSwg c28gdGhlIFVBIGlzDQo+ID4gPiBkaXNjYXJkZWQgYWxvbmcgd2l0aCB0aGUgcmVzdCBvZiB0aGUg c2Vzc2lvbiBzdGF0ZSBhbmQgdGhlIHRhcmdldA0KPiA+ID4gZmFpbHMgdG8gcmVjb2duaXplIHRo ZSBpbml0aWF0b3IncyBzdWJzZXF1ZW50IElTSUQgcmV1c2UgZXZlbiB0aG91Z2gNCj4gPiA+IHRo ZSBpbml0aWF0b3IgYmVsaWV2ZXMgdGhhdCBpdCdzIHJlLSBlc3RhYmxpc2hpbmcgdGhlIHNlc3Np b24uDQo+ID4gPiA+DQo+ID4gPiA+IFRoaXMgaXMgZnVydGhlciBjb21wbGljYXRlZCBieSB0aGUg VUEgdGhhdCBpbmRpY2F0ZXMgbmV4dXMgc3RhdGUNCj4gPiA+IHByZXNlcnZhdGlvbiBub3QgYmVp bmcgc3BlY2lmaWVkIGluIFNBTS0yLg0KPiA+ID4gPg0KPiA+ID4gPiBUaGFua3MsDQo+ID4gPiA+ IC0tRGF2aWQNCj4gPiA+ID4NCj4gPiA+ID4NCj4gPiA+ID4+IC0tLS0tT3JpZ2luYWwgTWVzc2Fn ZS0tLS0tDQo+ID4gPiA+PiBGcm9tOiBLbmlnaHQsIEZyZWRlcmljayBbbWFpbHRvOkZyZWRlcmlj ay5LbmlnaHRAbmV0YXBwLmNvbV0NCj4gPiA+ID4+IFNlbnQ6IFRodXJzZGF5LCBTZXB0ZW1iZXIg MjcsIDIwMTIgMTE6MTEgQU0NCj4gPiA+ID4+IFRvOiBCbGFjaywgRGF2aWQ7IEp1bGlhbiBTYXRy YW4NCj4gPiA+ID4+IENjOiBzdG9ybUBpZXRmLm9yZw0KPiA+ID4gPj4gU3ViamVjdDogUkU6IFtz dG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYNCj4gPiA+ID4+DQo+ ID4gPiA+PiBJIGFmcmFpZCBJIGhhdmUgdG8gY29udGVzdCBhIGxpdHRsZSBvbiB0aGlzIC0gaWYg SSB1bmRlcnN0YW5kDQo+ID4gPiA+PiB0aGlzIGRpc2N1c3Npb24gY29ycmVjdGx5Lg0KPiA+ID4g Pj4NCj4gPiA+ID4+IElmIGEgU0NTSSBkZXZpY2UgbG9zZXMgYW55IHN0YXRlIGFuZCBkb2VzIG5v dCByYWlzZSBhIFVOSVQNCj4gPiA+ID4+IEFUVEVOVElPTiwgdGhlbiBpdCBpcyBCUk9LRU4uICBJ dCBpcyBub3QgYSBtYXR0ZXIgb2YgYW55dGhpbmcgaW4NCj4gPiA+ID4+IGlTQ1NJICgzNzIwKTsg SSB0aGluayBTQU0gYW5kIFNQQyBhcmUgcHJldHR5IGNsZWFyIG9uIHRoaXMuICBJZg0KPiA+ID4g Pj4gdGhlcmUgaXMgbm8gVU5JVCBBVFRFTlRJT04sIHRoZW4gc3RhdGUgU0hBTEwgYmUgcHJlc2Vy dmVkOw0KPiA+ID4gPj4gb3RoZXJ3aXNlLCBhbGwgc29ydHMgb2Ygc3R1ZmYgd2lsbCBuZXZlciB3 b3JrIChtb2RlIHBhZ2UNCj4gPiA+ID4+IHNldHRpbmdzLCBSRVNFUlZFIGNvbW1hbmRzLCBhbGwg c29ydHMgb2YNCj4gPiA+IHN0dWZmKS4NCj4gPiA+ID4+DQo+ID4gPiA+PiBJcyBzb21lb25lIHJl YWxseSBzdWdnZXN0aW5nIHRoYXQgc29tZSBkZXZpY2VzIHdpbGwgZHJvcCBzdGF0ZQ0KPiA+ID4g Pj4gYW5kIE5PVCB0ZWxsIHRoZSBob3N0LCBhbmQgc29tZWhvdyB0aGF0IGl0IGlzIE9LIHRvIGRv IHRoYXQ/DQo+ID4gPiA+Pg0KPiA+ID4gPj4gVGhlIHN1Z2dlc3QgcmV3b3JkaW5nIHNheXMgdGhp cyBpcyBwb3NzaWJsZS4NCj4gPiA+ID4+DQo+ID4gPiA+PiAgICBGcmVkDQo+ID4gPiA+Pg0KPiA+ ID4gPj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPiA+ID4+IEZyb206IHN0b3JtLWJv dW5jZXNAaWV0Zi5vcmcgW21haWx0bzpzdG9ybS1ib3VuY2VzQGlldGYub3JnXSBPbg0KPiA+ID4g Pj4gQmVoYWxmIE9mIEJsYWNrLCBEYXZpZA0KPiA+ID4gPj4gU2VudDogVGh1cnNkYXksIFNlcHRl bWJlciAyNywgMjAxMiAxMDo0NyBBTQ0KPiA+ID4gPj4gVG86IEp1bGlhbiBTYXRyYW4NCj4gPiA+ ID4+IENjOiBzdG9ybUBpZXRmLm9yZw0KPiA+ID4gPj4gU3ViamVjdDogUmU6IFtzdG9ybV0gU1BD LTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9uIDYNCj4gPiA+ID4+DQo+ID4gPiA+PiBK dWxpYW4sDQo+ID4gPiA+Pg0KPiA+ID4gPj4+IFRoZSB0ZXh0IGluIDQuNC4zIHN0YXRlcyB0aGUg b2J2aW91cyAodGhlcmUgaXMgbm8gb3RoZXIgc3BlY2VkDQo+ID4gPiA+Pj4gbWV0aG9kIGZvciBj b250aW51YXRpb24gb3IgcmVpbnN0YXRlbWVudCkuIE1vcmVvdmVyIGlmIHRoZQ0KPiA+ID4gPj4+ IHNlc3Npb24gaXMgcmVpbnN0YXRlZCBhZnRlciB0aW1lMndhaXQgeW91IGNyZWF0ZSBhIG5lZWQg Zm9yIGEgVUENCj4gPiA+ID4+PiAob3IgYXQgbGVhc3Qgc3VnZ2VzdCBvbmUpIGFuZCB0aGF0IHdh cyBub3QgdGhlcmUuDQo+ID4gPiA+Pg0KPiA+ID4gPj4gVGhlIFVBcyBhcmUgY291cnRlc3kgb2Yg U0FNLTMgKGFuZCBTQU0tNCkuICBUbyBlbmNvbXBhc3Mgc3lzdGVtcw0KPiA+ID4gPj4gdGhhdCBt YXkgbm90IGRlbGl2ZXIgdGhlc2UgVUFzIGFuZCBzaXR1YXRpb25zIGluIHdoaWNoIHRoZXkgZG9u J3QNCj4gPiA+ID4+IG9jY3VyIChlLmcuLCB0aGUgdGFyZ2V0IGRvZXNuJ3QgcmVjb2duaXplIHdo YXQgaGFwcGVuZWQgYXMgbmV4dXMNCj4gPiA+ID4+IHJlZXN0YWJsaXNobWVudCwgaW5kZXBlbmRl bnQgb2Ygd2hhdCB0aGUgaW5pdGlhdG9yIHRoaW5rcyBpdCdzDQo+ID4gPiA+PiBkb2luZykgSQ0K PiA+ID4gdGhpbmsgdGhlIHRleHQgc2hvdWxkIGJlIHJlcGhyYXNlZCBhczoNCj4gPiA+ID4+DQo+ ID4gPiA+PiAgICAgSWYgYSBVbml0IEF0dGVudGlvbiBjb25kaXRpb24gcmVzdWx0cyBmcm9tIHNl c3Npb24gcmVlc3RhYmxpc2htZW50LA0KPiA+ID4gPj4gICAgIHRoZSBzcGVjaWZpYyBVbml0IEF0 dGVudGlvbiBjb25kaXRpb24gaW5kaWNhdGVzIHdoZXRoZXIgbmV4dXMgc3RhdGUNCj4gPiA+ID4+ ICAgICB3YXMgcHJlc2VydmVkLCBzZWUgU2VjdGlvbiA2LjMuNCBvZiBbU0FNNF0uDQo+ID4gPiA+ Pg0KPiA+ID4gPj4+IEp1c3Qgc3RhdGluZyB0aGF0IGNyZWF0aW9uL2NvbnRpbnVhdGlvbiBvZiBh IHNlc3Npb24gZG9uZQ0KPiA+ID4gPj4+IGFjY29yZGluZyA2LnggaW5jbHVkZXMgcHJlc2VydmF0 aW9uIG9mIHJlc2VydmUvcmVsZWFzZSBzdGF0ZQ0KPiA+ID4gPj4+IC10aGF0IHdhcyBhbHJlYWR5 IHRoZXJlIGltcGxpZWQgYnkgY29uZm9ybWFuY2UgdG8gc3BjMiBhbmQgdGhlDQo+ID4gPiA+Pj4g cmVsYXRpb24gYmV0d2VlbiBpdCBuZXh1cyBhbmQgc2Vzc2lvbi4NCj4gPiA+ID4+DQo+ID4gPiA+ PiBVbmZvcnR1bmF0ZWx5LCBTUEMtMiBpcyBzdWZmaWNpZW50bHkgaW1wcmVjaXNlIG9uIHRoaXMg c3ViamVjdA0KPiA+ID4gPj4gdGhhdCBzdWNoIGFuIGltcGxpY2F0aW9uIGlzIGF0IGJlc3QgYSAi cHJlZmVycmVkIGltcGxlbWVudGF0aW9uDQo+IGFwcHJvYWNoIi4NCj4gPiA+ID4+IEl0J3Mgbm90 IGEgcmVxdWlyZW1lbnQgdGhhdCBjYW4gYmUgc2FmZWx5IHN0YXRlZCBpbiBhbnkgc3Ryb25nZXIN Cj4gPiA+ID4+IGZhc2hpb24gd2l0aG91dCB0aGUgcmlzayBvZiByZW5kZXJpbmcgZXhpc3Rpbmcg aW1wbGVtZW50YXRpb25zDQo+ID4gPiA+PiBub24tDQo+ID4gPiBjb21wbGlhbnQuDQo+ID4gPiA+ Pg0KPiA+ID4gPj4+IFRoZSBuZXcgdGV4dCB5b3Ugc3VnZ2VzdCB0byA0LjQgaXMgbWVyZWx5IGEg YmVzdCBwcmFjdGljZXMNCj4gPiA+ID4+PiByZWNvbW1lbmRhdGlvbnMgYW5kIHdpbGwgbmVlZCBh ZGRpdGlvbnMgb25jZSB0aGUgbmV3ICJsb2NraW5nIg0KPiA+ID4gPj4+IHRocm91Z2ggY29tcGFy ZS1hbmQtc3dhcCBnZXQgaW4gd2lkZXNwcmVhZCB1c2UuDQo+ID4gPiA+Pg0KPiA+ID4gPj4gSXQn cyBkZWZpbml0ZWx5IGJlc3QgcHJhY3RpY2VzIHJlY29tbWVuZGF0aW9ucyAtIG5vIGFyZ3VtZW50 IHRoZXJlLg0KPiA+ID4gPj4NCj4gPiA+ID4+IEZvciBjb250ZXh0LCB3aGF0IEp1bGlhbiBpcyBy ZWZlcnJpbmcgdG8gaXMgdXNlIG9mIHRoZSBDT01QQVJFDQo+ID4gPiA+PiBBTkQgV1JJVEUgY29t bWFuZCB0aGF0IHBlcmZvcm1zIGFuIGF0b21pYyBvcGVyYXRpb24gYXQgdGhlIHRhcmdldA0KPiA+ ID4gPj4gaW5zdGVhZCBvZiB1c2luZyBSRVNFUlZFIGFuZCBSRUxFQVNFIHRvIGNyZWF0ZSBhIGNy aXRpY2FsIHNlY3Rpb24NCj4gPiA+ID4+IGF0IHRoZSBpbml0aWF0b3IgYmFzZWQgb24gcmVhZCBh bmQgd3JpdGUgY29tbWFuZHMuDQo+ID4gPiA+Pg0KPiA+ID4gPj4gQ09NUEFSRSBBTkQgV1JJVEUg aXMgaW4gdXNlIC0gYSBzZW50ZW5jZSBtZW50aW9uaW5nIHRoYXQgY29tbWFuZA0KPiA+ID4gPj4g Y291bGQgYmUgYWRkZWQsIGFsdGhvdWdoIHRoaXMgYmVzdCBwcmFjdGljZXMgdGV4dCBpcyBhaW1l ZA0KPiA+ID4gPj4gcHJpbWFyaWx5IGF0IHVzZXMgb2YgcmVzZXJ2ZS9yZWxlYXNlIGZvciBsb25n LXRlcm0gcmVzZXJ2YXRpb25zLg0KPiA+ID4gPj4gSSB0aGluayB0aGVyZSdzIGFuIGltcGxpY2l0 ICJ3aGVuIGEgcmVzZXJ2YXRpb24gaXMgdGhlIGRlc2lyZWQNCj4gYmVoYXZpb3IiDQo+ID4gPiA+ PiBpbXBsaWNhdGlvbiB0byB0aGUgInBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIFNIT1VMRCBiZSB1 c2VkIGluc3RlYWQiDQo+ID4gPiA+PiB0ZXh0IHRoYXQgSSBkb24ndCB0aGluayBuZWVkcyB0byBi ZSBtYWRlIGV4cGxpY2l0LiAgT1RPSCwgdGhhdCAiU0hPVUxEIg0KPiA+ID4gPj4gY291bGQgYmUg Y2hhbmdlZCB0byBhICJzaG91bGQiLCBvciBpdCBtaWdodCBiZSBldmVuIGJldHRlciB0bw0KPiA+ ID4gPj4gcmV3cml0ZSB0aGF0IHRleHQNCj4gPiA+ID4+IGFzOg0KPiA+ID4gPj4NCj4gPiA+ID4+ ICAgIHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIGFyZSBzdWdnZXN0ZWQgYXMgYW4gYWx0ZXJuYXRp dmUsIHNlZSBBbm5leCBCDQo+ID4gPiA+PiAgICBvZiBbU1BDNF0uDQo+ID4gPiA+Pg0KPiA+ID4g Pj4gSW4gY29udHJhc3QsICJTSE9VTEQgTk9UIGJlIHVzZWQiIGlzIGNsZWFybHkganVzdGlmaWVk IGZvcg0KPiA+ID4gPj4gcmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9ucyBieSBbU1BDM10uDQo+ ID4gPiA+Pg0KPiA+ID4gPj4gVGhhbmtzLA0KPiA+ID4gPj4gLS1EYXZpZA0KPiA+ID4gPj4NCj4g PiA+ID4+DQo+ID4gPiA+Pj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPiA+ID4+PiBG cm9tOiBKdWxpYW4gU2F0cmFuIFttYWlsdG86anVsaWFuQHNhdHJhbi5uZXRdDQo+ID4gPiA+Pj4g U2VudDogV2VkbmVzZGF5LCBTZXB0ZW1iZXIgMjYsIDIwMTIgMTA6MjUgUE0NCj4gPiA+ID4+PiBU bzogQmxhY2ssIERhdmlkDQo+ID4gPiA+Pj4gQ2M6IHN0b3JtQGlldGYub3JnDQo+ID4gPiA+Pj4g U3ViamVjdDogUmU6IFtzdG9ybV0gU1BDLTIgcmVzZXJ2ZS9yZWxlYXNlIHRleHQgLSB2ZXJzaW9u IDYNCj4gPiA+ID4+Pg0KPiA+ID4gPj4+IERhdmlkLA0KPiA+ID4gPj4+DQo+ID4gPiA+Pj4gVGhl IHRleHQgaW4gNC40LjMgc3RhdGVzIHRoZSBvYnZpb3VzICh0aGVyZSBpcyBubyBvdGhlciBzcGVj ZWQNCj4gPiA+ID4+PiBtZXRob2QgZm9yIGNvbnRpbnVhdGlvbiBvciByZWluc3RhdGVtZW50KS4g TW9yZW92ZXIgaWYgdGhlIHRoZQ0KPiA+ID4gPj4+IHNlc3Npb24gaXMgcmVpbnN0YXRlZCBhZnRl ciB0aW1lMndhaXQgeW91IGNyZWF0ZSBhIG5lZWQgZm9yIGEgVUENCj4gPiA+ID4+PiAob3IgYXQg bGVhc3Qgc3VnZ2VzdCBvbmUpIGFuZCB0aGF0IHdhcyBub3QgdGhlcmUuIEp1c3Qgc3RhdGluZw0K PiA+ID4gPj4+IHRoYXQgY3JlYXRpb24vY29udGludWF0aW9uIG9mIGEgc2Vzc2lvbiBkb25lIGFj Y29yZGluZyA2LngNCj4gPiA+ID4+PiBpbmNsdWRlcyBwcmVzZXJ2YXRpb24gb2YgcmVzZXJ2ZS9y ZWxlYXNlIHN0YXRlIC10aGF0IHdhcyBhbHJlYWR5DQo+ID4gPiA+Pj4gdGhlcmUgaW1wbGllZCBi eSBjb25mb3JtYW5jZSB0byBzcGMyIGFuZCB0aGUgcmVsYXRpb24gYmV0d2VlbiBpdA0KPiA+ID4g Pj4+IG5leHVzIGFuZA0KPiA+ID4gc2Vzc2lvbi4NCj4gPiA+ID4+Pg0KPiA+ID4gPj4+IFRoZSBu ZXcgdGV4dCB5b3Ugc3VnZ2VzdCB0byA0LjQgaXMgbWVyZWx5IGEgYmVzdCBwcmFjdGljZXMNCj4g PiA+ID4+PiByZWNvbWVuZGF0aW9ucyBhbmQgd2lsbCBuZWVkIGFkZGl0aW9ucyBvbmNlIHRoZSBu ZXcgImxvY2tpbmciDQo+ID4gPiA+Pj4gdGhyb3VnaCBjb21wYXJlLWFuZC1zd2FwIGdldCBpbiB3 aWRlc3ByZWFkIHVzZS4NCj4gPiA+ID4+PiBJZiBpbnN0ZWFkIHdlIHJlZnJhaW4gZnJvbSBzdGF0 aW5nIGFueXRoaW5nIHNwZWNpZmljIGFuZCBzdGF0ZQ0KPiA+ID4gPj4+IHRoYXQgdGFyZ2V0IHNo b3VsZCBtYWludGFpbiB3aGF0ZXZlciBzdGF0ZSBpcyByZXF1aXJlZCBieSBTUEMNCj4gPiA+ID4+ PiB3aGlsZSBhIHNlc3Npb24gaXMgY29udGludWVkIGFuZCBtYXkgZGlzY2FyZCBzdGF0ZSB3aGVu IGENCj4gPiA+ID4+PiBzZXNzaW9uIGlzIHRlcm1pbmF0ZWQgYW5kIG1heSBpbmRpY2F0ZSB0aGF0 IGl0IGRpc2NhcmRlZCBzdGF0ZQ0KPiA+ID4gPj4+IHRocm91Z2ggYSB1YSB5b3UgYXJlIG9uIHNh ZmVyIGdyb3VuZCBhbmQgZG8gbm90IGNyZWF0ZSBuZXcNCj4gcmVxdWlyZW1lbnQuDQo+ID4gPiA+ Pj4NCj4gPiA+ID4+PiBTZW50IGZyb20gbXkgaVBhZA0KPiA+ID4gPj4+DQo+ID4gPiA+Pj4gT24g Mjcg15HXodek15ggMjAxMiwgYXQgMDA6MjUsICJCbGFjaywgRGF2aWQiIDxkYXZpZC5ibGFja0Bl bWMuY29tPiB3cm90ZToNCj4gPiA+ID4+Pg0KPiA+ID4gPj4+PiBIZXJlJ3Mgd2hlcmUgSSB0aGlu ayB3ZSd2ZSBhcnJpdmVkIChJJ3ZlIG1hZGUgYSBmZXcgbWlub3INCj4gPiA+ID4+Pj4gZWRpdG9y aWFsIHR3ZWFrcyB0byB3aGF0J3MgYmVlbiBkaXNjdXNzZWQpLiAgRGVzcGl0ZSBhbGwgdGhlDQo+ ID4gPiA+Pj4+IGRldGFpbGVkIGRpc2N1c3Npb24sIEkgdGhpbmsgdGhpcyB2ZXJzaW9uIDYgY29t ZXMgY2xvc2UgdG8NCj4gPiA+ID4+Pj4gSnVsaWFuJ3Mgc3VnZ2VzdGlvbg0KPiA+ID4gPj4gdGhh dDoNCj4gPiA+ID4+Pj4NCj4gPiA+ID4+Pj4+IGkgd291bGQgbWVudGlvbiBvbmx5IHRoYXQgdGhl IHN0YXRlIHRoYXQgbXVzdCBiZSBtYWludGFpbmVkDQo+ID4gPiA+Pj4+PiB3aGlsZSB0aGUgc2Vz c2lvbiBpcyBtYWludGFpbmVkIGluY2x1ZGVzIHRoZSByZXNlcnZlL3JlbGVhc2UNCj4gPiA+ID4+ Pj4+IGFuZCBsZWF2ZSBpdCB0bw0KPiA+ID4gPj4+IHRoYXQuDQo+ID4gPiA+Pj4+DQo+ID4gPiA+ Pj4+IEFsdGhvdWdoIHdlIGNhbid0IGFjdHVhbGx5IHNheSAibXVzdCIgYXMgdGhhdCB3b3VsZCBi ZSBhIG5ldw0KPiA+ID4gPj4+PiByZXF1aXJlbWVudCwgYW5kIGltcGxlbWVudGF0aW9ucyBkaWZm ZXIgaW4gd2hhdCB0aGV5IGRvIGFib3V0DQo+ID4gPiA+Pj4+IHJlc2VydmUvcmVsZWFzZSByZXNl cnZhdGlvbiBzdGF0ZS4gIFRoZSBjdXJyZW50IGxhbmd1YWdlDQo+ID4gPiA+Pj4+IGNoYXJhY3Rl cml6ZXMgcmV0ZW50aW9uIG9mIHRoYXQgc3RhdGUgYXMgdGhlICJwcmVmZXJyZWQNCj4gPiA+ID4+ Pj4gaW1wbGVtZW50YXRpb24NCj4gPiA+ID4+IGFwcHJvYWNoIi4NCj4gPiA+ID4+Pj4NCj4gPiA+ ID4+Pj4gQmV5b25kIHRoYXQsIEkgdGhpbmsgdGhlIHR3byBhZGRpdGlvbmFsIGNhdmVhdHMgYWJv dXQNCj4gPiA+ID4+Pj4gcmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9ucyAocG9zc2libGUgbmVl ZCBmb3IgYSByZXNldCwNCj4gPiA+ID4+Pj4gcG90ZW50aWFsbHkgdW5leHBlY3RlZCBpbnRlcmFj dGlvbiB3aXRoIHBlcnNpc3RlbnQNCj4gPiA+ID4+Pj4gcmVzZXJ2YXRpb25zLCBzZWUgYmVsb3cp IGFyZSB1c2VmdWwNCj4gPiA+ID4+IHRvIGluY2x1ZGUuDQo+ID4gPiA+Pj4+DQo+ID4gPiA+Pj4+ IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ID4gPiA+Pj4+DQo+ ID4gPiA+Pj4+IFsxXSBGb3IgY2xhcml0eSwgdGhlIGZvbGxvd2luZyBjaGFuZ2Ugc2hvdWxkIGJl IG1hZGUgaW4NCj4gPiA+ID4+Pj4gNC40LjMuMToNCj4gPiA+ID4+Pj4NCj4gPiA+ID4+Pj4gT0xE DQo+ID4gPiA+Pj4+IFRoYXQgaXMsIGl0IHNob3VsZCBwZXJmb3JtIHNlc3Npb24gcmVjb3Zlcnkg YXMgIGRlc2NyaWJlZCBpbg0KPiA+ID4gPj4+PiBDaGFwdGVyIDYuDQo+ID4gPiA+Pj4+IE5FVw0K PiA+ID4gPj4+PiBUaGF0IGlzLCBpdCBzaG91bGQgcmVpbnN0YXRlIHRoZSBzZXNzaW9uIHZpYSBp U0NTSSBzZXNzaW9uDQo+ID4gPiA+Pj4+IHJlaW5zdGF0ZW1lbnQgKFNlY3Rpb24gNi4zLjUpIG9y IGNvbnRpbnVlIHZpYSBzZXNzaW9uDQo+ID4gPiA+Pj4+IGNvbnRpbnVhdGlvbiAoU2VjdGlvbiA2 LjMuNikuDQo+ID4gPiA+Pj4+IEVORA0KPiA+ID4gPj4+Pg0KPiA+ID4gPj4+PiBbMl0gQWRkIHRo ZSBmb2xsb3dpbmcgc2VudGVuY2UgdG8gdGhlIGVuZCBvZiA0LjQuMy4xOg0KPiA+ID4gPj4+Pg0K PiA+ID4gPj4+PiAgICBUaGUgc3BlY2lmaWMgVW5pdCBBdHRlbnRpb24gY29uZGl0aW9uIHRoYXQg cmVzdWx0cyBmcm9tIHNlc3Npb24NCj4gPiA+ID4+Pj4gICAgcmVlc3RhYmxpc2htZW50IGluZGlj YXRlcyB3aGV0aGVyIG5leHVzIHN0YXRlIHdhcyBwcmVzZXJ2ZWQsDQo+ID4gPiA+Pj4+ICAgIHNl ZSBTZWN0aW9uIDYuMy40IG9mIFtTQU00XS4NCj4gPiA+ID4+Pj4NCj4gPiA+ID4+Pj4gWzNdIE5F VyBURVhUOg0KPiA+ID4gPj4+Pg0KPiA+ID4gPj4+PiA0LjQuMy4yLiBSZXNlcnZhdGlvbnMNCj4g PiA+ID4+Pj4NCj4gPiA+ID4+Pj4gVGhlcmUgYXJlIHR3byByZXNlcnZhdGlvbiBtYW5hZ2VtZW50 IG1ldGhvZHMgZGVmaW5lZCBpbiB0aGUNCj4gPiA+ID4+Pj4gU0NTSSBzdGFuZGFyZHMsIHJlc2Vy dmUvcmVsZWFzZSByZXNlcnZhdGlvbnMsIGJhc2VkIG9uIHRoZQ0KPiA+ID4gPj4+PiBSRVNFUlZF IGFuZCBSRUxFQVNFIGNvbW1hbmRzIFtTUEMyXSwgYW5kIHBlcnNpc3RlbnQNCj4gPiA+ID4+Pj4g cmVzZXJ2YXRpb25zLCBiYXNlZCBvbiB0aGUgUEVSU0lTVEVOVCBSRVNFUlZFIElOIGFuZCBQRVJT SVNURU5UDQo+IFJFU0VSVkUgT1VUIGNvbW1hbmRzIFtTUEMzXS4NCj4gPiA+ID4+Pj4gUmVzZXJ2 ZS9yZWxlYXNlIHJlc2VydmF0aW9ucyBhcmUgb2Jzb2xldGUgW1NQQzNdIGFuZCBTSE9VTEQgTk9U DQo+ID4gPiA+Pj4+IGJlIHVzZWQ7IHBlcnNpc3RlbnQgcmVzZXJ2YXRpb25zIFNIT1VMRCBiZSB1 c2VkIGluc3RlYWQuDQo+ID4gPiA+Pj4+DQo+ID4gPiA+Pj4+IFN0YXRlIGZvciBwZXJzaXN0ZW50 IHJlc2VydmF0aW9ucyBpcyByZXF1aXJlZCB0byBwZXJzaXN0DQo+ID4gPiA+Pj4+IHRocm91Z2gg Y2hhbmdlcyBhbmQgZmFpbHVyZXMgYXQgdGhlIGlTQ1NJIGxheWVyIHRoYXQgcmVzdWx0IGluDQo+ ID4gPiA+Pj4+IElfVCBOZXh1cyBmYWlsdXJlcywgc2VlIFtTUEMzXSBmb3IgZGV0YWlscyBhbmQg c3BlY2lmaWMgcmVxdWlyZW1lbnRzLg0KPiA+ID4gPj4+Pg0KPiA+ID4gPj4+PiBJbiBjb250cmFz dCwgW1NQQzJdIGRvZXMgbm90IHNwZWNpZnkgZGV0YWlsZWQgcGVyc2lzdGVuY2UNCj4gPiA+ID4+ Pj4gcmVxdWlyZW1lbnRzIGZvciByZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb24gc3RhdGUgYWZ0 ZXIgYW4gSV9UDQo+ID4gPiA+Pj4+IE5leHVzIGZhaWx1cmUuICBOb25ldGhlbGVzcywgd2hlbiBy ZXNlcnZlL3JlbGVhc2UgcmVzZXJ2YXRpb25zDQo+ID4gPiA+Pj4+IGFyZSBzdXBwb3J0ZWQgYnkg YW4gaVNDU0kgdGFyZ2V0LCB0aGUgcHJlZmVycmVkIGltcGxlbWVudGF0aW9uDQo+ID4gPiA+Pj4+ IGFwcHJvYWNoIGlzIHRvIHByZXNlcnZlIHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbiBzdGF0 ZSBmb3INCj4gPiA+ID4+Pj4gaVNDU0kgc2Vzc2lvbiByZWluc3RhdGVtZW50IChzZWUgU2VjdGlv biA2LjMuNSkgb3Igc2Vzc2lvbg0KPiA+ID4gPj4+PiBjb250aW51YXRpb24gIChzZWUgU2VjdGlv biA2LjMuNikuDQo+ID4gPiA+Pj4+DQo+ID4gPiA+Pj4+IFR3byBhZGRpdGlvbmFsIGNhdmVhdHMg YXBwbHkgdG8gcmVzZXJ2ZS9yZWxlYXNlIHJlc2VydmF0aW9uczoNCj4gPiA+ID4+Pj4NCj4gPiA+ ID4+Pj4gLSBXaGVuIGNvbm5lY3Rpb24gZmFpbHVyZSBjYXVzZXMgdGhlIGlTQ1NJIHNlc3Npb24g dG8gZmFpbCwgYW5kDQo+ID4gPiA+Pj4+ICAgICB0aGUgc2Vzc2lvbiBpcyBub3QgcmVpbnN0YXRl ZCBvciBjb250aW51ZWQsIHRhcmdldCByZXRlbnRpb24NCj4gPiA+ID4+Pj4gICAgIG9mIHRoYXQg c2Vzc2lvbidzIHJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbiBzdGF0ZSBtYXkNCj4gPiA+ID4+ Pj4gICAgIHJlcXVpcmUgYW4gaW5pdGlhdG9yIHRvIGlzc3VlIGEgcmVzZXQgKGUuZy4sIExPR0lD QUwgVU5JVCBSRVNFVCwNCj4gPiA+ID4+Pj4gICAgIHNlZSBzZWN0aW9uIDExLjUpIGluIG9yZGVy IHRvIHJlbW92ZSB0aGF0IHJlc2VydmF0aW9uIHN0YXRlLg0KPiA+ID4gPj4+Pg0KPiA+ID4gPj4+ PiAtIFJlc2VydmUvcmVsZWFzZSByZXNlcnZhdGlvbnMgbWF5IG5vdCBiZWhhdmUgYXMgZXhwZWN0 ZWQgd2hlbg0KPiA+ID4gPj4+PiAgICAgcGVyc2lzdGVudCByZXNlcnZhdGlvbnMgYXJlIGFsc28g dXNlZCBvbiB0aGUgc2FtZSBsb2dpY2FsIHVuaXQ7DQo+ID4gPiA+Pj4+ICAgICBzZWUgdGhlIGRp c2N1c3Npb24gb2YgIkV4Y2VwdGlvbnMgdG8gU1BDLTIgUkVTRVJWRSBhbmQgUkVMRUFTRQ0KPiA+ ID4gPj4+PiAgICAgYmVoYXZpb3IiIGluIFtTUEM0XS4NCj4gPiA+ID4+Pj4NCj4gPiA+ID4+Pj4g VGhhbmtzLA0KPiA+ID4gPj4+PiAtLURhdmlkDQo+ID4gPiA+Pj4+IC0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gPiA+ID4+Pj4gRGF2aWQgTC4g QmxhY2ssIERpc3Rpbmd1aXNoZWQgRW5naW5lZXIgRU1DIENvcnBvcmF0aW9uLCAxNzYNCj4gPiA+ ID4+Pj4gU291dGggU3QuLCBIb3BraW50b24sIE1BICAwMTc0OA0KPiA+ID4gPj4+PiArMSAoNTA4 KSAyOTMtNzk1MyAgICAgICAgICAgICBGQVg6ICsxICg1MDgpIDI5My03Nzg2DQo+ID4gPiA+Pj4+ IGRhdmlkLmJsYWNrQGVtYy5jb20gICAgICAgIE1vYmlsZTogKzEgKDk3OCkgMzk0LTc3NTQNCj4g PiA+ID4+Pj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLQ0KPiA+ID4gPj4+Pg0KPiA+ID4gPj4+Pg0KPiA+ID4gPj4+PiBfX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiA+ID4gPj4+PiBzdG9ybSBtYWlsaW5n IGxpc3QNCj4gPiA+ID4+Pj4gc3Rvcm1AaWV0Zi5vcmcNCj4gPiA+ID4+Pj4gaHR0cHM6Ly93d3cu aWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9zdG9ybQ0KPiA+ID4gPj4NCj4gPiA+ID4+IF9fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+ID4gPiA+PiBzdG9y bSBtYWlsaW5nIGxpc3QNCj4gPiA+ID4+IHN0b3JtQGlldGYub3JnDQo+ID4gPiA+PiBodHRwczov L3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3N0b3JtDQo+ID4gPiA+IF9fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+ID4gPiA+IHN0b3JtIG1haWxp bmcgbGlzdA0KPiA+ID4gPiBzdG9ybUBpZXRmLm9yZw0KPiA+ID4gPiBodHRwczovL3d3dy5pZXRm Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3N0b3JtDQo= From david.black@emc.com Fri Sep 28 09:48:13 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8AC8E21F8456 for ; Fri, 28 Sep 2012 09:48:13 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.585 X-Spam-Level: X-Spam-Status: No, score=-102.585 tagged_above=-999 required=5 tests=[AWL=0.014, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2zHphTB6dt88 for ; Fri, 28 Sep 2012 09:48:13 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id D93D821F8455 for ; Fri, 28 Sep 2012 09:48:12 -0700 (PDT) Received: from hop04-l1d11-si02.isus.emc.com (HOP04-L1D11-SI02.isus.emc.com [10.254.111.55]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8SGmBWJ015748 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 28 Sep 2012 12:48:11 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.221.145]) by hop04-l1d11-si02.isus.emc.com (RSA Interceptor) for ; Fri, 28 Sep 2012 12:47:53 -0400 Received: from mxhub17.corp.emc.com (mxhub17.corp.emc.com [10.254.93.46]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8SGlrwp028523 for ; Fri, 28 Sep 2012 12:47:53 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub17.corp.emc.com ([10.254.93.46]) with mapi; Fri, 28 Sep 2012 12:47:52 -0400 From: "Black, David" To: "storm@ietf.org" Date: Fri, 28 Sep 2012 12:47:51 -0400 Thread-Topic: SPC-2 reserve/release text - version 7 (final, I hope) Thread-Index: Ac2cNcys4MzRK8PXQu651fVKNuEIxQBYniZg Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DF10E46@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: [storm] SPC-2 reserve/release text - version 7 (final, I hope) X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Sep 2012 16:48:13 -0000 Two more changes (UA text for session reestablishment, persistent recommendations recommendation text), and=20 I think we're finally (!) done ... ----------------------------------------- [1] For clarity, the following change should be made in 4.4.3.1: OLD That is, it should perform session recovery as described in Chapter 6. NEW That is, it should reinstate the session via iSCSI session reinstatement (Section 6.3.5) or continue via session continuation (Section 6.3.6). END [2] Add the following sentence to the end of 4.4.3.1: Unit Attention conditions that result from session reestablishment indicate whether nexus state was preserved, see Section 6.3.4 of [SAM4]. [3] NEW TEXT: 4.4.3.2. Reservations There are two reservation management methods defined in the SCSI standards, reserve/release reservations, based on the RESERVE and RELEASE commands [SPC2], and persistent reservations, based on the PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be used; persistent reservations are suggested as an alternative, see Annex B of [SPC4]. State for persistent reservations is required to persist through changes and failures at the iSCSI layer that result in I_T Nexus failures, see [SPC3] for details and specific requirements. In contrast, [SPC2] does not specify detailed persistence requirements for reserve/release reservation state after an I_T Nexus failure. Nonetheless, when reserve/release reservations are supported by an iSCSI target, the preferred implementation approach is to preserve reserve/release reservation state for iSCSI session reinstatement (see Section 6.3.5) or session continuation (see Section 6.3.6). Two additional caveats apply to reserve/release reservations: - When connection failure causes the iSCSI session to fail, and the session is not reinstated or continued, target retention of that session's reserve/release reservation state may require an initiator to issue a reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state. - Reserve/release reservations may not behave as expected when persistent reservations are also used on the same logical unit; see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE behavior" in [SPC4]. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- From cbm@chadalapaka.com Fri Sep 28 16:23:51 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DF79321F84FF for ; Fri, 28 Sep 2012 16:23:51 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OzjO+nkPlKk5 for ; Fri, 28 Sep 2012 16:23:51 -0700 (PDT) Received: from co1outboundpool.messaging.microsoft.com (co1ehsobe002.messaging.microsoft.com [216.32.180.185]) by ietfa.amsl.com (Postfix) with ESMTP id 2D1BE21F84FD for ; Fri, 28 Sep 2012 16:23:50 -0700 (PDT) Received: from mail96-co1-R.bigfish.com (10.243.78.247) by CO1EHSOBE011.bigfish.com (10.243.66.74) with Microsoft SMTP Server id 14.1.225.23; Fri, 28 Sep 2012 23:23:50 +0000 Received: from mail96-co1 (localhost [127.0.0.1]) by mail96-co1-R.bigfish.com (Postfix) with ESMTP id E7819A401F2 for ; Fri, 28 Sep 2012 23:23:49 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.240.117; KIP:(null); UIP:(null); IPV:NLI; H:BL2PRD0610HT005.namprd06.prod.outlook.com; RD:none; EFVD:NLI X-SpamScore: -26 X-BigFish: PS-26(zz9371I542M4015Id6f1izz1202h1d1ah1d2ahzz1033IL8275dhz2fh2a8h668h839hd25hf0ah107ah1288h12a5h12a9h12bdh137ah13b6h1155h) Received-SPF: pass (mail96-co1: domain of chadalapaka.com designates 157.56.240.117 as permitted sender) client-ip=157.56.240.117; envelope-from=cbm@chadalapaka.com; helo=BL2PRD0610HT005.namprd06.prod.outlook.com ; .outlook.com ; Received: from mail96-co1 (localhost.localdomain [127.0.0.1]) by mail96-co1 (MessageSwitch) id 1348874627388727_12699; Fri, 28 Sep 2012 23:23:47 +0000 (UTC) Received: from CO1EHSMHS015.bigfish.com (unknown [10.243.78.241]) by mail96-co1.bigfish.com (Postfix) with ESMTP id 5A2148C0159; Fri, 28 Sep 2012 23:23:47 +0000 (UTC) Received: from BL2PRD0610HT005.namprd06.prod.outlook.com (157.56.240.117) by CO1EHSMHS015.bigfish.com (10.243.66.25) with Microsoft SMTP Server (TLS) id 14.1.225.23; Fri, 28 Sep 2012 23:23:47 +0000 Received: from BL2PRD0610MB361.namprd06.prod.outlook.com ([169.254.12.31]) by BL2PRD0610HT005.namprd06.prod.outlook.com ([10.255.101.40]) with mapi id 14.16.0190.008; Fri, 28 Sep 2012 23:23:44 +0000 From: Mallikarjun Chadalapaka To: "Black, David" , "storm@ietf.org" Thread-Topic: SPC-2 reserve/release text - version 7 (final, I hope) Thread-Index: Ac2cNcys4MzRK8PXQu651fVKNuEIxQBYniZgAAz+S9A= Date: Fri, 28 Sep 2012 23:23:43 +0000 Message-ID: References: <8D3D17ACE214DC429325B2B98F3AE7120DF10E46@MX15A.corp.emc.com> In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DF10E46@MX15A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [71.197.226.181] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: chadalapaka.com Subject: Re: [storm] SPC-2 reserve/release text - version 7 (final, I hope) X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Sep 2012 23:23:52 -0000 Hi David, Thanks for diligently driving these text updates . Sorry, but I think we st= ill need a couple of tweaks: Thinking more about it, I don't think iSCSI protocol spec can place a norma= tive requirement on the overall solution outside of iSCSI. IOW, an iSCSI pr= otocol compliance test cannot really test a "SHOULD NOT" on a SCSI feature = usage (or lack thereof). So IMO, this should be a lower-case. Change: Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be used To: Reserve/release reservations are obsolete [SPC3] and should not be used And one editorial (because connection failure isn't the only reason for a s= ession failure; "target retention" can be phrased better): Change: =20 - When connection failure causes the iSCSI session to fail, and the session is not reinstated or continued, target retention of that session's reserve/release reservation state may require an initiator to issue a reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state. To: - Retention of a failed session's reserve/release reservation state by an= iSCSI target, even after that failed iSCSI session is not reinstated or continued, m= ay require an initiator to issue a reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state.=20 Thanks. Mallikarjun -----Original Message----- From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of B= lack, David Sent: Friday, September 28, 2012 9:48 AM To: storm@ietf.org Subject: [storm] SPC-2 reserve/release text - version 7 (final, I hope) Two more changes (UA text for session reestablishment, persistent recommend= ations recommendation text), and I think we're finally (!) done ... ----------------------------------------- [1] For clarity, the following change should be made in 4.4.3.1: OLD That is, it should perform session recovery as described in Chapter 6. NEW That is, it should reinstate the session via iSCSI session reinstatement (Section 6.3.5) or continue via session continuation (Section 6.3.6). END [2] Add the following sentence to the end of 4.4.3.1: Unit Attention conditions that result from session reestablishment indicate whether nexus state was preserved, see Section 6.3.4 of [SAM4]. [3] NEW TEXT: 4.4.3.2. Reservations There are two reservation management methods defined in the SCSI standards, reserve/release reservations, based on the RESERVE and RELEASE commands [SPC2], and persistent reservations, based on the PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be used; persistent reservations are suggested as an alternative, see Annex B of [SPC4]. State for persistent reservations is required to persist through changes and failures at the iSCSI layer that result in I_T Nexus failures, see [SPC3] for details and specific requirements. In contrast, [SPC2] does not specify detailed persistence requirements for reserve/release reservation state after an I_T Nexus failure. Nonetheless, when reserve/release reservations are supported by an iSCSI target, the preferred implementation approach is to preserve reserve/release reservation state for iSCSI session reinstatement (see Section 6.3.5) or session continuation (see Section 6.3.6). Two additional caveats apply to reserve/release reservations: - When connection failure causes the iSCSI session to fail, and the session is not reinstated or continued, target retention of that session's reserve/release reservation state may require an initiator to issue a reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state. - Reserve/release reservations may not behave as expected when persistent reservations are also used on the same logical unit; see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE behavior" in [SPC4]. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm From david.black@emc.com Fri Sep 28 16:49:38 2012 Return-Path: X-Original-To: storm@ietfa.amsl.com Delivered-To: storm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A303821F859B for ; Fri, 28 Sep 2012 16:49:38 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.586 X-Spam-Level: X-Spam-Status: No, score=-102.586 tagged_above=-999 required=5 tests=[AWL=0.013, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4k7bMKFlHKAv for ; Fri, 28 Sep 2012 16:49:38 -0700 (PDT) Received: from mexforward.lss.emc.com (hop-nat-141.emc.com [168.159.213.141]) by ietfa.amsl.com (Postfix) with ESMTP id E22A921F8528 for ; Fri, 28 Sep 2012 16:49:37 -0700 (PDT) Received: from hop04-l1d11-si02.isus.emc.com (HOP04-L1D11-SI02.isus.emc.com [10.254.111.55]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8SNnYpZ021500 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 28 Sep 2012 19:49:35 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.222.226]) by hop04-l1d11-si02.isus.emc.com (RSA Interceptor); Fri, 28 Sep 2012 19:49:18 -0400 Received: from mxhub10.corp.emc.com (mxhub10.corp.emc.com [10.254.92.105]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q8SNnH4W001992; Fri, 28 Sep 2012 19:49:18 -0400 Received: from mx15a.corp.emc.com ([169.254.1.83]) by mxhub10.corp.emc.com ([10.254.92.105]) with mapi; Fri, 28 Sep 2012 19:49:17 -0400 From: "Black, David" To: "'cbm@chadalapaka.com'" , "'storm@ietf.org'" Date: Fri, 28 Sep 2012 19:49:17 -0400 Thread-Topic: SPC-2 reserve/release text - version 7 (final, I hope) Thread-Index: Ac2cNcys4MzRK8PXQu651fVKNuEIxQBYniZgAAz+S9AAAeg68w== Message-ID: <8D3D17ACE214DC429325B2B98F3AE7120DD6459E@MX15A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: Re: [storm] SPC-2 reserve/release text - version 7 (final, I hope) X-BeenThere: storm@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Storage Maintenance WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Sep 2012 23:49:38 -0000 Both changes are fine with me. Thanks, --David +++Sent from Blackberry ----- Original Message ----- From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] Sent: Friday, September 28, 2012 07:23 PM=0A= To: Black, David; storm@ietf.org Subject: RE: SPC-2 reserve/release text - version 7 (final, I hope) Hi David, Thanks for diligently driving these text updates . Sorry, but I think we st= ill need a couple of tweaks: Thinking more about it, I don't think iSCSI protocol spec can place a norma= tive requirement on the overall solution outside of iSCSI. IOW, an iSCSI pr= otocol compliance test cannot really test a "SHOULD NOT" on a SCSI feature = usage (or lack thereof). So IMO, this should be a lower-case. Change: Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be used To: Reserve/release reservations are obsolete [SPC3] and should not be used And one editorial (because connection failure isn't the only reason for a s= ession failure; "target retention" can be phrased better): Change: =20 - When connection failure causes the iSCSI session to fail, and the session is not reinstated or continued, target retention of that session's reserve/release reservation state may require an initiator to issue a reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state. To: - Retention of a failed session's reserve/release reservation state by an= iSCSI target, even after that failed iSCSI session is not reinstated or continued, m= ay require an initiator to issue a reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state.=20 Thanks. Mallikarjun -----Original Message----- From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of B= lack, David Sent: Friday, September 28, 2012 9:48 AM To: storm@ietf.org Subject: [storm] SPC-2 reserve/release text - version 7 (final, I hope) Two more changes (UA text for session reestablishment, persistent recommend= ations recommendation text), and I think we're finally (!) done ... ----------------------------------------- [1] For clarity, the following change should be made in 4.4.3.1: OLD That is, it should perform session recovery as described in Chapter 6. NEW That is, it should reinstate the session via iSCSI session reinstatement (Section 6.3.5) or continue via session continuation (Section 6.3.6). END [2] Add the following sentence to the end of 4.4.3.1: Unit Attention conditions that result from session reestablishment indicate whether nexus state was preserved, see Section 6.3.4 of [SAM4]. [3] NEW TEXT: 4.4.3.2. Reservations There are two reservation management methods defined in the SCSI standards, reserve/release reservations, based on the RESERVE and RELEASE commands [SPC2], and persistent reservations, based on the PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3]. Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be used; persistent reservations are suggested as an alternative, see Annex B of [SPC4]. State for persistent reservations is required to persist through changes and failures at the iSCSI layer that result in I_T Nexus failures, see [SPC3] for details and specific requirements. In contrast, [SPC2] does not specify detailed persistence requirements for reserve/release reservation state after an I_T Nexus failure. Nonetheless, when reserve/release reservations are supported by an iSCSI target, the preferred implementation approach is to preserve reserve/release reservation state for iSCSI session reinstatement (see Section 6.3.5) or session continuation (see Section 6.3.6). Two additional caveats apply to reserve/release reservations: - When connection failure causes the iSCSI session to fail, and the session is not reinstated or continued, target retention of that session's reserve/release reservation state may require an initiator to issue a reset (e.g., LOGICAL UNIT RESET, see section 11.5) in order to remove that reservation state. - Reserve/release reservations may not behave as expected when persistent reservations are also used on the same logical unit; see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE behavior" in [SPC4]. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ storm mailing list storm@ietf.org https://www.ietf.org/mailman/listinfo/storm