From arkady.kanevsky@gmail.com  Fri Jun  3 18:29:21 2011
Return-Path: <arkady.kanevsky@gmail.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6B10DE0752 for <storm@ietfa.amsl.com>; Fri,  3 Jun 2011 18:29:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.598
X-Spam-Level: 
X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ka41xtykdxZA for <storm@ietfa.amsl.com>; Fri,  3 Jun 2011 18:29:19 -0700 (PDT)
Received: from mail-iw0-f172.google.com (mail-iw0-f172.google.com [209.85.214.172]) by ietfa.amsl.com (Postfix) with ESMTP id 7F949E071A for <storm@ietf.org>; Fri,  3 Jun 2011 18:29:19 -0700 (PDT)
Received: by iwn39 with SMTP id 39so2596880iwn.31 for <multiple recipients>; Fri, 03 Jun 2011 18:29:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=l6pkQxkYV+nKLAd6FETz8KnzuEdCIvbCOyijd2SZp/M=; b=sF3HQnvf3FTTOt58oYMVWJuxOzKWVHRGn9cF5ExLNEd2nZztyppFb4E8cKLBs1Lpu/ NZdlDc7Ueo5tW6rpWlmYXdG3o1BFpQM1PS4dPovgh4mCUBo5QHtxpAbgIFX8flUaiHjo 0OovOHr6Cf3LEe1kRviElDU0C/PNlFL2OW4uc=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=BiJXxFpvHZJuUfIISq4ByV2u+dXL+sSO7RE8PO9LXOZYymaZnDx7gtMjEYGvhXVMSc WhFbF1zIb/dxpncU8T7wnRo9PK2IJT98k5J0n2TNjhM4bVKVUWOrnrGbXWC2OOtiwoxd jItSZ36ijnFRSFhPnCi+eElr3DB6r5L3FB2zc=
MIME-Version: 1.0
Received: by 10.231.74.7 with SMTP id s7mr3511150ibj.172.1307150958793; Fri, 03 Jun 2011 18:29:18 -0700 (PDT)
Received: by 10.231.199.132 with HTTP; Fri, 3 Jun 2011 18:29:18 -0700 (PDT)
In-Reply-To: <B7F58B09EC4B43D7BF8BFA9FA95A8061@davidPC>
References: <AcwW/3OnDAJozBeCSrOODdtKJWAnhg==> <B7F58B09EC4B43D7BF8BFA9FA95A8061@davidPC>
Date: Fri, 3 Jun 2011 21:29:18 -0400
Message-ID: <BANLkTi=sUfHteXgDyHf1cngh3cVpu=YEwQ@mail.gmail.com>
From: arkady kanevsky <arkady.kanevsky@gmail.com>
To: David Harrington <ietfdbh@comcast.net>
Content-Type: multipart/alternative; boundary=000e0cd6b24491159804a4d8cc39
Cc: storm-chairs@ietf.org, draft-ietf-storm-mpa-peer-connect@tools.ietf.org, storm@ietf.org
Subject: Re: [storm] AD Review of draft-ietf-storm-mpa-peer-connect
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 04 Jun 2011 01:29:21 -0000

--000e0cd6b24491159804a4d8cc39
Content-Type: text/plain; charset=ISO-8859-1

David,
Thank you for a very good comments and careful reading of a draft.
We are working on the next version which address your comments, while tardy
on responding to the email.
Thanks,
Arkady

On Fri, May 20, 2011 at 11:06 AM, David Harrington <ietfdbh@comcast.net>wrote:

> AD Review of draft-ietf-storm-mpa-peer-connect
>
> I have reviewed this document and have comments. I think this work is
> important, but the way this document explains it could be better.
>
> My goal as AD is to have this document go through IETF Last Call and
> IESG review without any issues delaying advancement.
>
> Some comments are technical/process comments that refer to text that
> will likely cause delays during IESG Evaluation.
>
> Some are stylistic, such as capitalization and wording consistency,
> that are distracting and are likely to make readers wonder "is there
> something special here that I need to understand?" and to waste time
> trying to determine if there is or is not something special that led
> to your capitalizing the text, or using different wording than
> elsewhere. These could lead to IESG members raising unnecessary
> questions that would delay advancement.
>
> Others are editorial in nature, mostly about writing clear and
> unambiguous text. Generally, if it is simply a small editorial thing,
> I ignored it, expecting the RFC Editor to clean up the text a bit. But
> sometimes, the lack of clarity could lead to differing interpretations
> of the text, or lack of understanding of the text, and those should be
> fixed before I bring this to the IESG.
>
> In some cases, I have asked a question. If I had to ask a question,
> then the document apparently didn't explain it clearly enough. The
> answer to the question usually needs to be explained "in the
> document", not just in a private email to me.
>
> 1) idnits complains about unreachable reference:
> http://www.rdmaconsortium.org/home/draft-hilland-iwarp-verbs-v1.0-RDMA
> C.pdf
>
> 2) abstracts cannot contain references.
>
> 3) in 4.3.2, the conclusion could be stated more clearly, such as "If
> a nop approach was used, it would require lower layers to know the
> usage model, and would disrupt the calculations ..."
>
> 4) in 4.3.3, I have no idea what this text is saying or why it is
> relevant to the problem at hand. I don't know what an untagged message
> is; I don't know why a WC can only be generated by an untagged
> message. Can you expand on this so readers that are not RDMA experts
> understand what you're talking about? The IESG will need to review and
> approve this document, and to my knowledge none of them are RDMA
> experts.
>
> 5) It would be helpful to explain why taking the bit from RES is
> backwards compatible.
> Explain why older implementation will not have a problem with the new
> bit, and why new implemntations will be able to benefit from the new
> bit.
>
> 6) Rev "MUST be set to two"; would this be better as "two or higher?"
> to allow for a version three+ that would be backward compatible with
> the enhanced features of version two?
>
> 7) I don't understand "The Enhanced Reject function code SHOULD be
> used to indicate a configuration that would have been accepted." Would
> have been accepted except was not because of what condition? Why is
> this only a SHOULD and not a MUST?
>  8) How can the private data be zero length if it must begin with
> Enhanced RDMA Connection establishment data?
> 9) sometimes "Enhanced RDMA Connection Establishment Data" is
> capitalized, sometimes not. Be consistent.
> 10) Received extended ... message SHOULD be reported to the ULP. Why
> is this not a MUST? Anytime there is a SHOULD, the text should
> identify when the rule is not mandatory - what are the valid
> exceptions that make this not a MUST?
> 11) What is "enhanced DDP stream management"?
> 12) "It should be noted that" - you're noting it, so this lead-in
> phrase is totally unnecessary. If you leave it as "It should be noted
> that", then explain WHY it should be noted. If "it is especially
> important to recognize that" then say that, and explain WHY it is
> important to recognize.
> It appears that it is important only because you've made a case for
> peer-to-peer needing to initiate from either side, and the SCTP
> adaption layer allows for that already. So you should probably explain
> why you're duplicating that capability.
> 13) additional legal sequences - but then you don't seem to define any
> **sequences**. Is "Legal Sequences" a special term? If so, please
> explain where it is defined. if not, why is it capitalized?
> 14) I cannot parse "The protocol layers on which RDMA connection
> establishment is layered upon [RFC5040] and [RFC5041] define layers
> and error types."
> 15) if 5043 and 5044 do not define error codes, then how are the error
> codes used? are they carried in messages? which messages?
> 16) In looking for the answer to 15, I found that error codes ARE
> defined in RFC5044 section 8. So your text is apparently wrong.
> 17) in section 9, the responder SHOULD NOT set its IRD higher than
> Initiator ORD. Why is this not a MUST? Under what conditions is it
> acceptable to set this higher than Initiator ORD? What are the
> implications of doing so?
> the responder SHOULD set its ORD lower. What are the valid exceptions
> to this?
> 18) the Initiator MUST set its IRD to a value at least as large as the
> responder ORD if it has sufficient resources
> "MUST if" means this is a SHOULD, and lack of resources is the
> exception. If this is truly a MUST (which the following sentences seem
> to say, requiring a termination if ti doesn't do so, then I would
> remove the if clause from the first sentence, making it a definite
> MUST.
> 19) "Control flag for using" doesn't discuss whatthe values are. Does
> this mean that if A has a value 1, then FULPDU has a zero length?
> This section might be better organized by defining what the fields are
> (control flags and depths), then describing how the values are set.
> You do that for A,B,C but describe IRD and ORD in-line.
> 20) In section 9, "If there is no Standard RDMAP Configuration Data in
> the MPA Reply Frame, and the Enhanced Connection Establishment version
> number is used, it is the equivalent of setting 'A', 'B' and 'C'. "
> Why do we need this extra complexity? Why not require that these be
> set ONE way, not two? I also have concerns about "THE enhanced
> connectiosn establshment version number"; will this document need to
> be updated if a version three is ever developed?
> 21) "Setting no Control Flags in the MPA Reply indicates that an RDMA
> Send message will be required. As this option will require the
> initiator ULP to be involved it SHOULD NOT be used unless necessary. "
> Please define what necessary is. I think this would be better as a
> MUST NOT.
> 22) "Initiator or Responder SHOULD generate the TERM message" - why is
> thi snot a MUST? What are the valid exceptions to this?
> in section 10:
> 23) "An initiator SHOULD NOT use the Enhanced DDP Connection
> Establishment formats or function codes when no enhanced functionality
> is desired. " what are the valid exceptions that make this a SHOULD
> rather than a MUST?
> 24) the paragraph containing "If a responder does not support received
> extended MPA message, then it MUST close the TCP connection and exit
> MPA since MPA frame is improperly formatted for it as stated in
> [RFC5044]." needs a bit of English cleanup.
> 25) section11 This document defines error codes; so does RFC5044.
> Implementers and operators will need to hunt through documents to find
> the various error codes. This document apparently overlooks the error
> codes defined in 5044 section 8. Should there be a registry,
> maintained by IANA, for the MPA error codes?
> section 12:
> 26) I am always concerned when a security coinsiderations section says
> this document does not introduce any new security considerations. This
> is also a red flag during IESG Evaluation. I suggest you document WHY
> these changes do not impact security compared to 5043 and 5044.
> Demonstrate that the WG actually **considered** the security issues.
>
> 27) expand acronyms on first use (MPA, RDMA, RDDP, ULP, iWARP, etc.)
>
> 28) references for sctp, infiniband,
>
> 29) in 4.3.1, "required only for one transport" - mention the one
> transport?
>
> 30) an explanation/definition of MPA Fencing would help readers. Maybe
> a terminology section would be helpful, given all the acronyms, etc.
>
> 31) The information in 4.4 would be helful in the Introduction
> section. This would help to organize the document into "Here's how it
> works now; here is why the client/server model is undesirable in a P2P
> environment; and this is how to modify the existing standard to
> address this problem."
>
> 32) Is section 5 a description of how things currently work, or the
> way this document proposes they work? Section 5 mentions "Enhanced
> Connection Setup"; is that the same as "Enhanced RDMA Connection
> Establishment"? if so, then naming section 5 "Enhanced RDMA Connection
> Establishment" or "Enhanced MPA Connection Setup" might be helful. Use
> terminology consistently, please.
>
> I hope these suggestions help improve the document so we can advance
> it through the approval process quickly,
>
>
>
>
>
>
>
>
>
> David Harrington
> Director, IETF Transport Area
> ietfdbh@comcast.net (preferred for ietf)
> dbharrington@huaweisymantec.com
> +1 603 828 1401 (cell)
>
> _______________________________________________
> storm mailing list
> storm@ietf.org
> https://www.ietf.org/mailman/listinfo/storm
>



-- 
Cheers,
Arkady Kanevsky

--000e0cd6b24491159804a4d8cc39
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

David,<br>Thank you for a very good comments and careful reading of a draft=
.<br>We are working on the next version which address your comments, while =
tardy on responding to the email.<br>Thanks,<br>Arkady<br><br><div class=3D=
"gmail_quote">
On Fri, May 20, 2011 at 11:06 AM, David Harrington <span dir=3D"ltr">&lt;<a=
 href=3D"mailto:ietfdbh@comcast.net">ietfdbh@comcast.net</a>&gt;</span> wro=
te:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-=
left:1px #ccc solid;padding-left:1ex;">
AD Review of draft-ietf-storm-mpa-peer-connect<br>
<br>
I have reviewed this document and have comments. I think this work is<br>
important, but the way this document explains it could be better.<br>
<br>
My goal as AD is to have this document go through IETF Last Call and<br>
IESG review without any issues delaying advancement.<br>
<br>
Some comments are technical/process comments that refer to text that<br>
will likely cause delays during IESG Evaluation.<br>
<br>
Some are stylistic, such as capitalization and wording consistency,<br>
that are distracting and are likely to make readers wonder &quot;is there<b=
r>
something special here that I need to understand?&quot; and to waste time<b=
r>
trying to determine if there is or is not something special that led<br>
to your capitalizing the text, or using different wording than<br>
elsewhere. These could lead to IESG members raising unnecessary<br>
questions that would delay advancement.<br>
<br>
Others are editorial in nature, mostly about writing clear and<br>
unambiguous text. Generally, if it is simply a small editorial thing,<br>
I ignored it, expecting the RFC Editor to clean up the text a bit. But<br>
sometimes, the lack of clarity could lead to differing interpretations<br>
of the text, or lack of understanding of the text, and those should be<br>
fixed before I bring this to the IESG.<br>
<br>
In some cases, I have asked a question. If I had to ask a question,<br>
then the document apparently didn&#39;t explain it clearly enough. The<br>
answer to the question usually needs to be explained &quot;in the<br>
document&quot;, not just in a private email to me.<br>
<br>
1) idnits complains about unreachable reference:<br>
<a href=3D"http://www.rdmaconsortium.org/home/draft-hilland-iwarp-verbs-v1.=
0-RDMA%0AC.pdf" target=3D"_blank">http://www.rdmaconsortium.org/home/draft-=
hilland-iwarp-verbs-v1.0-RDMA<br>
C.pdf</a><br>
<br>
2) abstracts cannot contain references.<br>
<br>
3) in 4.3.2, the conclusion could be stated more clearly, such as &quot;If<=
br>
a nop approach was used, it would require lower layers to know the<br>
usage model, and would disrupt the calculations ...&quot;<br>
<br>
4) in 4.3.3, I have no idea what this text is saying or why it is<br>
relevant to the problem at hand. I don&#39;t know what an untagged message<=
br>
is; I don&#39;t know why a WC can only be generated by an untagged<br>
message. Can you expand on this so readers that are not RDMA experts<br>
understand what you&#39;re talking about? The IESG will need to review and<=
br>
approve this document, and to my knowledge none of them are RDMA<br>
experts.<br>
<br>
5) It would be helpful to explain why taking the bit from RES is<br>
backwards compatible.<br>
Explain why older implementation will not have a problem with the new<br>
bit, and why new implemntations will be able to benefit from the new<br>
bit.<br>
<br>
6) Rev &quot;MUST be set to two&quot;; would this be better as &quot;two or=
 higher?&quot;<br>
to allow for a version three+ that would be backward compatible with<br>
the enhanced features of version two?<br>
<br>
7) I don&#39;t understand &quot;The Enhanced Reject function code SHOULD be=
<br>
used to indicate a configuration that would have been accepted.&quot; Would=
<br>
have been accepted except was not because of what condition? Why is<br>
this only a SHOULD and not a MUST?<br>
=A08) How can the private data be zero length if it must begin with<br>
Enhanced RDMA Connection establishment data?<br>
9) sometimes &quot;Enhanced RDMA Connection Establishment Data&quot; is<br>
capitalized, sometimes not. Be consistent.<br>
10) Received extended ... message SHOULD be reported to the ULP. Why<br>
is this not a MUST? Anytime there is a SHOULD, the text should<br>
identify when the rule is not mandatory - what are the valid<br>
exceptions that make this not a MUST?<br>
11) What is &quot;enhanced DDP stream management&quot;?<br>
12) &quot;It should be noted that&quot; - you&#39;re noting it, so this lea=
d-in<br>
phrase is totally unnecessary. If you leave it as &quot;It should be noted<=
br>
that&quot;, then explain WHY it should be noted. If &quot;it is especially<=
br>
important to recognize that&quot; then say that, and explain WHY it is<br>
important to recognize.<br>
It appears that it is important only because you&#39;ve made a case for<br>
peer-to-peer needing to initiate from either side, and the SCTP<br>
adaption layer allows for that already. So you should probably explain<br>
why you&#39;re duplicating that capability.<br>
13) additional legal sequences - but then you don&#39;t seem to define any<=
br>
**sequences**. Is &quot;Legal Sequences&quot; a special term? If so, please=
<br>
explain where it is defined. if not, why is it capitalized?<br>
14) I cannot parse &quot;The protocol layers on which RDMA connection<br>
establishment is layered upon [RFC5040] and [RFC5041] define layers<br>
and error types.&quot;<br>
15) if 5043 and 5044 do not define error codes, then how are the error<br>
codes used? are they carried in messages? which messages?<br>
16) In looking for the answer to 15, I found that error codes ARE<br>
defined in RFC5044 section 8. So your text is apparently wrong.<br>
17) in section 9, the responder SHOULD NOT set its IRD higher than<br>
Initiator ORD. Why is this not a MUST? Under what conditions is it<br>
acceptable to set this higher than Initiator ORD? What are the<br>
implications of doing so?<br>
the responder SHOULD set its ORD lower. What are the valid exceptions<br>
to this?<br>
18) the Initiator MUST set its IRD to a value at least as large as the<br>
responder ORD if it has sufficient resources<br>
&quot;MUST if&quot; means this is a SHOULD, and lack of resources is the<br=
>
exception. If this is truly a MUST (which the following sentences seem<br>
to say, requiring a termination if ti doesn&#39;t do so, then I would<br>
remove the if clause from the first sentence, making it a definite<br>
MUST.<br>
19) &quot;Control flag for using&quot; doesn&#39;t discuss whatthe values a=
re. Does<br>
this mean that if A has a value 1, then FULPDU has a zero length?<br>
This section might be better organized by defining what the fields are<br>
(control flags and depths), then describing how the values are set.<br>
You do that for A,B,C but describe IRD and ORD in-line.<br>
20) In section 9, &quot;If there is no Standard RDMAP Configuration Data in=
<br>
the MPA Reply Frame, and the Enhanced Connection Establishment version<br>
number is used, it is the equivalent of setting &#39;A&#39;, &#39;B&#39; an=
d &#39;C&#39;. &quot;<br>
Why do we need this extra complexity? Why not require that these be<br>
set ONE way, not two? I also have concerns about &quot;THE enhanced<br>
connectiosn establshment version number&quot;; will this document need to<b=
r>
be updated if a version three is ever developed?<br>
21) &quot;Setting no Control Flags in the MPA Reply indicates that an RDMA<=
br>
Send message will be required. As this option will require the<br>
initiator ULP to be involved it SHOULD NOT be used unless necessary. &quot;=
<br>
Please define what necessary is. I think this would be better as a<br>
MUST NOT.<br>
22) &quot;Initiator or Responder SHOULD generate the TERM message&quot; - w=
hy is<br>
thi snot a MUST? What are the valid exceptions to this?<br>
in section 10:<br>
23) &quot;An initiator SHOULD NOT use the Enhanced DDP Connection<br>
Establishment formats or function codes when no enhanced functionality<br>
is desired. &quot; what are the valid exceptions that make this a SHOULD<br=
>
rather than a MUST?<br>
24) the paragraph containing &quot;If a responder does not support received=
<br>
extended MPA message, then it MUST close the TCP connection and exit<br>
MPA since MPA frame is improperly formatted for it as stated in<br>
[RFC5044].&quot; needs a bit of English cleanup.<br>
25) section11 This document defines error codes; so does RFC5044.<br>
Implementers and operators will need to hunt through documents to find<br>
the various error codes. This document apparently overlooks the error<br>
codes defined in 5044 section 8. Should there be a registry,<br>
maintained by IANA, for the MPA error codes?<br>
section 12:<br>
26) I am always concerned when a security coinsiderations section says<br>
this document does not introduce any new security considerations. This<br>
is also a red flag during IESG Evaluation. I suggest you document WHY<br>
these changes do not impact security compared to 5043 and 5044.<br>
Demonstrate that the WG actually **considered** the security issues.<br>
<br>
27) expand acronyms on first use (MPA, RDMA, RDDP, ULP, iWARP, etc.)<br>
<br>
28) references for sctp, infiniband,<br>
<br>
29) in 4.3.1, &quot;required only for one transport&quot; - mention the one=
<br>
transport?<br>
<br>
30) an explanation/definition of MPA Fencing would help readers. Maybe<br>
a terminology section would be helpful, given all the acronyms, etc.<br>
<br>
31) The information in 4.4 would be helful in the Introduction<br>
section. This would help to organize the document into &quot;Here&#39;s how=
 it<br>
works now; here is why the client/server model is undesirable in a P2P<br>
environment; and this is how to modify the existing standard to<br>
address this problem.&quot;<br>
<br>
32) Is section 5 a description of how things currently work, or the<br>
way this document proposes they work? Section 5 mentions &quot;Enhanced<br>
Connection Setup&quot;; is that the same as &quot;Enhanced RDMA Connection<=
br>
Establishment&quot;? if so, then naming section 5 &quot;Enhanced RDMA Conne=
ction<br>
Establishment&quot; or &quot;Enhanced MPA Connection Setup&quot; might be h=
elful. Use<br>
terminology consistently, please.<br>
<br>
I hope these suggestions help improve the document so we can advance<br>
it through the approval process quickly,<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
David Harrington<br>
Director, IETF Transport Area<br>
<a href=3D"mailto:ietfdbh@comcast.net">ietfdbh@comcast.net</a> (preferred f=
or ietf)<br>
<a href=3D"mailto:dbharrington@huaweisymantec.com">dbharrington@huaweisyman=
tec.com</a><br>
<a href=3D"tel:%2B1%20603%20828%201401" value=3D"+16038281401">+1 603 828 1=
401</a>=A0(cell)<br>
<br>
_______________________________________________<br>
storm mailing list<br>
<a href=3D"mailto:storm@ietf.org">storm@ietf.org</a><br>
<a href=3D"https://www.ietf.org/mailman/listinfo/storm" target=3D"_blank">h=
ttps://www.ietf.org/mailman/listinfo/storm</a><br>
</blockquote></div><br><br clear=3D"all"><br>-- <br>Cheers,<br>Arkady Kanev=
sky<br>

--000e0cd6b24491159804a4d8cc39--

From hemal@broadcom.com  Thu Jun  9 15:59:31 2011
Return-Path: <hemal@broadcom.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3545F11E80F5 for <storm@ietfa.amsl.com>; Thu,  9 Jun 2011 15:59:31 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.598
X-Spam-Level: 
X-Spam-Status: No, score=-2.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xUdW2J6qvg2N for <storm@ietfa.amsl.com>; Thu,  9 Jun 2011 15:59:30 -0700 (PDT)
Received: from MMS3.broadcom.com (mms3.broadcom.com [216.31.210.19]) by ietfa.amsl.com (Postfix) with ESMTP id EC7BF11E807A for <storm@ietf.org>; Thu,  9 Jun 2011 15:59:29 -0700 (PDT)
Received: from [10.9.200.133] by MMS3.broadcom.com with ESMTP (Broadcom SMTP Relay (Email Firewall v6.3.2)); Thu, 09 Jun 2011 16:03:20 -0700
X-Server-Uuid: B55A25B1-5D7D-41F8-BC53-C57E7AD3C201
Received: from IRVEXCHCCR02.corp.ad.broadcom.com ([10.9.200.183]) by IRVEXCHHUB02.corp.ad.broadcom.com ([10.9.200.133]) with mapi; Thu, 9 Jun 2011 15:59:10 -0700
From: "Hemal Shah" <hemal@broadcom.com>
To: "Mallikarjun Chadalapaka" <cbm@chadalapaka.com>, "david.black@emc.com" <david.black@emc.com>, "storm@ietf.org" <storm@ietf.org>
Date: Thu, 9 Jun 2011 15:56:38 -0700
Thread-Topic: [storm] Plan for iSCSI work
Thread-Index: AQCUfOqLFecrTE8dOe+/hpcfY+kOJQIRzTq9AgfEPyGW79UNkIAUSvPw
Message-ID: <76DBE161893C324BA6D4BB507EE4C38495124C4828@IRVEXCHCCR02.corp.ad.broadcom.com>
References: <7C4DFCE962635144B8FAE8CA11D0BF1E0588F426C5@MX14A.corp.emc.com> <SNT131-ds2DE0F0F2022D63A4AD747A0760@phx.gbl> <76DBE161893C324BA6D4BB507EE4C384951235CB38@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds117065D22D7BF8583485DBA0790@phx.gbl>
In-Reply-To: <SNT131-ds117065D22D7BF8583485DBA0790@phx.gbl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
MIME-Version: 1.0
X-WSS-ID: 61EF8EB24NS18610548-01-01
Content-Type: multipart/alternative; boundary=_000_76DBE161893C324BA6D4BB507EE4C38495124C4828IRVEXCHCCR02c_
Subject: Re: [storm] Plan for iSCSI work
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 09 Jun 2011 22:59:31 -0000

--_000_76DBE161893C324BA6D4BB507EE4C38495124C4828IRVEXCHCCR02c_
Content-Type: text/plain;
 charset=us-ascii
Content-Transfer-Encoding: quoted-printable

Thanks Mallikarjun!

The way I read Section 4.1.3 of RFC5048 and Section 4.2.3.4 of the consolid=
ated draft, the implementation of "FastAbort" feature is a MUST. From the f=
irst paragraph text from Section 4.1.3 of RFC5048, the first sentence manda=
tes that all iSCSI implementation comply to this section but the second sen=
tence say that the requirement in this section is conditional to the negoti=
ation of "FastAbort" key.

Protocol behavior defined in this section MUST be implemented by all iSCSI =
implementations complying with this document. Protocol behavior defined in =
this section MUST be exhibited by iSCSI implementations on an iSCSI session=
 when they negotiate the TaskReporting (Section 9.1) key to "FastAbort" on =
that session.


This is confusing. I think it will be better to remove the first sentence f=
rom 4.2.3.4 and clarify that the requirements in this section is conditiona=
l. I suggest rewording the first two sentences to below:

iSCSI implementations may negotiate the TaskReporting (Section 9.1) key to =
"FastAbort" on an iSCSI session. Protocol behavior
defined in this section MUST be exhibited by iSCSI implementations on an iS=
CSI session when they negotiate the TaskReporting (Section 9.1) key to "Fas=
tAbort" on that session.

Hemal


-----Original Message-----
From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
Sent: Friday, May 27, 2011 5:29 PM
To: Hemal Shah; david.black@emc.com; storm@ietf.org
Subject: RE: [storm] Plan for iSCSI work

Hi Hemal,

The text in this draft came verbatim from section 4.1.3 of RFC 5048.  There
have been no changes in this area.

The new text (as well as the old text) requires the TaskReporting key to be
negotiated to "FastAbort" before the multi-task abort semantics can be used
on an iSCSI session.

Thanks.

Mallikarjun




  -----Original Message-----
  From: Hemal Shah [mailto:hemal@broadcom.com]
  Sent: Friday, May 27, 2011 4:51 PM
  To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org
  Subject: RE: [storm] Plan for iSCSI work

  Mallikarjun and David,

  I noticed one problematic item in the consolidated draft. This item is th=
e
  requirement to support FastAbort. This feature was already defined in the
  implementation guide, but it was optional. In this draft, it became a
  required feature MUST - see in section 4.2.3.4.

  Do you know why the requirement was changed in the consolidated draft?

  I would like to keep the requirement optional as stated in the
  implementation guide and not break backward compatibility.

  Hemal

  -----Original Message-----
  From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf
  Of Mallikarjun Chadalapaka
  Sent: Thursday, May 26, 2011 6:50 PM
  To: david.black@emc.com; storm@ietf.org
  Subject: Re: [storm] Plan for iSCSI work

  Hi David,

  The list of changes you have called out are already done in the latest
draft.  I
  assume then that you are suggesting that the list itself should be
included
  in the next revision of the draft.

  Here's what I recall we have done so far:
  1)  iSCSIProtocolLevel specified as "1", and added a related normative
  reference to iSCSI-SAM draft
  2)  Markers and related keys were removed
  3)  SPKM authentication and related keys were removed
  4)  Added a new section on responding to obsoleted keys
  5)  Have explicitly allowed initiator+target implementations throughout
the
  text
  6)  Clarified that implementations SHOULD NOT rely on SLP-based discovery
  7)  Added UML diagrams, and related conventions

  The above is of course in addition to consolidating the different RFCs,
and
  making the related editorial changes.

  Thanks.

  Mallikarjun




    -----Original Message-----
    From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf
    Of david.black@emc.com
    Sent: Friday, May 20, 2011 2:49 PM
    To: storm@ietf.org
    Subject: [storm] Plan for iSCSI work

    I thought I'd offer some advance planning/warning on this, as the
    consolidated iSCSI draft (draft-ietf-storm-iscsi-cons) is large (over
300
    pages).  The current plan is to run a simultaneous WG Last Call on both
  this
    draft and the new features draft (draft-ietf-storm-iscsi-sam) starting
in
  mid-
    June (probably the week of June 13, after I get back from a badly neede=
d
    vacation).  That WG Last Call will run longer than the typical 2-week
time
    period, due to the total size of the drafts, but will end by July 5th a=
t
the
    latest so that the status of the drafts and the next steps are known
prior
  to
    the T10 (SCSI standards) meetings during the week of July 11.  As July
11th
    is also the draft cutoff deadline for the Quebec City IETF meetings,
revised
    draft versions may not show up until that meeting week (week of July
    24th).

    This is also a good point to announce that the storm WG will meet in
    Quebec City.
    I've only requested a 1-hour session, as we get most of our work done o=
n
    the mailing list.  Among the items for that meeting will be figuring ou=
t
  what
    to do with the RDMA extensions draft (despite its name,
draft-ietf-storm-
    rdmap-ext-00, it's not currently an official work item for the storm
WG).

    One thing that's missing from the consolidated iSCSI draft (and is a
reason
    why we're going to need a -03 version) is the changes that it makes to
the
    RFCs that it consolidates.  Off the top of my head, the major changes
are:
        - Removal of SPKM authentication
        - Removal of the Marker appendix
        - Removal of the SHOULD requirement for SLP implementation.
    Have I missed anything significant?  The summary of this will need to b=
e
    added to that draft.

    WG Last Call will be an opportunity (in fact the final opportunity) to
  discuss
    whether anything else should be removed from iSCSI, but there's no need
    to wait
    - I encourage people to review both drafts and post comments whenever
    they can.

    In parallel, work will get started on any iSCSI MIB changes that are
needed.
    So far, I only see one MIB change - the iSCSIProtocolLevel from the new
    features draft needs to be added to the MIB, probably with a structure
    analogous to the iSCSI version support that's already in the MIB.

    Thanks,
    --David (storm WG co-chair)
    ----------------------------------------------------
    David L. Black, Distinguished Engineer
    EMC Corporation, 176 South St., Hopkinton, MA  01748
    +1 (508) 293-7953             FAX: +1 (508) 293-7786
    david.black@emc.com        Mobile: +1 (978) 394-7754
    ----------------------------------------------------

    _______________________________________________
    storm mailing list
    storm@ietf.org
    https://www.ietf.org/mailman/listinfo/storm

  _______________________________________________
  storm mailing list
  storm@ietf.org
  https://www.ietf.org/mailman/listinfo/storm






--_000_76DBE161893C324BA6D4BB507EE4C38495124C4828IRVEXCHCCR02c_
Content-Type: text/html;
 charset=us-ascii
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left:=
 #800000 2px solid; } --></style>
</head>
<body>
<font face=3D"Consolas, monospace" size=3D"2">
<div>Thanks Mallikarjun!</div>
<div>&nbsp;</div>
<div>The way I read Section 4.1.3 of RFC5048 and Section 4.2.3.4 of the con=
solidated draft, the implementation of &quot;FastAbort&quot; feature is a M=
UST. From the first paragraph text from Section 4.1.3 of RFC5048, the first=
 sentence mandates that all iSCSI implementation
comply to this section but the second sentence say that the requirement in =
this section is conditional to the negotiation of &quot;FastAbort&quot; key=
. </div>
<div>&nbsp;</div>
<div style=3D"padding-left: 36pt; "><font face=3D"Courier, monospace" size=
=3D"2">Protocol behavior defined in this section MUST be implemented by all=
 iSCSI implementations complying with this document. Protocol behavior defi=
ned in this section MUST be exhibited by
iSCSI implementations on an iSCSI session when they negotiate the<font face=
=3D"Consolas, monospace" size=3D"2"> </font>TaskReporting (Section 9.1) key=
 to &quot;FastAbort&quot; on that session.</font></div>
<div style=3D"padding-left: 36pt; ">&nbsp;</div>
<div>&nbsp;</div>
<div>This is confusing. I think it will be better to remove the first sente=
nce from 4.2.3.4 and clarify that the requirements in this section is condi=
tional. I suggest rewording the first two sentences to below:</div>
<div>&nbsp;</div>
<div style=3D"text-indent: 36pt; "><font face=3D"Courier, monospace" size=
=3D"2">iSCSI implementations may negotiate the TaskReporting (Section 9.1) =
key to &quot;FastAbort&quot; on an iSCSI session. Protocol behavior</font><=
/div>
<div style=3D"padding-left: 36pt; "><font face=3D"Courier, monospace" size=
=3D"2">defined in this section MUST be exhibited by iSCSI implementations o=
n an iSCSI session when they negotiate the<font face=3D"Calibri, sans-serif=
" size=3D"2"> </font>TaskReporting (Section
9.1) key to &quot;FastAbort&quot; on that session.</font></div>
<div>&nbsp;</div>
<div>Hemal</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>-----Original Message-----<br>

From: Mallikarjun Chadalapaka [<a href=3D"mailto:cbm@chadalapaka.com">mailt=
o:cbm@chadalapaka.com</a>]
<br>

Sent: Friday, May 27, 2011 5:29 PM<br>

To: Hemal Shah; david.black@emc.com; storm@ietf.org<br>

Subject: RE: [storm] Plan for iSCSI work</div>
<div>&nbsp;</div>
<div>Hi Hemal,</div>
<div>&nbsp;</div>
<div>The text in this draft came verbatim from section 4.1.3 of RFC 5048.&n=
bsp; There</div>
<div>have been no changes in this area.</div>
<div>&nbsp;</div>
<div>The new text (as well as the old text) requires the TaskReporting key =
to be</div>
<div>negotiated to &quot;FastAbort&quot; before the multi-task abort semant=
ics can be used</div>
<div>on an iSCSI session.</div>
<div>&nbsp;</div>
<div>Thanks.</div>
<div>&nbsp;</div>
<div>Mallikarjun</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp; -----Original Message-----</div>
<div>&nbsp; From: Hemal Shah [<a href=3D"mailto:hemal@broadcom.com">mailto:=
hemal@broadcom.com</a>]</div>
<div>&nbsp; Sent: Friday, May 27, 2011 4:51 PM</div>
<div>&nbsp; To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.or=
g</div>
<div>&nbsp; Subject: RE: [storm] Plan for iSCSI work</div>
<div>&nbsp;</div>
<div>&nbsp; Mallikarjun and David,</div>
<div>&nbsp;</div>
<div>&nbsp; I noticed one problematic item in the consolidated draft. This =
item is the</div>
<div>&nbsp; requirement to support FastAbort. This feature was already defi=
ned in the</div>
<div>&nbsp; implementation guide, but it was optional. In this draft, it be=
came a</div>
<div>&nbsp; required feature MUST - see in section 4.2.3.4.</div>
<div>&nbsp;</div>
<div>&nbsp; Do you know why the requirement was changed in the consolidated=
 draft?</div>
<div>&nbsp;</div>
<div>&nbsp; I would like to keep the requirement optional as stated in the<=
/div>
<div>&nbsp; implementation guide and not break backward compatibility.</div=
>
<div>&nbsp;</div>
<div>&nbsp; Hemal</div>
<div>&nbsp;</div>
<div>&nbsp; -----Original Message-----</div>
<div>&nbsp; From: storm-bounces@ietf.org [<a href=3D"mailto:storm-bounces@i=
etf.org">mailto:storm-bounces@ietf.org</a>] On Behalf</div>
<div>&nbsp; Of Mallikarjun Chadalapaka</div>
<div>&nbsp; Sent: Thursday, May 26, 2011 6:50 PM</div>
<div>&nbsp; To: david.black@emc.com; storm@ietf.org</div>
<div>&nbsp; Subject: Re: [storm] Plan for iSCSI work</div>
<div>&nbsp;</div>
<div>&nbsp; Hi David,</div>
<div>&nbsp;</div>
<div>&nbsp; The list of changes you have called out are already done in the=
 latest</div>
<div>draft.&nbsp; I</div>
<div>&nbsp; assume then that you are suggesting that the list itself should=
 be</div>
<div>included</div>
<div>&nbsp; in the next revision of the draft.</div>
<div>&nbsp;</div>
<div>&nbsp; Here's what I recall we have done so far:</div>
<div>&nbsp; 1)&nbsp; iSCSIProtocolLevel specified as &quot;1&quot;, and add=
ed a related normative</div>
<div>&nbsp; reference to iSCSI-SAM draft</div>
<div>&nbsp; 2)&nbsp; Markers and related keys were removed</div>
<div>&nbsp; 3)&nbsp; SPKM authentication and related keys were removed</div=
>
<div>&nbsp; 4)&nbsp; Added a new section on responding to obsoleted keys</d=
iv>
<div>&nbsp; 5)&nbsp; Have explicitly allowed initiator&#43;target implement=
ations throughout</div>
<div>the</div>
<div>&nbsp; text</div>
<div>&nbsp; 6)&nbsp; Clarified that implementations SHOULD NOT rely on SLP-=
based discovery</div>
<div>&nbsp; 7)&nbsp; Added UML diagrams, and related conventions</div>
<div>&nbsp;</div>
<div>&nbsp; The above is of course in addition to consolidating the differe=
nt RFCs,</div>
<div>and</div>
<div>&nbsp; making the related editorial changes.</div>
<div>&nbsp;</div>
<div>&nbsp; Thanks.</div>
<div>&nbsp;</div>
<div>&nbsp; Mallikarjun</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; -----Original Message-----</div>
<div>&nbsp;&nbsp;&nbsp; From: storm-bounces@ietf.org [<a href=3D"mailto:sto=
rm-bounces@ietf.org">mailto:storm-bounces@ietf.org</a>] On Behalf</div>
<div>&nbsp;&nbsp;&nbsp; Of david.black@emc.com</div>
<div>&nbsp;&nbsp;&nbsp; Sent: Friday, May 20, 2011 2:49 PM</div>
<div>&nbsp;&nbsp;&nbsp; To: storm@ietf.org</div>
<div>&nbsp;&nbsp;&nbsp; Subject: [storm] Plan for iSCSI work</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; I thought I'd offer some advance planning/warning o=
n this, as the</div>
<div>&nbsp;&nbsp;&nbsp; consolidated iSCSI draft (draft-ietf-storm-iscsi-co=
ns) is large (over</div>
<div>300</div>
<div>&nbsp;&nbsp;&nbsp; pages).&nbsp; The current plan is to run a simultan=
eous WG Last Call on both</div>
<div>&nbsp; this</div>
<div>&nbsp;&nbsp;&nbsp; draft and the new features draft (draft-ietf-storm-=
iscsi-sam) starting</div>
<div>in</div>
<div>&nbsp; mid-</div>
<div>&nbsp;&nbsp;&nbsp; June (probably the week of June 13, after I get bac=
k from a badly needed</div>
<div>&nbsp;&nbsp;&nbsp; vacation).&nbsp; That WG Last Call will run longer =
than the typical 2-week</div>
<div>time</div>
<div>&nbsp;&nbsp;&nbsp; period, due to the total size of the drafts, but wi=
ll end by July 5th at</div>
<div>the</div>
<div>&nbsp;&nbsp;&nbsp; latest so that the status of the drafts and the nex=
t steps are known</div>
<div>prior</div>
<div>&nbsp; to</div>
<div>&nbsp;&nbsp;&nbsp; the T10 (SCSI standards) meetings during the week o=
f July 11.&nbsp; As July</div>
<div>11th</div>
<div>&nbsp;&nbsp;&nbsp; is also the draft cutoff deadline for the Quebec Ci=
ty IETF meetings,</div>
<div>revised</div>
<div>&nbsp;&nbsp;&nbsp; draft versions may not show up until that meeting w=
eek (week of July</div>
<div>&nbsp;&nbsp;&nbsp; 24th).</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; This is also a good point to announce that the stor=
m WG will meet in</div>
<div>&nbsp;&nbsp;&nbsp; Quebec City.</div>
<div>&nbsp;&nbsp;&nbsp; I've only requested a 1-hour session, as we get mos=
t of our work done on</div>
<div>&nbsp;&nbsp;&nbsp; the mailing list.&nbsp; Among the items for that me=
eting will be figuring out</div>
<div>&nbsp; what</div>
<div>&nbsp;&nbsp;&nbsp; to do with the RDMA extensions draft (despite its n=
ame,</div>
<div>draft-ietf-storm-</div>
<div>&nbsp;&nbsp;&nbsp; rdmap-ext-00, it's not currently an official work i=
tem for the storm</div>
<div>WG).</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; One thing that's missing from the consolidated iSCS=
I draft (and is a</div>
<div>reason</div>
<div>&nbsp;&nbsp;&nbsp; why we're going to need a -03 version) is the chang=
es that it makes to</div>
<div>the</div>
<div>&nbsp;&nbsp;&nbsp; RFCs that it consolidates.&nbsp; Off the top of my =
head, the major changes</div>
<div>are:</div>
<div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Removal of SPKM aut=
hentication</div>
<div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Removal of the Mark=
er appendix</div>
<div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Removal of the SHOU=
LD requirement for SLP implementation.</div>
<div>&nbsp;&nbsp;&nbsp; Have I missed anything significant?&nbsp; The summa=
ry of this will need to be</div>
<div>&nbsp;&nbsp;&nbsp; added to that draft.</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; WG Last Call will be an opportunity (in fact the fi=
nal opportunity) to</div>
<div>&nbsp; discuss</div>
<div>&nbsp;&nbsp;&nbsp; whether anything else should be removed from iSCSI,=
 but there's no need</div>
<div>&nbsp;&nbsp;&nbsp; to wait</div>
<div>&nbsp;&nbsp;&nbsp; - I encourage people to review both drafts and post=
 comments whenever</div>
<div>&nbsp;&nbsp;&nbsp; they can.</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; In parallel, work will get started on any iSCSI MIB=
 changes that are</div>
<div>needed.</div>
<div>&nbsp;&nbsp;&nbsp; So far, I only see one MIB change - the iSCSIProtoc=
olLevel from the new</div>
<div>&nbsp;&nbsp;&nbsp; features draft needs to be added to the MIB, probab=
ly with a structure</div>
<div>&nbsp;&nbsp;&nbsp; analogous to the iSCSI version support that's alrea=
dy in the MIB.</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; Thanks,</div>
<div>&nbsp;&nbsp;&nbsp; --David (storm WG co-chair)</div>
<div>&nbsp;&nbsp;&nbsp; ---------------------------------------------------=
-</div>
<div>&nbsp;&nbsp;&nbsp; David L. Black, Distinguished Engineer</div>
<div>&nbsp;&nbsp;&nbsp; EMC Corporation, 176 South St., Hopkinton, MA&nbsp;=
 01748</div>
<div>&nbsp;&nbsp;&nbsp; &#43;1 (508) 293-7953&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; FAX: &#43;1 (508) 293-7786</div>
<div>&nbsp;&nbsp;&nbsp; david.black@emc.com&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; Mobile: &#43;1 (978) 394-7754</div>
<div>&nbsp;&nbsp;&nbsp; ---------------------------------------------------=
-</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; _______________________________________________</di=
v>
<div>&nbsp;&nbsp;&nbsp; storm mailing list</div>
<div>&nbsp;&nbsp;&nbsp; storm@ietf.org</div>
<div>&nbsp;&nbsp;&nbsp; <a href=3D"https://www.ietf.org/mailman/listinfo/st=
orm">https://www.ietf.org/mailman/listinfo/storm</a></div>
<div>&nbsp;</div>
<div>&nbsp; _______________________________________________</div>
<div>&nbsp; storm mailing list</div>
<div>&nbsp; storm@ietf.org</div>
<div>&nbsp; <a href=3D"https://www.ietf.org/mailman/listinfo/storm">https:/=
/www.ietf.org/mailman/listinfo/storm</a></div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
</font>
</body>
</html>

--_000_76DBE161893C324BA6D4BB507EE4C38495124C4828IRVEXCHCCR02c_--


From internet-drafts@ietf.org  Tue Jun 14 09:55:56 2011
Return-Path: <internet-drafts@ietf.org>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1893F11E80AF; Tue, 14 Jun 2011 09:55:56 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -102.456
X-Spam-Level: 
X-Spam-Status: No, score=-102.456 tagged_above=-999 required=5 tests=[AWL=0.143, BAYES_00=-2.599, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id H1Asiuu7XxAS; Tue, 14 Jun 2011 09:55:55 -0700 (PDT)
Received: from ietfa.amsl.com (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7F4CF11E807F; Tue, 14 Jun 2011 09:55:55 -0700 (PDT)
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
From: internet-drafts@ietf.org
To: i-d-announce@ietf.org
X-Test-IDTracker: no
X-IETF-IDTracker: 3.55
Message-ID: <20110614165555.7027.65843.idtracker@ietfa.amsl.com>
Date: Tue, 14 Jun 2011 09:55:55 -0700
Cc: storm@ietf.org
Subject: [storm] I-D Action: draft-ietf-storm-mpa-peer-connect-05.txt
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 14 Jun 2011 16:55:56 -0000

A New Internet-Draft is available from the on-line Internet-Drafts director=
ies. This draft is a work item of the STORage Maintenance Working Group of =
the IETF.

	Title           : Enhanced RDMA Connection Establishment
	Author(s)       : Arkady Kanevsky
                          Caitlin Bestler
                          Robert Sharp
                          Steve Wise
	Filename        : draft-ietf-storm-mpa-peer-connect-05.txt
	Pages           : 20
	Date            : 2011-06-14

   This document updates RFC5043 and RFC5044 by extending MPA
   negotiation for RDMA connection establishment.  The first enhancement
   extends RFC5043, enabling peer-to-peer connection establishment over
   MPA/TCP.  The second enhancement extends both RFC5043 and RFC5044, by
   providing an option for standardized exchange of RDMA-layer
   connection configuration.


A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-storm-mpa-peer-connect-05.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

This Internet-Draft can be retrieved at:
ftp://ftp.ietf.org/internet-drafts/draft-ietf-storm-mpa-peer-connect-05.txt

From arkady.kanevsky@gmail.com  Tue Jun 14 09:57:59 2011
Return-Path: <arkady.kanevsky@gmail.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 67BD411E807F for <storm@ietfa.amsl.com>; Tue, 14 Jun 2011 09:57:59 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.598
X-Spam-Level: 
X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NfyY4E5wE-C0 for <storm@ietfa.amsl.com>; Tue, 14 Jun 2011 09:57:57 -0700 (PDT)
Received: from mail-px0-f182.google.com (mail-px0-f182.google.com [209.85.212.182]) by ietfa.amsl.com (Postfix) with ESMTP id 999ED11E808A for <storm@ietf.org>; Tue, 14 Jun 2011 09:57:57 -0700 (PDT)
Received: by pxi20 with SMTP id 20so4341993pxi.27 for <multiple recipients>; Tue, 14 Jun 2011 09:57:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=iT2WU+gB1N+WB1K3uRM1nFfqrmBKjM5p743fea2ZCbs=; b=D8dJEfhp/HCyEImIWsEvMwvoD9i2inNAHcVIfbAlB6PVZBilpUy9UTTDSlciUrFGMw A0ZmDiozAX5C8lZ1eIEOldHokVcYK+8ai0cvHANnHRpjONwp2OwEhieK8zMotYBhxrTj 5QhWhXgM8KoN5ORtp5n6RzI1meigKds5I4S7Q=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=l1X06IiMFVOyZ0FrKHo+gWewlVUvy5VF5IRcBiQxbmpQ7nooH28J2RIXolzT9x7yMH Qo+hPeCnumSYpYmtUe/Veh5LXwKLm/uQAeVD/j/T5e5OLmBhWLLyc5ijNVrQ0vhvDo1h NECN1hTB8vFxeiMpus1J+h5PBjwwkNMs258xc=
MIME-Version: 1.0
Received: by 10.142.55.16 with SMTP id d16mr1177617wfa.27.1308070676551; Tue, 14 Jun 2011 09:57:56 -0700 (PDT)
Received: by 10.142.143.3 with HTTP; Tue, 14 Jun 2011 09:57:56 -0700 (PDT)
In-Reply-To: <B7F58B09EC4B43D7BF8BFA9FA95A8061@davidPC>
References: <AcwW/3OnDAJozBeCSrOODdtKJWAnhg==> <B7F58B09EC4B43D7BF8BFA9FA95A8061@davidPC>
Date: Tue, 14 Jun 2011 12:57:56 -0400
Message-ID: <BANLkTikJ+P_oBJqMay-GkpkPYUz0F0zUGw@mail.gmail.com>
From: arkady kanevsky <arkady.kanevsky@gmail.com>
To: David Harrington <ietfdbh@comcast.net>
Content-Type: multipart/alternative; boundary=001636ef0485045abe04a5aef01f
Cc: storm-chairs@ietf.org, draft-ietf-storm-mpa-peer-connect@tools.ietf.org, storm@ietf.org
Subject: Re: [storm] AD Review of draft-ietf-storm-mpa-peer-connect
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 14 Jun 2011 16:57:59 -0000

--001636ef0485045abe04a5aef01f
Content-Type: text/plain; charset=ISO-8859-1

David,
version 5 of the draft is posted that responds to your comments.
Again, thank you very much for detailed comments and suggestions.
Arkady

On Fri, May 20, 2011 at 11:06 AM, David Harrington <ietfdbh@comcast.net>wrote:

> AD Review of draft-ietf-storm-mpa-peer-connect
>
> I have reviewed this document and have comments. I think this work is
> important, but the way this document explains it could be better.
>
> My goal as AD is to have this document go through IETF Last Call and
> IESG review without any issues delaying advancement.
>
> Some comments are technical/process comments that refer to text that
> will likely cause delays during IESG Evaluation.
>
> Some are stylistic, such as capitalization and wording consistency,
> that are distracting and are likely to make readers wonder "is there
> something special here that I need to understand?" and to waste time
> trying to determine if there is or is not something special that led
> to your capitalizing the text, or using different wording than
> elsewhere. These could lead to IESG members raising unnecessary
> questions that would delay advancement.
>
> Others are editorial in nature, mostly about writing clear and
> unambiguous text. Generally, if it is simply a small editorial thing,
> I ignored it, expecting the RFC Editor to clean up the text a bit. But
> sometimes, the lack of clarity could lead to differing interpretations
> of the text, or lack of understanding of the text, and those should be
> fixed before I bring this to the IESG.
>
> In some cases, I have asked a question. If I had to ask a question,
> then the document apparently didn't explain it clearly enough. The
> answer to the question usually needs to be explained "in the
> document", not just in a private email to me.
>
> 1) idnits complains about unreachable reference:
> http://www.rdmaconsortium.org/home/draft-hilland-iwarp-verbs-v1.0-RDMA
> C.pdf
>
> 2) abstracts cannot contain references.
>
> 3) in 4.3.2, the conclusion could be stated more clearly, such as "If
> a nop approach was used, it would require lower layers to know the
> usage model, and would disrupt the calculations ..."
>
> 4) in 4.3.3, I have no idea what this text is saying or why it is
> relevant to the problem at hand. I don't know what an untagged message
> is; I don't know why a WC can only be generated by an untagged
> message. Can you expand on this so readers that are not RDMA experts
> understand what you're talking about? The IESG will need to review and
> approve this document, and to my knowledge none of them are RDMA
> experts.
>
> 5) It would be helpful to explain why taking the bit from RES is
> backwards compatible.
> Explain why older implementation will not have a problem with the new
> bit, and why new implemntations will be able to benefit from the new
> bit.
>
> 6) Rev "MUST be set to two"; would this be better as "two or higher?"
> to allow for a version three+ that would be backward compatible with
> the enhanced features of version two?
>
> 7) I don't understand "The Enhanced Reject function code SHOULD be
> used to indicate a configuration that would have been accepted." Would
> have been accepted except was not because of what condition? Why is
> this only a SHOULD and not a MUST?
>  8) How can the private data be zero length if it must begin with
> Enhanced RDMA Connection establishment data?
> 9) sometimes "Enhanced RDMA Connection Establishment Data" is
> capitalized, sometimes not. Be consistent.
> 10) Received extended ... message SHOULD be reported to the ULP. Why
> is this not a MUST? Anytime there is a SHOULD, the text should
> identify when the rule is not mandatory - what are the valid
> exceptions that make this not a MUST?
> 11) What is "enhanced DDP stream management"?
> 12) "It should be noted that" - you're noting it, so this lead-in
> phrase is totally unnecessary. If you leave it as "It should be noted
> that", then explain WHY it should be noted. If "it is especially
> important to recognize that" then say that, and explain WHY it is
> important to recognize.
> It appears that it is important only because you've made a case for
> peer-to-peer needing to initiate from either side, and the SCTP
> adaption layer allows for that already. So you should probably explain
> why you're duplicating that capability.
> 13) additional legal sequences - but then you don't seem to define any
> **sequences**. Is "Legal Sequences" a special term? If so, please
> explain where it is defined. if not, why is it capitalized?
> 14) I cannot parse "The protocol layers on which RDMA connection
> establishment is layered upon [RFC5040] and [RFC5041] define layers
> and error types."
> 15) if 5043 and 5044 do not define error codes, then how are the error
> codes used? are they carried in messages? which messages?
> 16) In looking for the answer to 15, I found that error codes ARE
> defined in RFC5044 section 8. So your text is apparently wrong.
> 17) in section 9, the responder SHOULD NOT set its IRD higher than
> Initiator ORD. Why is this not a MUST? Under what conditions is it
> acceptable to set this higher than Initiator ORD? What are the
> implications of doing so?
> the responder SHOULD set its ORD lower. What are the valid exceptions
> to this?
> 18) the Initiator MUST set its IRD to a value at least as large as the
> responder ORD if it has sufficient resources
> "MUST if" means this is a SHOULD, and lack of resources is the
> exception. If this is truly a MUST (which the following sentences seem
> to say, requiring a termination if ti doesn't do so, then I would
> remove the if clause from the first sentence, making it a definite
> MUST.
> 19) "Control flag for using" doesn't discuss whatthe values are. Does
> this mean that if A has a value 1, then FULPDU has a zero length?
> This section might be better organized by defining what the fields are
> (control flags and depths), then describing how the values are set.
> You do that for A,B,C but describe IRD and ORD in-line.
> 20) In section 9, "If there is no Standard RDMAP Configuration Data in
> the MPA Reply Frame, and the Enhanced Connection Establishment version
> number is used, it is the equivalent of setting 'A', 'B' and 'C'. "
> Why do we need this extra complexity? Why not require that these be
> set ONE way, not two? I also have concerns about "THE enhanced
> connectiosn establshment version number"; will this document need to
> be updated if a version three is ever developed?
> 21) "Setting no Control Flags in the MPA Reply indicates that an RDMA
> Send message will be required. As this option will require the
> initiator ULP to be involved it SHOULD NOT be used unless necessary. "
> Please define what necessary is. I think this would be better as a
> MUST NOT.
> 22) "Initiator or Responder SHOULD generate the TERM message" - why is
> thi snot a MUST? What are the valid exceptions to this?
> in section 10:
> 23) "An initiator SHOULD NOT use the Enhanced DDP Connection
> Establishment formats or function codes when no enhanced functionality
> is desired. " what are the valid exceptions that make this a SHOULD
> rather than a MUST?
> 24) the paragraph containing "If a responder does not support received
> extended MPA message, then it MUST close the TCP connection and exit
> MPA since MPA frame is improperly formatted for it as stated in
> [RFC5044]." needs a bit of English cleanup.
> 25) section11 This document defines error codes; so does RFC5044.
> Implementers and operators will need to hunt through documents to find
> the various error codes. This document apparently overlooks the error
> codes defined in 5044 section 8. Should there be a registry,
> maintained by IANA, for the MPA error codes?
> section 12:
> 26) I am always concerned when a security coinsiderations section says
> this document does not introduce any new security considerations. This
> is also a red flag during IESG Evaluation. I suggest you document WHY
> these changes do not impact security compared to 5043 and 5044.
> Demonstrate that the WG actually **considered** the security issues.
>
> 27) expand acronyms on first use (MPA, RDMA, RDDP, ULP, iWARP, etc.)
>
> 28) references for sctp, infiniband,
>
> 29) in 4.3.1, "required only for one transport" - mention the one
> transport?
>
> 30) an explanation/definition of MPA Fencing would help readers. Maybe
> a terminology section would be helpful, given all the acronyms, etc.
>
> 31) The information in 4.4 would be helful in the Introduction
> section. This would help to organize the document into "Here's how it
> works now; here is why the client/server model is undesirable in a P2P
> environment; and this is how to modify the existing standard to
> address this problem."
>
> 32) Is section 5 a description of how things currently work, or the
> way this document proposes they work? Section 5 mentions "Enhanced
> Connection Setup"; is that the same as "Enhanced RDMA Connection
> Establishment"? if so, then naming section 5 "Enhanced RDMA Connection
> Establishment" or "Enhanced MPA Connection Setup" might be helful. Use
> terminology consistently, please.
>
> I hope these suggestions help improve the document so we can advance
> it through the approval process quickly,
>
>
>
>
>
>
>
>
>
> David Harrington
> Director, IETF Transport Area
> ietfdbh@comcast.net (preferred for ietf)
> dbharrington@huaweisymantec.com
> +1 603 828 1401 (cell)
>
> _______________________________________________
> storm mailing list
> storm@ietf.org
> https://www.ietf.org/mailman/listinfo/storm
>



-- 
Cheers,
Arkady Kanevsky

--001636ef0485045abe04a5aef01f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

David,<br>version 5 of the draft is posted that responds to your comments.<=
br>Again, thank you very much for detailed comments and suggestions.<br>Ark=
ady<br><br><div class=3D"gmail_quote">On Fri, May 20, 2011 at 11:06 AM, Dav=
id Harrington <span dir=3D"ltr">&lt;<a href=3D"mailto:ietfdbh@comcast.net">=
ietfdbh@comcast.net</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex;">AD Review of draft-ietf-storm-mpa-peer-conn=
ect<br>
<br>
I have reviewed this document and have comments. I think this work is<br>
important, but the way this document explains it could be better.<br>
<br>
My goal as AD is to have this document go through IETF Last Call and<br>
IESG review without any issues delaying advancement.<br>
<br>
Some comments are technical/process comments that refer to text that<br>
will likely cause delays during IESG Evaluation.<br>
<br>
Some are stylistic, such as capitalization and wording consistency,<br>
that are distracting and are likely to make readers wonder &quot;is there<b=
r>
something special here that I need to understand?&quot; and to waste time<b=
r>
trying to determine if there is or is not something special that led<br>
to your capitalizing the text, or using different wording than<br>
elsewhere. These could lead to IESG members raising unnecessary<br>
questions that would delay advancement.<br>
<br>
Others are editorial in nature, mostly about writing clear and<br>
unambiguous text. Generally, if it is simply a small editorial thing,<br>
I ignored it, expecting the RFC Editor to clean up the text a bit. But<br>
sometimes, the lack of clarity could lead to differing interpretations<br>
of the text, or lack of understanding of the text, and those should be<br>
fixed before I bring this to the IESG.<br>
<br>
In some cases, I have asked a question. If I had to ask a question,<br>
then the document apparently didn&#39;t explain it clearly enough. The<br>
answer to the question usually needs to be explained &quot;in the<br>
document&quot;, not just in a private email to me.<br>
<br>
1) idnits complains about unreachable reference:<br>
<a href=3D"http://www.rdmaconsortium.org/home/draft-hilland-iwarp-verbs-v1.=
0-RDMA%0AC.pdf" target=3D"_blank">http://www.rdmaconsortium.org/home/draft-=
hilland-iwarp-verbs-v1.0-RDMA<br>
C.pdf</a><br>
<br>
2) abstracts cannot contain references.<br>
<br>
3) in 4.3.2, the conclusion could be stated more clearly, such as &quot;If<=
br>
a nop approach was used, it would require lower layers to know the<br>
usage model, and would disrupt the calculations ...&quot;<br>
<br>
4) in 4.3.3, I have no idea what this text is saying or why it is<br>
relevant to the problem at hand. I don&#39;t know what an untagged message<=
br>
is; I don&#39;t know why a WC can only be generated by an untagged<br>
message. Can you expand on this so readers that are not RDMA experts<br>
understand what you&#39;re talking about? The IESG will need to review and<=
br>
approve this document, and to my knowledge none of them are RDMA<br>
experts.<br>
<br>
5) It would be helpful to explain why taking the bit from RES is<br>
backwards compatible.<br>
Explain why older implementation will not have a problem with the new<br>
bit, and why new implemntations will be able to benefit from the new<br>
bit.<br>
<br>
6) Rev &quot;MUST be set to two&quot;; would this be better as &quot;two or=
 higher?&quot;<br>
to allow for a version three+ that would be backward compatible with<br>
the enhanced features of version two?<br>
<br>
7) I don&#39;t understand &quot;The Enhanced Reject function code SHOULD be=
<br>
used to indicate a configuration that would have been accepted.&quot; Would=
<br>
have been accepted except was not because of what condition? Why is<br>
this only a SHOULD and not a MUST?<br>
=A08) How can the private data be zero length if it must begin with<br>
Enhanced RDMA Connection establishment data?<br>
9) sometimes &quot;Enhanced RDMA Connection Establishment Data&quot; is<br>
capitalized, sometimes not. Be consistent.<br>
10) Received extended ... message SHOULD be reported to the ULP. Why<br>
is this not a MUST? Anytime there is a SHOULD, the text should<br>
identify when the rule is not mandatory - what are the valid<br>
exceptions that make this not a MUST?<br>
11) What is &quot;enhanced DDP stream management&quot;?<br>
12) &quot;It should be noted that&quot; - you&#39;re noting it, so this lea=
d-in<br>
phrase is totally unnecessary. If you leave it as &quot;It should be noted<=
br>
that&quot;, then explain WHY it should be noted. If &quot;it is especially<=
br>
important to recognize that&quot; then say that, and explain WHY it is<br>
important to recognize.<br>
It appears that it is important only because you&#39;ve made a case for<br>
peer-to-peer needing to initiate from either side, and the SCTP<br>
adaption layer allows for that already. So you should probably explain<br>
why you&#39;re duplicating that capability.<br>
13) additional legal sequences - but then you don&#39;t seem to define any<=
br>
**sequences**. Is &quot;Legal Sequences&quot; a special term? If so, please=
<br>
explain where it is defined. if not, why is it capitalized?<br>
14) I cannot parse &quot;The protocol layers on which RDMA connection<br>
establishment is layered upon [RFC5040] and [RFC5041] define layers<br>
and error types.&quot;<br>
15) if 5043 and 5044 do not define error codes, then how are the error<br>
codes used? are they carried in messages? which messages?<br>
16) In looking for the answer to 15, I found that error codes ARE<br>
defined in RFC5044 section 8. So your text is apparently wrong.<br>
17) in section 9, the responder SHOULD NOT set its IRD higher than<br>
Initiator ORD. Why is this not a MUST? Under what conditions is it<br>
acceptable to set this higher than Initiator ORD? What are the<br>
implications of doing so?<br>
the responder SHOULD set its ORD lower. What are the valid exceptions<br>
to this?<br>
18) the Initiator MUST set its IRD to a value at least as large as the<br>
responder ORD if it has sufficient resources<br>
&quot;MUST if&quot; means this is a SHOULD, and lack of resources is the<br=
>
exception. If this is truly a MUST (which the following sentences seem<br>
to say, requiring a termination if ti doesn&#39;t do so, then I would<br>
remove the if clause from the first sentence, making it a definite<br>
MUST.<br>
19) &quot;Control flag for using&quot; doesn&#39;t discuss whatthe values a=
re. Does<br>
this mean that if A has a value 1, then FULPDU has a zero length?<br>
This section might be better organized by defining what the fields are<br>
(control flags and depths), then describing how the values are set.<br>
You do that for A,B,C but describe IRD and ORD in-line.<br>
20) In section 9, &quot;If there is no Standard RDMAP Configuration Data in=
<br>
the MPA Reply Frame, and the Enhanced Connection Establishment version<br>
number is used, it is the equivalent of setting &#39;A&#39;, &#39;B&#39; an=
d &#39;C&#39;. &quot;<br>
Why do we need this extra complexity? Why not require that these be<br>
set ONE way, not two? I also have concerns about &quot;THE enhanced<br>
connectiosn establshment version number&quot;; will this document need to<b=
r>
be updated if a version three is ever developed?<br>
21) &quot;Setting no Control Flags in the MPA Reply indicates that an RDMA<=
br>
Send message will be required. As this option will require the<br>
initiator ULP to be involved it SHOULD NOT be used unless necessary. &quot;=
<br>
Please define what necessary is. I think this would be better as a<br>
MUST NOT.<br>
22) &quot;Initiator or Responder SHOULD generate the TERM message&quot; - w=
hy is<br>
thi snot a MUST? What are the valid exceptions to this?<br>
in section 10:<br>
23) &quot;An initiator SHOULD NOT use the Enhanced DDP Connection<br>
Establishment formats or function codes when no enhanced functionality<br>
is desired. &quot; what are the valid exceptions that make this a SHOULD<br=
>
rather than a MUST?<br>
24) the paragraph containing &quot;If a responder does not support received=
<br>
extended MPA message, then it MUST close the TCP connection and exit<br>
MPA since MPA frame is improperly formatted for it as stated in<br>
[RFC5044].&quot; needs a bit of English cleanup.<br>
25) section11 This document defines error codes; so does RFC5044.<br>
Implementers and operators will need to hunt through documents to find<br>
the various error codes. This document apparently overlooks the error<br>
codes defined in 5044 section 8. Should there be a registry,<br>
maintained by IANA, for the MPA error codes?<br>
section 12:<br>
26) I am always concerned when a security coinsiderations section says<br>
this document does not introduce any new security considerations. This<br>
is also a red flag during IESG Evaluation. I suggest you document WHY<br>
these changes do not impact security compared to 5043 and 5044.<br>
Demonstrate that the WG actually **considered** the security issues.<br>
<br>
27) expand acronyms on first use (MPA, RDMA, RDDP, ULP, iWARP, etc.)<br>
<br>
28) references for sctp, infiniband,<br>
<br>
29) in 4.3.1, &quot;required only for one transport&quot; - mention the one=
<br>
transport?<br>
<br>
30) an explanation/definition of MPA Fencing would help readers. Maybe<br>
a terminology section would be helpful, given all the acronyms, etc.<br>
<br>
31) The information in 4.4 would be helful in the Introduction<br>
section. This would help to organize the document into &quot;Here&#39;s how=
 it<br>
works now; here is why the client/server model is undesirable in a P2P<br>
environment; and this is how to modify the existing standard to<br>
address this problem.&quot;<br>
<br>
32) Is section 5 a description of how things currently work, or the<br>
way this document proposes they work? Section 5 mentions &quot;Enhanced<br>
Connection Setup&quot;; is that the same as &quot;Enhanced RDMA Connection<=
br>
Establishment&quot;? if so, then naming section 5 &quot;Enhanced RDMA Conne=
ction<br>
Establishment&quot; or &quot;Enhanced MPA Connection Setup&quot; might be h=
elful. Use<br>
terminology consistently, please.<br>
<br>
I hope these suggestions help improve the document so we can advance<br>
it through the approval process quickly,<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
David Harrington<br>
Director, IETF Transport Area<br>
<a href=3D"mailto:ietfdbh@comcast.net">ietfdbh@comcast.net</a> (preferred f=
or ietf)<br>
<a href=3D"mailto:dbharrington@huaweisymantec.com">dbharrington@huaweisyman=
tec.com</a><br>
<a href=3D"tel:%2B1%20603%20828%201401" value=3D"+16038281401">+1 603 828 1=
401</a>=A0(cell)<br>
<br>
_______________________________________________<br>
storm mailing list<br>
<a href=3D"mailto:storm@ietf.org">storm@ietf.org</a><br>
<a href=3D"https://www.ietf.org/mailman/listinfo/storm" target=3D"_blank">h=
ttps://www.ietf.org/mailman/listinfo/storm</a><br>
</blockquote></div><br><br clear=3D"all"><br>-- <br>Cheers,<br>Arkady Kanev=
sky<br>

--001636ef0485045abe04a5aef01f--

From pmacarth@iol.unh.edu  Fri Jun 17 07:38:19 2011
Return-Path: <pmacarth@iol.unh.edu>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B03D211E8182 for <storm@ietfa.amsl.com>; Fri, 17 Jun 2011 07:38:19 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.392
X-Spam-Level: 
X-Spam-Status: No, score=-5.392 tagged_above=-999 required=5 tests=[AWL=1.207,  BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nrLujgCzLLEv for <storm@ietfa.amsl.com>; Fri, 17 Jun 2011 07:38:19 -0700 (PDT)
Received: from exprod5og112.obsmtp.com (exprod5og112.obsmtp.com [64.18.0.24]) by ietfa.amsl.com (Postfix) with SMTP id 87A7111E8087 for <storm@ietf.org>; Fri, 17 Jun 2011 07:38:18 -0700 (PDT)
Received: from postal.iol.unh.edu ([132.177.123.84]) by exprod5ob112.postini.com ([64.18.4.12]) with SMTP ID DSNKTftm2cfsbcnKSwZnNhfZaKH8WqF9bVro@postini.com; Fri, 17 Jun 2011 07:38:18 PDT
Received: from [172.16.0.7] (ofa.iol.unh.edu [132.177.125.245]) (using SSLv3 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by postal.iol.unh.edu (Postfix) with ESMTPSA id 746098F0079; Fri, 17 Jun 2011 10:38:17 -0400 (EDT)
From: Patrick MacArthur <pmacarth@iol.unh.edu>
To: storm@ietf.org
Content-Type: text/plain
Organization: UNH Interoperability Laboratory
Date: Fri, 17 Jun 2011 10:38:17 -0400
Message-Id: <1308321497.3882.10.camel@chiron.ofa>
Mime-Version: 1.0
X-Mailer: Evolution 2.12.3 (2.12.3-19.el5) 
Content-Transfer-Encoding: 7bit
Subject: [storm] Consolidated iSCSI draft and TaskReporting
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 17 Jun 2011 14:38:19 -0000

Hi,

I wanted to ask for clarification on an item appearing in the
consolidated iSCSI draft.

    "All keys defined in [RFC3720] MUST be supported by all
    compliant implementations; a NotUnderstood answer on any of the
    [RFC3720] keys therefore MUST be considered a protocol error and
    handled accordingly. For all other later keys, a NotUnderstood
    answer concludes the negotiation for a negotiated key whereas for
    a declarative key, a NotUnderstood answer simply informs the
    declarer of a lack of comprehension by the receiver."

I would like to request that the references to RFC3720 above be changed
to RFC3720 and RFC5048.  The comments on this list seem to indicate that
the baseline for the iSCSI work is on the combination of RFC 3720 and
RFC 5048, which I interpret to mean that it would be applicable only to
implementations compliant with both RFCs.  An implementation that claims
to be RFC 5048-compliant and responds with TaskReporting=NotUnderstood
is broken IMHO. In that case, is there a reason that we allow a
NotUnderstood response on an RFC 5048 key (TaskReporting)?

If the STORM WG is trying to ensure compatibility with the requirements
of RFC 3720 and RFC 5048 combined, it seems strange to be allowing
implementers not to implement the key requirement of RFC 5048.

Thanks,

-- 
Patrick MacArthur
Research and Development
iSCSI/OFA/IPv6 Consortia
UNH InterOperability Laboratory


From cbm@chadalapaka.com  Fri Jun 17 15:32:15 2011
Return-Path: <cbm@chadalapaka.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 50B0811E80A9 for <storm@ietfa.amsl.com>; Fri, 17 Jun 2011 15:32:15 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id j5rY2iNB0jTx for <storm@ietfa.amsl.com>; Fri, 17 Jun 2011 15:32:14 -0700 (PDT)
Received: from snt0-omc1-s49.snt0.hotmail.com (snt0-omc1-s49.snt0.hotmail.com [65.54.61.86]) by ietfa.amsl.com (Postfix) with ESMTP id 2F9CA11E807F for <storm@ietf.org>; Fri, 17 Jun 2011 15:32:14 -0700 (PDT)
Received: from SNT131-DS21 ([65.55.90.7]) by snt0-omc1-s49.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);  Fri, 17 Jun 2011 15:32:14 -0700
X-Originating-IP: [131.107.0.71]
X-Originating-Email: [cbm@chadalapaka.com]
Message-ID: <SNT131-ds219B987FD6FA0437A497FDA06D0@phx.gbl>
From: Mallikarjun Chadalapaka <cbm@chadalapaka.com>
To: "'Hemal Shah'" <hemal@broadcom.com>, <david.black@emc.com>, <storm@ietf.org>
References: <7C4DFCE962635144B8FAE8CA11D0BF1E0588F426C5@MX14A.corp.emc.com> <SNT131-ds2DE0F0F2022D63A4AD747A0760@phx.gbl> <76DBE161893C324BA6D4BB507EE4C384951235CB38@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds117065D22D7BF8583485DBA0790@phx.gbl> <76DBE161893C324BA6D4BB507EE4C38495124C4828@IRVEXCHCCR02.corp.ad.broadcom.com>
In-Reply-To: <76DBE161893C324BA6D4BB507EE4C38495124C4828@IRVEXCHCCR02.corp.ad.broadcom.com>
Date: Fri, 17 Jun 2011 15:32:12 -0700
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
X-Mailer: Microsoft Outlook 14.0
Thread-Index: AQCUfOqLFecrTE8dOe+/hpcfY+kOJQIRzTq9AgfEPyEC0X+ZOALMKXeKluOzE3A=
Content-Language: en-us
X-OriginalArrivalTime: 17 Jun 2011 22:32:14.0160 (UTC) FILETIME=[67EB5500:01CC2D3E]
Subject: Re: [storm] Plan for iSCSI work
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 17 Jun 2011 22:32:15 -0000

Hi Hemal,

During RFC 5048 effort, I recall we settled on the =93MUST implement, =
but MUST
negotiate=94 formulation after some list deliberation. =20

With plain RFC 3720 semantics, there are some multi-initiator scenarios
where multi-task aborts could essentially lead to target deadlocks, =
waiting
on initiators on third-party sessions that may never respond.  With the
=93Clarified=94 semantics in RFC 5048, the third-party deadlocks =
aren=92t a
problem anymore, but issuing initiator could still experience timeouts =
and
escalate error recovery =96 depending on the number of LUs, sessions,
connections and tasks affected with the issued TMF.    Escalated error
recovery is usually something we want to avoid as it adds more
delays/alerts/logs/failovers/failbacks etc.  Finally, =93Updated=94 =
semantics in
RFC 5048 are meant to address this timeout problem =96 they permit =
targets to
provide accelerated responses and allow them to deal with
book-keeping/quiescing operations in a lazy fashion (which are the real
culprits that trigger timeouts). =20

Given all the above, the WG settled on FastAbort as a MUST-implement.  =
The
"MUST negotiate" part then is to enable backwards-compatibility with RFC
3720 implementations.  At this point, I'd rather not loosen the =
Consolidated
iSCSI draft requirement from what it is in RFC 5048.  I suggest we leave =
it
the way it is.

Mallikarjun






From: Hemal Shah [mailto:hemal@broadcom.com]=20
Sent: Thursday, June 09, 2011 3:57 PM
To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org
Subject: RE: [storm] Plan for iSCSI work

Thanks Mallikarjun!
=A0
The way I read Section 4.1.3 of RFC5048 and Section 4.2.3.4 of the
consolidated draft, the implementation of "FastAbort" feature is a MUST.
>From the first paragraph text from Section 4.1.3 of RFC5048, the first
sentence mandates that all iSCSI implementation comply to this section =
but
the second sentence say that the requirement in this section is =
conditional
to the negotiation of "FastAbort" key.=20
=A0
Protocol behavior defined in this section MUST be implemented by all =
iSCSI
implementations complying with this document. Protocol behavior defined =
in
this section MUST be exhibited by iSCSI implementations on an iSCSI =
session
when they negotiate the TaskReporting (Section 9.1) key to "FastAbort" =
on
that session.
=A0
=A0
This is confusing. I think it will be better to remove the first =
sentence
from 4.2.3.4 and clarify that the requirements in this section is
conditional. I suggest rewording the first two sentences to below:
=A0
       iSCSI implementations may negotiate the TaskReporting (Section =
9.1)
key to "FastAbort" on an iSCSI session. Protocol behavior
defined in this section MUST be exhibited by iSCSI implementations on an
iSCSI session when they negotiate the TaskReporting (Section 9.1) key to
"FastAbort" on that session.
=A0
Hemal
=A0
=A0
-----Original Message-----
From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]=20
Sent: Friday, May 27, 2011 5:29 PM
To: Hemal Shah; david.black@emc.com; storm@ietf.org
Subject: RE: [storm] Plan for iSCSI work
=A0
Hi Hemal,
=A0
The text in this draft came verbatim from section 4.1.3 of RFC 5048.=A0 =
There
have been no changes in this area.
=A0
The new text (as well as the old text) requires the TaskReporting key to =
be
negotiated to "FastAbort" before the multi-task abort semantics can be =
used
on an iSCSI session.
=A0
Thanks.
=A0
Mallikarjun
=A0
=A0
=A0
=A0
=A0 -----Original Message-----
=A0 From: Hemal Shah [mailto:hemal@broadcom.com]
=A0 Sent: Friday, May 27, 2011 4:51 PM
=A0 To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org
=A0 Subject: RE: [storm] Plan for iSCSI work
=A0
=A0 Mallikarjun and David,
=A0
=A0 I noticed one problematic item in the consolidated draft. This item =
is the
=A0 requirement to support FastAbort. This feature was already defined =
in the
=A0 implementation guide, but it was optional. In this draft, it became =
a
=A0 required feature MUST - see in section 4.2.3.4.
=A0
=A0 Do you know why the requirement was changed in the consolidated =
draft?
=A0
=A0 I would like to keep the requirement optional as stated in the
=A0 implementation guide and not break backward compatibility.
=A0
=A0 Hemal
=A0
=A0 -----Original Message-----
=A0 From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On =
Behalf
=A0 Of Mallikarjun Chadalapaka
=A0 Sent: Thursday, May 26, 2011 6:50 PM
=A0 To: david.black@emc.com; storm@ietf.org
=A0 Subject: Re: [storm] Plan for iSCSI work
=A0
=A0 Hi David,
=A0
=A0 The list of changes you have called out are already done in the =
latest
draft.=A0 I
=A0 assume then that you are suggesting that the list itself should be
included
=A0 in the next revision of the draft.
=A0
=A0 Here's what I recall we have done so far:
=A0 1)=A0 iSCSIProtocolLevel specified as "1", and added a related =
normative
=A0 reference to iSCSI-SAM draft
=A0 2)=A0 Markers and related keys were removed
=A0 3)=A0 SPKM authentication and related keys were removed
=A0 4)=A0 Added a new section on responding to obsoleted keys
=A0 5)=A0 Have explicitly allowed initiator+target implementations =
throughout
the
=A0 text
=A0 6)=A0 Clarified that implementations SHOULD NOT rely on SLP-based =
discovery
=A0 7)=A0 Added UML diagrams, and related conventions
=A0
=A0 The above is of course in addition to consolidating the different =
RFCs,
and
=A0 making the related editorial changes.
=A0
=A0 Thanks.
=A0
=A0 Mallikarjun
=A0
=A0
=A0
=A0
=A0=A0=A0 -----Original Message-----
=A0=A0=A0 From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] =
On Behalf
=A0=A0=A0 Of david.black@emc.com
=A0=A0=A0 Sent: Friday, May 20, 2011 2:49 PM
=A0=A0=A0 To: storm@ietf.org
=A0=A0=A0 Subject: [storm] Plan for iSCSI work
=A0
=A0=A0=A0 I thought I'd offer some advance planning/warning on this, as =
the
=A0=A0=A0 consolidated iSCSI draft (draft-ietf-storm-iscsi-cons) is =
large (over
300
=A0=A0=A0 pages).=A0 The current plan is to run a simultaneous WG Last =
Call on both
=A0 this
=A0=A0=A0 draft and the new features draft (draft-ietf-storm-iscsi-sam) =
starting
in
=A0 mid-
=A0=A0=A0 June (probably the week of June 13, after I get back from a =
badly needed
=A0=A0=A0 vacation).=A0 That WG Last Call will run longer than the =
typical 2-week
time
=A0=A0=A0 period, due to the total size of the drafts, but will end by =
July 5th at
the
=A0=A0=A0 latest so that the status of the drafts and the next steps are =
known
prior
=A0 to
=A0=A0=A0 the T10 (SCSI standards) meetings during the week of July =
11.=A0 As July
11th
=A0=A0=A0 is also the draft cutoff deadline for the Quebec City IETF =
meetings,
revised
=A0=A0=A0 draft versions may not show up until that meeting week (week =
of July
=A0=A0=A0 24th).
=A0
=A0=A0=A0 This is also a good point to announce that the storm WG will =
meet in
=A0=A0=A0 Quebec City.
=A0=A0=A0 I've only requested a 1-hour session, as we get most of our =
work done on
=A0=A0=A0 the mailing list.=A0 Among the items for that meeting will be =
figuring out
=A0 what
=A0=A0=A0 to do with the RDMA extensions draft (despite its name,
draft-ietf-storm-
=A0=A0=A0 rdmap-ext-00, it's not currently an official work item for the =
storm
WG).
=A0
=A0=A0=A0 One thing that's missing from the consolidated iSCSI draft =
(and is a
reason
=A0=A0=A0 why we're going to need a -03 version) is the changes that it =
makes to
the
=A0=A0=A0 RFCs that it consolidates.=A0 Off the top of my head, the =
major changes
are:
=A0=A0=A0=A0=A0=A0=A0=A0 - Removal of SPKM authentication
=A0=A0=A0=A0=A0=A0=A0=A0 - Removal of the Marker appendix
=A0=A0=A0=A0=A0=A0=A0=A0 - Removal of the SHOULD requirement for SLP =
implementation.
=A0=A0=A0 Have I missed anything significant?=A0 The summary of this =
will need to be
=A0=A0=A0 added to that draft.
=A0
=A0=A0=A0 WG Last Call will be an opportunity (in fact the final =
opportunity) to
=A0 discuss
=A0=A0=A0 whether anything else should be removed from iSCSI, but =
there's no need
=A0=A0=A0 to wait
=A0=A0=A0 - I encourage people to review both drafts and post comments =
whenever
=A0=A0=A0 they can.
=A0
=A0=A0=A0 In parallel, work will get started on any iSCSI MIB changes =
that are
needed.
=A0=A0=A0 So far, I only see one MIB change - the iSCSIProtocolLevel =
from the new
=A0=A0=A0 features draft needs to be added to the MIB, probably with a =
structure
=A0=A0=A0 analogous to the iSCSI version support that's already in the =
MIB.
=A0
=A0=A0=A0 Thanks,
=A0=A0=A0 --David (storm WG co-chair)
=A0=A0=A0 ----------------------------------------------------
=A0=A0=A0 David L. Black, Distinguished Engineer
=A0=A0=A0 EMC Corporation, 176 South St., Hopkinton, MA=A0 01748
=A0=A0=A0 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 =
(508) 293-7786
=A0=A0=A0 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) =
394-7754
=A0=A0=A0 ----------------------------------------------------
=A0
=A0=A0=A0 _______________________________________________
=A0=A0=A0 storm mailing list
=A0=A0=A0 storm@ietf.org
=A0=A0=A0 https://www.ietf.org/mailman/listinfo/storm
=A0
=A0 _______________________________________________
=A0 storm mailing list
=A0 storm@ietf.org
=A0 https://www.ietf.org/mailman/listinfo/storm
=A0
=A0
=A0
=A0
=A0


From cbm@chadalapaka.com  Fri Jun 17 17:23:58 2011
Return-Path: <cbm@chadalapaka.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3CFBC11E8118 for <storm@ietfa.amsl.com>; Fri, 17 Jun 2011 17:23:58 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wv2Bvxe8eWvU for <storm@ietfa.amsl.com>; Fri, 17 Jun 2011 17:23:57 -0700 (PDT)
Received: from snt0-omc3-s1.snt0.hotmail.com (snt0-omc3-s1.snt0.hotmail.com [65.55.90.140]) by ietfa.amsl.com (Postfix) with ESMTP id 882A611E8089 for <storm@ietf.org>; Fri, 17 Jun 2011 17:23:57 -0700 (PDT)
Received: from SNT131-DS21 ([65.55.90.136]) by snt0-omc3-s1.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);  Fri, 17 Jun 2011 17:23:57 -0700
X-Originating-IP: [131.107.0.94]
X-Originating-Email: [cbm@chadalapaka.com]
Message-ID: <SNT131-ds212BA8CFEE311B42259821A06C0@phx.gbl>
From: Mallikarjun Chadalapaka <cbm@chadalapaka.com>
To: "'Patrick MacArthur'" <pmacarth@iol.unh.edu>, <storm@ietf.org>
References: <1308321497.3882.10.camel@chiron.ofa>
In-Reply-To: <1308321497.3882.10.camel@chiron.ofa>
Date: Fri, 17 Jun 2011 17:23:56 -0700
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Outlook 14.0
Thread-Index: AQDeOsqbPwyi4a8ddBcbQxUWgjUBFZaeCrnw
Content-Language: en-us
X-OriginalArrivalTime: 18 Jun 2011 00:23:57.0238 (UTC) FILETIME=[0343DD60:01CC2D4E]
Subject: Re: [storm] Consolidated iSCSI draft and TaskReporting
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 18 Jun 2011 00:23:58 -0000

Hi Patrick,

I tend to agree with you, you raise a valid point.

The problem lies in special-casing RFC 3720.  I think the right way to
address your comment is to require that an implementation's "NotUnderstood"
response should correlate to the highest protocol version that it has
offered (or would have offered) for the iSCSIProtocolLevel key in a
negotiation.   E.g., let's say if an iSCSI implementation has *offered* a
value of 2 in a negotiation (even though the other side has negotiated it
down to 1), it means that the offering implementation can comprehend all
text keys defined through the spec with iSCSIProtocolLevel=2.  So it must
not respond to any of those text keys with a "NotUnderstood".

So the formal verbiage would look something like the following:

An iSCSI implementation MUST comprehend all text keys defined in iSCSI
Protocol specifications from [RFC3720] through the RFC keyed to the highest
protocol version that the implementation has offered (or would have offered)
for the iSCSIProtocolLevel key in a negotiation.  Returning a NotUnderstood
response on any of these text keys therefore MUST be considered a protocol
error and handled accordingly.  For all other "later" keys, i.e. text keys
defined in specifications keyed to higher values of iSCSIProtocolLevel, a
NotUnderstood  answer concludes the negotiation for a negotiated key whereas
for a declarative key, a NotUnderstood answer simply informs the declarer of
a lack of comprehension by the receiver.

Please let me know if you have comments or questions.  Thanks.

Mallikarjun



  -----Original Message-----
  From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf
  Of Patrick MacArthur
  Sent: Friday, June 17, 2011 7:38 AM
  To: storm@ietf.org
  Subject: [storm] Consolidated iSCSI draft and TaskReporting
  
  Hi,
  
  I wanted to ask for clarification on an item appearing in the consolidated
  iSCSI draft.
  
      "All keys defined in [RFC3720] MUST be supported by all
      compliant implementations; a NotUnderstood answer on any of the
      [RFC3720] keys therefore MUST be considered a protocol error and
      handled accordingly. For all other later keys, a NotUnderstood
      answer concludes the negotiation for a negotiated key whereas for
      a declarative key, a NotUnderstood answer simply informs the
      declarer of a lack of comprehension by the receiver."
  
  I would like to request that the references to RFC3720 above be changed to
  RFC3720 and RFC5048.  The comments on this list seem to indicate that the
  baseline for the iSCSI work is on the combination of RFC 3720 and RFC
  5048, which I interpret to mean that it would be applicable only to
  implementations compliant with both RFCs.  An implementation that
  claims to be RFC 5048-compliant and responds with
  TaskReporting=NotUnderstood is broken IMHO. In that case, is there a
  reason that we allow a NotUnderstood response on an RFC 5048 key
  (TaskReporting)?
  
  If the STORM WG is trying to ensure compatibility with the requirements of
  RFC 3720 and RFC 5048 combined, it seems strange to be allowing
  implementers not to implement the key requirement of RFC 5048.
  
  Thanks,
  
  --
  Patrick MacArthur
  Research and Development
  iSCSI/OFA/IPv6 Consortia
  UNH InterOperability Laboratory
  
  _______________________________________________
  storm mailing list
  storm@ietf.org
  https://www.ietf.org/mailman/listinfo/storm


From david.black@emc.com  Fri Jun 24 18:15:08 2011
Return-Path: <david.black@emc.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 15CCC11E80A4 for <storm@ietfa.amsl.com>; Fri, 24 Jun 2011 18:15:07 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -106.599
X-Spam-Level: 
X-Spam-Status: No, score=-106.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4nzOQn6QHjVd for <storm@ietfa.amsl.com>; Fri, 24 Jun 2011 18:15:05 -0700 (PDT)
Received: from mexforward.lss.emc.com (mexforward.lss.emc.com [128.222.32.20]) by ietfa.amsl.com (Postfix) with ESMTP id A62A711E809C for <storm@ietf.org>; Fri, 24 Jun 2011 18:15:04 -0700 (PDT)
Received: from hop04-l1d11-si01.isus.emc.com (HOP04-L1D11-SI01.isus.emc.com [10.254.111.54]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id p5P1F2tH002726 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for <storm@ietf.org>; Fri, 24 Jun 2011 21:15:02 -0400
Received: from mailhub.lss.emc.com (mailhubhoprd03.lss.emc.com [10.254.221.145]) by hop04-l1d11-si01.isus.emc.com (RSA Interceptor) for <storm@ietf.org>; Fri, 24 Jun 2011 21:14:55 -0400
Received: from mxhub09.corp.emc.com (mxhub09.corp.emc.com [10.254.92.104]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id p5P1DhhS015330 for <storm@ietf.org>; Fri, 24 Jun 2011 21:13:55 -0400
Received: from mx14a.corp.emc.com ([169.254.1.245]) by mxhub09.corp.emc.com ([10.254.92.104]) with mapi; Fri, 24 Jun 2011 21:13:45 -0400
From: <david.black@emc.com>
To: <storm@ietf.org>
Date: Fri, 24 Jun 2011 21:13:43 -0400
Thread-Topic: STORM - Requested session has been scheduled for IETF 81 
Thread-Index: Acwx/tvv878BEVR9Tx2xo5ODrvoaVQA1gJUw
Message-ID: <7C4DFCE962635144B8FAE8CA11D0BF1E058930F6AB@MX14A.corp.emc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-EMM-MHVC: 1
Subject: [storm] FW: STORM - Requested session has been scheduled for IETF 81
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 25 Jun 2011 01:15:08 -0000

QWdlbmRhIHByZXAgY29taW5nIG5leHQgd2Vlaywgc3RheSB0dW5lZCAuLi4gdGhlIGVudGlyZSBh
Z2VuZGEgZm9yIHRoZSBRdWViZWMgQ2l0eSBtZWV0aW5nIGlzIG9uIHRoZSBJRVRGIHdlYiBzaXRl
Lg0KDQpUaGFua3MsDQotLURhdmlkDQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9t
OiBJRVRGIFNlY3JldGFyaWF0IFttYWlsdG86YWdlbmRhQGlldGYub3JnXSANClNlbnQ6IFRodXJz
ZGF5LCBKdW5lIDIzLCAyMDExIDc6MzggUE0NClRvOiBCbGFjaywgRGF2aWQNCkNjOiB0dGFscGV5
QG1pY3Jvc29mdC5jb207IGlldGZkYmhAY29tY2FzdC5uZXQ7IHdlc0BtdGktc3lzdGVtcy5jb207
IHNlc3Npb24tcmVxdWVzdEBpZXRmLm9yZw0KU3ViamVjdDogU1RPUk0gLSBSZXF1ZXN0ZWQgc2Vz
c2lvbiBoYXMgYmVlbiBzY2hlZHVsZWQgZm9yIElFVEYgODEgDQoNCkRlYXIgRGF2aWQgQmxhY2ss
DQoNClRoZSBzZXNzaW9ucyB0aGF0IHlvdSBoYXZlIHJlcXVlc3RlZCBoYXZlIGJlZW4gc2NoZWR1
bGVkLg0KQmVsb3cgaXMgdGhlIHNjaGVkdWxlZCBzZXNzaW9uIGluZm9ybWF0aW9uIGZvbGxvd2Vk
IGJ5IA0KdGhlIGluZm9ybWF0aW9uIG9mIHNlc3Npb25zIHRoYXQgeW91IGhhdmUgcmVxdWVzdGVk
Lg0KDQpTVE9STSBTZXNzaW9uIDEgKDEgaG91cikNClR1ZXNkYXksIEFmdGVybm9vbiBTZXNzaW9u
IElJSSAxNzEwLTE4MTANClJvb20gTmFtZTogMjEwMQ0KLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KDQo=

From hemal@broadcom.com  Tue Jun 28 11:34:10 2011
Return-Path: <hemal@broadcom.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D0C6521F8686 for <storm@ietfa.amsl.com>; Tue, 28 Jun 2011 11:34:10 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.598
X-Spam-Level: 
X-Spam-Status: No, score=-2.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fCnO3HUawH+i for <storm@ietfa.amsl.com>; Tue, 28 Jun 2011 11:34:05 -0700 (PDT)
Received: from mms1.broadcom.com (mms1.broadcom.com [216.31.210.17]) by ietfa.amsl.com (Postfix) with ESMTP id 5021E21F8684 for <storm@ietf.org>; Tue, 28 Jun 2011 11:34:02 -0700 (PDT)
Received: from [10.9.200.133] by mms1.broadcom.com with ESMTP (Broadcom SMTP Relay (Email Firewall v6.3.2)); Tue, 28 Jun 2011 11:38:35 -0700
X-Server-Uuid: 02CED230-5797-4B57-9875-D5D2FEE4708A
Received: from IRVEXCHCCR02.corp.ad.broadcom.com ([10.9.200.183]) by IRVEXCHHUB02.corp.ad.broadcom.com ([10.9.200.133]) with mapi; Tue, 28 Jun 2011 11:33:31 -0700
From: "Hemal Shah" <hemal@broadcom.com>
To: "Mallikarjun Chadalapaka" <cbm@chadalapaka.com>, "david.black@emc.com" <david.black@emc.com>, "storm@ietf.org" <storm@ietf.org>
Date: Tue, 28 Jun 2011 11:32:39 -0700
Thread-Topic: [storm] Plan for iSCSI work
Thread-Index: AQCUfOqLFecrTE8dOe+/hpcfY+kOJQIRzTq9AgfEPyEC0X+ZOALMKXeKluOzE3CAEQD0gA==
Message-ID: <76DBE161893C324BA6D4BB507EE4C3849513370AB5@IRVEXCHCCR02.corp.ad.broadcom.com>
References: <7C4DFCE962635144B8FAE8CA11D0BF1E0588F426C5@MX14A.corp.emc.com> <SNT131-ds2DE0F0F2022D63A4AD747A0760@phx.gbl> <76DBE161893C324BA6D4BB507EE4C384951235CB38@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds117065D22D7BF8583485DBA0790@phx.gbl> <76DBE161893C324BA6D4BB507EE4C38495124C4828@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds219B987FD6FA0437A497FDA06D0@phx.gbl>
In-Reply-To: <SNT131-ds219B987FD6FA0437A497FDA06D0@phx.gbl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
MIME-Version: 1.0
X-WSS-ID: 6214C0213B410223401-01-01
Content-Type: multipart/alternative; boundary=_000_76DBE161893C324BA6D4BB507EE4C3849513370AB5IRVEXCHCCR02c_
Subject: Re: [storm] Plan for iSCSI work
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 28 Jun 2011 18:34:10 -0000

--_000_76DBE161893C324BA6D4BB507EE4C3849513370AB5IRVEXCHCCR02c_
Content-Type: text/plain;
 charset=us-ascii
Content-Transfer-Encoding: quoted-printable

Thanks Mallikarjun! I would still like to confirm my understanding of what =
you describe as "MUST implement, but MUST negotiate".

Here is my understanding:

1.      The implementation of TaskReporting is required and NotUnderstood r=
esponse for TaskReporting key is not permitted based on the following text =
from RFC5048 and Section 6.2 of the consolidated draft:
                All keys defined in [RFC3720] MUST be supported by all
            compliant implementations; a NotUnderstood answer on any of the
            [RFC3720] keys therefore MUST be considered a protocol error an=
d
            handled accordingly.
2.      To enable FastAbort on a session, the negotiation of TaskReporting =
key is required.
3.      Only when the TaskReporting=3DFastAbort functionality is supported,=
 the protocol behavior specified in Section 4.2.3.4 is required to be imple=
mented and exhibited. There are several places in the spec that indicate th=
is type of conditionality.
a.      Section 4.2.3.5: If an iSCSI target implementation is capable of su=
pporting
TaskReporting=3DFastAbort functionality (Section 13.23)...
b.      Section 4.2.2.3.3: Whenever the TaskReporting key (Section 12.23 "T=
ask Reporting") is negotiated to ResponseFence or FastAbort for an iSCSI se=
ssion...
c.      Section 4.2.2.3.4: Whenever the TaskReporting key (Section 13.23) i=
s negotiated to ResponseFence or FastAbort for an iSCSI session...
4.      If TaskReporting=3DFastAbort functionality is not supported, then t=
he protocol behavior specified in Section 4.2.3.4 is not required to be imp=
lemented or exhibited. If this is true, then the first sentence in section =
4.2.3.4 is misleading and should be removed or prefixed with some text like=
 "Only when the TaskReporting=3DFastAbort functionality is supported..".

Hemal


-----Original Message-----
From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
Sent: Friday, June 17, 2011 3:32 PM
To: Hemal Shah; david.black@emc.com; storm@ietf.org
Subject: RE: [storm] Plan for iSCSI work

Hi Hemal,

During RFC 5048 effort, I recall we settled on the "MUST implement, but MUS=
T
negotiate" formulation after some list deliberation.

With plain RFC 3720 semantics, there are some multi-initiator scenarios
where multi-task aborts could essentially lead to target deadlocks, waiting
on initiators on third-party sessions that may never respond.  With the
"Clarified" semantics in RFC 5048, the third-party deadlocks aren't a
problem anymore, but issuing initiator could still experience timeouts and
escalate error recovery - depending on the number of LUs, sessions,
connections and tasks affected with the issued TMF.    Escalated error
recovery is usually something we want to avoid as it adds more
delays/alerts/logs/failovers/failbacks etc.  Finally, "Updated" semantics i=
n
RFC 5048 are meant to address this timeout problem - they permit targets to
provide accelerated responses and allow them to deal with
book-keeping/quiescing operations in a lazy fashion (which are the real
culprits that trigger timeouts).

Given all the above, the WG settled on FastAbort as a MUST-implement.  The
"MUST negotiate" part then is to enable backwards-compatibility with RFC
3720 implementations.  At this point, I'd rather not loosen the Consolidate=
d
iSCSI draft requirement from what it is in RFC 5048.  I suggest we leave it
the way it is.

Mallikarjun






From: Hemal Shah [mailto:hemal@broadcom.com]
Sent: Thursday, June 09, 2011 3:57 PM
To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org
Subject: RE: [storm] Plan for iSCSI work

Thanks Mallikarjun!

The way I read Section 4.1.3 of RFC5048 and Section 4.2.3.4 of the
consolidated draft, the implementation of "FastAbort" feature is a MUST.
>From the first paragraph text from Section 4.1.3 of RFC5048, the first
sentence mandates that all iSCSI implementation comply to this section but
the second sentence say that the requirement in this section is conditional
to the negotiation of "FastAbort" key.

Protocol behavior defined in this section MUST be implemented by all iSCSI
implementations complying with this document. Protocol behavior defined in
this section MUST be exhibited by iSCSI implementations on an iSCSI session
when they negotiate the TaskReporting (Section 9.1) key to "FastAbort" on
that session.


This is confusing. I think it will be better to remove the first sentence
from 4.2.3.4 and clarify that the requirements in this section is
conditional. I suggest rewording the first two sentences to below:

       iSCSI implementations may negotiate the TaskReporting (Section 9.1)
key to "FastAbort" on an iSCSI session. Protocol behavior
defined in this section MUST be exhibited by iSCSI implementations on an
iSCSI session when they negotiate the TaskReporting (Section 9.1) key to
"FastAbort" on that session.

Hemal


-----Original Message-----
From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
Sent: Friday, May 27, 2011 5:29 PM
To: Hemal Shah; david.black@emc.com; storm@ietf.org
Subject: RE: [storm] Plan for iSCSI work

Hi Hemal,

The text in this draft came verbatim from section 4.1.3 of RFC 5048.  There
have been no changes in this area.

The new text (as well as the old text) requires the TaskReporting key to be
negotiated to "FastAbort" before the multi-task abort semantics can be used
on an iSCSI session.

Thanks.

Mallikarjun




  -----Original Message-----
  From: Hemal Shah [mailto:hemal@broadcom.com]
  Sent: Friday, May 27, 2011 4:51 PM
  To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org
  Subject: RE: [storm] Plan for iSCSI work

  Mallikarjun and David,

  I noticed one problematic item in the consolidated draft. This item is th=
e
  requirement to support FastAbort. This feature was already defined in the
  implementation guide, but it was optional. In this draft, it became a
  required feature MUST - see in section 4.2.3.4.

  Do you know why the requirement was changed in the consolidated draft?

  I would like to keep the requirement optional as stated in the
  implementation guide and not break backward compatibility.

  Hemal

  -----Original Message-----
  From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf
  Of Mallikarjun Chadalapaka
  Sent: Thursday, May 26, 2011 6:50 PM
  To: david.black@emc.com; storm@ietf.org
  Subject: Re: [storm] Plan for iSCSI work

  Hi David,

  The list of changes you have called out are already done in the latest
draft.  I
  assume then that you are suggesting that the list itself should be
included
  in the next revision of the draft.

  Here's what I recall we have done so far:
  1)  iSCSIProtocolLevel specified as "1", and added a related normative
  reference to iSCSI-SAM draft
  2)  Markers and related keys were removed
  3)  SPKM authentication and related keys were removed
  4)  Added a new section on responding to obsoleted keys
  5)  Have explicitly allowed initiator+target implementations throughout
the
  text
  6)  Clarified that implementations SHOULD NOT rely on SLP-based discovery
  7)  Added UML diagrams, and related conventions

  The above is of course in addition to consolidating the different RFCs,
and
  making the related editorial changes.

  Thanks.

  Mallikarjun




    -----Original Message-----
    From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf
    Of david.black@emc.com
    Sent: Friday, May 20, 2011 2:49 PM
    To: storm@ietf.org
    Subject: [storm] Plan for iSCSI work

    I thought I'd offer some advance planning/warning on this, as the
    consolidated iSCSI draft (draft-ietf-storm-iscsi-cons) is large (over
300
    pages).  The current plan is to run a simultaneous WG Last Call on both
  this
    draft and the new features draft (draft-ietf-storm-iscsi-sam) starting
in
  mid-
    June (probably the week of June 13, after I get back from a badly neede=
d
    vacation).  That WG Last Call will run longer than the typical 2-week
time
    period, due to the total size of the drafts, but will end by July 5th a=
t
the
    latest so that the status of the drafts and the next steps are known
prior
  to
    the T10 (SCSI standards) meetings during the week of July 11.  As July
11th
    is also the draft cutoff deadline for the Quebec City IETF meetings,
revised
    draft versions may not show up until that meeting week (week of July
    24th).

    This is also a good point to announce that the storm WG will meet in
    Quebec City.
    I've only requested a 1-hour session, as we get most of our work done o=
n
    the mailing list.  Among the items for that meeting will be figuring ou=
t
  what
    to do with the RDMA extensions draft (despite its name,
draft-ietf-storm-
    rdmap-ext-00, it's not currently an official work item for the storm
WG).

    One thing that's missing from the consolidated iSCSI draft (and is a
reason
    why we're going to need a -03 version) is the changes that it makes to
the
    RFCs that it consolidates.  Off the top of my head, the major changes
are:
         - Removal of SPKM authentication
         - Removal of the Marker appendix
         - Removal of the SHOULD requirement for SLP implementation.
    Have I missed anything significant?  The summary of this will need to b=
e
    added to that draft.

    WG Last Call will be an opportunity (in fact the final opportunity) to
  discuss
    whether anything else should be removed from iSCSI, but there's no need
    to wait
    - I encourage people to review both drafts and post comments whenever
    they can.

    In parallel, work will get started on any iSCSI MIB changes that are
needed.
    So far, I only see one MIB change - the iSCSIProtocolLevel from the new
    features draft needs to be added to the MIB, probably with a structure
    analogous to the iSCSI version support that's already in the MIB.

    Thanks,
    --David (storm WG co-chair)
    ----------------------------------------------------
    David L. Black, Distinguished Engineer
    EMC Corporation, 176 South St., Hopkinton, MA  01748
    +1 (508) 293-7953             FAX: +1 (508) 293-7786
    david.black@emc.com        Mobile: +1 (978) 394-7754
    ----------------------------------------------------

    _______________________________________________
    storm mailing list
    storm@ietf.org
    https://www.ietf.org/mailman/listinfo/storm

  _______________________________________________
  storm mailing list
  storm@ietf.org
  https://www.ietf.org/mailman/listinfo/storm









--_000_76DBE161893C324BA6D4BB507EE4C3849513370AB5IRVEXCHCCR02c_
Content-Type: text/html;
 charset=us-ascii
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left:=
 #800000 2px solid; } --></style>
</head>
<body>
<font face=3D"Consolas, monospace" size=3D"2">
<div>Thanks Mallikarjun! I would still like to confirm my understanding of =
what you describe as &quot;MUST implement, but MUST negotiate&quot;.</div>
<div>&nbsp;</div>
<div>Here is my understanding:</div>
<div>&nbsp;</div>
<ol style=3D"margin-top: 0pt; margin-bottom: 0pt; margin-left: 36pt; ">
<li>The implementation of TaskReporting is required and NotUnderstood respo=
nse for TaskReporting key is not permitted based on the following text from=
 RFC5048 and Section 6.2 of the consolidated draft:</li></ol>
<div><font face=3D"Courier New, monospace" size=3D"2">&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
 All keys defined in [RFC3720] MUST be supported by all</font></div>
<div><font face=3D"Courier New, monospace" size=3D"2">&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;compliant implementation=
s; a NotUnderstood answer on any of the</font></div>
<div><font face=3D"Courier New, monospace" size=3D"2">&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[RFC3720] keys therefore=
 MUST be considered a protocol error and</font></div>
<div><font face=3D"Courier New, monospace" size=3D"2">&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handled accordingly.</fo=
nt></div>
<ol start=3D"2" style=3D"margin-top: 0pt; margin-bottom: 0pt; margin-left: =
36pt; ">
<li>To enable FastAbort on a session, the negotiation of TaskReporting key =
is required.</li><li>Only when the TaskReporting=3DFastAbort functionality =
is supported, the protocol behavior specified in Section 4.2.3.4 is require=
d to be implemented and exhibited. There are several places in the spec tha=
t indicate this type of conditionality.</li></ol>
<ol type=3D"a" style=3D"margin-top: 0pt; margin-bottom: 0pt; margin-left: 7=
2pt; ">
<font face=3D"Courier New, monospace" size=3D"2">
<li>Section 4.2.3.5: If an iSCSI target implementation is capable of suppor=
ting</li></font>
</ol>
<div style=3D"padding-left: 72pt; "><font face=3D"Courier New, monospace" s=
ize=3D"2">TaskReporting=3DFastAbort functionality (Section 13.23)&#8230;</f=
ont></div>
<ol type=3D"a" start=3D"2" style=3D"margin-top: 0pt; margin-bottom: 0pt; ma=
rgin-left: 72pt; ">
<font face=3D"Courier New, monospace" size=3D"2">
<li>Section 4.2.2.3.3: Whenever the TaskReporting key (Section 12.23 &quot;=
Task Reporting&quot;) is negotiated to ResponseFence or FastAbort for an iS=
CSI session&#8230;</li><li>Section 4.2.2.3.4: Whenever the TaskReporting ke=
y (Section 13.23) is negotiated to ResponseFence or FastAbort for an iSCSI =
session&#8230;</li></font>
</ol>
<ol start=3D"4" style=3D"margin-top: 0pt; margin-bottom: 0pt; margin-left: =
36pt; ">
<li>If TaskReporting=3DFastAbort functionality is not supported, then the p=
rotocol behavior specified in Section 4.2.3.4 is not required to be impleme=
nted or exhibited. If this is true, then the first sentence in section 4.2.=
3.4 is misleading and should be removed
or prefixed with some text like &#8220;Only when the TaskReporting=3DFastAb=
ort functionality is supported..&#8221;.</li></ol>
<div>&nbsp;</div>
<div>Hemal</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>-----Original Message-----<br>

From: Mallikarjun Chadalapaka [<a href=3D"mailto:cbm@chadalapaka.com">mailt=
o:cbm@chadalapaka.com</a>]
<br>

Sent: Friday, June 17, 2011 3:32 PM<br>

To: Hemal Shah; david.black@emc.com; storm@ietf.org<br>

Subject: RE: [storm] Plan for iSCSI work</div>
<div>&nbsp;</div>
<div>Hi Hemal,</div>
<div>&nbsp;</div>
<div>During RFC 5048 effort, I recall we settled on the &#8220;MUST impleme=
nt, but MUST</div>
<div>negotiate&#8221; formulation after some list deliberation.&nbsp; </div=
>
<div>&nbsp;</div>
<div>With plain RFC 3720 semantics, there are some multi-initiator scenario=
s</div>
<div>where multi-task aborts could essentially lead to target deadlocks, wa=
iting</div>
<div>on initiators on third-party sessions that may never respond.&nbsp; Wi=
th the</div>
<div>&#8220;Clarified&#8221; semantics in RFC 5048, the third-party deadloc=
ks aren&#8217;t a</div>
<div>problem anymore, but issuing initiator could still experience timeouts=
 and</div>
<div>escalate error recovery &#8211; depending on the number of LUs, sessio=
ns,</div>
<div>connections and tasks affected with the issued TMF.&nbsp;&nbsp;&nbsp; =
Escalated error</div>
<div>recovery is usually something we want to avoid as it adds more</div>
<div>delays/alerts/logs/failovers/failbacks etc.&nbsp; Finally, &#8220;Upda=
ted&#8221; semantics in</div>
<div>RFC 5048 are meant to address this timeout problem &#8211; they permit=
 targets to</div>
<div>provide accelerated responses and allow them to deal with</div>
<div>book-keeping/quiescing operations in a lazy fashion (which are the rea=
l</div>
<div>culprits that trigger timeouts).&nbsp; </div>
<div>&nbsp;</div>
<div>Given all the above, the WG settled on FastAbort as a MUST-implement.&=
nbsp; The</div>
<div>&quot;MUST negotiate&quot; part then is to enable backwards-compatibil=
ity with RFC</div>
<div>3720 implementations.&nbsp; At this point, I'd rather not loosen the C=
onsolidated</div>
<div>iSCSI draft requirement from what it is in RFC 5048.&nbsp; I suggest w=
e leave it</div>
<div>the way it is.</div>
<div>&nbsp;</div>
<div>Mallikarjun</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>From: Hemal Shah [<a href=3D"mailto:hemal@broadcom.com">mailto:hemal@b=
roadcom.com</a>] </div>
<div>Sent: Thursday, June 09, 2011 3:57 PM</div>
<div>To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org</div>
<div>Subject: RE: [storm] Plan for iSCSI work</div>
<div>&nbsp;</div>
<div>Thanks Mallikarjun!</div>
<div>&nbsp;</div>
<div>The way I read Section 4.1.3 of RFC5048 and Section 4.2.3.4 of the</di=
v>
<div>consolidated draft, the implementation of &quot;FastAbort&quot; featur=
e is a MUST.</div>
<div>From the first paragraph text from Section 4.1.3 of RFC5048, the first=
</div>
<div>sentence mandates that all iSCSI implementation comply to this section=
 but</div>
<div>the second sentence say that the requirement in this section is condit=
ional</div>
<div>to the negotiation of &quot;FastAbort&quot; key. </div>
<div>&nbsp;</div>
<div>Protocol behavior defined in this section MUST be implemented by all i=
SCSI</div>
<div>implementations complying with this document. Protocol behavior define=
d in</div>
<div>this section MUST be exhibited by iSCSI implementations on an iSCSI se=
ssion</div>
<div>when they negotiate the TaskReporting (Section 9.1) key to &quot;FastA=
bort&quot; on</div>
<div>that session.</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>This is confusing. I think it will be better to remove the first sente=
nce</div>
<div>from 4.2.3.4 and clarify that the requirements in this section is</div=
>
<div>conditional. I suggest rewording the first two sentences to below:</di=
v>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; iSCSI implementations may negotia=
te the TaskReporting (Section 9.1)</div>
<div>key to &quot;FastAbort&quot; on an iSCSI session. Protocol behavior</d=
iv>
<div>defined in this section MUST be exhibited by iSCSI implementations on =
an</div>
<div>iSCSI session when they negotiate the TaskReporting (Section 9.1) key =
to</div>
<div>&quot;FastAbort&quot; on that session.</div>
<div>&nbsp;</div>
<div>Hemal</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>-----Original Message-----</div>
<div>From: Mallikarjun Chadalapaka [<a href=3D"mailto:cbm@chadalapaka.com">=
mailto:cbm@chadalapaka.com</a>] </div>
<div>Sent: Friday, May 27, 2011 5:29 PM</div>
<div>To: Hemal Shah; david.black@emc.com; storm@ietf.org</div>
<div>Subject: RE: [storm] Plan for iSCSI work</div>
<div>&nbsp;</div>
<div>Hi Hemal,</div>
<div>&nbsp;</div>
<div>The text in this draft came verbatim from section 4.1.3 of RFC 5048.&n=
bsp; There</div>
<div>have been no changes in this area.</div>
<div>&nbsp;</div>
<div>The new text (as well as the old text) requires the TaskReporting key =
to be</div>
<div>negotiated to &quot;FastAbort&quot; before the multi-task abort semant=
ics can be used</div>
<div>on an iSCSI session.</div>
<div>&nbsp;</div>
<div>Thanks.</div>
<div>&nbsp;</div>
<div>Mallikarjun</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp; -----Original Message-----</div>
<div>&nbsp; From: Hemal Shah [<a href=3D"mailto:hemal@broadcom.com">mailto:=
hemal@broadcom.com</a>]</div>
<div>&nbsp; Sent: Friday, May 27, 2011 4:51 PM</div>
<div>&nbsp; To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.or=
g</div>
<div>&nbsp; Subject: RE: [storm] Plan for iSCSI work</div>
<div>&nbsp;</div>
<div>&nbsp; Mallikarjun and David,</div>
<div>&nbsp;</div>
<div>&nbsp; I noticed one problematic item in the consolidated draft. This =
item is the</div>
<div>&nbsp; requirement to support FastAbort. This feature was already defi=
ned in the</div>
<div>&nbsp; implementation guide, but it was optional. In this draft, it be=
came a</div>
<div>&nbsp; required feature MUST - see in section 4.2.3.4.</div>
<div>&nbsp;</div>
<div>&nbsp; Do you know why the requirement was changed in the consolidated=
 draft?</div>
<div>&nbsp;</div>
<div>&nbsp; I would like to keep the requirement optional as stated in the<=
/div>
<div>&nbsp; implementation guide and not break backward compatibility.</div=
>
<div>&nbsp;</div>
<div>&nbsp; Hemal</div>
<div>&nbsp;</div>
<div>&nbsp; -----Original Message-----</div>
<div>&nbsp; From: storm-bounces@ietf.org [<a href=3D"mailto:storm-bounces@i=
etf.org">mailto:storm-bounces@ietf.org</a>] On Behalf</div>
<div>&nbsp; Of Mallikarjun Chadalapaka</div>
<div>&nbsp; Sent: Thursday, May 26, 2011 6:50 PM</div>
<div>&nbsp; To: david.black@emc.com; storm@ietf.org</div>
<div>&nbsp; Subject: Re: [storm] Plan for iSCSI work</div>
<div>&nbsp;</div>
<div>&nbsp; Hi David,</div>
<div>&nbsp;</div>
<div>&nbsp; The list of changes you have called out are already done in the=
 latest</div>
<div>draft.&nbsp; I</div>
<div>&nbsp; assume then that you are suggesting that the list itself should=
 be</div>
<div>included</div>
<div>&nbsp; in the next revision of the draft.</div>
<div>&nbsp;</div>
<div>&nbsp; Here's what I recall we have done so far:</div>
<div>&nbsp; 1)&nbsp; iSCSIProtocolLevel specified as &quot;1&quot;, and add=
ed a related normative</div>
<div>&nbsp; reference to iSCSI-SAM draft</div>
<div>&nbsp; 2)&nbsp; Markers and related keys were removed</div>
<div>&nbsp; 3)&nbsp; SPKM authentication and related keys were removed</div=
>
<div>&nbsp; 4)&nbsp; Added a new section on responding to obsoleted keys</d=
iv>
<div>&nbsp; 5)&nbsp; Have explicitly allowed initiator&#43;target implement=
ations throughout</div>
<div>the</div>
<div>&nbsp; text</div>
<div>&nbsp; 6)&nbsp; Clarified that implementations SHOULD NOT rely on SLP-=
based discovery</div>
<div>&nbsp; 7)&nbsp; Added UML diagrams, and related conventions</div>
<div>&nbsp;</div>
<div>&nbsp; The above is of course in addition to consolidating the differe=
nt RFCs,</div>
<div>and</div>
<div>&nbsp; making the related editorial changes.</div>
<div>&nbsp;</div>
<div>&nbsp; Thanks.</div>
<div>&nbsp;</div>
<div>&nbsp; Mallikarjun</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; -----Original Message-----</div>
<div>&nbsp;&nbsp;&nbsp; From: storm-bounces@ietf.org [<a href=3D"mailto:sto=
rm-bounces@ietf.org">mailto:storm-bounces@ietf.org</a>] On Behalf</div>
<div>&nbsp;&nbsp;&nbsp; Of david.black@emc.com</div>
<div>&nbsp;&nbsp;&nbsp; Sent: Friday, May 20, 2011 2:49 PM</div>
<div>&nbsp;&nbsp;&nbsp; To: storm@ietf.org</div>
<div>&nbsp;&nbsp;&nbsp; Subject: [storm] Plan for iSCSI work</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; I thought I'd offer some advance planning/warning o=
n this, as the</div>
<div>&nbsp;&nbsp;&nbsp; consolidated iSCSI draft (draft-ietf-storm-iscsi-co=
ns) is large (over</div>
<div>300</div>
<div>&nbsp;&nbsp;&nbsp; pages).&nbsp; The current plan is to run a simultan=
eous WG Last Call on both</div>
<div>&nbsp; this</div>
<div>&nbsp;&nbsp;&nbsp; draft and the new features draft (draft-ietf-storm-=
iscsi-sam) starting</div>
<div>in</div>
<div>&nbsp; mid-</div>
<div>&nbsp;&nbsp;&nbsp; June (probably the week of June 13, after I get bac=
k from a badly needed</div>
<div>&nbsp;&nbsp;&nbsp; vacation).&nbsp; That WG Last Call will run longer =
than the typical 2-week</div>
<div>time</div>
<div>&nbsp;&nbsp;&nbsp; period, due to the total size of the drafts, but wi=
ll end by July 5th at</div>
<div>the</div>
<div>&nbsp;&nbsp;&nbsp; latest so that the status of the drafts and the nex=
t steps are known</div>
<div>prior</div>
<div>&nbsp; to</div>
<div>&nbsp;&nbsp;&nbsp; the T10 (SCSI standards) meetings during the week o=
f July 11.&nbsp; As July</div>
<div>11th</div>
<div>&nbsp;&nbsp;&nbsp; is also the draft cutoff deadline for the Quebec Ci=
ty IETF meetings,</div>
<div>revised</div>
<div>&nbsp;&nbsp;&nbsp; draft versions may not show up until that meeting w=
eek (week of July</div>
<div>&nbsp;&nbsp;&nbsp; 24th).</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; This is also a good point to announce that the stor=
m WG will meet in</div>
<div>&nbsp;&nbsp;&nbsp; Quebec City.</div>
<div>&nbsp;&nbsp;&nbsp; I've only requested a 1-hour session, as we get mos=
t of our work done on</div>
<div>&nbsp;&nbsp;&nbsp; the mailing list.&nbsp; Among the items for that me=
eting will be figuring out</div>
<div>&nbsp; what</div>
<div>&nbsp;&nbsp;&nbsp; to do with the RDMA extensions draft (despite its n=
ame,</div>
<div>draft-ietf-storm-</div>
<div>&nbsp;&nbsp;&nbsp; rdmap-ext-00, it's not currently an official work i=
tem for the storm</div>
<div>WG).</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; One thing that's missing from the consolidated iSCS=
I draft (and is a</div>
<div>reason</div>
<div>&nbsp;&nbsp;&nbsp; why we're going to need a -03 version) is the chang=
es that it makes to</div>
<div>the</div>
<div>&nbsp;&nbsp;&nbsp; RFCs that it consolidates.&nbsp; Off the top of my =
head, the major changes</div>
<div>are:</div>
<div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Removal of SPKM aut=
hentication</div>
<div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Removal of the Mark=
er appendix</div>
<div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Removal of the SHOU=
LD requirement for SLP implementation.</div>
<div>&nbsp;&nbsp;&nbsp; Have I missed anything significant?&nbsp; The summa=
ry of this will need to be</div>
<div>&nbsp;&nbsp;&nbsp; added to that draft.</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; WG Last Call will be an opportunity (in fact the fi=
nal opportunity) to</div>
<div>&nbsp; discuss</div>
<div>&nbsp;&nbsp;&nbsp; whether anything else should be removed from iSCSI,=
 but there's no need</div>
<div>&nbsp;&nbsp;&nbsp; to wait</div>
<div>&nbsp;&nbsp;&nbsp; - I encourage people to review both drafts and post=
 comments whenever</div>
<div>&nbsp;&nbsp;&nbsp; they can.</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; In parallel, work will get started on any iSCSI MIB=
 changes that are</div>
<div>needed.</div>
<div>&nbsp;&nbsp;&nbsp; So far, I only see one MIB change - the iSCSIProtoc=
olLevel from the new</div>
<div>&nbsp;&nbsp;&nbsp; features draft needs to be added to the MIB, probab=
ly with a structure</div>
<div>&nbsp;&nbsp;&nbsp; analogous to the iSCSI version support that's alrea=
dy in the MIB.</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; Thanks,</div>
<div>&nbsp;&nbsp;&nbsp; --David (storm WG co-chair)</div>
<div>&nbsp;&nbsp;&nbsp; ---------------------------------------------------=
-</div>
<div>&nbsp;&nbsp;&nbsp; David L. Black, Distinguished Engineer</div>
<div>&nbsp;&nbsp;&nbsp; EMC Corporation, 176 South St., Hopkinton, MA&nbsp;=
 01748</div>
<div>&nbsp;&nbsp;&nbsp; &#43;1 (508) 293-7953&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; FAX: &#43;1 (508) 293-7786</div>
<div>&nbsp;&nbsp;&nbsp; david.black@emc.com&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; Mobile: &#43;1 (978) 394-7754</div>
<div>&nbsp;&nbsp;&nbsp; ---------------------------------------------------=
-</div>
<div>&nbsp;</div>
<div>&nbsp;&nbsp;&nbsp; _______________________________________________</di=
v>
<div>&nbsp;&nbsp;&nbsp; storm mailing list</div>
<div>&nbsp;&nbsp;&nbsp; storm@ietf.org</div>
<div>&nbsp;&nbsp;&nbsp; <a href=3D"https://www.ietf.org/mailman/listinfo/st=
orm">https://www.ietf.org/mailman/listinfo/storm</a></div>
<div>&nbsp;</div>
<div>&nbsp; _______________________________________________</div>
<div>&nbsp; storm mailing list</div>
<div>&nbsp; storm@ietf.org</div>
<div>&nbsp; <a href=3D"https://www.ietf.org/mailman/listinfo/storm">https:/=
/www.ietf.org/mailman/listinfo/storm</a></div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
</font>
</body>
</html>

--_000_76DBE161893C324BA6D4BB507EE4C3849513370AB5IRVEXCHCCR02c_--


From nezhinsky@gmail.com  Wed Jun 29 04:58:45 2011
Return-Path: <nezhinsky@gmail.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 60C4B21F86E0 for <storm@ietfa.amsl.com>; Wed, 29 Jun 2011 04:58:45 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.598
X-Spam-Level: 
X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id e-QthgmyclCY for <storm@ietfa.amsl.com>; Wed, 29 Jun 2011 04:58:43 -0700 (PDT)
Received: from mail-gx0-f172.google.com (mail-gx0-f172.google.com [209.85.161.172]) by ietfa.amsl.com (Postfix) with ESMTP id 6774F21F86CE for <storm@ietf.org>; Wed, 29 Jun 2011 04:58:43 -0700 (PDT)
Received: by gxk19 with SMTP id 19so558817gxk.31 for <storm@ietf.org>; Wed, 29 Jun 2011 04:58:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=B2jQtCSre3/4HQQ1mdORE9Ge9ZeHmCWjYq0DaO4auOo=; b=NvFqme57SljSX0YgHGqZ1ymG0nZK3n+mP7OK+r8D70IA4n4/AbzUq5FT35TfreMVyb 6lLnHwMa3hscjw8f3vBuqY9X0qdzRFFBRhzWDQBH/MVPUtfIQnN41HoHxXqoGFgG0T5z QsAIBU4OHJ1P5fLlNiuaFk2iLaVjbEKNyHYN4=
MIME-Version: 1.0
Received: by 10.236.116.101 with SMTP id f65mr802005yhh.32.1309348722487; Wed, 29 Jun 2011 04:58:42 -0700 (PDT)
Received: by 10.236.203.196 with HTTP; Wed, 29 Jun 2011 04:58:42 -0700 (PDT)
Date: Wed, 29 Jun 2011 14:58:42 +0300
Message-ID: <BANLkTi=GVHUBdsZiwsRXxETd_rU=F8f85w@mail.gmail.com>
From: Alexander Nezhinsky <nezhinsky@gmail.com>
To: storm@ietf.org, david.black@emc.com, michael@huaweisymantec.com
Content-Type: multipart/alternative; boundary=20cf303ddb567db33a04a6d88190
Subject: [storm] iSER draft
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 29 Jun 2011 11:58:45 -0000

--20cf303ddb567db33a04a6d88190
Content-Type: text/plain; charset=UTF-8

Hi,

I would like to post some comments and suggestions regarding the latest iSER
draft:
http://www.watersprings.org/pub/id/draft-ietf-storm-iser-01.txt

Unfortunately no comments have been posted since it expired. I'm acting both
as a
contributor and a practical implementer.
Earlier this year i submitted a new stable implementation of iSER, which was
accepted
by the open source STGT project. As a part of their official release it
already made its way
to a few Linux distributions.

There is a gap between the existing implementation and the original RFC,
which is mostly
(but not exclusively) related to the possibility of implementation over
Infiniband.
This gap was almost completely covered by the recent
draft-ietf-storm-iser-01.
My comments below strive to eliminate the remaining discrepancies.
The general idea is to either reflect some implemented changes in the spec,
or to pave a comprehensive way for the implementation to become spec
compliant.

Alexander Nezhinsky

---

1.* Account for VA in buffer advertisement*

Add VA to the list of attributes which identify advertized buffers and
change related definitions and statements.

* Add a definition of VA to section 1.1. For example:
"Virtual Address (VA) - an identifier supplied by RCaP together with STag
when registering a memory buffer. In some modes it is not used,
and VA=0 is implied. These modes are known as ZBVA (Zero Based VA).
In such cases VA should be set to zero when buffer advertisements
are communicated."

* Definition of Remote Mapping:
"Remote Mapping - A task state record maintained by the iSER Layer
 that associates the Initiator Task Tag to the Advertised STag(s)..."

should read:

"...that associates the Initiator Task Tag to the Advertised STag(s)
and Virtual Address(es)..."

* Definition of Tagged Buffer:

"Tagged Buffer - A buffer that is explicitly Advertised to the iSER Layer
at the remote node  through the exchange of an STag, Tagged Offset, and
length."

should read:

"...the exchange of an STag, VA, Tagged Offset, and length."

2.* iser message format*

The headers are now defined for using with iWARP. Actually, they contain an
iWARP
prefix followed by a generic part.

Define a generic iser header as mandatory, while stating that the underlying
transport
protocols may add their own header sections before the iser header
and provide iWARP as the example of such prefix.

Read and write buffers are identified by the combination of STag and VA.
Both fields should be required for any transport. This requirement does not
preclude
using ZBVA where possible. When VA is not used (as in iWARP) it is set to 0.

Below is the actually used format:

struct iser_hdr {
  uint8_t   flags;
  uint8_t   rsvd[3];
  uint32_t  write_stag; /* write rkey in IB */
  uint64_t  write_va;
  uint32_t  read_stag;  /* read rkey in IB */
  uint64_t  read_va;
}

3.* Use RCaP as a general term, iWARP and IB as particular cases*

The language of the current draft is iWARP-dominated in many places. We need
to
make the text to refer to RCaP as a common denominator, stressing the
differences
between protocols where necessary.

* section 2 :

"The iWARP protocol stack provides direct data placement
functionality that is usable in practice, and in addition, there is
also interest in using iSCSI with other RDMA protocol stacks that
support direct data placement, such as the one provided by
InfiniBand.  The generic term RDMA-Capable Protocol (RCaP) is used
to refer to the RDMA functionality provided by such protocol stacks."

should read:

"The generic term RDMA-Capable Protocol (RCaP) is used to refer to
the RDMA functionality provided by the underlying protocol stack.
Two practical examples of protocol stacks providing direct data placement
functionality are iWARP and InfiniBand."

* section 5.1.1 :

 "Among the connection resources allocated at the initiator is the
Inbound RDMA Read Queue Depth (IRD) ...
... Initially, the iSER-IRD value at the initiator SHOULD
be set to the IRD value at the initiator and MUST NOT be more than
the IRD value."

should be prefixed with:

"For iWARP based implementations, ..."

4. *Mark iWARP-specific sections as such*

Some protocol-specific statements, like in above, are scattered through the
text, but there are entire sections which are iWARP-specific (RDMAP features
having no paralle in IB, etc.) We should explicitly state this.
The same goes for IB, of course, although right now there are no such
clauses.

In particular, sections 8.2. and 8.3. with sub-sections 8.3.x
should be either grouped as the subsections of
 "8.2. iWARP considerations in Flow Control"
or retain their numbers and have "(iWARP specific)" added to their titles

5. *Fix a typo in section numbering*

Appendix A should be section 15.

6.* iSER Hello-HelloReply*

Hello-HelloReply should be made optional and transport dependent.

Right now the IB based target's implementation terminates the connection if
it receives Hello.
IB-based implementation does not need any Hello exchange, so the implemented
behavior may be
turned into standard. iWARP does need Hello-HelloReply and its format is
provided.

We can define a new key called iSERHelloRequired with the default being
"No", so
that the current implementation becomes compliant.

Need to update some statements regarding IserHello, like in 5.1.1 and 5.1.2
alleviating some MUSTs with "if relevant for transport"-style of comments.

7.* Default values for MaxOutstandingUnexpectedPDUs and MaxAHSLength*

The defaults for MaxOutstandingUnexpectedPDUs and MaxAHSLength
should change from 0 (unlimited) to some small reasonable values
to protect a side not sending these keys from potentially harmful assumption

by its peer about unlimited resources available, so that buffer overflow
(MaxAHSLength)
or sending more buffers than actually posted (MaxOutstandingUnexpectedPDUs)
may result.

proposed default values:
MaxOutstandingUnexpectedPDUs = 4
MaxAHSLength = 256 (accounts for 1 Var len CDB + 1 bidir read-length)

8. *Ignoring MaxRecvDataSegmentLength key*

It should be clarified that MaxRecvDataSegmentLength may be sent but MUST be
ignored,
because {Initiator|Target}RecvDataSegmentLength have their own defaults.

If {Initiator|Target}RecvDataSegmentLength are sent, the declared values
would override
MaxRecvDataSegmentLength anyway, and if they are not sent, the default
values
must be used to avoid confusing scenarios.

--20cf303ddb567db33a04a6d88190
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi,<br><br>I would like to post some comments and sug=
gestions regarding the latest iSER draft: <br><a href=3D"http://www.watersp=
rings.org/pub/id/draft-ietf-storm-iser-01.txt">http://www.watersprings.org/=
pub/id/draft-ietf-storm-iser-01.txt</a><br>
<br>Unfortunately no comments have been posted since it expired. I&#39;m ac=
ting both as a <br>contributor and a practical implementer.<br>Earlier this=
 year i submitted a new stable implementation of iSER, which was accepted<b=
r>
by the open source STGT=C2=A0project. As a part of their official release i=
t already made its way <br>to a few Linux distributions.<br><br>There is a =
gap between the existing implementation and the original RFC, which is most=
ly<br>
(but not exclusively) related to the possibility of implementation over Inf=
iniband.<br>This gap was almost completely covered by the recent draft-ietf=
-storm-iser-01.<br>My comments below strive to eliminate the remaining disc=
repancies.<br>
The general idea is to either reflect some implemented changes in the spec,=
<br>or to pave a comprehensive way for the implementation to become spec co=
mpliant.<br><br>Alexander Nezhinsky<br><br>---<br><br>1.<b> Account for VA =
in buffer advertisement</b><br>
<br>Add VA to the list of attributes which identify advertized buffers and =
change related definitions and statements.<br><br>* Add a definition of VA =
to section 1.1. For example:<br><div style=3D"font-family: courier new,mono=
space; margin-left: 40px;">
&quot;Virtual Address (VA) - an identifier supplied by RCaP together with S=
Tag <br>when<span style=3D"font-family: courier new,monospace;"> registerin=
g a memory buffer. In some modes it is not used,</span><br style=3D"font-fa=
mily: courier new,monospace;">
<span style=3D"font-family: courier new,monospace;">and VA=3D0 is implied. =
These modes are known as ZBVA (Zero Based VA).</span><br style=3D"font-fami=
ly: courier new,monospace;"><span style=3D"font-family: courier new,monospa=
ce;">In such cases VA should be set to zero when buffer advertisements</spa=
n><br style=3D"font-family: courier new,monospace;">
<span style=3D"font-family: courier new,monospace;">are communicated.&quot;=
</span><br></div><br></div><div>* Definition of Remote Mapping:<div style=
=3D"margin-left: 40px; font-family: times new roman,serif;"><span style=3D"=
font-family: courier new,monospace;">&quot;Remote Mapping - A task state re=
cord maintained by the iSER Layer</span><br style=3D"font-family: courier n=
ew,monospace;">
<span style=3D"font-family: courier new,monospace;">=C2=A0that associates t=
he Initiator Task Tag to the Advertised STag(s)...&quot;</span><br></div><b=
r>should read:<br><br>
<div style=3D"margin-left: 40px;"><span style=3D"font-family: courier new,m=
onospace;">&quot;...that associates the Initiator Task Tag to the Advertise=
d STag(s)<br>and Virtual Address(es)...&quot;</span><br>
</div>
<br>* Definition of Tagged Buffer:<br><br><div style=3D"margin-left: 40px;"=
><span style=3D"font-family: courier new,monospace;">&quot;Tagged Buffer - =
A buffer that is explicitly Advertised to the iSER Layer<br>at the remote n=
ode</span>=C2=A0 <span style=3D"font-family: courier new,monospace;">throug=
h the exchange of an STag, Tagged Offset, and length.&quot;</span><br>
</div>
<br>
should read:<br><br>
<div style=3D"margin-left: 40px;"><span style=3D"font-family: courier new,m=
onospace;">&quot;...the exchange of an STag, VA, Tagged Offset, and length.=
&quot;</span><br>
</div>
<br>2.<b> iser message format</b><br><br>The headers are now defined for us=
ing with iWARP. Actually, they contain an iWARP<br>prefix followed by a gen=
eric part. <br><br>Define a generic iser header as mandatory, while stating=
 that the underlying transport<br>
protocols may add their own header sections before the iser header<br>and p=
rovide iWARP as the example of such prefix.<br><br>Read and write buffers a=
re identified by the combination of STag and VA.<br>Both fields should be r=
equired for any transport. This requirement does not preclude<br>
using ZBVA where possible. When VA is not used (as in iWARP) it is set to 0=
.<br><br>Below is the actually used format:<br><br><span style=3D"font-fami=
ly: courier new,monospace;">struct iser_hdr {</span><br style=3D"font-famil=
y: courier new,monospace;">
<span style=3D"font-family: courier new,monospace;">=C2=A0 uint8_t=C2=A0=C2=
=A0 flags;</span><br style=3D"font-family: courier new,monospace;"><span st=
yle=3D"font-family: courier new,monospace;">=C2=A0 uint8_t=C2=A0=C2=A0 rsvd=
[3];</span><br style=3D"font-family: courier new,monospace;">
<span style=3D"font-family: courier new,monospace;">=C2=A0 uint32_t=C2=A0 w=
rite_stag; /* write rkey in IB */</span><br style=3D"font-family: courier n=
ew,monospace;"><span style=3D"font-family: courier new,monospace;">=C2=A0 u=
int64_t=C2=A0 write_va;</span><br style=3D"font-family: courier new,monospa=
ce;">
<span style=3D"font-family: courier new,monospace;">=C2=A0 uint32_t=C2=A0 r=
ead_stag;=C2=A0 /* read rkey in IB */</span><br style=3D"font-family: couri=
er new,monospace;"><span style=3D"font-family: courier new,monospace;">=C2=
=A0 uint64_t=C2=A0 read_va;</span><br style=3D"font-family: courier new,mon=
ospace;">
<span style=3D"font-family: courier new,monospace;">}</span><br style=3D"fo=
nt-family: courier new,monospace;"><br>3.<b> Use RCaP as a general term, iW=
ARP and IB as particular cases</b><br>
<br>The language of the current draft is iWARP-dominated in many places. We=
 need to<br>make the text to refer to RCaP as a common denominator, stressi=
ng the differences<br>between protocols where necessary.<br><br>* section 2=
 :<br>

=C2=A0<br>
<div style=3D"margin-left: 40px;"><span style=3D"font-family: courier new,m=
onospace;">&quot;The iWARP protocol stack provides direct data placement</s=
pan><br style=3D"font-family: courier new,monospace;">
  <span style=3D"font-family: courier new,monospace;">functionality that is=
 usable in practice, and in addition, there is</span><br style=3D"font-fami=
ly: courier new,monospace;">
  <span style=3D"font-family: courier new,monospace;">also interest in usin=
g iSCSI with other RDMA protocol stacks that</span><br style=3D"font-family=
: courier new,monospace;">
  <span style=3D"font-family: courier new,monospace;">support direct data p=
lacement, such as the one provided by</span><br style=3D"font-family: couri=
er new,monospace;">
  <span style=3D"font-family: courier new,monospace;">InfiniBand.=C2=A0 The=
 generic term RDMA-Capable Protocol (RCaP) is used</span><br style=3D"font-=
family: courier new,monospace;">
  <span style=3D"font-family: courier new,monospace;">to refer to the RDMA =
functionality provided by such protocol stacks.&quot;</span><br style=3D"fo=
nt-family: courier new,monospace;">
</div>

<br>
should read:<br>
<br><div style=3D"margin-left: 40px; font-family: courier new,monospace;">
&quot;The generic term RDMA-Capable Protocol (RCaP) is used to refer to<br>
the RDMA functionality provided by the underlying protocol stack.<br>
Two practical examples of protocol stacks providing direct data placement<b=
r>
functionality are iWARP and InfiniBand.&quot;<br></div>
<br>* section 5.1.1 :<br>
<br><div style=3D"margin-left: 40px; font-family: times new roman,serif;">=
=C2=A0<span style=3D"font-family: courier new,monospace;">&quot;Among the c=
onnection resources allocated at the initiator is the</span><br style=3D"fo=
nt-family: courier new,monospace;">
<span style=3D"font-family: courier new,monospace;">Inbound RDMA Read Queue=
 Depth (IRD) ...</span><br style=3D"font-family: courier new,monospace;"><s=
pan style=3D"font-family: courier new,monospace;">... Initially, the iSER-I=
RD value at the initiator SHOULD</span><br style=3D"font-family: courier ne=
w,monospace;">
<span style=3D"font-family: courier new,monospace;">be set to the IRD value=
 at the initiator and MUST NOT be more than</span><br style=3D"font-family:=
 courier new,monospace;"><span style=3D"font-family: courier new,monospace;=
">the IRD value.&quot;</span><br>
</div>
<br>
should be prefixed with:<br><br><div style=3D"margin-left: 40px;"><span sty=
le=3D"font-family: courier new,monospace;">&quot;For iWARP based implementa=
tions, ...&quot;</span><br></div>
<br>4. <b>Mark iWARP-specific sections as such</b><br>
<br>
Some protocol-specific statements, like in above, are scattered through the=
<br>text, but there are entire sections which are iWARP-specific (RDMAP fea=
tures<br>having no paralle in IB, etc.) We should explicitly state this. <b=
r>

The same goes for IB, of course, although right now there are no such claus=
es.<br>
<br>
In particular, sections 8.2. and 8.3. with sub-sections 8.3.x<br>
should be either grouped as the subsections of<br>
=C2=A0&quot;8.2. iWARP considerations in Flow Control&quot;<br>
or retain their numbers and have &quot;(iWARP specific)&quot; added to thei=
r titles<br>

<br>5. <b>Fix a typo in section numbering</b><br></div><br><div><div><div><=
div><div><div><div>Appendix A should be section 15.<br><br>6.<b> iSER Hello=
-HelloReply</b><br><br>Hello-HelloReply should be made optional and transpo=
rt dependent.<br>
<br>Right now the IB based target&#39;s implementation terminates the conne=
ction if it receives Hello.<br>IB-based implementation does not need any He=
llo exchange, so the implemented behavior may be<br>turned into standard. i=
WARP does need Hello-HelloReply and its format is provided.<br>
<br>We can define a new key called iSERHelloRequired with the default being=
 &quot;No&quot;, so<br>that the current implementation becomes compliant.<b=
r><br>Need to update some statements regarding IserHello, like in 5.1.1 and=
 5.1.2<br>
alleviating some MUSTs with &quot;if relevant for transport&quot;-style of =
comments.<br><br>7.<b> Default values for MaxOutstandingUnexpectedPDUs and =
MaxAHSLength</b><br><br>The defaults for MaxOutstandingUnexpectedPDUs and M=
axAHSLength<br>
should change from 0 (unlimited) to some small reasonable values<br>to prot=
ect a side not sending these keys from potentially harmful assumption <br>b=
y its peer about unlimited resources available, so that buffer overflow (Ma=
xAHSLength)<br>
or sending more buffers than actually posted (MaxOutstandingUnexpectedPDUs)=
 may result.<br><br>proposed default values:<br><span style=3D"font-family:=
 courier new,monospace;">MaxOutstandingUnexpectedPDUs =3D 4</span><br style=
=3D"font-family: courier new,monospace;">
<span style=3D"font-family: courier new,monospace;">MaxAHSLength =3D 256</s=
pan> (accounts for 1 Var len CDB + 1 bidir read-length)<br><br>8. <b>Ignori=
ng MaxRecvDataSegmentLength key</b><br><br>It should be clarified that MaxR=
ecvDataSegmentLength may be sent but MUST be ignored,<br>
because {Initiator|Target}RecvDataSegmentLength have their own defaults. <b=
r><br>
If {Initiator|Target}RecvDataSegmentLength are sent, the declared values wo=
uld override <br>MaxRecvDataSegmentLength anyway, and if they are not sent,=
 the default values<br>must be used to avoid confusing scenarios.<br><br>
<br></div></div></div></div></div></div></div></div>

--20cf303ddb567db33a04a6d88190--

From ttalpey@microsoft.com  Wed Jun 29 08:30:11 2011
Return-Path: <ttalpey@microsoft.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 14EDC11E807F for <storm@ietfa.amsl.com>; Wed, 29 Jun 2011 08:30:11 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -110.598
X-Spam-Level: 
X-Spam-Status: No, score=-110.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AoL6Oav+2eeD for <storm@ietfa.amsl.com>; Wed, 29 Jun 2011 08:30:08 -0700 (PDT)
Received: from smtp.microsoft.com (smtp.microsoft.com [131.107.115.214]) by ietfa.amsl.com (Postfix) with ESMTP id 8310E11E8076 for <storm@ietf.org>; Wed, 29 Jun 2011 08:30:08 -0700 (PDT)
Received: from TK5EX14MLTC102.redmond.corp.microsoft.com (157.54.79.180) by TK5-EXGWY-E803.partners.extranet.microsoft.com (10.251.56.169) with Microsoft SMTP Server (TLS) id 8.2.176.0; Wed, 29 Jun 2011 08:30:07 -0700
Received: from TK5EX14MBXC118.redmond.corp.microsoft.com ([169.254.9.75]) by TK5EX14MLTC102.redmond.corp.microsoft.com ([157.54.79.180]) with mapi id 14.01.0289.008; Wed, 29 Jun 2011 08:30:07 -0700
From: Tom Talpey <ttalpey@microsoft.com>
To: Alexander Nezhinsky <nezhinsky@gmail.com>, "storm@ietf.org" <storm@ietf.org>
Thread-Topic: [storm] iSER draft
Thread-Index: AQHMNlPpov6tD7xTq0WTfzkPhGmP2JTUcDZA
Date: Wed, 29 Jun 2011 15:30:06 +0000
Message-ID: <F83812DF4B59B9499C1BC978336D91745EDE207F@TK5EX14MBXC118.redmond.corp.microsoft.com>
References: <BANLkTi=GVHUBdsZiwsRXxETd_rU=F8f85w@mail.gmail.com>
In-Reply-To: <BANLkTi=GVHUBdsZiwsRXxETd_rU=F8f85w@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [157.54.51.34]
Content-Type: multipart/alternative; boundary="_000_F83812DF4B59B9499C1BC978336D91745EDE207FTK5EX14MBXC118r_"
MIME-Version: 1.0
Subject: Re: [storm] iSER draft
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 29 Jun 2011 15:30:11 -0000

--_000_F83812DF4B59B9499C1BC978336D91745EDE207FTK5EX14MBXC118r_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

V29ya2luZyBHcm91cCBDby1jaGFpciBoYXQgb2ZmLg0KDQpBbGV4YW5kZXIsIHRoYW5rcyBmb3Ig
Y29udHJpYnV0aW5nIHRoZXNlIHVwZGF0ZXMuIEkgaGF2ZSBzb21lIGdlbmVyYWwgY29tbWVudHMg
b24gdHdvIGl0ZW1zLg0KDQo+ICogQWRkIGEgZGVmaW5pdGlvbiBvZiBWQSB0byBzZWN0aW9uIDEu
MS4gRm9yIGV4YW1wbGU6DQo+IlZpcnR1YWwgQWRkcmVzcyAoVkEpIC0gYW4gaWRlbnRpZmllciBz
dXBwbGllZCBieSBSQ2FQIHRvZ2V0aGVyIHdpdGggU1RhZw0KPndoZW4gcmVnaXN0ZXJpbmcgYSBt
ZW1vcnkgYnVmZmVyLiBJbiBzb21lIG1vZGVzIGl0IGlzIG5vdCB1c2VkLA0KPmFuZCBWQT0wIGlz
IGltcGxpZWQuIFRoZXNlIG1vZGVzIGFyZSBrbm93biBhcyBaQlZBIChaZXJvIEJhc2VkIFZBKS4N
Cj5JbiBzdWNoIGNhc2VzIFZBIHNob3VsZCBiZSBzZXQgdG8gemVybyB3aGVuIGJ1ZmZlciBhZHZl
cnRpc2VtZW50cw0KPmFyZSBjb21tdW5pY2F0ZWQuIg0KDQpJIHRoaW5rIGl0IG1heSBiZSBwcm9i
bGVtYXRpYyB0byBpbnRyb2R1Y2UgdGhlIG5ldyB0ZXJtIOKAnFZpcnR1YWwgQWRkcmVzc+KAnSB3
aXRob3V0IGRpc2FtYmlndWF0aW5nIGl0IGZyb20gdGhlIGV4aXN0aW5nIHRlcm0g4oCcVGFnZ2Vk
IE9mZnNldOKAnS4gVGhlIHRlcm0gVkEgZG9lcyBub3QgYXBwZWFyIGluIFJGQzUwNDYsIGFuZCBp
biB0aGUgdXBkYXRlLCB3YXMgaW50cm9kdWNlZCBpbiBvbmx5IG9uZSBtZXNzYWdlIHR5cGUgKGNv
bnRyb2wgUERVIHNlY3Rpb25zIDYuOSBhbmQgOS4yKS4gQWxzbywgeW91ciBkZWZpbml0aW9uIGlt
cGxpZXMgdGhhdCBpdCBpcyBwcm92aWRlZCBsb2NhbGx5IHRvIGFuIHVuc3BlY2lmaWVkIEFQSS4g
SWYgaXQgaXMgbm90IHBhc3NlZCBvbiB0aGUgd2lyZSwgdGhlbiBpdCBzZWVtcyBzdXBlcmZsdW91
cyB0byB0aGUgcHJvdG9jb2wuDQoNCkVpdGhlciB3YXksIHRoZSBkZWZpbml0aW9uIG5lZWRzIHRv
IGF2b2lkIHRoZSBzdGF0ZW1lbnRzIOKAnEluIHNvbWUgbW9kZXPigJ0gYW5kIOKAnHNob3VsZCBi
ZSBzZXQgdG8gemVyb+KAnS4gVGhlc2UgYXJlIHByb3RvY29sIGJlaGF2aW9yIGFuZCBzdWNoIGRp
c2N1c3Npb24gaXMgYmUgYXZvaWRlZCBpbiB0aGUgZ2xvc3NhcnkuIEkgd291bGQgc3VnZ2VzdCBk
ZWxldGluZyBhbGwgYnV0IHRoZSBmaXJzdCBzZW50ZW5jZSwgYWZ0ZXIgcmVzb2x2aW5nIHRoZSB1
c2Ugb2YgVkEgb2YgY291cnNlLg0KDQo+KiBEZWZpbml0aW9uIG9mIFRhZ2dlZCBCdWZmZXI6DQo+
IlRhZ2dlZCBCdWZmZXIgLSBBIGJ1ZmZlciB0aGF0IGlzIGV4cGxpY2l0bHkgQWR2ZXJ0aXNlZCB0
byB0aGUgaVNFUiBMYXllcg0KPmF0IHRoZSByZW1vdGUgbm9kZSAgdGhyb3VnaCB0aGUgZXhjaGFu
Z2Ugb2YgYW4gU1RhZywgVGFnZ2VkIE9mZnNldCwgYW5kIGxlbmd0aC4iDQo+DQo+c2hvdWxkIHJl
YWQ6DQo+Ii4uLnRoZSBleGNoYW5nZSBvZiBhbiBTVGFnLCBWQSwgVGFnZ2VkIE9mZnNldCwgYW5k
IGxlbmd0aC4iDQoNCk5vdyBJ4oCZbSByZWFsbHkgY29uZnVzZWQuIElzIHRoZSBUYWdnZWQgQnVm
ZmVyIHJlcHJlc2VudGVkIGFzIGEgNC10dXBsZSBpbiB0aGUgbmV3IHByb3RvY29sPyBJZiBzbywg
d2hlcmUgd2lsbCB0aGlzIGJlIGRlc2NyaWJlZD8NCg0KDQoNCkZyb206IHN0b3JtLWJvdW5jZXNA
aWV0Zi5vcmcgW21haWx0bzpzdG9ybS1ib3VuY2VzQGlldGYub3JnXSBPbiBCZWhhbGYgT2YgQWxl
eGFuZGVyIE5lemhpbnNreQ0KU2VudDogV2VkbmVzZGF5LCBKdW5lIDI5LCAyMDExIDc6NTkgQU0N
ClRvOiBzdG9ybUBpZXRmLm9yZzsgZGF2aWQuYmxhY2tAZW1jLmNvbTsgbWljaGFlbEBodWF3ZWlz
eW1hbnRlYy5jb20NClN1YmplY3Q6IFtzdG9ybV0gaVNFUiBkcmFmdA0KDQpIaSwNCg0KSSB3b3Vs
ZCBsaWtlIHRvIHBvc3Qgc29tZSBjb21tZW50cyBhbmQgc3VnZ2VzdGlvbnMgcmVnYXJkaW5nIHRo
ZSBsYXRlc3QgaVNFUiBkcmFmdDoNCmh0dHA6Ly93d3cud2F0ZXJzcHJpbmdzLm9yZy9wdWIvaWQv
ZHJhZnQtaWV0Zi1zdG9ybS1pc2VyLTAxLnR4dA0KDQpVbmZvcnR1bmF0ZWx5IG5vIGNvbW1lbnRz
IGhhdmUgYmVlbiBwb3N0ZWQgc2luY2UgaXQgZXhwaXJlZC4gSSdtIGFjdGluZyBib3RoIGFzIGEN
CmNvbnRyaWJ1dG9yIGFuZCBhIHByYWN0aWNhbCBpbXBsZW1lbnRlci4NCkVhcmxpZXIgdGhpcyB5
ZWFyIGkgc3VibWl0dGVkIGEgbmV3IHN0YWJsZSBpbXBsZW1lbnRhdGlvbiBvZiBpU0VSLCB3aGlj
aCB3YXMgYWNjZXB0ZWQNCmJ5IHRoZSBvcGVuIHNvdXJjZSBTVEdUIHByb2plY3QuIEFzIGEgcGFy
dCBvZiB0aGVpciBvZmZpY2lhbCByZWxlYXNlIGl0IGFscmVhZHkgbWFkZSBpdHMgd2F5DQp0byBh
IGZldyBMaW51eCBkaXN0cmlidXRpb25zLg0KDQpUaGVyZSBpcyBhIGdhcCBiZXR3ZWVuIHRoZSBl
eGlzdGluZyBpbXBsZW1lbnRhdGlvbiBhbmQgdGhlIG9yaWdpbmFsIFJGQywgd2hpY2ggaXMgbW9z
dGx5DQooYnV0IG5vdCBleGNsdXNpdmVseSkgcmVsYXRlZCB0byB0aGUgcG9zc2liaWxpdHkgb2Yg
aW1wbGVtZW50YXRpb24gb3ZlciBJbmZpbmliYW5kLg0KVGhpcyBnYXAgd2FzIGFsbW9zdCBjb21w
bGV0ZWx5IGNvdmVyZWQgYnkgdGhlIHJlY2VudCBkcmFmdC1pZXRmLXN0b3JtLWlzZXItMDEuDQpN
eSBjb21tZW50cyBiZWxvdyBzdHJpdmUgdG8gZWxpbWluYXRlIHRoZSByZW1haW5pbmcgZGlzY3Jl
cGFuY2llcy4NClRoZSBnZW5lcmFsIGlkZWEgaXMgdG8gZWl0aGVyIHJlZmxlY3Qgc29tZSBpbXBs
ZW1lbnRlZCBjaGFuZ2VzIGluIHRoZSBzcGVjLA0Kb3IgdG8gcGF2ZSBhIGNvbXByZWhlbnNpdmUg
d2F5IGZvciB0aGUgaW1wbGVtZW50YXRpb24gdG8gYmVjb21lIHNwZWMgY29tcGxpYW50Lg0KDQpB
bGV4YW5kZXIgTmV6aGluc2t5DQoNCi0tLQ0KDQoxLiBBY2NvdW50IGZvciBWQSBpbiBidWZmZXIg
YWR2ZXJ0aXNlbWVudA0KDQpBZGQgVkEgdG8gdGhlIGxpc3Qgb2YgYXR0cmlidXRlcyB3aGljaCBp
ZGVudGlmeSBhZHZlcnRpemVkIGJ1ZmZlcnMgYW5kIGNoYW5nZSByZWxhdGVkIGRlZmluaXRpb25z
IGFuZCBzdGF0ZW1lbnRzLg0KDQoqIEFkZCBhIGRlZmluaXRpb24gb2YgVkEgdG8gc2VjdGlvbiAx
LjEuIEZvciBleGFtcGxlOg0KIlZpcnR1YWwgQWRkcmVzcyAoVkEpIC0gYW4gaWRlbnRpZmllciBz
dXBwbGllZCBieSBSQ2FQIHRvZ2V0aGVyIHdpdGggU1RhZw0Kd2hlbiByZWdpc3RlcmluZyBhIG1l
bW9yeSBidWZmZXIuIEluIHNvbWUgbW9kZXMgaXQgaXMgbm90IHVzZWQsDQphbmQgVkE9MCBpcyBp
bXBsaWVkLiBUaGVzZSBtb2RlcyBhcmUga25vd24gYXMgWkJWQSAoWmVybyBCYXNlZCBWQSkuDQpJ
biBzdWNoIGNhc2VzIFZBIHNob3VsZCBiZSBzZXQgdG8gemVybyB3aGVuIGJ1ZmZlciBhZHZlcnRp
c2VtZW50cw0KYXJlIGNvbW11bmljYXRlZC4iDQoNCiogRGVmaW5pdGlvbiBvZiBSZW1vdGUgTWFw
cGluZzoNCiJSZW1vdGUgTWFwcGluZyAtIEEgdGFzayBzdGF0ZSByZWNvcmQgbWFpbnRhaW5lZCBi
eSB0aGUgaVNFUiBMYXllcg0KIHRoYXQgYXNzb2NpYXRlcyB0aGUgSW5pdGlhdG9yIFRhc2sgVGFn
IHRvIHRoZSBBZHZlcnRpc2VkIFNUYWcocykuLi4iDQoNCnNob3VsZCByZWFkOg0KIi4uLnRoYXQg
YXNzb2NpYXRlcyB0aGUgSW5pdGlhdG9yIFRhc2sgVGFnIHRvIHRoZSBBZHZlcnRpc2VkIFNUYWco
cykNCmFuZCBWaXJ0dWFsIEFkZHJlc3MoZXMpLi4uIg0KDQoqIERlZmluaXRpb24gb2YgVGFnZ2Vk
IEJ1ZmZlcjoNCiJUYWdnZWQgQnVmZmVyIC0gQSBidWZmZXIgdGhhdCBpcyBleHBsaWNpdGx5IEFk
dmVydGlzZWQgdG8gdGhlIGlTRVIgTGF5ZXINCmF0IHRoZSByZW1vdGUgbm9kZSAgdGhyb3VnaCB0
aGUgZXhjaGFuZ2Ugb2YgYW4gU1RhZywgVGFnZ2VkIE9mZnNldCwgYW5kIGxlbmd0aC4iDQoNCnNo
b3VsZCByZWFkOg0KIi4uLnRoZSBleGNoYW5nZSBvZiBhbiBTVGFnLCBWQSwgVGFnZ2VkIE9mZnNl
dCwgYW5kIGxlbmd0aC4iDQoNCjIuIGlzZXIgbWVzc2FnZSBmb3JtYXQNCg0KVGhlIGhlYWRlcnMg
YXJlIG5vdyBkZWZpbmVkIGZvciB1c2luZyB3aXRoIGlXQVJQLiBBY3R1YWxseSwgdGhleSBjb250
YWluIGFuIGlXQVJQDQpwcmVmaXggZm9sbG93ZWQgYnkgYSBnZW5lcmljIHBhcnQuDQoNCkRlZmlu
ZSBhIGdlbmVyaWMgaXNlciBoZWFkZXIgYXMgbWFuZGF0b3J5LCB3aGlsZSBzdGF0aW5nIHRoYXQg
dGhlIHVuZGVybHlpbmcgdHJhbnNwb3J0DQpwcm90b2NvbHMgbWF5IGFkZCB0aGVpciBvd24gaGVh
ZGVyIHNlY3Rpb25zIGJlZm9yZSB0aGUgaXNlciBoZWFkZXINCmFuZCBwcm92aWRlIGlXQVJQIGFz
IHRoZSBleGFtcGxlIG9mIHN1Y2ggcHJlZml4Lg0KDQpSZWFkIGFuZCB3cml0ZSBidWZmZXJzIGFy
ZSBpZGVudGlmaWVkIGJ5IHRoZSBjb21iaW5hdGlvbiBvZiBTVGFnIGFuZCBWQS4NCkJvdGggZmll
bGRzIHNob3VsZCBiZSByZXF1aXJlZCBmb3IgYW55IHRyYW5zcG9ydC4gVGhpcyByZXF1aXJlbWVu
dCBkb2VzIG5vdCBwcmVjbHVkZQ0KdXNpbmcgWkJWQSB3aGVyZSBwb3NzaWJsZS4gV2hlbiBWQSBp
cyBub3QgdXNlZCAoYXMgaW4gaVdBUlApIGl0IGlzIHNldCB0byAwLg0KDQpCZWxvdyBpcyB0aGUg
YWN0dWFsbHkgdXNlZCBmb3JtYXQ6DQoNCnN0cnVjdCBpc2VyX2hkciB7DQogIHVpbnQ4X3QgICBm
bGFnczsNCiAgdWludDhfdCAgIHJzdmRbM107DQogIHVpbnQzMl90ICB3cml0ZV9zdGFnOyAvKiB3
cml0ZSBya2V5IGluIElCICovDQogIHVpbnQ2NF90ICB3cml0ZV92YTsNCiAgdWludDMyX3QgIHJl
YWRfc3RhZzsgIC8qIHJlYWQgcmtleSBpbiBJQiAqLw0KICB1aW50NjRfdCAgcmVhZF92YTsNCn0N
Cg0KMy4gVXNlIFJDYVAgYXMgYSBnZW5lcmFsIHRlcm0sIGlXQVJQIGFuZCBJQiBhcyBwYXJ0aWN1
bGFyIGNhc2VzDQoNClRoZSBsYW5ndWFnZSBvZiB0aGUgY3VycmVudCBkcmFmdCBpcyBpV0FSUC1k
b21pbmF0ZWQgaW4gbWFueSBwbGFjZXMuIFdlIG5lZWQgdG8NCm1ha2UgdGhlIHRleHQgdG8gcmVm
ZXIgdG8gUkNhUCBhcyBhIGNvbW1vbiBkZW5vbWluYXRvciwgc3RyZXNzaW5nIHRoZSBkaWZmZXJl
bmNlcw0KYmV0d2VlbiBwcm90b2NvbHMgd2hlcmUgbmVjZXNzYXJ5Lg0KDQoqIHNlY3Rpb24gMiA6
DQoNCiJUaGUgaVdBUlAgcHJvdG9jb2wgc3RhY2sgcHJvdmlkZXMgZGlyZWN0IGRhdGEgcGxhY2Vt
ZW50DQpmdW5jdGlvbmFsaXR5IHRoYXQgaXMgdXNhYmxlIGluIHByYWN0aWNlLCBhbmQgaW4gYWRk
aXRpb24sIHRoZXJlIGlzDQphbHNvIGludGVyZXN0IGluIHVzaW5nIGlTQ1NJIHdpdGggb3RoZXIg
UkRNQSBwcm90b2NvbCBzdGFja3MgdGhhdA0Kc3VwcG9ydCBkaXJlY3QgZGF0YSBwbGFjZW1lbnQs
IHN1Y2ggYXMgdGhlIG9uZSBwcm92aWRlZCBieQ0KSW5maW5pQmFuZC4gIFRoZSBnZW5lcmljIHRl
cm0gUkRNQS1DYXBhYmxlIFByb3RvY29sIChSQ2FQKSBpcyB1c2VkDQp0byByZWZlciB0byB0aGUg
UkRNQSBmdW5jdGlvbmFsaXR5IHByb3ZpZGVkIGJ5IHN1Y2ggcHJvdG9jb2wgc3RhY2tzLiINCg0K
c2hvdWxkIHJlYWQ6DQoiVGhlIGdlbmVyaWMgdGVybSBSRE1BLUNhcGFibGUgUHJvdG9jb2wgKFJD
YVApIGlzIHVzZWQgdG8gcmVmZXIgdG8NCnRoZSBSRE1BIGZ1bmN0aW9uYWxpdHkgcHJvdmlkZWQg
YnkgdGhlIHVuZGVybHlpbmcgcHJvdG9jb2wgc3RhY2suDQpUd28gcHJhY3RpY2FsIGV4YW1wbGVz
IG9mIHByb3RvY29sIHN0YWNrcyBwcm92aWRpbmcgZGlyZWN0IGRhdGEgcGxhY2VtZW50DQpmdW5j
dGlvbmFsaXR5IGFyZSBpV0FSUCBhbmQgSW5maW5pQmFuZC4iDQoNCiogc2VjdGlvbiA1LjEuMSA6
DQogIkFtb25nIHRoZSBjb25uZWN0aW9uIHJlc291cmNlcyBhbGxvY2F0ZWQgYXQgdGhlIGluaXRp
YXRvciBpcyB0aGUNCkluYm91bmQgUkRNQSBSZWFkIFF1ZXVlIERlcHRoIChJUkQpIC4uLg0KLi4u
IEluaXRpYWxseSwgdGhlIGlTRVItSVJEIHZhbHVlIGF0IHRoZSBpbml0aWF0b3IgU0hPVUxEDQpi
ZSBzZXQgdG8gdGhlIElSRCB2YWx1ZSBhdCB0aGUgaW5pdGlhdG9yIGFuZCBNVVNUIE5PVCBiZSBt
b3JlIHRoYW4NCnRoZSBJUkQgdmFsdWUuIg0KDQpzaG91bGQgYmUgcHJlZml4ZWQgd2l0aDoNCiJG
b3IgaVdBUlAgYmFzZWQgaW1wbGVtZW50YXRpb25zLCAuLi4iDQoNCjQuIE1hcmsgaVdBUlAtc3Bl
Y2lmaWMgc2VjdGlvbnMgYXMgc3VjaA0KDQpTb21lIHByb3RvY29sLXNwZWNpZmljIHN0YXRlbWVu
dHMsIGxpa2UgaW4gYWJvdmUsIGFyZSBzY2F0dGVyZWQgdGhyb3VnaCB0aGUNCnRleHQsIGJ1dCB0
aGVyZSBhcmUgZW50aXJlIHNlY3Rpb25zIHdoaWNoIGFyZSBpV0FSUC1zcGVjaWZpYyAoUkRNQVAg
ZmVhdHVyZXMNCmhhdmluZyBubyBwYXJhbGxlIGluIElCLCBldGMuKSBXZSBzaG91bGQgZXhwbGlj
aXRseSBzdGF0ZSB0aGlzLg0KVGhlIHNhbWUgZ29lcyBmb3IgSUIsIG9mIGNvdXJzZSwgYWx0aG91
Z2ggcmlnaHQgbm93IHRoZXJlIGFyZSBubyBzdWNoIGNsYXVzZXMuDQoNCkluIHBhcnRpY3VsYXIs
IHNlY3Rpb25zIDguMi4gYW5kIDguMy4gd2l0aCBzdWItc2VjdGlvbnMgOC4zLngNCnNob3VsZCBi
ZSBlaXRoZXIgZ3JvdXBlZCBhcyB0aGUgc3Vic2VjdGlvbnMgb2YNCiAiOC4yLiBpV0FSUCBjb25z
aWRlcmF0aW9ucyBpbiBGbG93IENvbnRyb2wiDQpvciByZXRhaW4gdGhlaXIgbnVtYmVycyBhbmQg
aGF2ZSAiKGlXQVJQIHNwZWNpZmljKSIgYWRkZWQgdG8gdGhlaXIgdGl0bGVzDQoNCjUuIEZpeCBh
IHR5cG8gaW4gc2VjdGlvbiBudW1iZXJpbmcNCg0KQXBwZW5kaXggQSBzaG91bGQgYmUgc2VjdGlv
biAxNS4NCg0KNi4gaVNFUiBIZWxsby1IZWxsb1JlcGx5DQoNCkhlbGxvLUhlbGxvUmVwbHkgc2hv
dWxkIGJlIG1hZGUgb3B0aW9uYWwgYW5kIHRyYW5zcG9ydCBkZXBlbmRlbnQuDQoNClJpZ2h0IG5v
dyB0aGUgSUIgYmFzZWQgdGFyZ2V0J3MgaW1wbGVtZW50YXRpb24gdGVybWluYXRlcyB0aGUgY29u
bmVjdGlvbiBpZiBpdCByZWNlaXZlcyBIZWxsby4NCklCLWJhc2VkIGltcGxlbWVudGF0aW9uIGRv
ZXMgbm90IG5lZWQgYW55IEhlbGxvIGV4Y2hhbmdlLCBzbyB0aGUgaW1wbGVtZW50ZWQgYmVoYXZp
b3IgbWF5IGJlDQp0dXJuZWQgaW50byBzdGFuZGFyZC4gaVdBUlAgZG9lcyBuZWVkIEhlbGxvLUhl
bGxvUmVwbHkgYW5kIGl0cyBmb3JtYXQgaXMgcHJvdmlkZWQuDQoNCldlIGNhbiBkZWZpbmUgYSBu
ZXcga2V5IGNhbGxlZCBpU0VSSGVsbG9SZXF1aXJlZCB3aXRoIHRoZSBkZWZhdWx0IGJlaW5nICJO
byIsIHNvDQp0aGF0IHRoZSBjdXJyZW50IGltcGxlbWVudGF0aW9uIGJlY29tZXMgY29tcGxpYW50
Lg0KDQpOZWVkIHRvIHVwZGF0ZSBzb21lIHN0YXRlbWVudHMgcmVnYXJkaW5nIElzZXJIZWxsbywg
bGlrZSBpbiA1LjEuMSBhbmQgNS4xLjINCmFsbGV2aWF0aW5nIHNvbWUgTVVTVHMgd2l0aCAiaWYg
cmVsZXZhbnQgZm9yIHRyYW5zcG9ydCItc3R5bGUgb2YgY29tbWVudHMuDQoNCjcuIERlZmF1bHQg
dmFsdWVzIGZvciBNYXhPdXRzdGFuZGluZ1VuZXhwZWN0ZWRQRFVzIGFuZCBNYXhBSFNMZW5ndGgN
Cg0KVGhlIGRlZmF1bHRzIGZvciBNYXhPdXRzdGFuZGluZ1VuZXhwZWN0ZWRQRFVzIGFuZCBNYXhB
SFNMZW5ndGgNCnNob3VsZCBjaGFuZ2UgZnJvbSAwICh1bmxpbWl0ZWQpIHRvIHNvbWUgc21hbGwg
cmVhc29uYWJsZSB2YWx1ZXMNCnRvIHByb3RlY3QgYSBzaWRlIG5vdCBzZW5kaW5nIHRoZXNlIGtl
eXMgZnJvbSBwb3RlbnRpYWxseSBoYXJtZnVsIGFzc3VtcHRpb24NCmJ5IGl0cyBwZWVyIGFib3V0
IHVubGltaXRlZCByZXNvdXJjZXMgYXZhaWxhYmxlLCBzbyB0aGF0IGJ1ZmZlciBvdmVyZmxvdyAo
TWF4QUhTTGVuZ3RoKQ0Kb3Igc2VuZGluZyBtb3JlIGJ1ZmZlcnMgdGhhbiBhY3R1YWxseSBwb3N0
ZWQgKE1heE91dHN0YW5kaW5nVW5leHBlY3RlZFBEVXMpIG1heSByZXN1bHQuDQoNCnByb3Bvc2Vk
IGRlZmF1bHQgdmFsdWVzOg0KTWF4T3V0c3RhbmRpbmdVbmV4cGVjdGVkUERVcyA9IDQNCk1heEFI
U0xlbmd0aCA9IDI1NiAoYWNjb3VudHMgZm9yIDEgVmFyIGxlbiBDREIgKyAxIGJpZGlyIHJlYWQt
bGVuZ3RoKQ0KDQo4LiBJZ25vcmluZyBNYXhSZWN2RGF0YVNlZ21lbnRMZW5ndGgga2V5DQoNCkl0
IHNob3VsZCBiZSBjbGFyaWZpZWQgdGhhdCBNYXhSZWN2RGF0YVNlZ21lbnRMZW5ndGggbWF5IGJl
IHNlbnQgYnV0IE1VU1QgYmUgaWdub3JlZCwNCmJlY2F1c2Uge0luaXRpYXRvcnxUYXJnZXR9UmVj
dkRhdGFTZWdtZW50TGVuZ3RoIGhhdmUgdGhlaXIgb3duIGRlZmF1bHRzLg0KDQpJZiB7SW5pdGlh
dG9yfFRhcmdldH1SZWN2RGF0YVNlZ21lbnRMZW5ndGggYXJlIHNlbnQsIHRoZSBkZWNsYXJlZCB2
YWx1ZXMgd291bGQgb3ZlcnJpZGUNCk1heFJlY3ZEYXRhU2VnbWVudExlbmd0aCBhbnl3YXksIGFu
ZCBpZiB0aGV5IGFyZSBub3Qgc2VudCwgdGhlIGRlZmF1bHQgdmFsdWVzDQptdXN0IGJlIHVzZWQg
dG8gYXZvaWQgY29uZnVzaW5nIHNjZW5hcmlvcy4NCg0K

--_000_F83812DF4B59B9499C1BC978336D91745EDE207FTK5EX14MBXC118r_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64

PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy
bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt
YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj
cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv
VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg
Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRv
ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTQgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxl
PjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6
Q2FsaWJyaTsNCglwYW5vc2UtMToyIDE1IDUgMiAyIDIgNCAzIDIgNDt9DQpAZm9udC1mYWNlDQoJ
e2ZvbnQtZmFtaWx5OlRhaG9tYTsNCglwYW5vc2UtMToyIDExIDYgNCAzIDUgNCA0IDIgNDt9DQov
KiBTdHlsZSBEZWZpbml0aW9ucyAqLw0KcC5Nc29Ob3JtYWwsIGxpLk1zb05vcm1hbCwgZGl2Lk1z
b05vcm1hbA0KCXttYXJnaW46MGluOw0KCW1hcmdpbi1ib3R0b206LjAwMDFwdDsNCglmb250LXNp
emU6MTIuMHB0Ow0KCWZvbnQtZmFtaWx5OiJUaW1lcyBOZXcgUm9tYW4iLCJzZXJpZiI7fQ0KYTps
aW5rLCBzcGFuLk1zb0h5cGVybGluaw0KCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6
Ymx1ZTsNCgl0ZXh0LWRlY29yYXRpb246dW5kZXJsaW5lO30NCmE6dmlzaXRlZCwgc3Bhbi5Nc29I
eXBlcmxpbmtGb2xsb3dlZA0KCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6cHVycGxl
Ow0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxpbmU7fQ0KcC5Nc29MaXN0UGFyYWdyYXBoLCBsaS5N
c29MaXN0UGFyYWdyYXBoLCBkaXYuTXNvTGlzdFBhcmFncmFwaA0KCXttc28tc3R5bGUtcHJpb3Jp
dHk6MzQ7DQoJbWFyZ2luLXRvcDowaW47DQoJbWFyZ2luLXJpZ2h0OjBpbjsNCgltYXJnaW4tYm90
dG9tOjBpbjsNCgltYXJnaW4tbGVmdDouNWluOw0KCW1hcmdpbi1ib3R0b206LjAwMDFwdDsNCglm
b250LXNpemU6MTIuMHB0Ow0KCWZvbnQtZmFtaWx5OiJUaW1lcyBOZXcgUm9tYW4iLCJzZXJpZiI7
fQ0Kc3Bhbi5FbWFpbFN0eWxlMTcNCgl7bXNvLXN0eWxlLXR5cGU6cGVyc29uYWwtcmVwbHk7DQoJ
Zm9udC1mYW1pbHk6IkNhbGlicmkiLCJzYW5zLXNlcmlmIjsNCgljb2xvcjojMUY0OTdEO30NCi5N
c29DaHBEZWZhdWx0DQoJe21zby1zdHlsZS10eXBlOmV4cG9ydC1vbmx5Ow0KCWZvbnQtZmFtaWx5
OiJDYWxpYnJpIiwic2Fucy1zZXJpZiI7fQ0KQHBhZ2UgV29yZFNlY3Rpb24xDQoJe3NpemU6OC41
aW4gMTEuMGluOw0KCW1hcmdpbjoxLjBpbiAxLjBpbiAxLjBpbiAxLjBpbjt9DQpkaXYuV29yZFNl
Y3Rpb24xDQoJe3BhZ2U6V29yZFNlY3Rpb24xO30NCi0tPjwvc3R5bGU+PCEtLVtpZiBndGUgbXNv
IDldPjx4bWw+DQo8bzpzaGFwZWRlZmF1bHRzIHY6ZXh0PSJlZGl0IiBzcGlkbWF4PSIxMDI2IiAv
Pg0KPC94bWw+PCFbZW5kaWZdLS0+PCEtLVtpZiBndGUgbXNvIDldPjx4bWw+DQo8bzpzaGFwZWxh
eW91dCB2OmV4dD0iZWRpdCI+DQo8bzppZG1hcCB2OmV4dD0iZWRpdCIgZGF0YT0iMSIgLz4NCjwv
bzpzaGFwZWxheW91dD48L3htbD48IVtlbmRpZl0tLT4NCjwvaGVhZD4NCjxib2R5IGxhbmc9IkVO
LVVTIiBsaW5rPSJibHVlIiB2bGluaz0icHVycGxlIj4NCjxkaXYgY2xhc3M9IldvcmRTZWN0aW9u
MSI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtm
b250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29s
b3I6IzFGNDk3RCI+V29ya2luZyBHcm91cCBDby1jaGFpciBoYXQgb2ZmLjxvOnA+PC9vOnA+PC9z
cGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEu
MHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90
Oztjb2xvcjojMUY0OTdEIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0i
TXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVv
dDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+QWxl
eGFuZGVyLCB0aGFua3MgZm9yIGNvbnRyaWJ1dGluZyB0aGVzZSB1cGRhdGVzLiBJIGhhdmUgc29t
ZSBnZW5lcmFsIGNvbW1lbnRzIG9uIHR3byBpdGVtcy48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8
cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZh
bWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFG
NDk3RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+
Jmd0OyAqIEFkZCBhIGRlZmluaXRpb24gb2YgVkEgdG8gc2VjdGlvbiAxLjEuIEZvciBleGFtcGxl
OjxvOnA+PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQt
ZmFtaWx5OiZxdW90O0NvdXJpZXIgTmV3JnF1b3Q7Ij4mZ3Q7JnF1b3Q7VmlydHVhbCBBZGRyZXNz
IChWQSkgLSBhbiBpZGVudGlmaWVyIHN1cHBsaWVkIGJ5IFJDYVAgdG9nZXRoZXIgd2l0aCBTVGFn
DQo8YnI+DQomZ3Q7d2hlbiByZWdpc3RlcmluZyBhIG1lbW9yeSBidWZmZXIuIEluIHNvbWUgbW9k
ZXMgaXQgaXMgbm90IHVzZWQsPGJyPg0KJmd0O2FuZCBWQT0wIGlzIGltcGxpZWQuIFRoZXNlIG1v
ZGVzIGFyZSBrbm93biBhcyBaQlZBIChaZXJvIEJhc2VkIFZBKS48YnI+DQomZ3Q7SW4gc3VjaCBj
YXNlcyBWQSBzaG91bGQgYmUgc2V0IHRvIHplcm8gd2hlbiBidWZmZXIgYWR2ZXJ0aXNlbWVudHM8
YnI+DQomZ3Q7YXJlIGNvbW11bmljYXRlZC4mcXVvdDs8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8
cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZh
bWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFG
NDk3RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+
PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZx
dW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPkkgdGhpbmsgaXQgbWF5
IGJlIHByb2JsZW1hdGljIHRvIGludHJvZHVjZSB0aGUgbmV3IHRlcm0g4oCcVmlydHVhbCBBZGRy
ZXNz4oCdIHdpdGhvdXQgZGlzYW1iaWd1YXRpbmcgaXQgZnJvbSB0aGUgZXhpc3RpbmcgdGVybSDi
gJxUYWdnZWQgT2Zmc2V04oCdLiBUaGUgdGVybSBWQSBkb2VzDQogbm90IGFwcGVhciBpbiBSRkM1
MDQ2LCBhbmQgaW4gdGhlIHVwZGF0ZSwgd2FzIGludHJvZHVjZWQgaW4gb25seSBvbmUgbWVzc2Fn
ZSB0eXBlIChjb250cm9sIFBEVSBzZWN0aW9ucyA2LjkgYW5kIDkuMikuIEFsc28sIHlvdXIgZGVm
aW5pdGlvbiBpbXBsaWVzIHRoYXQgaXQgaXMgcHJvdmlkZWQgbG9jYWxseSB0byBhbiB1bnNwZWNp
ZmllZCBBUEkuIElmIGl0IGlzIG5vdCBwYXNzZWQgb24gdGhlIHdpcmUsIHRoZW4gaXQgc2VlbXMg
c3VwZXJmbHVvdXMNCiB0byB0aGUgcHJvdG9jb2wuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAg
Y2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1p
bHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMxRjQ5
N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxz
cGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVv
dDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMUY0OTdEIj5FaXRoZXIgd2F5LCB0aGUg
ZGVmaW5pdGlvbiBuZWVkcyB0byBhdm9pZCB0aGUgc3RhdGVtZW50cyDigJxJbiBzb21lIG1vZGVz
4oCdIGFuZCDigJxzaG91bGQgYmUgc2V0IHRvIHplcm/igJ0uIFRoZXNlIGFyZSBwcm90b2NvbCBi
ZWhhdmlvciBhbmQgc3VjaCBkaXNjdXNzaW9uIGlzIGJlDQogYXZvaWRlZCBpbiB0aGUgZ2xvc3Nh
cnkuIEkgd291bGQgc3VnZ2VzdCBkZWxldGluZyBhbGwgYnV0IHRoZSBmaXJzdCBzZW50ZW5jZSwg
YWZ0ZXIgcmVzb2x2aW5nIHRoZSB1c2Ugb2YgVkEgb2YgY291cnNlLjxvOnA+PC9vOnA+PC9zcGFu
PjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0
O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztj
b2xvcjojMUY0OTdEIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNv
Tm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJvdHRvbToxMi4wcHQiPiZndDsqIERlZmluaXRpb24gb2Yg
VGFnZ2VkIEJ1ZmZlcjo8bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFu
IHN0eWxlPSJmb250LWZhbWlseTomcXVvdDtDb3VyaWVyIE5ldyZxdW90OyI+Jmd0OyZxdW90O1Rh
Z2dlZCBCdWZmZXIgLSBBIGJ1ZmZlciB0aGF0IGlzIGV4cGxpY2l0bHkgQWR2ZXJ0aXNlZCB0byB0
aGUgaVNFUiBMYXllcjxicj4NCiZndDthdCB0aGUgcmVtb3RlIG5vZGU8L3NwYW4+Jm5ic3A7IDxz
cGFuIHN0eWxlPSJmb250LWZhbWlseTomcXVvdDtDb3VyaWVyIE5ldyZxdW90OyI+dGhyb3VnaCB0
aGUgZXhjaGFuZ2Ugb2YgYW4gU1RhZywgVGFnZ2VkIE9mZnNldCwgYW5kIGxlbmd0aC4mcXVvdDs8
L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2lu
LWJvdHRvbToxMi4wcHQiPiZndDs8YnI+DQomZ3Q7c2hvdWxkIHJlYWQ6PG86cD48L286cD48L3A+
DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6JnF1b3Q7Q291
cmllciBOZXcmcXVvdDsiPiZndDsmcXVvdDsuLi50aGUgZXhjaGFuZ2Ugb2YgYW4gU1RhZywgVkEs
IFRhZ2dlZCBPZmZzZXQsIGFuZCBsZW5ndGguJnF1b3Q7PC9zcGFuPjxvOnA+PC9vOnA+PC9wPg0K
PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1m
YW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMx
RjQ5N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi
PjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkm
cXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMUY0OTdEIj5Ob3cgSeKAmW0gcmVh
bGx5IGNvbmZ1c2VkLiBJcyB0aGUgVGFnZ2VkIEJ1ZmZlciByZXByZXNlbnRlZCBhcyBhIDQtdHVw
bGUgaW4gdGhlIG5ldyBwcm90b2NvbD8gSWYgc28sIHdoZXJlIHdpbGwgdGhpcyBiZSBkZXNjcmli
ZWQ/PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5
bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVv
dDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFu
PjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0
O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztj
b2xvcjojMUY0OTdEIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNv
Tm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVvdDtD
YWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+PG86cD4m
bmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PGI+PHNwYW4gc3R5
bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGFob21hJnF1b3Q7LCZxdW90
O3NhbnMtc2VyaWYmcXVvdDsiPkZyb206PC9zcGFuPjwvYj48c3BhbiBzdHlsZT0iZm9udC1zaXpl
OjEwLjBwdDtmb250LWZhbWlseTomcXVvdDtUYWhvbWEmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZx
dW90OyI+IHN0b3JtLWJvdW5jZXNAaWV0Zi5vcmcgW21haWx0bzpzdG9ybS1ib3VuY2VzQGlldGYu
b3JnXQ0KPGI+T24gQmVoYWxmIE9mIDwvYj5BbGV4YW5kZXIgTmV6aGluc2t5PGJyPg0KPGI+U2Vu
dDo8L2I+IFdlZG5lc2RheSwgSnVuZSAyOSwgMjAxMSA3OjU5IEFNPGJyPg0KPGI+VG86PC9iPiBz
dG9ybUBpZXRmLm9yZzsgZGF2aWQuYmxhY2tAZW1jLmNvbTsgbWljaGFlbEBodWF3ZWlzeW1hbnRl
Yy5jb208YnI+DQo8Yj5TdWJqZWN0OjwvYj4gW3N0b3JtXSBpU0VSIGRyYWZ0PG86cD48L286cD48
L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+DQo8
ZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPkhpLDxicj4NCjxicj4NCkkgd291bGQg
bGlrZSB0byBwb3N0IHNvbWUgY29tbWVudHMgYW5kIHN1Z2dlc3Rpb25zIHJlZ2FyZGluZyB0aGUg
bGF0ZXN0IGlTRVIgZHJhZnQ6DQo8YnI+DQo8YSBocmVmPSJodHRwOi8vd3d3LndhdGVyc3ByaW5n
cy5vcmcvcHViL2lkL2RyYWZ0LWlldGYtc3Rvcm0taXNlci0wMS50eHQiPmh0dHA6Ly93d3cud2F0
ZXJzcHJpbmdzLm9yZy9wdWIvaWQvZHJhZnQtaWV0Zi1zdG9ybS1pc2VyLTAxLnR4dDwvYT48YnI+
DQo8YnI+DQpVbmZvcnR1bmF0ZWx5IG5vIGNvbW1lbnRzIGhhdmUgYmVlbiBwb3N0ZWQgc2luY2Ug
aXQgZXhwaXJlZC4gSSdtIGFjdGluZyBib3RoIGFzIGENCjxicj4NCmNvbnRyaWJ1dG9yIGFuZCBh
IHByYWN0aWNhbCBpbXBsZW1lbnRlci48YnI+DQpFYXJsaWVyIHRoaXMgeWVhciBpIHN1Ym1pdHRl
ZCBhIG5ldyBzdGFibGUgaW1wbGVtZW50YXRpb24gb2YgaVNFUiwgd2hpY2ggd2FzIGFjY2VwdGVk
PGJyPg0KYnkgdGhlIG9wZW4gc291cmNlIFNUR1QmbmJzcDtwcm9qZWN0LiBBcyBhIHBhcnQgb2Yg
dGhlaXIgb2ZmaWNpYWwgcmVsZWFzZSBpdCBhbHJlYWR5IG1hZGUgaXRzIHdheQ0KPGJyPg0KdG8g
YSBmZXcgTGludXggZGlzdHJpYnV0aW9ucy48YnI+DQo8YnI+DQpUaGVyZSBpcyBhIGdhcCBiZXR3
ZWVuIHRoZSBleGlzdGluZyBpbXBsZW1lbnRhdGlvbiBhbmQgdGhlIG9yaWdpbmFsIFJGQywgd2hp
Y2ggaXMgbW9zdGx5PGJyPg0KKGJ1dCBub3QgZXhjbHVzaXZlbHkpIHJlbGF0ZWQgdG8gdGhlIHBv
c3NpYmlsaXR5IG9mIGltcGxlbWVudGF0aW9uIG92ZXIgSW5maW5pYmFuZC48YnI+DQpUaGlzIGdh
cCB3YXMgYWxtb3N0IGNvbXBsZXRlbHkgY292ZXJlZCBieSB0aGUgcmVjZW50IGRyYWZ0LWlldGYt
c3Rvcm0taXNlci0wMS48YnI+DQpNeSBjb21tZW50cyBiZWxvdyBzdHJpdmUgdG8gZWxpbWluYXRl
IHRoZSByZW1haW5pbmcgZGlzY3JlcGFuY2llcy48YnI+DQpUaGUgZ2VuZXJhbCBpZGVhIGlzIHRv
IGVpdGhlciByZWZsZWN0IHNvbWUgaW1wbGVtZW50ZWQgY2hhbmdlcyBpbiB0aGUgc3BlYyw8YnI+
DQpvciB0byBwYXZlIGEgY29tcHJlaGVuc2l2ZSB3YXkgZm9yIHRoZSBpbXBsZW1lbnRhdGlvbiB0
byBiZWNvbWUgc3BlYyBjb21wbGlhbnQuPGJyPg0KPGJyPg0KQWxleGFuZGVyIE5lemhpbnNreTxi
cj4NCjxicj4NCi0tLTxicj4NCjxicj4NCjEuPGI+IEFjY291bnQgZm9yIFZBIGluIGJ1ZmZlciBh
ZHZlcnRpc2VtZW50PC9iPjxicj4NCjxicj4NCkFkZCBWQSB0byB0aGUgbGlzdCBvZiBhdHRyaWJ1
dGVzIHdoaWNoIGlkZW50aWZ5IGFkdmVydGl6ZWQgYnVmZmVycyBhbmQgY2hhbmdlIHJlbGF0ZWQg
ZGVmaW5pdGlvbnMgYW5kIHN0YXRlbWVudHMuPGJyPg0KPGJyPg0KKiBBZGQgYSBkZWZpbml0aW9u
IG9mIFZBIHRvIHNlY3Rpb24gMS4xLiBGb3IgZXhhbXBsZTo8bzpwPjwvbzpwPjwvcD4NCjxkaXYg
c3R5bGU9Im1hcmdpbi1sZWZ0OjMwLjBwdCI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBz
dHlsZT0iZm9udC1mYW1pbHk6JnF1b3Q7Q291cmllciBOZXcmcXVvdDsiPiZxdW90O1ZpcnR1YWwg
QWRkcmVzcyAoVkEpIC0gYW4gaWRlbnRpZmllciBzdXBwbGllZCBieSBSQ2FQIHRvZ2V0aGVyIHdp
dGggU1RhZw0KPGJyPg0Kd2hlbiByZWdpc3RlcmluZyBhIG1lbW9yeSBidWZmZXIuIEluIHNvbWUg
bW9kZXMgaXQgaXMgbm90IHVzZWQsPGJyPg0KYW5kIFZBPTAgaXMgaW1wbGllZC4gVGhlc2UgbW9k
ZXMgYXJlIGtub3duIGFzIFpCVkEgKFplcm8gQmFzZWQgVkEpLjxicj4NCkluIHN1Y2ggY2FzZXMg
VkEgc2hvdWxkIGJlIHNldCB0byB6ZXJvIHdoZW4gYnVmZmVyIGFkdmVydGlzZW1lbnRzPGJyPg0K
YXJlIGNvbW11bmljYXRlZC4mcXVvdDs8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxw
IGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0K
PHAgY2xhc3M9Ik1zb05vcm1hbCI+KiBEZWZpbml0aW9uIG9mIFJlbW90ZSBNYXBwaW5nOjxvOnA+
PC9vOnA+PC9wPg0KPGRpdiBzdHlsZT0ibWFyZ2luLWxlZnQ6MzAuMHB0Ij4NCjxwIGNsYXNzPSJN
c29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTomcXVvdDtDb3VyaWVyIE5ldyZxdW90
OyI+JnF1b3Q7UmVtb3RlIE1hcHBpbmcgLSBBIHRhc2sgc3RhdGUgcmVjb3JkIG1haW50YWluZWQg
YnkgdGhlIGlTRVIgTGF5ZXI8YnI+DQombmJzcDt0aGF0IGFzc29jaWF0ZXMgdGhlIEluaXRpYXRv
ciBUYXNrIFRhZyB0byB0aGUgQWR2ZXJ0aXNlZCBTVGFnKHMpLi4uJnF1b3Q7PC9zcGFuPjxvOnA+
PC9vOnA+PC9wPg0KPC9kaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJv
dHRvbToxMi4wcHQiPjxicj4NCnNob3VsZCByZWFkOjxvOnA+PC9vOnA+PC9wPg0KPGRpdiBzdHls
ZT0ibWFyZ2luLWxlZnQ6MzAuMHB0Ij4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxl
PSJmb250LWZhbWlseTomcXVvdDtDb3VyaWVyIE5ldyZxdW90OyI+JnF1b3Q7Li4udGhhdCBhc3Nv
Y2lhdGVzIHRoZSBJbml0aWF0b3IgVGFzayBUYWcgdG8gdGhlIEFkdmVydGlzZWQgU1RhZyhzKTxi
cj4NCmFuZCBWaXJ0dWFsIEFkZHJlc3MoZXMpLi4uJnF1b3Q7PC9zcGFuPjxvOnA+PC9vOnA+PC9w
Pg0KPC9kaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJvdHRvbToxMi4w
cHQiPjxicj4NCiogRGVmaW5pdGlvbiBvZiBUYWdnZWQgQnVmZmVyOjxvOnA+PC9vOnA+PC9wPg0K
PGRpdiBzdHlsZT0ibWFyZ2luLWxlZnQ6MzAuMHB0Ij4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxz
cGFuIHN0eWxlPSJmb250LWZhbWlseTomcXVvdDtDb3VyaWVyIE5ldyZxdW90OyI+JnF1b3Q7VGFn
Z2VkIEJ1ZmZlciAtIEEgYnVmZmVyIHRoYXQgaXMgZXhwbGljaXRseSBBZHZlcnRpc2VkIHRvIHRo
ZSBpU0VSIExheWVyPGJyPg0KYXQgdGhlIHJlbW90ZSBub2RlPC9zcGFuPiZuYnNwOyA8c3BhbiBz
dHlsZT0iZm9udC1mYW1pbHk6JnF1b3Q7Q291cmllciBOZXcmcXVvdDsiPnRocm91Z2ggdGhlIGV4
Y2hhbmdlIG9mIGFuIFNUYWcsIFRhZ2dlZCBPZmZzZXQsIGFuZCBsZW5ndGguJnF1b3Q7PC9zcGFu
PjxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFy
Z2luLWJvdHRvbToxMi4wcHQiPjxicj4NCnNob3VsZCByZWFkOjxvOnA+PC9vOnA+PC9wPg0KPGRp
diBzdHlsZT0ibWFyZ2luLWxlZnQ6MzAuMHB0Ij4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFu
IHN0eWxlPSJmb250LWZhbWlseTomcXVvdDtDb3VyaWVyIE5ldyZxdW90OyI+JnF1b3Q7Li4udGhl
IGV4Y2hhbmdlIG9mIGFuIFNUYWcsIFZBLCBUYWdnZWQgT2Zmc2V0LCBhbmQgbGVuZ3RoLiZxdW90
Ozwvc3Bhbj48bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PGJy
Pg0KMi48Yj4gaXNlciBtZXNzYWdlIGZvcm1hdDwvYj48YnI+DQo8YnI+DQpUaGUgaGVhZGVycyBh
cmUgbm93IGRlZmluZWQgZm9yIHVzaW5nIHdpdGggaVdBUlAuIEFjdHVhbGx5LCB0aGV5IGNvbnRh
aW4gYW4gaVdBUlA8YnI+DQpwcmVmaXggZm9sbG93ZWQgYnkgYSBnZW5lcmljIHBhcnQuIDxicj4N
Cjxicj4NCkRlZmluZSBhIGdlbmVyaWMgaXNlciBoZWFkZXIgYXMgbWFuZGF0b3J5LCB3aGlsZSBz
dGF0aW5nIHRoYXQgdGhlIHVuZGVybHlpbmcgdHJhbnNwb3J0PGJyPg0KcHJvdG9jb2xzIG1heSBh
ZGQgdGhlaXIgb3duIGhlYWRlciBzZWN0aW9ucyBiZWZvcmUgdGhlIGlzZXIgaGVhZGVyPGJyPg0K
YW5kIHByb3ZpZGUgaVdBUlAgYXMgdGhlIGV4YW1wbGUgb2Ygc3VjaCBwcmVmaXguPGJyPg0KPGJy
Pg0KUmVhZCBhbmQgd3JpdGUgYnVmZmVycyBhcmUgaWRlbnRpZmllZCBieSB0aGUgY29tYmluYXRp
b24gb2YgU1RhZyBhbmQgVkEuPGJyPg0KQm90aCBmaWVsZHMgc2hvdWxkIGJlIHJlcXVpcmVkIGZv
ciBhbnkgdHJhbnNwb3J0LiBUaGlzIHJlcXVpcmVtZW50IGRvZXMgbm90IHByZWNsdWRlPGJyPg0K
dXNpbmcgWkJWQSB3aGVyZSBwb3NzaWJsZS4gV2hlbiBWQSBpcyBub3QgdXNlZCAoYXMgaW4gaVdB
UlApIGl0IGlzIHNldCB0byAwLjxicj4NCjxicj4NCkJlbG93IGlzIHRoZSBhY3R1YWxseSB1c2Vk
IGZvcm1hdDo8YnI+DQo8YnI+DQo8c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6JnF1b3Q7Q291cmll
ciBOZXcmcXVvdDsiPnN0cnVjdCBpc2VyX2hkciB7PGJyPg0KJm5ic3A7IHVpbnQ4X3QmbmJzcDsm
bmJzcDsgZmxhZ3M7PGJyPg0KJm5ic3A7IHVpbnQ4X3QmbmJzcDsmbmJzcDsgcnN2ZFszXTs8YnI+
DQombmJzcDsgdWludDMyX3QmbmJzcDsgd3JpdGVfc3RhZzsgLyogd3JpdGUgcmtleSBpbiBJQiAq
Lzxicj4NCiZuYnNwOyB1aW50NjRfdCZuYnNwOyB3cml0ZV92YTs8YnI+DQombmJzcDsgdWludDMy
X3QmbmJzcDsgcmVhZF9zdGFnOyZuYnNwOyAvKiByZWFkIHJrZXkgaW4gSUIgKi88YnI+DQombmJz
cDsgdWludDY0X3QmbmJzcDsgcmVhZF92YTs8YnI+DQp9PGJyPg0KPC9zcGFuPjxicj4NCjMuPGI+
IFVzZSBSQ2FQIGFzIGEgZ2VuZXJhbCB0ZXJtLCBpV0FSUCBhbmQgSUIgYXMgcGFydGljdWxhciBj
YXNlczwvYj48YnI+DQo8YnI+DQpUaGUgbGFuZ3VhZ2Ugb2YgdGhlIGN1cnJlbnQgZHJhZnQgaXMg
aVdBUlAtZG9taW5hdGVkIGluIG1hbnkgcGxhY2VzLiBXZSBuZWVkIHRvPGJyPg0KbWFrZSB0aGUg
dGV4dCB0byByZWZlciB0byBSQ2FQIGFzIGEgY29tbW9uIGRlbm9taW5hdG9yLCBzdHJlc3Npbmcg
dGhlIGRpZmZlcmVuY2VzPGJyPg0KYmV0d2VlbiBwcm90b2NvbHMgd2hlcmUgbmVjZXNzYXJ5Ljxi
cj4NCjxicj4NCiogc2VjdGlvbiAyIDo8YnI+DQombmJzcDs8bzpwPjwvbzpwPjwvcD4NCjxkaXYg
c3R5bGU9Im1hcmdpbi1sZWZ0OjMwLjBwdCI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBz
dHlsZT0iZm9udC1mYW1pbHk6JnF1b3Q7Q291cmllciBOZXcmcXVvdDsiPiZxdW90O1RoZSBpV0FS
UCBwcm90b2NvbCBzdGFjayBwcm92aWRlcyBkaXJlY3QgZGF0YSBwbGFjZW1lbnQ8YnI+DQpmdW5j
dGlvbmFsaXR5IHRoYXQgaXMgdXNhYmxlIGluIHByYWN0aWNlLCBhbmQgaW4gYWRkaXRpb24sIHRo
ZXJlIGlzPGJyPg0KYWxzbyBpbnRlcmVzdCBpbiB1c2luZyBpU0NTSSB3aXRoIG90aGVyIFJETUEg
cHJvdG9jb2wgc3RhY2tzIHRoYXQ8YnI+DQpzdXBwb3J0IGRpcmVjdCBkYXRhIHBsYWNlbWVudCwg
c3VjaCBhcyB0aGUgb25lIHByb3ZpZGVkIGJ5PGJyPg0KSW5maW5pQmFuZC4mbmJzcDsgVGhlIGdl
bmVyaWMgdGVybSBSRE1BLUNhcGFibGUgUHJvdG9jb2wgKFJDYVApIGlzIHVzZWQ8YnI+DQp0byBy
ZWZlciB0byB0aGUgUkRNQSBmdW5jdGlvbmFsaXR5IHByb3ZpZGVkIGJ5IHN1Y2ggcHJvdG9jb2wg
c3RhY2tzLiZxdW90Ozwvc3Bhbj48bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPHAgY2xhc3M9Ik1z
b05vcm1hbCIgc3R5bGU9Im1hcmdpbi1ib3R0b206MTIuMHB0Ij48YnI+DQpzaG91bGQgcmVhZDo8
bzpwPjwvbzpwPjwvcD4NCjxkaXYgc3R5bGU9Im1hcmdpbi1sZWZ0OjMwLjBwdCI+DQo8cCBjbGFz
cz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6JnF1b3Q7Q291cmllciBOZXcm
cXVvdDsiPiZxdW90O1RoZSBnZW5lcmljIHRlcm0gUkRNQS1DYXBhYmxlIFByb3RvY29sIChSQ2FQ
KSBpcyB1c2VkIHRvIHJlZmVyIHRvPGJyPg0KdGhlIFJETUEgZnVuY3Rpb25hbGl0eSBwcm92aWRl
ZCBieSB0aGUgdW5kZXJseWluZyBwcm90b2NvbCBzdGFjay48YnI+DQpUd28gcHJhY3RpY2FsIGV4
YW1wbGVzIG9mIHByb3RvY29sIHN0YWNrcyBwcm92aWRpbmcgZGlyZWN0IGRhdGEgcGxhY2VtZW50
PGJyPg0KZnVuY3Rpb25hbGl0eSBhcmUgaVdBUlAgYW5kIEluZmluaUJhbmQuJnF1b3Q7PG86cD48
L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFy
Z2luLWJvdHRvbToxMi4wcHQiPjxicj4NCiogc2VjdGlvbiA1LjEuMSA6PG86cD48L286cD48L3A+
DQo8ZGl2IHN0eWxlPSJtYXJnaW4tbGVmdDozMC4wcHQiPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+
Jm5ic3A7PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiZxdW90O0NvdXJpZXIgTmV3JnF1b3Q7Ij4m
cXVvdDtBbW9uZyB0aGUgY29ubmVjdGlvbiByZXNvdXJjZXMgYWxsb2NhdGVkIGF0IHRoZSBpbml0
aWF0b3IgaXMgdGhlPGJyPg0KSW5ib3VuZCBSRE1BIFJlYWQgUXVldWUgRGVwdGggKElSRCkgLi4u
PGJyPg0KLi4uIEluaXRpYWxseSwgdGhlIGlTRVItSVJEIHZhbHVlIGF0IHRoZSBpbml0aWF0b3Ig
U0hPVUxEPGJyPg0KYmUgc2V0IHRvIHRoZSBJUkQgdmFsdWUgYXQgdGhlIGluaXRpYXRvciBhbmQg
TVVTVCBOT1QgYmUgbW9yZSB0aGFuPGJyPg0KdGhlIElSRCB2YWx1ZS4mcXVvdDs8L3NwYW4+PG86
cD48L286cD48L3A+DQo8L2Rpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4t
Ym90dG9tOjEyLjBwdCI+PGJyPg0Kc2hvdWxkIGJlIHByZWZpeGVkIHdpdGg6PG86cD48L286cD48
L3A+DQo8ZGl2IHN0eWxlPSJtYXJnaW4tbGVmdDozMC4wcHQiPg0KPHAgY2xhc3M9Ik1zb05vcm1h
bCI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiZxdW90O0NvdXJpZXIgTmV3JnF1b3Q7Ij4mcXVv
dDtGb3IgaVdBUlAgYmFzZWQgaW1wbGVtZW50YXRpb25zLCAuLi4mcXVvdDs8L3NwYW4+PG86cD48
L286cD48L3A+DQo8L2Rpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxicj4NCjQuIDxiPk1hcmsg
aVdBUlAtc3BlY2lmaWMgc2VjdGlvbnMgYXMgc3VjaDwvYj48YnI+DQo8YnI+DQpTb21lIHByb3Rv
Y29sLXNwZWNpZmljIHN0YXRlbWVudHMsIGxpa2UgaW4gYWJvdmUsIGFyZSBzY2F0dGVyZWQgdGhy
b3VnaCB0aGU8YnI+DQp0ZXh0LCBidXQgdGhlcmUgYXJlIGVudGlyZSBzZWN0aW9ucyB3aGljaCBh
cmUgaVdBUlAtc3BlY2lmaWMgKFJETUFQIGZlYXR1cmVzPGJyPg0KaGF2aW5nIG5vIHBhcmFsbGUg
aW4gSUIsIGV0Yy4pIFdlIHNob3VsZCBleHBsaWNpdGx5IHN0YXRlIHRoaXMuIDxicj4NClRoZSBz
YW1lIGdvZXMgZm9yIElCLCBvZiBjb3Vyc2UsIGFsdGhvdWdoIHJpZ2h0IG5vdyB0aGVyZSBhcmUg
bm8gc3VjaCBjbGF1c2VzLjxicj4NCjxicj4NCkluIHBhcnRpY3VsYXIsIHNlY3Rpb25zIDguMi4g
YW5kIDguMy4gd2l0aCBzdWItc2VjdGlvbnMgOC4zLng8YnI+DQpzaG91bGQgYmUgZWl0aGVyIGdy
b3VwZWQgYXMgdGhlIHN1YnNlY3Rpb25zIG9mPGJyPg0KJm5ic3A7JnF1b3Q7OC4yLiBpV0FSUCBj
b25zaWRlcmF0aW9ucyBpbiBGbG93IENvbnRyb2wmcXVvdDs8YnI+DQpvciByZXRhaW4gdGhlaXIg
bnVtYmVycyBhbmQgaGF2ZSAmcXVvdDsoaVdBUlAgc3BlY2lmaWMpJnF1b3Q7IGFkZGVkIHRvIHRo
ZWlyIHRpdGxlczxicj4NCjxicj4NCjUuIDxiPkZpeCBhIHR5cG8gaW4gc2VjdGlvbiBudW1iZXJp
bmc8L2I+PG86cD48L286cD48L3A+DQo8L2Rpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+
Jm5ic3A7PC9vOnA+PC9wPg0KPGRpdj4NCjxkaXY+DQo8ZGl2Pg0KPGRpdj4NCjxkaXY+DQo8ZGl2
Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tYm90dG9tOjEyLjBw
dCI+QXBwZW5kaXggQSBzaG91bGQgYmUgc2VjdGlvbiAxNS48YnI+DQo8YnI+DQo2LjxiPiBpU0VS
IEhlbGxvLUhlbGxvUmVwbHk8L2I+PGJyPg0KPGJyPg0KSGVsbG8tSGVsbG9SZXBseSBzaG91bGQg
YmUgbWFkZSBvcHRpb25hbCBhbmQgdHJhbnNwb3J0IGRlcGVuZGVudC48YnI+DQo8YnI+DQpSaWdo
dCBub3cgdGhlIElCIGJhc2VkIHRhcmdldCdzIGltcGxlbWVudGF0aW9uIHRlcm1pbmF0ZXMgdGhl
IGNvbm5lY3Rpb24gaWYgaXQgcmVjZWl2ZXMgSGVsbG8uPGJyPg0KSUItYmFzZWQgaW1wbGVtZW50
YXRpb24gZG9lcyBub3QgbmVlZCBhbnkgSGVsbG8gZXhjaGFuZ2UsIHNvIHRoZSBpbXBsZW1lbnRl
ZCBiZWhhdmlvciBtYXkgYmU8YnI+DQp0dXJuZWQgaW50byBzdGFuZGFyZC4gaVdBUlAgZG9lcyBu
ZWVkIEhlbGxvLUhlbGxvUmVwbHkgYW5kIGl0cyBmb3JtYXQgaXMgcHJvdmlkZWQuPGJyPg0KPGJy
Pg0KV2UgY2FuIGRlZmluZSBhIG5ldyBrZXkgY2FsbGVkIGlTRVJIZWxsb1JlcXVpcmVkIHdpdGgg
dGhlIGRlZmF1bHQgYmVpbmcgJnF1b3Q7Tm8mcXVvdDssIHNvPGJyPg0KdGhhdCB0aGUgY3VycmVu
dCBpbXBsZW1lbnRhdGlvbiBiZWNvbWVzIGNvbXBsaWFudC48YnI+DQo8YnI+DQpOZWVkIHRvIHVw
ZGF0ZSBzb21lIHN0YXRlbWVudHMgcmVnYXJkaW5nIElzZXJIZWxsbywgbGlrZSBpbiA1LjEuMSBh
bmQgNS4xLjI8YnI+DQphbGxldmlhdGluZyBzb21lIE1VU1RzIHdpdGggJnF1b3Q7aWYgcmVsZXZh
bnQgZm9yIHRyYW5zcG9ydCZxdW90Oy1zdHlsZSBvZiBjb21tZW50cy48YnI+DQo8YnI+DQo3Ljxi
PiBEZWZhdWx0IHZhbHVlcyBmb3IgTWF4T3V0c3RhbmRpbmdVbmV4cGVjdGVkUERVcyBhbmQgTWF4
QUhTTGVuZ3RoPC9iPjxicj4NCjxicj4NClRoZSBkZWZhdWx0cyBmb3IgTWF4T3V0c3RhbmRpbmdV
bmV4cGVjdGVkUERVcyBhbmQgTWF4QUhTTGVuZ3RoPGJyPg0Kc2hvdWxkIGNoYW5nZSBmcm9tIDAg
KHVubGltaXRlZCkgdG8gc29tZSBzbWFsbCByZWFzb25hYmxlIHZhbHVlczxicj4NCnRvIHByb3Rl
Y3QgYSBzaWRlIG5vdCBzZW5kaW5nIHRoZXNlIGtleXMgZnJvbSBwb3RlbnRpYWxseSBoYXJtZnVs
IGFzc3VtcHRpb24gPGJyPg0KYnkgaXRzIHBlZXIgYWJvdXQgdW5saW1pdGVkIHJlc291cmNlcyBh
dmFpbGFibGUsIHNvIHRoYXQgYnVmZmVyIG92ZXJmbG93IChNYXhBSFNMZW5ndGgpPGJyPg0Kb3Ig
c2VuZGluZyBtb3JlIGJ1ZmZlcnMgdGhhbiBhY3R1YWxseSBwb3N0ZWQgKE1heE91dHN0YW5kaW5n
VW5leHBlY3RlZFBEVXMpIG1heSByZXN1bHQuPGJyPg0KPGJyPg0KcHJvcG9zZWQgZGVmYXVsdCB2
YWx1ZXM6PGJyPg0KPHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiZxdW90O0NvdXJpZXIgTmV3JnF1
b3Q7Ij5NYXhPdXRzdGFuZGluZ1VuZXhwZWN0ZWRQRFVzID0gNDxicj4NCk1heEFIU0xlbmd0aCA9
IDI1Njwvc3Bhbj4gKGFjY291bnRzIGZvciAxIFZhciBsZW4gQ0RCICYjNDM7IDEgYmlkaXIgcmVh
ZC1sZW5ndGgpPGJyPg0KPGJyPg0KOC4gPGI+SWdub3JpbmcgTWF4UmVjdkRhdGFTZWdtZW50TGVu
Z3RoIGtleTwvYj48YnI+DQo8YnI+DQpJdCBzaG91bGQgYmUgY2xhcmlmaWVkIHRoYXQgTWF4UmVj
dkRhdGFTZWdtZW50TGVuZ3RoIG1heSBiZSBzZW50IGJ1dCBNVVNUIGJlIGlnbm9yZWQsPGJyPg0K
YmVjYXVzZSB7SW5pdGlhdG9yfFRhcmdldH1SZWN2RGF0YVNlZ21lbnRMZW5ndGggaGF2ZSB0aGVp
ciBvd24gZGVmYXVsdHMuIDxicj4NCjxicj4NCklmIHtJbml0aWF0b3J8VGFyZ2V0fVJlY3ZEYXRh
U2VnbWVudExlbmd0aCBhcmUgc2VudCwgdGhlIGRlY2xhcmVkIHZhbHVlcyB3b3VsZCBvdmVycmlk
ZQ0KPGJyPg0KTWF4UmVjdkRhdGFTZWdtZW50TGVuZ3RoIGFueXdheSwgYW5kIGlmIHRoZXkgYXJl
IG5vdCBzZW50LCB0aGUgZGVmYXVsdCB2YWx1ZXM8YnI+DQptdXN0IGJlIHVzZWQgdG8gYXZvaWQg
Y29uZnVzaW5nIHNjZW5hcmlvcy48YnI+DQo8YnI+DQo8bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0K
PC9kaXY+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9kaXY+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9kaXY+DQo8
L2Rpdj4NCjwvYm9keT4NCjwvaHRtbD4NCg==

--_000_F83812DF4B59B9499C1BC978336D91745EDE207FTK5EX14MBXC118r_--

From Michael@huaweisymantec.com  Wed Jun 29 14:26:21 2011
Return-Path: <Michael@huaweisymantec.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7DBA69E8063 for <storm@ietfa.amsl.com>; Wed, 29 Jun 2011 14:26:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.598
X-Spam-Level: 
X-Spam-Status: No, score=-2.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oDUdhF12x97k for <storm@ietfa.amsl.com>; Wed, 29 Jun 2011 14:26:19 -0700 (PDT)
Received: from mta2.huaweisymantec.com (mta2.huaweisymantec.com [218.17.155.15]) by ietfa.amsl.com (Postfix) with ESMTP id 1E1379E805D for <storm@ietf.org>; Wed, 29 Jun 2011 14:26:19 -0700 (PDT)
MIME-version: 1.0
Content-type: multipart/alternative; boundary="Boundary_(ID_HxQcbemlksbqBL6Hji2lNg)"
Received: from hstml01-in.huaweisymantec.com ([172.26.3.42]) by hstga02-in.huaweisymantec.com (Sun Java(tm) System Messaging Server 6.3-8.03 (built Apr 24 2009; 32bit)) with ESMTP id <0LNK00E9PM7S0Z70@hstga02-in.huaweisymantec.com> for storm@ietf.org; Thu, 30 Jun 2011 05:26:16 +0800 (CST)
Received: from m90003900a ([69.199.248.19]) by hstml01-in.huaweisymantec.com (Sun Java(tm) System Messaging Server 6.3-8.03 (built Apr 24 2009; 32bit)) with ESMTPA id <0LNK008W6M8T9D00@hstml01-in.huaweisymantec.com> for storm@ietf.org; Thu, 30 Jun 2011 05:26:57 +0800 (CST)
Message-id: <971779FD6B314A40B1A3658E521C0D4A@china.huawei.com>
From: Michael Ko <Michael@huaweisymantec.com>
To: Tom Talpey <ttalpey@microsoft.com>, Alexander Nezhinsky <nezhinsky@gmail.com>, storm@ietf.org
References: <BANLkTi=GVHUBdsZiwsRXxETd_rU=F8f85w@mail.gmail.com> <F83812DF4B59B9499C1BC978336D91745EDE207F@TK5EX14MBXC118.redmond.corp.microsoft.com>
Date: Wed, 29 Jun 2011 14:26:12 -0700
X-Priority: 3
X-MSMail-priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.5931
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2900.6109
Subject: Re: [storm] iSER draft
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 29 Jun 2011 21:26:21 -0000

This is a multi-part message in MIME format.

--Boundary_(ID_HxQcbemlksbqBL6Hji2lNg)
Content-type: text/plain; charset=UTF-8
Content-transfer-encoding: quoted-printable

Tom,

As originally written, the iSER spec always assumes ZBVA (zero-based =
virtual address).  But when iSER is used in conjunction with transports =
other than iWARP, there is a need to include the virtual address for the =
tagged buffer.  Such is the case for Infiniband.  Accordingly, the =
Virtual Address field has been added to the iSCSI Control Type PDUs =
(sec. 9.2).  Alex pointed out that the tagged buffer should now be =
identified by the 4-tuple (STag, VA, TO, and length) instead.  This =
should be updated everywhere where the term "tagged buffer" is mentioned =
in the spec.  For example, in sec. 1.1 under Advertisement, the sentence =
"A typical method would be for the iSER Layer to embed the Tagged =
Buffer's STag, TO, and buffer length in a Send Message destined for the =
remote iSER Layer" should be modified to "A typical method would be for =
the iSER Layer to embed the Tagged Buffer's STag, VA, TO, and buffer =
length in a Send Message destined for the remote iSER Layer."  BTW, this =
sentence also answers your question as to whether VA is passed on the =
wire.

The Virtual Address is literally the virtual address of the tagged =
buffer.  The tagged offset is the displacement relative to the beginning =
of the buffer starting at the virtual address.

We could rephrase the definition of Virtual Address in sec. 1.1 as =
follows:

"Virtual Address (VA) - This is the virtual address of the Tagged Buffer =
on a Node.  When set to 0, it indicates that a virtual address is not =
needed to locate the Tagged Buffer."

Mike

----- Original Message -----=20
  From: Tom Talpey=20
  To: Alexander Nezhinsky ; storm@ietf.org=20
  Sent: Wednesday, June 29, 2011 8:30 AM
  Subject: Re: [storm] iSER draft


  Working Group Co-chair hat off.

  =20

  Alexander, thanks for contributing these updates. I have some general =
comments on two items.

  =20

  > * Add a definition of VA to section 1.1. For example:

  >"Virtual Address (VA) - an identifier supplied by RCaP together with =
STag=20
  >when registering a memory buffer. In some modes it is not used,
  >and VA=3D0 is implied. These modes are known as ZBVA (Zero Based VA).
  >In such cases VA should be set to zero when buffer advertisements
  >are communicated."

  =20

  I think it may be problematic to introduce the new term =
=E2=80=9CVirtual Address=E2=80=9D without disambiguating it from the =
existing term =E2=80=9CTagged Offset=E2=80=9D. The term VA does not =
appear in RFC5046, and in the update, was introduced in only one message =
type (control PDU sections 6.9 and 9.2). Also, your definition implies =
that it is provided locally to an unspecified API. If it is not passed =
on the wire, then it seems superfluous to the protocol.

  =20

  Either way, the definition needs to avoid the statements =E2=80=9CIn =
some modes=E2=80=9D and =E2=80=9Cshould be set to zero=E2=80=9D. These =
are protocol behavior and such discussion is be avoided in the glossary. =
I would suggest deleting all but the first sentence, after resolving the =
use of VA of course.

  =20

  >* Definition of Tagged Buffer:

  >"Tagged Buffer - A buffer that is explicitly Advertised to the iSER =
Layer
  >at the remote node  through the exchange of an STag, Tagged Offset, =
and length."

  >
  >should read:

  >"...the exchange of an STag, VA, Tagged Offset, and length."

  =20

  Now I=E2=80=99m really confused. Is the Tagged Buffer represented as a =
4-tuple in the new protocol? If so, where will this be described?

  =20

  =20

  =20

  From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf =
Of Alexander Nezhinsky
  Sent: Wednesday, June 29, 2011 7:59 AM
  To: storm@ietf.org; david.black@emc.com; michael@huaweisymantec.com
  Subject: [storm] iSER draft

  =20

  Hi,

  I would like to post some comments and suggestions regarding the =
latest iSER draft:=20
  http://www.watersprings.org/pub/id/draft-ietf-storm-iser-01.txt

  Unfortunately no comments have been posted since it expired. I'm =
acting both as a=20
  contributor and a practical implementer.
  Earlier this year i submitted a new stable implementation of iSER, =
which was accepted
  by the open source STGT project. As a part of their official release =
it already made its way=20
  to a few Linux distributions.

  There is a gap between the existing implementation and the original =
RFC, which is mostly
  (but not exclusively) related to the possibility of implementation =
over Infiniband.
  This gap was almost completely covered by the recent =
draft-ietf-storm-iser-01.
  My comments below strive to eliminate the remaining discrepancies.
  The general idea is to either reflect some implemented changes in the =
spec,
  or to pave a comprehensive way for the implementation to become spec =
compliant.

  Alexander Nezhinsky

  ---

  1. Account for VA in buffer advertisement

  Add VA to the list of attributes which identify advertized buffers and =
change related definitions and statements.

  * Add a definition of VA to section 1.1. For example:

  "Virtual Address (VA) - an identifier supplied by RCaP together with =
STag=20
  when registering a memory buffer. In some modes it is not used,
  and VA=3D0 is implied. These modes are known as ZBVA (Zero Based VA).
  In such cases VA should be set to zero when buffer advertisements
  are communicated."

  =20

  * Definition of Remote Mapping:

  "Remote Mapping - A task state record maintained by the iSER Layer
   that associates the Initiator Task Tag to the Advertised STag(s)..."


  should read:

  "...that associates the Initiator Task Tag to the Advertised STag(s)
  and Virtual Address(es)..."


  * Definition of Tagged Buffer:

  "Tagged Buffer - A buffer that is explicitly Advertised to the iSER =
Layer
  at the remote node  through the exchange of an STag, Tagged Offset, =
and length."


  should read:

  "...the exchange of an STag, VA, Tagged Offset, and length."


  2. iser message format

  The headers are now defined for using with iWARP. Actually, they =
contain an iWARP
  prefix followed by a generic part.=20

  Define a generic iser header as mandatory, while stating that the =
underlying transport
  protocols may add their own header sections before the iser header
  and provide iWARP as the example of such prefix.

  Read and write buffers are identified by the combination of STag and =
VA.
  Both fields should be required for any transport. This requirement =
does not preclude
  using ZBVA where possible. When VA is not used (as in iWARP) it is set =
to 0.

  Below is the actually used format:

  struct iser_hdr {
    uint8_t   flags;
    uint8_t   rsvd[3];
    uint32_t  write_stag; /* write rkey in IB */
    uint64_t  write_va;
    uint32_t  read_stag;  /* read rkey in IB */
    uint64_t  read_va;
  }

  3. Use RCaP as a general term, iWARP and IB as particular cases

  The language of the current draft is iWARP-dominated in many places. =
We need to
  make the text to refer to RCaP as a common denominator, stressing the =
differences
  between protocols where necessary.

  * section 2 :
  =20

  "The iWARP protocol stack provides direct data placement
  functionality that is usable in practice, and in addition, there is
  also interest in using iSCSI with other RDMA protocol stacks that
  support direct data placement, such as the one provided by
  InfiniBand.  The generic term RDMA-Capable Protocol (RCaP) is used
  to refer to the RDMA functionality provided by such protocol stacks."


  should read:

  "The generic term RDMA-Capable Protocol (RCaP) is used to refer to
  the RDMA functionality provided by the underlying protocol stack.
  Two practical examples of protocol stacks providing direct data =
placement
  functionality are iWARP and InfiniBand."


  * section 5.1.1 :

   "Among the connection resources allocated at the initiator is the
  Inbound RDMA Read Queue Depth (IRD) ...
  ... Initially, the iSER-IRD value at the initiator SHOULD
  be set to the IRD value at the initiator and MUST NOT be more than
  the IRD value."


  should be prefixed with:

  "For iWARP based implementations, ..."


  4. Mark iWARP-specific sections as such

  Some protocol-specific statements, like in above, are scattered =
through the
  text, but there are entire sections which are iWARP-specific (RDMAP =
features
  having no paralle in IB, etc.) We should explicitly state this.=20
  The same goes for IB, of course, although right now there are no such =
clauses.

  In particular, sections 8.2. and 8.3. with sub-sections 8.3.x
  should be either grouped as the subsections of
   "8.2. iWARP considerations in Flow Control"
  or retain their numbers and have "(iWARP specific)" added to their =
titles

  5. Fix a typo in section numbering

  =20

  Appendix A should be section 15.

  6. iSER Hello-HelloReply

  Hello-HelloReply should be made optional and transport dependent.

  Right now the IB based target's implementation terminates the =
connection if it receives Hello.
  IB-based implementation does not need any Hello exchange, so the =
implemented behavior may be
  turned into standard. iWARP does need Hello-HelloReply and its format =
is provided.

  We can define a new key called iSERHelloRequired with the default =
being "No", so
  that the current implementation becomes compliant.

  Need to update some statements regarding IserHello, like in 5.1.1 and =
5.1.2
  alleviating some MUSTs with "if relevant for transport"-style of =
comments.

  7. Default values for MaxOutstandingUnexpectedPDUs and MaxAHSLength

  The defaults for MaxOutstandingUnexpectedPDUs and MaxAHSLength
  should change from 0 (unlimited) to some small reasonable values
  to protect a side not sending these keys from potentially harmful =
assumption=20
  by its peer about unlimited resources available, so that buffer =
overflow (MaxAHSLength)
  or sending more buffers than actually posted =
(MaxOutstandingUnexpectedPDUs) may result.

  proposed default values:
  MaxOutstandingUnexpectedPDUs =3D 4
  MaxAHSLength =3D 256 (accounts for 1 Var len CDB + 1 bidir =
read-length)

  8. Ignoring MaxRecvDataSegmentLength key

  It should be clarified that MaxRecvDataSegmentLength may be sent but =
MUST be ignored,
  because {Initiator|Target}RecvDataSegmentLength have their own =
defaults.=20

  If {Initiator|Target}RecvDataSegmentLength are sent, the declared =
values would override=20
  MaxRecvDataSegmentLength anyway, and if they are not sent, the default =
values
  must be used to avoid confusing scenarios.





-------------------------------------------------------------------------=
-----


  _______________________________________________
  storm mailing list
  storm@ietf.org
  https://www.ietf.org/mailman/listinfo/storm

--Boundary_(ID_HxQcbemlksbqBL6Hji2lNg)
Content-type: text/html; charset=UTF-8
Content-transfer-encoding: quoted-printable

=EF=BB=BF<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns=3D"http://www.w3.org/TR/REC-html40" xmlns:v =3D=20
"urn:schemas-microsoft-com:vml" xmlns:o =3D=20
"urn:schemas-microsoft-com:office:office" xmlns:w =3D=20
"urn:schemas-microsoft-com:office:word" xmlns:m =3D=20
"http://schemas.microsoft.com/office/2004/12/omml"><HEAD>
<META content=3D"text/html; charset=3Dutf-8" http-equiv=3DContent-Type>
<META name=3DGENERATOR content=3D"MSHTML 8.00.6001.19088">
<STYLE>@font-face {
	font-family: Calibri;
}
@font-face {
	font-family: Tahoma;
}
@page WordSection1 {size: 8.5in 11.0in; margin: 1.0in 1.0in 1.0in 1.0in; =
}
P.MsoNormal {
	MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif"; FONT-SIZE: =
12pt
}
LI.MsoNormal {
	MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif"; FONT-SIZE: =
12pt
}
DIV.MsoNormal {
	MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif"; FONT-SIZE: =
12pt
}
A:link {
	COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlink {
	COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
A:visited {
	COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
	COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
P.MsoListParagraph {
	MARGIN: 0in 0in 0pt 0.5in; FONT-FAMILY: "Times New Roman","serif"; =
FONT-SIZE: 12pt; mso-style-priority: 34
}
LI.MsoListParagraph {
	MARGIN: 0in 0in 0pt 0.5in; FONT-FAMILY: "Times New Roman","serif"; =
FONT-SIZE: 12pt; mso-style-priority: 34
}
DIV.MsoListParagraph {
	MARGIN: 0in 0in 0pt 0.5in; FONT-FAMILY: "Times New Roman","serif"; =
FONT-SIZE: 12pt; mso-style-priority: 34
}
SPAN.EmailStyle17 {
	FONT-FAMILY: "Calibri","sans-serif"; COLOR: #1f497d; mso-style-type: =
personal-reply
}
.MsoChpDefault {
	FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: export-only
}
DIV.WordSection1 {
	page: WordSection1
}
</STYLE>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]--></HEAD>
<BODY lang=3DEN-US link=3Dblue bgColor=3D#ffffff vLink=3Dpurple>
<DIV><FONT size=3D2>Tom,</FONT></DIV>
<DIV><FONT size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2>As originally written, the iSER spec always assumes =
ZBVA=20
(zero-based virtual address).&nbsp; But when iSER is used in conjunction =
with=20
transports other than iWARP, there is a need to include the virtual =
address for=20
the tagged buffer.&nbsp; Such is the case for Infiniband.&nbsp; =
Accordingly, the=20
Virtual Address field has been added to the iSCSI Control Type PDUs =
(sec.=20
9.2).&nbsp; Alex pointed out that the tagged buffer should now be =
identified by=20
the 4-tuple (STag, VA, TO, and length) instead.&nbsp; This should be =
updated=20
everywhere where the term "tagged buffer" is mentioned in the =
spec.&nbsp; For=20
example, in sec. 1.1 under Advertisement,&nbsp;the sentence "A typical =
method=20
would be for the iSER Layer to embed the Tagged Buffer's STag, TO, and =
buffer=20
length in a Send Message destined for the remote iSER Layer" should be =
modified=20
to "A typical method would be for the iSER Layer to embed the Tagged =
Buffer's=20
STag, VA, TO, and buffer length in a Send Message destined for the =
remote iSER=20
Layer."&nbsp; BTW, this sentence also answers your question as to =
whether VA is=20
passed on the wire.</FONT></DIV>
<DIV><FONT size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2>The Virtual Address is literally the virtual address =
of the=20
tagged buffer.&nbsp; The tagged offset is the displacement relative to =
the=20
beginning of the buffer starting at the virtual address.</FONT></DIV>
<DIV><FONT size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2>We could rephrase the definition of Virtual Address =
in sec.=20
1.1 as follows:</FONT></DIV>
<DIV><FONT size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2>"Virtual Address (VA) - This is the virtual address =
of=20
the&nbsp;Tagged Buffer on a Node.&nbsp; When set to 0, it indicates that =
a=20
virtual address is not needed to locate the Tagged Buffer."</FONT></DIV>
<DIV><FONT size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2>Mike</FONT></DIV>
<DIV><FONT size=3D2></FONT>&nbsp;</DIV>
<DIV>----- Original Message ----- </DIV>
<BLOCKQUOTE=20
style=3D"BORDER-LEFT: #000000 2px solid; PADDING-LEFT: 5px; =
PADDING-RIGHT: 0px; MARGIN-LEFT: 5px; MARGIN-RIGHT: 0px">
  <DIV=20
  style=3D"FONT: 10pt arial; BACKGROUND: #e4e4e4; font-color: =
black"><B>From:</B>=20
  <A title=3Dttalpey@microsoft.com =
href=3D"mailto:ttalpey@microsoft.com">Tom=20
  Talpey</A> </DIV>
  <DIV style=3D"FONT: 10pt arial"><B>To:</B> <A =
title=3Dnezhinsky@gmail.com=20
  href=3D"mailto:nezhinsky@gmail.com">Alexander Nezhinsky</A> ; <A=20
  title=3Dstorm@ietf.org =
href=3D"mailto:storm@ietf.org">storm@ietf.org</A> </DIV>
  <DIV style=3D"FONT: 10pt arial"><B>Sent:</B> Wednesday, June 29, 2011 =
8:30=20
  AM</DIV>
  <DIV style=3D"FONT: 10pt arial"><B>Subject:</B> Re: [storm] iSER =
draft</DIV>
  <DIV><BR></DIV>
  <DIV class=3DWordSection1>
  <P class=3DMsoNormal><SPAN=20
  style=3D"FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; =
FONT-SIZE: 11pt">Working=20
  Group Co-chair hat off.<o:p></o:p></SPAN></P>
  <P class=3DMsoNormal><SPAN=20
  style=3D"FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; =
FONT-SIZE: 11pt"><o:p>&nbsp;</o:p></SPAN></P>
  <P class=3DMsoNormal><SPAN=20
  style=3D"FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; =
FONT-SIZE: 11pt">Alexander,=20
  thanks for contributing these updates. I have some general comments on =
two=20
  items.<o:p></o:p></SPAN></P>
  <P class=3DMsoNormal><SPAN=20
  style=3D"FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; =
FONT-SIZE: 11pt"><o:p>&nbsp;</o:p></SPAN></P>
  <P class=3DMsoNormal>&gt; * Add a definition of VA to section 1.1. For =

  example:<o:p></o:p></P>
  <P class=3DMsoNormal><SPAN style=3D"FONT-FAMILY: 'Courier =
New'">&gt;"Virtual=20
  Address (VA) - an identifier supplied by RCaP together with STag =
<BR>&gt;when=20
  registering a memory buffer. In some modes it is not used,<BR>&gt;and =
VA=3D0 is=20
  implied. These modes are known as ZBVA (Zero Based VA).<BR>&gt;In such =
cases=20
  VA should be set to zero when buffer advertisements<BR>&gt;are=20
  communicated."<o:p></o:p></SPAN></P>
  <P class=3DMsoNormal><SPAN=20
  style=3D"FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; =
FONT-SIZE: 11pt"><o:p>&nbsp;</o:p></SPAN></P>
  <P class=3DMsoNormal><SPAN=20
  style=3D"FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; =
FONT-SIZE: 11pt">I=20
  think it may be problematic to introduce the new term =E2=80=9CVirtual =
Address=E2=80=9D=20
  without disambiguating it from the existing term =E2=80=9CTagged =
Offset=E2=80=9D. The term VA=20
  does not appear in RFC5046, and in the update, was introduced in only =
one=20
  message type (control PDU sections 6.9 and 9.2). Also, your definition =
implies=20
  that it is provided locally to an unspecified API. If it is not passed =
on the=20
  wire, then it seems superfluous to the protocol.<o:p></o:p></SPAN></P>
  <P class=3DMsoNormal><SPAN=20
  style=3D"FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; =
FONT-SIZE: 11pt"><o:p>&nbsp;</o:p></SPAN></P>
  <P class=3DMsoNormal><SPAN=20
  style=3D"FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; =
FONT-SIZE: 11pt">Either=20
  way, the definition needs to avoid the statements =E2=80=9CIn some =
modes=E2=80=9D and =E2=80=9Cshould=20
  be set to zero=E2=80=9D. These are protocol behavior and such =
discussion is be avoided=20
  in the glossary. I would suggest deleting all but the first sentence, =
after=20
  resolving the use of VA of course.<o:p></o:p></SPAN></P>
  <P class=3DMsoNormal><SPAN=20
  style=3D"FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; =
FONT-SIZE: 11pt"><o:p>&nbsp;</o:p></SPAN></P>
  <P style=3D"MARGIN-BOTTOM: 12pt" class=3DMsoNormal>&gt;* Definition of =
Tagged=20
  Buffer:<o:p></o:p></P>
  <P class=3DMsoNormal><SPAN style=3D"FONT-FAMILY: 'Courier =
New'">&gt;"Tagged Buffer=20
  - A buffer that is explicitly Advertised to the iSER Layer<BR>&gt;at =
the=20
  remote node</SPAN>&nbsp; <SPAN style=3D"FONT-FAMILY: 'Courier =
New'">through the=20
  exchange of an STag, Tagged Offset, and length."</SPAN><o:p></o:p></P>
  <P style=3D"MARGIN-BOTTOM: 12pt" class=3DMsoNormal>&gt;<BR>&gt;should=20
  read:<o:p></o:p></P>
  <P class=3DMsoNormal><SPAN style=3D"FONT-FAMILY: 'Courier =
New'">&gt;"...the=20
  exchange of an STag, VA, Tagged Offset, and =
length."</SPAN><o:p></o:p></P>
  <P class=3DMsoNormal><SPAN=20
  style=3D"FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; =
FONT-SIZE: 11pt"><o:p>&nbsp;</o:p></SPAN></P>
  <P class=3DMsoNormal><SPAN=20
  style=3D"FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; =
FONT-SIZE: 11pt">Now=20
  I=E2=80=99m really confused. Is the Tagged Buffer represented as a =
4-tuple in the new=20
  protocol? If so, where will this be described?<o:p></o:p></SPAN></P>
  <P class=3DMsoNormal><SPAN=20
  style=3D"FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; =
FONT-SIZE: 11pt"><o:p>&nbsp;</o:p></SPAN></P>
  <P class=3DMsoNormal><SPAN=20
  style=3D"FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; =
FONT-SIZE: 11pt"><o:p>&nbsp;</o:p></SPAN></P>
  <P class=3DMsoNormal><SPAN=20
  style=3D"FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; =
FONT-SIZE: 11pt"><o:p>&nbsp;</o:p></SPAN></P>
  <P class=3DMsoNormal><B><SPAN=20
  style=3D"FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: =
10pt">From:</SPAN></B><SPAN=20
  style=3D"FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt">=20
  storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] <B>On Behalf Of =

  </B>Alexander Nezhinsky<BR><B>Sent:</B> Wednesday, June 29, 2011 7:59=20
  AM<BR><B>To:</B> storm@ietf.org; david.black@emc.com;=20
  michael@huaweisymantec.com<BR><B>Subject:</B> [storm] iSER=20
  draft<o:p></o:p></SPAN></P>
  <P class=3DMsoNormal><o:p>&nbsp;</o:p></P>
  <DIV>
  <DIV>
  <P class=3DMsoNormal>Hi,<BR><BR>I would like to post some comments and =

  suggestions regarding the latest iSER draft: <BR><A=20
  =
href=3D"http://www.watersprings.org/pub/id/draft-ietf-storm-iser-01.txt">=
http://www.watersprings.org/pub/id/draft-ietf-storm-iser-01.txt</A><BR><B=
R>Unfortunately=20
  no comments have been posted since it expired. I'm acting both as a=20
  <BR>contributor and a practical implementer.<BR>Earlier this year i =
submitted=20
  a new stable implementation of iSER, which was accepted<BR>by the open =
source=20
  STGT&nbsp;project. As a part of their official release it already made =
its way=20
  <BR>to a few Linux distributions.<BR><BR>There is a gap between the =
existing=20
  implementation and the original RFC, which is mostly<BR>(but not =
exclusively)=20
  related to the possibility of implementation over Infiniband.<BR>This =
gap was=20
  almost completely covered by the recent =
draft-ietf-storm-iser-01.<BR>My=20
  comments below strive to eliminate the remaining discrepancies.<BR>The =
general=20
  idea is to either reflect some implemented changes in the spec,<BR>or =
to pave=20
  a comprehensive way for the implementation to become spec=20
  compliant.<BR><BR>Alexander Nezhinsky<BR><BR>---<BR><BR>1.<B> Account =
for VA=20
  in buffer advertisement</B><BR><BR>Add VA to the list of attributes =
which=20
  identify advertized buffers and change related definitions and=20
  statements.<BR><BR>* Add a definition of VA to section 1.1. For=20
  example:<o:p></o:p></P>
  <DIV style=3D"MARGIN-LEFT: 30pt">
  <P class=3DMsoNormal><SPAN style=3D"FONT-FAMILY: 'Courier =
New'">"Virtual Address=20
  (VA) - an identifier supplied by RCaP together with STag <BR>when =
registering=20
  a memory buffer. In some modes it is not used,<BR>and VA=3D0 is =
implied. These=20
  modes are known as ZBVA (Zero Based VA).<BR>In such cases VA should be =
set to=20
  zero when buffer advertisements<BR>are=20
  communicated."<o:p></o:p></SPAN></P></DIV>
  <P class=3DMsoNormal><o:p>&nbsp;</o:p></P></DIV>
  <DIV>
  <P class=3DMsoNormal>* Definition of Remote Mapping:<o:p></o:p></P>
  <DIV style=3D"MARGIN-LEFT: 30pt">
  <P class=3DMsoNormal><SPAN style=3D"FONT-FAMILY: 'Courier =
New'">"Remote Mapping -=20
  A task state record maintained by the iSER Layer<BR>&nbsp;that =
associates the=20
  Initiator Task Tag to the Advertised =
STag(s)..."</SPAN><o:p></o:p></P></DIV>
  <P style=3D"MARGIN-BOTTOM: 12pt" class=3DMsoNormal><BR>should =
read:<o:p></o:p></P>
  <DIV style=3D"MARGIN-LEFT: 30pt">
  <P class=3DMsoNormal><SPAN style=3D"FONT-FAMILY: 'Courier =
New'">"...that=20
  associates the Initiator Task Tag to the Advertised STag(s)<BR>and =
Virtual=20
  Address(es)..."</SPAN><o:p></o:p></P></DIV>
  <P style=3D"MARGIN-BOTTOM: 12pt" class=3DMsoNormal><BR>* Definition of =
Tagged=20
  Buffer:<o:p></o:p></P>
  <DIV style=3D"MARGIN-LEFT: 30pt">
  <P class=3DMsoNormal><SPAN style=3D"FONT-FAMILY: 'Courier =
New'">"Tagged Buffer - A=20
  buffer that is explicitly Advertised to the iSER Layer<BR>at the =
remote=20
  node</SPAN>&nbsp; <SPAN style=3D"FONT-FAMILY: 'Courier New'">through =
the=20
  exchange of an STag, Tagged Offset, and =
length."</SPAN><o:p></o:p></P></DIV>
  <P style=3D"MARGIN-BOTTOM: 12pt" class=3DMsoNormal><BR>should =
read:<o:p></o:p></P>
  <DIV style=3D"MARGIN-LEFT: 30pt">
  <P class=3DMsoNormal><SPAN style=3D"FONT-FAMILY: 'Courier =
New'">"...the exchange=20
  of an STag, VA, Tagged Offset, and =
length."</SPAN><o:p></o:p></P></DIV>
  <P class=3DMsoNormal><BR>2.<B> iser message format</B><BR><BR>The =
headers are=20
  now defined for using with iWARP. Actually, they contain an =
iWARP<BR>prefix=20
  followed by a generic part. <BR><BR>Define a generic iser header as =
mandatory,=20
  while stating that the underlying transport<BR>protocols may add their =
own=20
  header sections before the iser header<BR>and provide iWARP as the =
example of=20
  such prefix.<BR><BR>Read and write buffers are identified by the =
combination=20
  of STag and VA.<BR>Both fields should be required for any transport. =
This=20
  requirement does not preclude<BR>using ZBVA where possible. When VA is =
not=20
  used (as in iWARP) it is set to 0.<BR><BR>Below is the actually used=20
  format:<BR><BR><SPAN style=3D"FONT-FAMILY: 'Courier New'">struct =
iser_hdr=20
  {<BR>&nbsp; uint8_t&nbsp;&nbsp; flags;<BR>&nbsp; uint8_t&nbsp;&nbsp;=20
  rsvd[3];<BR>&nbsp; uint32_t&nbsp; write_stag; /* write rkey in IB =
*/<BR>&nbsp;=20
  uint64_t&nbsp; write_va;<BR>&nbsp; uint32_t&nbsp; read_stag;&nbsp; /* =
read=20
  rkey in IB */<BR>&nbsp; uint64_t&nbsp; =
read_va;<BR>}<BR></SPAN><BR>3.<B> Use=20
  RCaP as a general term, iWARP and IB as particular =
cases</B><BR><BR>The=20
  language of the current draft is iWARP-dominated in many places. We =
need=20
  to<BR>make the text to refer to RCaP as a common denominator, =
stressing the=20
  differences<BR>between protocols where necessary.<BR><BR>* section 2=20
  :<BR>&nbsp;<o:p></o:p></P>
  <DIV style=3D"MARGIN-LEFT: 30pt">
  <P class=3DMsoNormal><SPAN style=3D"FONT-FAMILY: 'Courier New'">"The =
iWARP=20
  protocol stack provides direct data placement<BR>functionality that is =
usable=20
  in practice, and in addition, there is<BR>also interest in using iSCSI =
with=20
  other RDMA protocol stacks that<BR>support direct data placement, such =
as the=20
  one provided by<BR>InfiniBand.&nbsp; The generic term RDMA-Capable =
Protocol=20
  (RCaP) is used<BR>to refer to the RDMA functionality provided by such =
protocol=20
  stacks."</SPAN><o:p></o:p></P></DIV>
  <P style=3D"MARGIN-BOTTOM: 12pt" class=3DMsoNormal><BR>should =
read:<o:p></o:p></P>
  <DIV style=3D"MARGIN-LEFT: 30pt">
  <P class=3DMsoNormal><SPAN style=3D"FONT-FAMILY: 'Courier New'">"The =
generic term=20
  RDMA-Capable Protocol (RCaP) is used to refer to<BR>the RDMA =
functionality=20
  provided by the underlying protocol stack.<BR>Two practical examples =
of=20
  protocol stacks providing direct data placement<BR>functionality are =
iWARP and=20
  InfiniBand."<o:p></o:p></SPAN></P></DIV>
  <P style=3D"MARGIN-BOTTOM: 12pt" class=3DMsoNormal><BR>* section 5.1.1 =

  :<o:p></o:p></P>
  <DIV style=3D"MARGIN-LEFT: 30pt">
  <P class=3DMsoNormal>&nbsp;<SPAN style=3D"FONT-FAMILY: 'Courier =
New'">"Among the=20
  connection resources allocated at the initiator is the<BR>Inbound RDMA =
Read=20
  Queue Depth (IRD) ...<BR>... Initially, the iSER-IRD value at the =
initiator=20
  SHOULD<BR>be set to the IRD value at the initiator and MUST NOT be =
more=20
  than<BR>the IRD value."</SPAN><o:p></o:p></P></DIV>
  <P style=3D"MARGIN-BOTTOM: 12pt" class=3DMsoNormal><BR>should be =
prefixed=20
  with:<o:p></o:p></P>
  <DIV style=3D"MARGIN-LEFT: 30pt">
  <P class=3DMsoNormal><SPAN style=3D"FONT-FAMILY: 'Courier New'">"For =
iWARP based=20
  implementations, ..."</SPAN><o:p></o:p></P></DIV>
  <P class=3DMsoNormal><BR>4. <B>Mark iWARP-specific sections as=20
  such</B><BR><BR>Some protocol-specific statements, like in above, are=20
  scattered through the<BR>text, but there are entire sections which are =

  iWARP-specific (RDMAP features<BR>having no paralle in IB, etc.) We =
should=20
  explicitly state this. <BR>The same goes for IB, of course, although =
right now=20
  there are no such clauses.<BR><BR>In particular, sections 8.2. and =
8.3. with=20
  sub-sections 8.3.x<BR>should be either grouped as the subsections=20
  of<BR>&nbsp;"8.2. iWARP considerations in Flow Control"<BR>or retain =
their=20
  numbers and have "(iWARP specific)" added to their titles<BR><BR>5. =
<B>Fix a=20
  typo in section numbering</B><o:p></o:p></P></DIV>
  <P class=3DMsoNormal><o:p>&nbsp;</o:p></P>
  <DIV>
  <DIV>
  <DIV>
  <DIV>
  <DIV>
  <DIV>
  <DIV>
  <P style=3D"MARGIN-BOTTOM: 12pt" class=3DMsoNormal>Appendix A should =
be section=20
  15.<BR><BR>6.<B> iSER Hello-HelloReply</B><BR><BR>Hello-HelloReply =
should be=20
  made optional and transport dependent.<BR><BR>Right now the IB based =
target's=20
  implementation terminates the connection if it receives =
Hello.<BR>IB-based=20
  implementation does not need any Hello exchange, so the implemented =
behavior=20
  may be<BR>turned into standard. iWARP does need Hello-HelloReply and =
its=20
  format is provided.<BR><BR>We can define a new key called =
iSERHelloRequired=20
  with the default being "No", so<BR>that the current implementation =
becomes=20
  compliant.<BR><BR>Need to update some statements regarding IserHello, =
like in=20
  5.1.1 and 5.1.2<BR>alleviating some MUSTs with "if relevant for=20
  transport"-style of comments.<BR><BR>7.<B> Default values for=20
  MaxOutstandingUnexpectedPDUs and MaxAHSLength</B><BR><BR>The defaults =
for=20
  MaxOutstandingUnexpectedPDUs and MaxAHSLength<BR>should change from 0=20
  (unlimited) to some small reasonable values<BR>to protect a side not =
sending=20
  these keys from potentially harmful assumption <BR>by its peer about =
unlimited=20
  resources available, so that buffer overflow (MaxAHSLength)<BR>or =
sending more=20
  buffers than actually posted (MaxOutstandingUnexpectedPDUs) may=20
  result.<BR><BR>proposed default values:<BR><SPAN=20
  style=3D"FONT-FAMILY: 'Courier New'">MaxOutstandingUnexpectedPDUs =3D=20
  4<BR>MaxAHSLength =3D 256</SPAN> (accounts for 1 Var len CDB + 1 bidir =

  read-length)<BR><BR>8. <B>Ignoring MaxRecvDataSegmentLength =
key</B><BR><BR>It=20
  should be clarified that MaxRecvDataSegmentLength may be sent but MUST =
be=20
  ignored,<BR>because {Initiator|Target}RecvDataSegmentLength have their =
own=20
  defaults. <BR><BR>If {Initiator|Target}RecvDataSegmentLength are sent, =
the=20
  declared values would override <BR>MaxRecvDataSegmentLength anyway, =
and if=20
  they are not sent, the default values<BR>must be used to avoid =
confusing=20
  =
scenarios.<BR><BR><o:p></o:p></P></DIV></DIV></DIV></DIV></DIV></DIV></DI=
V></DIV></DIV>
  <P>
  <HR>

  <P></P>_______________________________________________<BR>storm =
mailing=20
  =
list<BR>storm@ietf.org<BR>https://www.ietf.org/mailman/listinfo/storm<BR>=
</BLOCKQUOTE></BODY></HTML>

--Boundary_(ID_HxQcbemlksbqBL6Hji2lNg)--

From cbm@chadalapaka.com  Wed Jun 29 15:15:58 2011
Return-Path: <cbm@chadalapaka.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AF88311E80A6 for <storm@ietfa.amsl.com>; Wed, 29 Jun 2011 15:15:58 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 84b3eib-WUqi for <storm@ietfa.amsl.com>; Wed, 29 Jun 2011 15:15:57 -0700 (PDT)
Received: from snt0-omc1-s24.snt0.hotmail.com (snt0-omc1-s24.snt0.hotmail.com [65.55.90.35]) by ietfa.amsl.com (Postfix) with ESMTP id 5857C11E807B for <storm@ietf.org>; Wed, 29 Jun 2011 15:15:57 -0700 (PDT)
Received: from SNT131-DS1 ([65.55.90.8]) by snt0-omc1-s24.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);  Wed, 29 Jun 2011 15:15:57 -0700
X-Originating-IP: [131.107.0.94]
X-Originating-Email: [cbm@chadalapaka.com]
Message-ID: <SNT131-ds1709A999A40FA52B3C247A0590@phx.gbl>
From: Mallikarjun Chadalapaka <cbm@chadalapaka.com>
To: "'Hemal Shah'" <hemal@broadcom.com>, <david.black@emc.com>, <storm@ietf.org>
References: <7C4DFCE962635144B8FAE8CA11D0BF1E0588F426C5@MX14A.corp.emc.com> <SNT131-ds2DE0F0F2022D63A4AD747A0760@phx.gbl> <76DBE161893C324BA6D4BB507EE4C384951235CB38@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds117065D22D7BF8583485DBA0790@phx.gbl> <76DBE161893C324BA6D4BB507EE4C38495124C4828@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds219B987FD6FA0437A497FDA06D0@phx.gbl> <76DBE161893C324BA6D4BB507EE4C3849513370AB5@IRVEXCHCCR02.corp.ad.broadcom.com>
In-Reply-To: <76DBE161893C324BA6D4BB507EE4C3849513370AB5@IRVEXCHCCR02.corp.ad.broadcom.com>
Date: Wed, 29 Jun 2011 15:15:56 -0700
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Outlook 14.0
Thread-Index: AQCUfOqLFecrTE8dOe+/hpcfY+kOJQIRzTq9AgfEPyEC0X+ZOALMKXeKAZ2F7W4CMc4IEZbYF5gw
Content-Language: en-us
X-OriginalArrivalTime: 29 Jun 2011 22:15:57.0144 (UTC) FILETIME=[1E878580:01CC36AA]
Subject: Re: [storm] Plan for iSCSI work
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 29 Jun 2011 22:15:58 -0000

Hi Hemal,

>1.	The implementation of TaskReporting is required and NotUnderstood
response for TaskReporting key is not permitted

No.  Note that TaskReporting is not defined in RFC 3720.  Patrick MacArthur
proposed on this list a couple of weeks ago that we should fix this
disconnect in the consolidated draft - which I agreed with because I think
it's a good suggestion.  

 >2.	To enable FastAbort on a session, the negotiation of TaskReporting
key is required.

Correct, specifically "TaskReporting=FastAbort" must be negotiated.

>3.	Only when the TaskReporting=FastAbort functionality is supported,
the protocol behavior specified in Section 4.2.3.4 is required to be
implemented and exhibited.

If I were to attempt re-writing your idea, I would say:

When the protocol behavior specified in Section 4.2.3.4 is implemented and
TaskReporting=FastAbort is operational on a session, then the FastAbort
functionality is supported/exhibited on that session by the implementation.

>a.	Section 4.2.3.5: If an iSCSI target implementation is capable of
supporting
>TaskReporting=FastAbort functionality (Section 13.23).
 
"Supporting" = implemented and negotiated to "TaskReporting=FastAbort".
Note that if even if an implementation has implemented this functionality
and wants to use it, it will not be able to negotiate it on any session if
it's only communicating with a bunch of RFC 3720 implementations.  

 >b.	Section 4.2.2.3.3: Whenever the TaskReporting key (Section 12.23
"Task Reporting") is negotiated to ResponseFence or FastAbort for an iSCSI
session.
>c.	Section 4.2.2.3.4: Whenever the TaskReporting key (Section 13.23) is
negotiated to ResponseFence or FastAbort for an iSCSI session.

I don't see an issue.  Note that the discussion in these two sections is on
response fencing, not fast abort per se.

>4.	...prefixed with some text like "Only when the
TaskReporting=FastAbort functionality is supported..".

Please see the above responses.  Your wording is vague for me, as I don't
comprehend a standalone notion of "supporting" a feature, other than
implementing a feature and negotiating the feature usage.   Current RFC 5048
requirement is that implementation is MUST, and negotiation is MUST (may
result in the feature to be used, or not).  So as such, there is already
ample flexibility for implementations.  

I think you are restating your earlier suggestion to loosen the "MUST
implement" requirement in RFC5048.  And I've already shared how I feel about
it, :)  I think it would be a step in the wrong direction, when faster error
recovery with multi-task aborts with large # of sessions/LUs/initiators is
even more critical, due to increased consolidation in data centers.  

At this point, I think it would be good to also hear from the rest of the
list.

Thanks.

Mallikarjun





>Thanks Mallikarjun! I would still like to confirm my understanding of what
you describe as "MUST implement, but MUST negotiate".
> 
>Here is my understanding:
> 
>1.	The implementation of TaskReporting is required and NotUnderstood
response for TaskReporting key is not permitted based on the following text
from RFC5048 and Section 6.2 of the consolidated draft:
>                 All keys defined in [RFC3720] MUST be supported by all
>            compliant implementations; a NotUnderstood answer on any of the
>            [RFC3720] keys therefore MUST be considered a protocol error
and
>            handled accordingly.
>2.	To enable FastAbort on a session, the negotiation of TaskReporting
key is required.
>3.	Only when the TaskReporting=FastAbort functionality is supported,
the protocol behavior specified in Section 4.2.3.4 is required to be
implemented and exhibited. There are several places in the spec that
indicate this type of conditionality.
>a.	Section 4.2.3.5: If an iSCSI target implementation is capable of
supporting
>TaskReporting=FastAbort functionality (Section 13.23).
>b.	Section 4.2.2.3.3: Whenever the TaskReporting key (Section 12.23
"Task Reporting") is negotiated to ResponseFence or FastAbort for an iSCSI
session.
>c.	Section 4.2.2.3.4: Whenever the TaskReporting key (Section 13.23) is
negotiated to ResponseFence or FastAbort for an iSCSI session.
>4.	If TaskReporting=FastAbort functionality is not supported, then the
protocol behavior specified in Section 4.2.3.4 is not required to be
implemented or exhibited. If this is true, then the first sentence in
section 4.2.3.4 is misleading and should be removed or prefixed with some
text like "Only when the TaskReporting=FastAbort functionality is
supported..".
> 
>Hemal
> 
> 
>-----Original Message-----
>From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] 
>Sent: Friday, June 17, 2011 3:32 PM
>To: Hemal Shah; david.black@emc.com; storm@ietf.org
>Subject: RE: [storm] Plan for iSCSI work
> 
>Hi Hemal,
> 
>During RFC 5048 effort, I recall we settled on the "MUST implement, but
MUST
>negotiate" formulation after some list deliberation.  
> 
>With plain RFC 3720 semantics, there are some multi-initiator scenarios
>where multi-task aborts could essentially lead to target deadlocks, waiting
>on initiators on third-party sessions that may never respond.  With the
>"Clarified" semantics in RFC 5048, the third-party deadlocks aren't a
>problem anymore, but issuing initiator could still experience timeouts and
>escalate error recovery - depending on the number of LUs, sessions,
>connections and tasks affected with the issued TMF.    Escalated error
>recovery is usually something we want to avoid as it adds more
>delays/alerts/logs/failovers/failbacks etc.  Finally, "Updated" semantics
in
>RFC 5048 are meant to address this timeout problem - they permit targets to
>provide accelerated responses and allow them to deal with
>book-keeping/quiescing operations in a lazy fashion (which are the real
>culprits that trigger timeouts).  
> 
>Given all the above, the WG settled on FastAbort as a MUST-implement.  The
>"MUST negotiate" part then is to enable backwards-compatibility with RFC
>3720 implementations.  At this point, I'd rather not loosen the
Consolidated
>iSCSI draft requirement from what it is in RFC 5048.  I suggest we leave it
>the way it is.
> 
>Mallikarjun
> 
> 
> 
> 
> 
> 
>From: Hemal Shah [mailto:hemal@broadcom.com] 
>Sent: Thursday, June 09, 2011 3:57 PM
>To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org
>Subject: RE: [storm] Plan for iSCSI work
> 
>Thanks Mallikarjun!
> 
>The way I read Section 4.1.3 of RFC5048 and Section 4.2.3.4 of the
>consolidated draft, the implementation of "FastAbort" feature is a MUST.
>From the first paragraph text from Section 4.1.3 of RFC5048, the first
>sentence mandates that all iSCSI implementation comply to this section but
>the second sentence say that the requirement in this section is conditional
>to the negotiation of "FastAbort" key. 
> 
>Protocol behavior defined in this section MUST be implemented by all iSCSI
>implementations complying with this document. Protocol behavior defined in
>this section MUST be exhibited by iSCSI implementations on an iSCSI session
>when they negotiate the TaskReporting (Section 9.1) key to "FastAbort" on
>that session.
> 
> 
>This is confusing. I think it will be better to remove the first sentence
>from 4.2.3.4 and clarify that the requirements in this section is
>conditional. I suggest rewording the first two sentences to below:
> 
>       iSCSI implementations may negotiate the TaskReporting (Section 9.1)
>key to "FastAbort" on an iSCSI session. Protocol behavior
>defined in this section MUST be exhibited by iSCSI implementations on an
>iSCSI session when they negotiate the TaskReporting (Section 9.1) key to
>"FastAbort" on that session.
> 
>Hemal
> 
> 
>-----Original Message-----
>From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com] 
>Sent: Friday, May 27, 2011 5:29 PM
>To: Hemal Shah; david.black@emc.com; storm@ietf.org
>Subject: RE: [storm] Plan for iSCSI work
> 
>Hi Hemal,
> 
>The text in this draft came verbatim from section 4.1.3 of RFC 5048.  There
>have been no changes in this area.
> 
>The new text (as well as the old text) requires the TaskReporting key to be
>negotiated to "FastAbort" before the multi-task abort semantics can be used
>on an iSCSI session.
> 
>Thanks.
> 
>Mallikarjun
> 
> 
> 
> 
>  -----Original Message-----
>  From: Hemal Shah [mailto:hemal@broadcom.com]
>  Sent: Friday, May 27, 2011 4:51 PM
>  To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org
>  Subject: RE: [storm] Plan for iSCSI work
> 
>  Mallikarjun and David,
> 
>  I noticed one problematic item in the consolidated draft. This item is
the
>  requirement to support FastAbort. This feature was already defined in the
>  implementation guide, but it was optional. In this draft, it became a
>  required feature MUST - see in section 4.2.3.4.
> 
>  Do you know why the requirement was changed in the consolidated draft?
> 
>  I would like to keep the requirement optional as stated in the
>  implementation guide and not break backward compatibility.
> 
>  Hemal
> 
>  -----Original Message-----
>  From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf
>  Of Mallikarjun Chadalapaka
>  Sent: Thursday, May 26, 2011 6:50 PM
>  To: david.black@emc.com; storm@ietf.org
>  Subject: Re: [storm] Plan for iSCSI work
> 
>  Hi David,
> 
>  The list of changes you have called out are already done in the latest
>draft.  I
>  assume then that you are suggesting that the list itself should be
>included
>  in the next revision of the draft.
> 
>  Here's what I recall we have done so far:
>  1)  iSCSIProtocolLevel specified as "1", and added a related normative
>  reference to iSCSI-SAM draft
>  2)  Markers and related keys were removed
>  3)  SPKM authentication and related keys were removed
>  4)  Added a new section on responding to obsoleted keys
>  5)  Have explicitly allowed initiator+target implementations throughout
>the
>  text
>  6)  Clarified that implementations SHOULD NOT rely on SLP-based discovery
>  7)  Added UML diagrams, and related conventions
> 
>  The above is of course in addition to consolidating the different RFCs,
>and
>  making the related editorial changes.
> 
>  Thanks.
> 
>  Mallikarjun
> 
> 
> 
> 
>    -----Original Message-----
>    From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf
>    Of david.black@emc.com
>    Sent: Friday, May 20, 2011 2:49 PM
>    To: storm@ietf.org
>    Subject: [storm] Plan for iSCSI work
> 
>    I thought I'd offer some advance planning/warning on this, as the
>    consolidated iSCSI draft (draft-ietf-storm-iscsi-cons) is large (over
>300
>    pages).  The current plan is to run a simultaneous WG Last Call on both
>  this
>    draft and the new features draft (draft-ietf-storm-iscsi-sam) starting
>in
>  mid-
>    June (probably the week of June 13, after I get back from a badly
needed
>    vacation).  That WG Last Call will run longer than the typical 2-week
>time
>    period, due to the total size of the drafts, but will end by July 5th
at
>the
>    latest so that the status of the drafts and the next steps are known
>prior
>  to
>    the T10 (SCSI standards) meetings during the week of July 11.  As July
>11th
>    is also the draft cutoff deadline for the Quebec City IETF meetings,
>revised
>    draft versions may not show up until that meeting week (week of July
>    24th).
> 
>    This is also a good point to announce that the storm WG will meet in
>    Quebec City.
>    I've only requested a 1-hour session, as we get most of our work done
on
>    the mailing list.  Among the items for that meeting will be figuring
out
>  what
>    to do with the RDMA extensions draft (despite its name,
>draft-ietf-storm-
>    rdmap-ext-00, it's not currently an official work item for the storm
>WG).
> 
>    One thing that's missing from the consolidated iSCSI draft (and is a
>reason
>    why we're going to need a -03 version) is the changes that it makes to
>the
>    RFCs that it consolidates.  Off the top of my head, the major changes
>are:
>         - Removal of SPKM authentication
>         - Removal of the Marker appendix
>         - Removal of the SHOULD requirement for SLP implementation.
>    Have I missed anything significant?  The summary of this will need to
be
>    added to that draft.
> 
>    WG Last Call will be an opportunity (in fact the final opportunity) to
>  discuss
>    whether anything else should be removed from iSCSI, but there's no need
>    to wait
>    - I encourage people to review both drafts and post comments whenever
>    they can.
> 
>    In parallel, work will get started on any iSCSI MIB changes that are
>needed.
>    So far, I only see one MIB change - the iSCSIProtocolLevel from the new
>    features draft needs to be added to the MIB, probably with a structure
>    analogous to the iSCSI version support that's already in the MIB.
> 
>    Thanks,
>    --David (storm WG co-chair)
>    ----------------------------------------------------
>    David L. Black, Distinguished Engineer
>    EMC Corporation, 176 South St., Hopkinton, MA  01748
>    +1 (508) 293-7953             FAX: +1 (508) 293-7786
>    david.black@emc.com        Mobile: +1 (978) 394-7754
>    ----------------------------------------------------
> 
>    _______________________________________________
>    storm mailing list
>    storm@ietf.org
>    https://www.ietf.org/mailman/listinfo/storm
> 
>  _______________________________________________
>  storm mailing list
>  storm@ietf.org
>  https://www.ietf.org/mailman/listinfo/storm
> 
> 
> 
> 
> 
> 
> 
> 
>


From pthaler@broadcom.com  Wed Jun 29 16:02:59 2011
Return-Path: <pthaler@broadcom.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 91AF611E8114 for <storm@ietfa.amsl.com>; Wed, 29 Jun 2011 16:02:59 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rWwPtZgXwiyo for <storm@ietfa.amsl.com>; Wed, 29 Jun 2011 16:02:58 -0700 (PDT)
Received: from mms1.broadcom.com (mms1.broadcom.com [216.31.210.17]) by ietfa.amsl.com (Postfix) with ESMTP id 8DAB611E80E3 for <storm@ietf.org>; Wed, 29 Jun 2011 16:02:57 -0700 (PDT)
Received: from [10.16.192.224] by mms1.broadcom.com with ESMTP (Broadcom SMTP Relay (Email Firewall v6.3.2)); Wed, 29 Jun 2011 16:07:31 -0700
X-Server-Uuid: 02CED230-5797-4B57-9875-D5D2FEE4708A
Received: from SJEXCHCCR01.corp.ad.broadcom.com ([10.252.49.130]) by SJEXCHHUB01.corp.ad.broadcom.com ([10.16.192.224]) with mapi; Wed, 29 Jun 2011 16:02:38 -0700
From: "Pat Thaler" <pthaler@broadcom.com>
To: "Mallikarjun Chadalapaka" <cbm@chadalapaka.com>, "Hemal Shah" <hemal@broadcom.com>, "david.black@emc.com" <david.black@emc.com>, "storm@ietf.org" <storm@ietf.org>
Date: Wed, 29 Jun 2011 16:02:35 -0700
Thread-Topic: [storm] Plan for iSCSI work
Thread-Index: AQCUfOqLFecrTE8dOe+/hpcfY+kOJQIRzTq9AgfEPyEC0X+ZOALMKXeKAZ2F7W4CMc4IEZbYF5gwgAAV2AA=
Message-ID: <DBFBCB90599B2C46992032279293D10020A74038C5@SJEXCHCCR01.corp.ad.broadcom.com>
References: <7C4DFCE962635144B8FAE8CA11D0BF1E0588F426C5@MX14A.corp.emc.com> <SNT131-ds2DE0F0F2022D63A4AD747A0760@phx.gbl> <76DBE161893C324BA6D4BB507EE4C384951235CB38@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds117065D22D7BF8583485DBA0790@phx.gbl> <76DBE161893C324BA6D4BB507EE4C38495124C4828@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds219B987FD6FA0437A497FDA06D0@phx.gbl> <76DBE161893C324BA6D4BB507EE4C3849513370AB5@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds1709A999A40FA52B3C247A0590@phx.gbl>
In-Reply-To: <SNT131-ds1709A999A40FA52B3C247A0590@phx.gbl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
MIME-Version: 1.0
X-WSS-ID: 62156FB93B411315367-01-01
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Re: [storm] Plan for iSCSI work
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 29 Jun 2011 23:02:59 -0000

Hi Malikarjun,

10.7 in the consolidated draft says that FastAbort is a SHOULD: "Therefore,=
 both iSCSI target and initiator implementations SHOULD support FastAbort m=
ulti-task abort semantics (Section 4.2.3.4)."

When 4.2.3.5 saying "If an iSCSI target implementation is capable of suppor=
ting TaskReporting=3DFastAbort functionality" implies that the implementati=
on was allowed to not implement FastAbort. You might interpret "supporting"=
 =3D (implemented and negotiated to TaskAbort) as you say below, but "capab=
le of supporting" still reads as implemented. When the standard puts an if =
in front of that, it implies that an implementation is allowed to be not ca=
pable of supporting FastAbort.

Neither of these is consistent with MUST implement for FastAbort. The ratio=
nale for requiring FastAbort doesn't seem strong and I'd prefer that the in=
consistency be resolved by not requiring its implementation.

Regards,
Pat

-----Original Message-----
From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of M=
allikarjun Chadalapaka
Sent: Wednesday, June 29, 2011 3:16 PM
To: Hemal Shah; david.black@emc.com; storm@ietf.org
Subject: Re: [storm] Plan for iSCSI work

Hi Hemal,

>1.     The implementation of TaskReporting is required and NotUnderstood
response for TaskReporting key is not permitted

No.  Note that TaskReporting is not defined in RFC 3720.  Patrick MacArthur
proposed on this list a couple of weeks ago that we should fix this
disconnect in the consolidated draft - which I agreed with because I think
it's a good suggestion.

 >2.    To enable FastAbort on a session, the negotiation of TaskReporting
key is required.

Correct, specifically "TaskReporting=3DFastAbort" must be negotiated.

>3.     Only when the TaskReporting=3DFastAbort functionality is supported,
the protocol behavior specified in Section 4.2.3.4 is required to be
implemented and exhibited.

If I were to attempt re-writing your idea, I would say:

When the protocol behavior specified in Section 4.2.3.4 is implemented and
TaskReporting=3DFastAbort is operational on a session, then the FastAbort
functionality is supported/exhibited on that session by the implementation.

>a.     Section 4.2.3.5: If an iSCSI target implementation is capable of
supporting
>TaskReporting=3DFastAbort functionality (Section 13.23).

"Supporting" =3D implemented and negotiated to "TaskReporting=3DFastAbort".
Note that if even if an implementation has implemented this functionality
and wants to use it, it will not be able to negotiate it on any session if
it's only communicating with a bunch of RFC 3720 implementations.

 >b.    Section 4.2.2.3.3: Whenever the TaskReporting key (Section 12.23
"Task Reporting") is negotiated to ResponseFence or FastAbort for an iSCSI
session.
>c.     Section 4.2.2.3.4: Whenever the TaskReporting key (Section 13.23) i=
s
negotiated to ResponseFence or FastAbort for an iSCSI session.

I don't see an issue.  Note that the discussion in these two sections is on
response fencing, not fast abort per se.

>4.     ...prefixed with some text like "Only when the
TaskReporting=3DFastAbort functionality is supported..".

Please see the above responses.  Your wording is vague for me, as I don't
comprehend a standalone notion of "supporting" a feature, other than
implementing a feature and negotiating the feature usage.   Current RFC 504=
8
requirement is that implementation is MUST, and negotiation is MUST (may
result in the feature to be used, or not).  So as such, there is already
ample flexibility for implementations.

I think you are restating your earlier suggestion to loosen the "MUST
implement" requirement in RFC5048.  And I've already shared how I feel abou=
t
it, :)  I think it would be a step in the wrong direction, when faster erro=
r
recovery with multi-task aborts with large # of sessions/LUs/initiators is
even more critical, due to increased consolidation in data centers.

At this point, I think it would be good to also hear from the rest of the
list.

Thanks.

Mallikarjun





>Thanks Mallikarjun! I would still like to confirm my understanding of what
you describe as "MUST implement, but MUST negotiate".
>
>Here is my understanding:
>
>1.     The implementation of TaskReporting is required and NotUnderstood
response for TaskReporting key is not permitted based on the following text
from RFC5048 and Section 6.2 of the consolidated draft:
>                 All keys defined in [RFC3720] MUST be supported by all
>            compliant implementations; a NotUnderstood answer on any of th=
e
>            [RFC3720] keys therefore MUST be considered a protocol error
and
>            handled accordingly.
>2.     To enable FastAbort on a session, the negotiation of TaskReporting
key is required.
>3.     Only when the TaskReporting=3DFastAbort functionality is supported,
the protocol behavior specified in Section 4.2.3.4 is required to be
implemented and exhibited. There are several places in the spec that
indicate this type of conditionality.
>a.     Section 4.2.3.5: If an iSCSI target implementation is capable of
supporting
>TaskReporting=3DFastAbort functionality (Section 13.23).
>b.     Section 4.2.2.3.3: Whenever the TaskReporting key (Section 12.23
"Task Reporting") is negotiated to ResponseFence or FastAbort for an iSCSI
session.
>c.     Section 4.2.2.3.4: Whenever the TaskReporting key (Section 13.23) i=
s
negotiated to ResponseFence or FastAbort for an iSCSI session.
>4.     If TaskReporting=3DFastAbort functionality is not supported, then t=
he
protocol behavior specified in Section 4.2.3.4 is not required to be
implemented or exhibited. If this is true, then the first sentence in
section 4.2.3.4 is misleading and should be removed or prefixed with some
text like "Only when the TaskReporting=3DFastAbort functionality is
supported..".
>
>Hemal
>
>
>-----Original Message-----
>From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
>Sent: Friday, June 17, 2011 3:32 PM
>To: Hemal Shah; david.black@emc.com; storm@ietf.org
>Subject: RE: [storm] Plan for iSCSI work
>
>Hi Hemal,
>
>During RFC 5048 effort, I recall we settled on the "MUST implement, but
MUST
>negotiate" formulation after some list deliberation.
>
>With plain RFC 3720 semantics, there are some multi-initiator scenarios
>where multi-task aborts could essentially lead to target deadlocks, waitin=
g
>on initiators on third-party sessions that may never respond.  With the
>"Clarified" semantics in RFC 5048, the third-party deadlocks aren't a
>problem anymore, but issuing initiator could still experience timeouts and
>escalate error recovery - depending on the number of LUs, sessions,
>connections and tasks affected with the issued TMF.    Escalated error
>recovery is usually something we want to avoid as it adds more
>delays/alerts/logs/failovers/failbacks etc.  Finally, "Updated" semantics
in
>RFC 5048 are meant to address this timeout problem - they permit targets t=
o
>provide accelerated responses and allow them to deal with
>book-keeping/quiescing operations in a lazy fashion (which are the real
>culprits that trigger timeouts).
>
>Given all the above, the WG settled on FastAbort as a MUST-implement.  The
>"MUST negotiate" part then is to enable backwards-compatibility with RFC
>3720 implementations.  At this point, I'd rather not loosen the
Consolidated
>iSCSI draft requirement from what it is in RFC 5048.  I suggest we leave i=
t
>the way it is.
>
>Mallikarjun
>
>
>
>
>
>
>From: Hemal Shah [mailto:hemal@broadcom.com]
>Sent: Thursday, June 09, 2011 3:57 PM
>To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org
>Subject: RE: [storm] Plan for iSCSI work
>
>Thanks Mallikarjun!
>
>The way I read Section 4.1.3 of RFC5048 and Section 4.2.3.4 of the
>consolidated draft, the implementation of "FastAbort" feature is a MUST.
>From the first paragraph text from Section 4.1.3 of RFC5048, the first
>sentence mandates that all iSCSI implementation comply to this section but
>the second sentence say that the requirement in this section is conditiona=
l
>to the negotiation of "FastAbort" key.
>
>Protocol behavior defined in this section MUST be implemented by all iSCSI
>implementations complying with this document. Protocol behavior defined in
>this section MUST be exhibited by iSCSI implementations on an iSCSI sessio=
n
>when they negotiate the TaskReporting (Section 9.1) key to "FastAbort" on
>that session.
>
>
>This is confusing. I think it will be better to remove the first sentence
>from 4.2.3.4 and clarify that the requirements in this section is
>conditional. I suggest rewording the first two sentences to below:
>
>       iSCSI implementations may negotiate the TaskReporting (Section 9.1)
>key to "FastAbort" on an iSCSI session. Protocol behavior
>defined in this section MUST be exhibited by iSCSI implementations on an
>iSCSI session when they negotiate the TaskReporting (Section 9.1) key to
>"FastAbort" on that session.
>
>Hemal
>
>
>-----Original Message-----
>From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
>Sent: Friday, May 27, 2011 5:29 PM
>To: Hemal Shah; david.black@emc.com; storm@ietf.org
>Subject: RE: [storm] Plan for iSCSI work
>
>Hi Hemal,
>
>The text in this draft came verbatim from section 4.1.3 of RFC 5048.  Ther=
e
>have been no changes in this area.
>
>The new text (as well as the old text) requires the TaskReporting key to b=
e
>negotiated to "FastAbort" before the multi-task abort semantics can be use=
d
>on an iSCSI session.
>
>Thanks.
>
>Mallikarjun
>
>
>
>
>  -----Original Message-----
>  From: Hemal Shah [mailto:hemal@broadcom.com]
>  Sent: Friday, May 27, 2011 4:51 PM
>  To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org
>  Subject: RE: [storm] Plan for iSCSI work
>
>  Mallikarjun and David,
>
>  I noticed one problematic item in the consolidated draft. This item is
the
>  requirement to support FastAbort. This feature was already defined in th=
e
>  implementation guide, but it was optional. In this draft, it became a
>  required feature MUST - see in section 4.2.3.4.
>
>  Do you know why the requirement was changed in the consolidated draft?
>
>  I would like to keep the requirement optional as stated in the
>  implementation guide and not break backward compatibility.
>
>  Hemal
>
>  -----Original Message-----
>  From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf
>  Of Mallikarjun Chadalapaka
>  Sent: Thursday, May 26, 2011 6:50 PM
>  To: david.black@emc.com; storm@ietf.org
>  Subject: Re: [storm] Plan for iSCSI work
>
>  Hi David,
>
>  The list of changes you have called out are already done in the latest
>draft.  I
>  assume then that you are suggesting that the list itself should be
>included
>  in the next revision of the draft.
>
>  Here's what I recall we have done so far:
>  1)  iSCSIProtocolLevel specified as "1", and added a related normative
>  reference to iSCSI-SAM draft
>  2)  Markers and related keys were removed
>  3)  SPKM authentication and related keys were removed
>  4)  Added a new section on responding to obsoleted keys
>  5)  Have explicitly allowed initiator+target implementations throughout
>the
>  text
>  6)  Clarified that implementations SHOULD NOT rely on SLP-based discover=
y
>  7)  Added UML diagrams, and related conventions
>
>  The above is of course in addition to consolidating the different RFCs,
>and
>  making the related editorial changes.
>
>  Thanks.
>
>  Mallikarjun
>
>
>
>
>    -----Original Message-----
>    From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf
>    Of david.black@emc.com
>    Sent: Friday, May 20, 2011 2:49 PM
>    To: storm@ietf.org
>    Subject: [storm] Plan for iSCSI work
>
>    I thought I'd offer some advance planning/warning on this, as the
>    consolidated iSCSI draft (draft-ietf-storm-iscsi-cons) is large (over
>300
>    pages).  The current plan is to run a simultaneous WG Last Call on bot=
h
>  this
>    draft and the new features draft (draft-ietf-storm-iscsi-sam) starting
>in
>  mid-
>    June (probably the week of June 13, after I get back from a badly
needed
>    vacation).  That WG Last Call will run longer than the typical 2-week
>time
>    period, due to the total size of the drafts, but will end by July 5th
at
>the
>    latest so that the status of the drafts and the next steps are known
>prior
>  to
>    the T10 (SCSI standards) meetings during the week of July 11.  As July
>11th
>    is also the draft cutoff deadline for the Quebec City IETF meetings,
>revised
>    draft versions may not show up until that meeting week (week of July
>    24th).
>
>    This is also a good point to announce that the storm WG will meet in
>    Quebec City.
>    I've only requested a 1-hour session, as we get most of our work done
on
>    the mailing list.  Among the items for that meeting will be figuring
out
>  what
>    to do with the RDMA extensions draft (despite its name,
>draft-ietf-storm-
>    rdmap-ext-00, it's not currently an official work item for the storm
>WG).
>
>    One thing that's missing from the consolidated iSCSI draft (and is a
>reason
>    why we're going to need a -03 version) is the changes that it makes to
>the
>    RFCs that it consolidates.  Off the top of my head, the major changes
>are:
>         - Removal of SPKM authentication
>         - Removal of the Marker appendix
>         - Removal of the SHOULD requirement for SLP implementation.
>    Have I missed anything significant?  The summary of this will need to
be
>    added to that draft.
>
>    WG Last Call will be an opportunity (in fact the final opportunity) to
>  discuss
>    whether anything else should be removed from iSCSI, but there's no nee=
d
>    to wait
>    - I encourage people to review both drafts and post comments whenever
>    they can.
>
>    In parallel, work will get started on any iSCSI MIB changes that are
>needed.
>    So far, I only see one MIB change - the iSCSIProtocolLevel from the ne=
w
>    features draft needs to be added to the MIB, probably with a structure
>    analogous to the iSCSI version support that's already in the MIB.
>
>    Thanks,
>    --David (storm WG co-chair)
>    ----------------------------------------------------
>    David L. Black, Distinguished Engineer
>    EMC Corporation, 176 South St., Hopkinton, MA  01748
>    +1 (508) 293-7953             FAX: +1 (508) 293-7786
>    david.black@emc.com        Mobile: +1 (978) 394-7754
>    ----------------------------------------------------
>
>    _______________________________________________
>    storm mailing list
>    storm@ietf.org
>    https://www.ietf.org/mailman/listinfo/storm
>
>  _______________________________________________
>  storm mailing list
>  storm@ietf.org
>  https://www.ietf.org/mailman/listinfo/storm
>
>
>
>
>
>
>
>
>

_______________________________________________
storm mailing list
storm@ietf.org
https://www.ietf.org/mailman/listinfo/storm



From cait@asomi.com  Wed Jun 29 20:34:56 2011
Return-Path: <cait@asomi.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 24E2B21F8635 for <storm@ietfa.amsl.com>; Wed, 29 Jun 2011 20:34:56 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ppl0g6Mw3Y4I for <storm@ietfa.amsl.com>; Wed, 29 Jun 2011 20:34:55 -0700 (PDT)
Received: from p3plsmtpa01-08.prod.phx3.secureserver.net (p3plsmtpa01-08.prod.phx3.secureserver.net [72.167.82.88]) by ietfa.amsl.com (Postfix) with SMTP id 306BE21F862A for <storm@ietf.org>; Wed, 29 Jun 2011 20:34:55 -0700 (PDT)
Received: (qmail 30008 invoked from network); 30 Jun 2011 03:34:54 -0000
Received: from unknown (108.80.57.164) by p3plsmtpa01-08.prod.phx3.secureserver.net (72.167.82.88) with ESMTP; 30 Jun 2011 03:34:54 -0000
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: text/plain; charset=us-ascii
From: Caitlin Bestler <cait@asomi.com>
X-Priority: 3
In-Reply-To: <971779FD6B314A40B1A3658E521C0D4A@china.huawei.com>
Date: Wed, 29 Jun 2011 20:34:52 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <707F30B0-06DD-4F1A-BBCB-229E9D3794AA@asomi.com>
References: <BANLkTi=GVHUBdsZiwsRXxETd_rU=F8f85w@mail.gmail.com> <F83812DF4B59B9499C1BC978336D91745EDE207F@TK5EX14MBXC118.redmond.corp.microsoft.com> <971779FD6B314A40B1A3658E521C0D4A@china.huawei.com>
To: Michael Ko <Michael@huaweisymantec.com>
X-Mailer: Apple Mail (2.1084)
Cc: storm@ietf.org
Subject: Re: [storm] iSER draft
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 30 Jun 2011 03:34:56 -0000

Let me deliberately mis-paraphrase your rationale here:

	We only need a three-tuple to specify an advertised buffer, but =
the IETF should define it to be a four-tuple
	in order to make it easier for some host implementations to =
share code with a non-IETF transport.

That's not much of a rationale for making an IETF protocol more complex =
than it has to be.

Could you at least be explicit and provide a neutral explanation as to =
why having a VA *and* a TO could make
a certain class of implementations easier? Something better than =
"InfiniBand did it that way?"



On Jun 29, 2011, at 2:26 PM, Michael Ko wrote:

> Tom,
> =20
> As originally written, the iSER spec always assumes ZBVA (zero-based =
virtual address).  But when iSER is used in conjunction with transports =
other than iWARP, there is a need to include the virtual address for the =
tagged buffer.  Such is the case for Infiniband.  Accordingly, the =
Virtual Address field has been added to the iSCSI Control Type PDUs =
(sec. 9.2).  Alex pointed out that the tagged buffer should now be =
identified by the 4-tuple (STag, VA, TO, and length) instead.  This =
should be updated everywhere where the term "tagged buffer" is mentioned =
in the spec.  For example, in sec. 1.1 under Advertisement, the sentence =
"A typical method would be for the iSER Layer to embed the Tagged =
Buffer's STag, TO, and buffer length in a Send Message destined for the =
remote iSER Layer" should be modified to "A typical method would be for =
the iSER Layer to embed the Tagged Buffer's STag, VA, TO, and buffer =
length in a Send Message destined for the remote iSER Layer."  BTW, this =
sentence also answers your question as to whether VA is passed on the =
wire.
> =20
> The Virtual Address is literally the virtual address of the tagged =
buffer.  The tagged offset is the displacement relative to the beginning =
of the buffer starting at the virtual address.
> =20
> We could rephrase the definition of Virtual Address in sec. 1.1 as =
follows:
> =20
> "Virtual Address (VA) - This is the virtual address of the Tagged =
Buffer on a Node.  When set to 0, it indicates that a virtual address is =
not needed to locate the Tagged Buffer."
> =20
> Mike
> =20


From Michael@huaweisymantec.com  Thu Jun 30 10:18:52 2011
Return-Path: <Michael@huaweisymantec.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C045A11E8271 for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 10:18:52 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.598
X-Spam-Level: 
X-Spam-Status: No, score=-2.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bBO3e5iCe53f for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 10:18:51 -0700 (PDT)
Received: from mta2.huaweisymantec.com (mta2.huaweisymantec.com [218.17.155.15]) by ietfa.amsl.com (Postfix) with ESMTP id 4B7C511E8229 for <storm@ietf.org>; Thu, 30 Jun 2011 10:18:51 -0700 (PDT)
MIME-version: 1.0
Content-type: multipart/alternative; boundary="Boundary_(ID_tnEtJjoP2s9Nffge6XKU9A)"
Received: from hstml01-in.huaweisymantec.com ([172.26.3.42]) by hstga02-in.huaweisymantec.com (Sun Java(tm) System Messaging Server 6.3-8.03 (built Apr 24 2009; 32bit)) with ESMTP id <0LNM0057M5FC2580@hstga02-in.huaweisymantec.com> for storm@ietf.org; Fri, 01 Jul 2011 01:18:48 +0800 (CST)
Received: from m90003900a ([69.199.248.19]) by hstml01-in.huaweisymantec.com (Sun Java(tm) System Messaging Server 6.3-8.03 (built Apr 24 2009; 32bit)) with ESMTPA id <0LNM00GB75GA3600@hstml01-in.huaweisymantec.com> for storm@ietf.org; Fri, 01 Jul 2011 01:19:29 +0800 (CST)
Message-id: <58D74E56251E4229833ED223E6D39E49@china.huawei.com>
From: Michael Ko <Michael@huaweisymantec.com>
To: Caitlin Bestler <cait@asomi.com>
References: <BANLkTi=GVHUBdsZiwsRXxETd_rU=F8f85w@mail.gmail.com> <F83812DF4B59B9499C1BC978336D91745EDE207F@TK5EX14MBXC118.redmond.corp.microsoft.com> <971779FD6B314A40B1A3658E521C0D4A@china.huawei.com> <707F30B0-06DD-4F1A-BBCB-229E9D3794AA@asomi.com>
Date: Thu, 30 Jun 2011 10:18:40 -0700
X-Priority: 3
X-MSMail-priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.5931
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2900.6109
Cc: storm@ietf.org
Subject: Re: [storm] iSER draft
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 30 Jun 2011 17:18:52 -0000

This is a multi-part message in MIME format.

--Boundary_(ID_tnEtJjoP2s9Nffge6XKU9A)
Content-type: text/plain; charset=iso-8859-1
Content-transfer-encoding: quoted-printable

In RFC 5046, we specifically created the term RDMA-Capable Protocol =
(RCaP) to refer to protocol stacks that provide the RDMA functionality.  =
We pointed to iWARP and Infiniband as examples of RCaP in the RFC, and =
recently RoCE (RDMA over Converged Ethernet) can also be added to the =
list.  The ideal situation is for iSER implementations to be usable in =
any RCaP.  This minimizes the development effort for iSER and =
facilitates the acceptance of iSER.  Bob Russell suggested back in =
3/22/2009 to incorporate the use of Virtual Address as this would allow =
implementations such as the OFED stack to be used for different RCaPs.  =
The initial iSER draft (draft-ietf-storm-iser-01) submitted to STORM =
reflected this change request.  Alex suggested further clarfiying the =
use of Virtual Address to clear up the ambiguity in the -01 version.

Mike
  ----- Original Message -----=20
  From: Caitlin Bestler=20
  To: Michael Ko=20
  Cc: Tom Talpey ; Alexander Nezhinsky ; storm@ietf.org=20
  Sent: Wednesday, June 29, 2011 8:34 PM
  Subject: Re: [storm] iSER draft


  Let me deliberately mis-paraphrase your rationale here:

  We only need a three-tuple to specify an advertised buffer, but the =
IETF should define it to be a four-tuple
  in order to make it easier for some host implementations to share code =
with a non-IETF transport.

  That's not much of a rationale for making an IETF protocol more =
complex than it has to be.

  Could you at least be explicit and provide a neutral explanation as to =
why having a VA *and* a TO could make
  a certain class of implementations easier? Something better than =
"InfiniBand did it that way?"



  On Jun 29, 2011, at 2:26 PM, Michael Ko wrote:

  > Tom,
  > =20
  > As originally written, the iSER spec always assumes ZBVA (zero-based =
virtual address).  But when iSER is used in conjunction with transports =
other than iWARP, there is a need to include the virtual address for the =
tagged buffer.  Such is the case for Infiniband.  Accordingly, the =
Virtual Address field has been added to the iSCSI Control Type PDUs =
(sec. 9.2).  Alex pointed out that the tagged buffer should now be =
identified by the 4-tuple (STag, VA, TO, and length) instead.  This =
should be updated everywhere where the term "tagged buffer" is mentioned =
in the spec.  For example, in sec. 1.1 under Advertisement, the sentence =
"A typical method would be for the iSER Layer to embed the Tagged =
Buffer's STag, TO, and buffer length in a Send Message destined for the =
remote iSER Layer" should be modified to "A typical method would be for =
the iSER Layer to embed the Tagged Buffer's STag, VA, TO, and buffer =
length in a Send Message destined for the remote iSER Layer."  BTW, this =
sentence also answers your question as to whether VA is passed on the =
wire.
  > =20
  > The Virtual Address is literally the virtual address of the tagged =
buffer.  The tagged offset is the displacement relative to the beginning =
of the buffer starting at the virtual address.
  > =20
  > We could rephrase the definition of Virtual Address in sec. 1.1 as =
follows:
  > =20
  > "Virtual Address (VA) - This is the virtual address of the Tagged =
Buffer on a Node.  When set to 0, it indicates that a virtual address is =
not needed to locate the Tagged Buffer."
  > =20
  > Mike
  > =20


--Boundary_(ID_tnEtJjoP2s9Nffge6XKU9A)
Content-type: text/html; charset=iso-8859-1
Content-transfer-encoding: 7BIT

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=iso-8859-1" http-equiv=Content-Type>
<META name=GENERATOR content="MSHTML 8.00.6001.19088">
<STYLE></STYLE>
</HEAD>
<BODY bgColor=#ffffff>
<DIV><FONT size=2>In RFC 5046, we specifically created the term RDMA-Capable 
Protocol (RCaP) to refer to&nbsp;protocol stacks that provide the RDMA 
functionality.&nbsp; We pointed to iWARP and Infiniband as examples of RCaP in 
the RFC, and recently RoCE (RDMA over Converged Ethernet) can also be added to 
the list.&nbsp; The ideal situation is for iSER implementations to be usable in 
any RCaP.&nbsp; This minimizes the development effort for iSER&nbsp;and 
facilitates the acceptance of iSER.&nbsp;&nbsp;Bob Russell suggested back in 
3/22/2009 to incorporate the use of Virtual Address as this would allow 
implementations such as the OFED stack to be used for different RCaPs.&nbsp; The 
initial iSER draft (draft-ietf-storm-iser-01) submitted to STORM reflected this 
change request.&nbsp; Alex suggested further clarfiying the use of Virtual 
Address to clear up the ambiguity in the -01 version.</FONT></DIV>
<DIV><FONT size=2></FONT>&nbsp;</DIV>
<DIV><FONT size=2>Mike</FONT></DIV>
<BLOCKQUOTE 
style="BORDER-LEFT: #000000 2px solid; PADDING-LEFT: 5px; PADDING-RIGHT: 0px; MARGIN-LEFT: 5px; MARGIN-RIGHT: 0px">
  <DIV style="FONT: 10pt arial">----- Original Message ----- </DIV>
  <DIV 
  style="FONT: 10pt arial; BACKGROUND: #e4e4e4; font-color: black"><B>From:</B> 
  <A title=cait@asomi.com href="mailto:cait@asomi.com">Caitlin Bestler</A> 
</DIV>
  <DIV style="FONT: 10pt arial"><B>To:</B> <A title=Michael@huaweisymantec.com 
  href="mailto:Michael@huaweisymantec.com">Michael Ko</A> </DIV>
  <DIV style="FONT: 10pt arial"><B>Cc:</B> <A title=ttalpey@microsoft.com 
  href="mailto:ttalpey@microsoft.com">Tom Talpey</A> ; <A 
  title=nezhinsky@gmail.com href="mailto:nezhinsky@gmail.com">Alexander 
  Nezhinsky</A> ; <A title=storm@ietf.org 
  href="mailto:storm@ietf.org">storm@ietf.org</A> </DIV>
  <DIV style="FONT: 10pt arial"><B>Sent:</B> Wednesday, June 29, 2011 8:34 
  PM</DIV>
  <DIV style="FONT: 10pt arial"><B>Subject:</B> Re: [storm] iSER draft</DIV>
  <DIV><BR></DIV>Let me deliberately mis-paraphrase your rationale 
  here:<BR><BR>We only need a three-tuple to specify an advertised buffer, but 
  the IETF should define it to be a four-tuple<BR>in order to make it easier for 
  some host implementations to share code with a non-IETF 
  transport.<BR><BR>That's not much of a rationale for making an IETF protocol 
  more complex than it has to be.<BR><BR>Could you at least be explicit and 
  provide a neutral explanation as to why having a VA *and* a TO could make<BR>a 
  certain class of implementations easier? Something better than "InfiniBand did 
  it that way?"<BR><BR><BR><BR>On Jun 29, 2011, at 2:26 PM, Michael Ko 
  wrote:<BR><BR>&gt; Tom,<BR>&gt;&nbsp; <BR>&gt; As originally written, the iSER 
  spec always assumes ZBVA (zero-based virtual address).&nbsp; But when iSER is 
  used in conjunction with transports other than iWARP, there is a need to 
  include the virtual address for the tagged buffer.&nbsp; Such is the case for 
  Infiniband.&nbsp; Accordingly, the Virtual Address field has been added to the 
  iSCSI Control Type PDUs (sec. 9.2).&nbsp; Alex pointed out that the tagged 
  buffer should now be identified by the 4-tuple (STag, VA, TO, and length) 
  instead.&nbsp; This should be updated everywhere where the term "tagged 
  buffer" is mentioned in the spec.&nbsp; For example, in sec. 1.1 under 
  Advertisement, the sentence "A typical method would be for the iSER Layer to 
  embed the Tagged Buffer's STag, TO, and buffer length in a Send Message 
  destined for the remote iSER Layer" should be modified to "A typical method 
  would be for the iSER Layer to embed the Tagged Buffer's STag, VA, TO, and 
  buffer length in a Send Message destined for the remote iSER Layer."&nbsp; 
  BTW, this sentence also answers your question as to whether VA is passed on 
  the wire.<BR>&gt;&nbsp; <BR>&gt; The Virtual Address is literally the virtual 
  address of the tagged buffer.&nbsp; The tagged offset is the displacement 
  relative to the beginning of the buffer starting at the virtual 
  address.<BR>&gt;&nbsp; <BR>&gt; We could rephrase the definition of Virtual 
  Address in sec. 1.1 as follows:<BR>&gt;&nbsp; <BR>&gt; "Virtual Address (VA) - 
  This is the virtual address of the Tagged Buffer on a Node.&nbsp; When set to 
  0, it indicates that a virtual address is not needed to locate the Tagged 
  Buffer."<BR>&gt;&nbsp; <BR>&gt; Mike<BR>&gt;&nbsp; 
<BR><BR></BLOCKQUOTE></BODY></HTML>

--Boundary_(ID_tnEtJjoP2s9Nffge6XKU9A)--

From cbm@chadalapaka.com  Thu Jun 30 11:06:11 2011
Return-Path: <cbm@chadalapaka.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2B62211E8195 for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 11:06:11 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XhFunfwTAYIj for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 11:06:09 -0700 (PDT)
Received: from snt0-omc2-s14.snt0.hotmail.com (snt0-omc2-s14.snt0.hotmail.com [65.55.90.89]) by ietfa.amsl.com (Postfix) with ESMTP id B69B311E80B5 for <storm@ietf.org>; Thu, 30 Jun 2011 11:06:09 -0700 (PDT)
Received: from SNT131-DS13 ([65.55.90.72]) by snt0-omc2-s14.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);  Thu, 30 Jun 2011 11:06:09 -0700
X-Originating-IP: [131.107.0.94]
X-Originating-Email: [cbm@chadalapaka.com]
Message-ID: <SNT131-ds13E5904E7EA7050ED37F89A0580@phx.gbl>
From: Mallikarjun Chadalapaka <cbm@chadalapaka.com>
To: "'Pat Thaler'" <pthaler@broadcom.com>, "'Hemal Shah'" <hemal@broadcom.com>, <david.black@emc.com>, <storm@ietf.org>
References: <7C4DFCE962635144B8FAE8CA11D0BF1E0588F426C5@MX14A.corp.emc.com> <SNT131-ds2DE0F0F2022D63A4AD747A0760@phx.gbl> <76DBE161893C324BA6D4BB507EE4C384951235CB38@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds117065D22D7BF8583485DBA0790@phx.gbl> <76DBE161893C324BA6D4BB507EE4C38495124C4828@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds219B987FD6FA0437A497FDA06D0@phx.gbl> <76DBE161893C324BA6D4BB507EE4C3849513370AB5@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds1709A999A40FA52B3C247A0590@phx.gbl> <DBFBCB90599B2C46992032279293D10020A74038C5@SJEXCHCCR01.corp.ad.broadcom.com>
In-Reply-To: <DBFBCB90599B2C46992032279293D10020A74038C5@SJEXCHCCR01.corp.ad.broadcom.com>
Date: Thu, 30 Jun 2011 11:06:08 -0700
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Outlook 14.0
Thread-Index: AQCUfOqLFecrTE8dOe+/hpcfY+kOJQIRzTq9AgfEPyEC0X+ZOALMKXeKAZ2F7W4CMc4IEQIKcUnlAchkSwmWus8xoA==
Content-Language: en-us
X-OriginalArrivalTime: 30 Jun 2011 18:06:09.0437 (UTC) FILETIME=[6392ACD0:01CC3750]
Subject: Re: [storm] Plan for iSCSI work
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 30 Jun 2011 18:06:11 -0000

Hi Pat,

Thanks for flagging the inconsistency in 10.7, that's a good catch.  Given
that and your/Hemal's feedback, I think turning the current MUST into a
SHOULD is probably the right resolution.   That syncs them both to SHOULD.
Unless I see other comments, I will get this into the next revision.

Thanks.

Mallikarjun


  -----Original Message-----
  From: Pat Thaler [mailto:pthaler@broadcom.com]
  Sent: Wednesday, June 29, 2011 4:03 PM
  To: Mallikarjun Chadalapaka; Hemal Shah; david.black@emc.com;
  storm@ietf.org
  Subject: RE: [storm] Plan for iSCSI work
  
  Hi Malikarjun,
  
  10.7 in the consolidated draft says that FastAbort is a SHOULD:
"Therefore,
  both iSCSI target and initiator implementations SHOULD support FastAbort
  multi-task abort semantics (Section 4.2.3.4)."
  
  When 4.2.3.5 saying "If an iSCSI target implementation is capable of
  supporting TaskReporting=FastAbort functionality" implies that the
  implementation was allowed to not implement FastAbort. You might
  interpret "supporting" = (implemented and negotiated to TaskAbort) as you
  say below, but "capable of supporting" still reads as implemented. When
  the standard puts an if in front of that, it implies that an
implementation is
  allowed to be not capable of supporting FastAbort.
  
  Neither of these is consistent with MUST implement for FastAbort. The
  rationale for requiring FastAbort doesn't seem strong and I'd prefer that
the
  inconsistency be resolved by not requiring its implementation.
  
  Regards,
  Pat
  
  -----Original Message-----
  From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf
  Of Mallikarjun Chadalapaka
  Sent: Wednesday, June 29, 2011 3:16 PM
  To: Hemal Shah; david.black@emc.com; storm@ietf.org
  Subject: Re: [storm] Plan for iSCSI work
  
  Hi Hemal,
  
  >1.     The implementation of TaskReporting is required and NotUnderstood
  response for TaskReporting key is not permitted
  
  No.  Note that TaskReporting is not defined in RFC 3720.  Patrick
MacArthur
  proposed on this list a couple of weeks ago that we should fix this
  disconnect in the consolidated draft - which I agreed with because I think
  it's a good suggestion.
  
   >2.    To enable FastAbort on a session, the negotiation of TaskReporting
  key is required.
  
  Correct, specifically "TaskReporting=FastAbort" must be negotiated.
  
  >3.     Only when the TaskReporting=FastAbort functionality is supported,
  the protocol behavior specified in Section 4.2.3.4 is required to be
  implemented and exhibited.
  
  If I were to attempt re-writing your idea, I would say:
  
  When the protocol behavior specified in Section 4.2.3.4 is implemented and
  TaskReporting=FastAbort is operational on a session, then the FastAbort
  functionality is supported/exhibited on that session by the
implementation.
  
  >a.     Section 4.2.3.5: If an iSCSI target implementation is capable of
  supporting
  >TaskReporting=FastAbort functionality (Section 13.23).
  
  "Supporting" = implemented and negotiated to "TaskReporting=FastAbort".
  Note that if even if an implementation has implemented this functionality
  and wants to use it, it will not be able to negotiate it on any session if
it's
  only communicating with a bunch of RFC 3720 implementations.
  
   >b.    Section 4.2.2.3.3: Whenever the TaskReporting key (Section 12.23
  "Task Reporting") is negotiated to ResponseFence or FastAbort for an iSCSI
  session.
  >c.     Section 4.2.2.3.4: Whenever the TaskReporting key (Section 13.23)
is
  negotiated to ResponseFence or FastAbort for an iSCSI session.
  
  I don't see an issue.  Note that the discussion in these two sections is
on
  response fencing, not fast abort per se.
  
  >4.     ...prefixed with some text like "Only when the
  TaskReporting=FastAbort functionality is supported..".
  
  Please see the above responses.  Your wording is vague for me, as I don't
  comprehend a standalone notion of "supporting" a feature, other than
  implementing a feature and negotiating the feature usage.   Current RFC
  5048
  requirement is that implementation is MUST, and negotiation is MUST
  (may result in the feature to be used, or not).  So as such, there is
already
  ample flexibility for implementations.
  
  I think you are restating your earlier suggestion to loosen the "MUST
  implement" requirement in RFC5048.  And I've already shared how I feel
  about it, :)  I think it would be a step in the wrong direction, when
faster
  error recovery with multi-task aborts with large # of
sessions/LUs/initiators
  is even more critical, due to increased consolidation in data centers.
  
  At this point, I think it would be good to also hear from the rest of the
list.
  
  Thanks.
  
  Mallikarjun
  
  
  
  
  
  >Thanks Mallikarjun! I would still like to confirm my understanding of
  >what
  you describe as "MUST implement, but MUST negotiate".
  >
  >Here is my understanding:
  >
  >1.     The implementation of TaskReporting is required and NotUnderstood
  response for TaskReporting key is not permitted based on the following
text
  from RFC5048 and Section 6.2 of the consolidated draft:
  >                 All keys defined in [RFC3720] MUST be supported by all
  >            compliant implementations; a NotUnderstood answer on any of
  the
  >            [RFC3720] keys therefore MUST be considered a protocol
  > error
  and
  >            handled accordingly.
  >2.     To enable FastAbort on a session, the negotiation of TaskReporting
  key is required.
  >3.     Only when the TaskReporting=FastAbort functionality is supported,
  the protocol behavior specified in Section 4.2.3.4 is required to be
  implemented and exhibited. There are several places in the spec that
  indicate this type of conditionality.
  >a.     Section 4.2.3.5: If an iSCSI target implementation is capable of
  supporting
  >TaskReporting=FastAbort functionality (Section 13.23).
  >b.     Section 4.2.2.3.3: Whenever the TaskReporting key (Section 12.23
  "Task Reporting") is negotiated to ResponseFence or FastAbort for an iSCSI
  session.
  >c.     Section 4.2.2.3.4: Whenever the TaskReporting key (Section 13.23)
is
  negotiated to ResponseFence or FastAbort for an iSCSI session.
  >4.     If TaskReporting=FastAbort functionality is not supported, then
the
  protocol behavior specified in Section 4.2.3.4 is not required to be
  implemented or exhibited. If this is true, then the first sentence in
section
  4.2.3.4 is misleading and should be removed or prefixed with some text
like
  "Only when the TaskReporting=FastAbort functionality is supported..".
  >
  >Hemal
  >
  >
  >-----Original Message-----
  >From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
  >Sent: Friday, June 17, 2011 3:32 PM
  >To: Hemal Shah; david.black@emc.com; storm@ietf.org
  >Subject: RE: [storm] Plan for iSCSI work
  >
  >Hi Hemal,
  >
  >During RFC 5048 effort, I recall we settled on the "MUST implement, but
  MUST
  >negotiate" formulation after some list deliberation.
  >
  >With plain RFC 3720 semantics, there are some multi-initiator scenarios
  >where multi-task aborts could essentially lead to target deadlocks,
  >waiting on initiators on third-party sessions that may never respond.
  >With the "Clarified" semantics in RFC 5048, the third-party deadlocks
  >aren't a problem anymore, but issuing initiator could still experience
  >timeouts and escalate error recovery - depending on the number of LUs,
  sessions,
  >connections and tasks affected with the issued TMF.    Escalated error
  >recovery is usually something we want to avoid as it adds more
  >delays/alerts/logs/failovers/failbacks etc.  Finally, "Updated"
  >semantics
  in
  >RFC 5048 are meant to address this timeout problem - they permit
  >targets to provide accelerated responses and allow them to deal with
  >book-keeping/quiescing operations in a lazy fashion (which are the real
  >culprits that trigger timeouts).
  >
  >Given all the above, the WG settled on FastAbort as a MUST-implement.
  >The "MUST negotiate" part then is to enable backwards-compatibility
  >with RFC
  >3720 implementations.  At this point, I'd rather not loosen the
  Consolidated
  >iSCSI draft requirement from what it is in RFC 5048.  I suggest we
  >leave it the way it is.
  >
  >Mallikarjun
  >
  >
  >
  >
  >
  >
  >From: Hemal Shah [mailto:hemal@broadcom.com]
  >Sent: Thursday, June 09, 2011 3:57 PM
  >To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org
  >Subject: RE: [storm] Plan for iSCSI work
  >
  >Thanks Mallikarjun!
  >
  >The way I read Section 4.1.3 of RFC5048 and Section 4.2.3.4 of the
  >consolidated draft, the implementation of "FastAbort" feature is a MUST.
  >From the first paragraph text from Section 4.1.3 of RFC5048, the first
  >sentence mandates that all iSCSI implementation comply to this section
  >but the second sentence say that the requirement in this section is
  >conditional to the negotiation of "FastAbort" key.
  >
  >Protocol behavior defined in this section MUST be implemented by all
  >iSCSI implementations complying with this document. Protocol behavior
  >defined in this section MUST be exhibited by iSCSI implementations on
  >an iSCSI session when they negotiate the TaskReporting (Section 9.1)
  >key to "FastAbort" on that session.
  >
  >
  >This is confusing. I think it will be better to remove the first
  >sentence from 4.2.3.4 and clarify that the requirements in this section
  >is conditional. I suggest rewording the first two sentences to below:
  >
  >       iSCSI implementations may negotiate the TaskReporting (Section
  >9.1) key to "FastAbort" on an iSCSI session. Protocol behavior defined
  >in this section MUST be exhibited by iSCSI implementations on an iSCSI
  >session when they negotiate the TaskReporting (Section 9.1) key to
  >"FastAbort" on that session.
  >
  >Hemal
  >
  >
  >-----Original Message-----
  >From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
  >Sent: Friday, May 27, 2011 5:29 PM
  >To: Hemal Shah; david.black@emc.com; storm@ietf.org
  >Subject: RE: [storm] Plan for iSCSI work
  >
  >Hi Hemal,
  >
  >The text in this draft came verbatim from section 4.1.3 of RFC 5048.
  >There have been no changes in this area.
  >
  >The new text (as well as the old text) requires the TaskReporting key
  >to be negotiated to "FastAbort" before the multi-task abort semantics
  >can be used on an iSCSI session.
  >
  >Thanks.
  >
  >Mallikarjun
  >
  >
  >
  >
  >  -----Original Message-----
  >  From: Hemal Shah [mailto:hemal@broadcom.com]
  >  Sent: Friday, May 27, 2011 4:51 PM
  >  To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org
  >  Subject: RE: [storm] Plan for iSCSI work
  >
  >  Mallikarjun and David,
  >
  >  I noticed one problematic item in the consolidated draft. This item
  > is
  the
  >  requirement to support FastAbort. This feature was already defined in
  > the  implementation guide, but it was optional. In this draft, it
  > became a  required feature MUST - see in section 4.2.3.4.
  >
  >  Do you know why the requirement was changed in the consolidated
  draft?
  >
  >  I would like to keep the requirement optional as stated in the
  > implementation guide and not break backward compatibility.
  >
  >  Hemal
  >
  >  -----Original Message-----
  >  From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On
  > Behalf  Of Mallikarjun Chadalapaka
  >  Sent: Thursday, May 26, 2011 6:50 PM
  >  To: david.black@emc.com; storm@ietf.org
  >  Subject: Re: [storm] Plan for iSCSI work
  >
  >  Hi David,
  >
  >  The list of changes you have called out are already done in the
  >latest draft.  I
  >  assume then that you are suggesting that the list itself should be
  >included
  >  in the next revision of the draft.
  >
  >  Here's what I recall we have done so far:
  >  1)  iSCSIProtocolLevel specified as "1", and added a related
  >normative
  >  reference to iSCSI-SAM draft
  >  2)  Markers and related keys were removed
  >  3)  SPKM authentication and related keys were removed
  >  4)  Added a new section on responding to obsoleted keys
  >  5)  Have explicitly allowed initiator+target implementations
  >throughout the
  >  text
  >  6)  Clarified that implementations SHOULD NOT rely on SLP-based
  >discovery
  >  7)  Added UML diagrams, and related conventions
  >
  >  The above is of course in addition to consolidating the different
  >RFCs, and
  >  making the related editorial changes.
  >
  >  Thanks.
  >
  >  Mallikarjun
  >
  >
  >
  >
  >    -----Original Message-----
  >    From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On
  Behalf
  >    Of david.black@emc.com
  >    Sent: Friday, May 20, 2011 2:49 PM
  >    To: storm@ietf.org
  >    Subject: [storm] Plan for iSCSI work
  >
  >    I thought I'd offer some advance planning/warning on this, as the
  >    consolidated iSCSI draft (draft-ietf-storm-iscsi-cons) is large
  >(over
  >300
  >    pages).  The current plan is to run a simultaneous WG Last Call on
  >both
  >  this
  >    draft and the new features draft (draft-ietf-storm-iscsi-sam)
  >starting in
  >  mid-
  >    June (probably the week of June 13, after I get back from a badly
  needed
  >    vacation).  That WG Last Call will run longer than the typical
  >2-week time
  >    period, due to the total size of the drafts, but will end by July
  >5th
  at
  >the
  >    latest so that the status of the drafts and the next steps are
  >known prior
  >  to
  >    the T10 (SCSI standards) meetings during the week of July 11.  As
  >July 11th
  >    is also the draft cutoff deadline for the Quebec City IETF
  >meetings, revised
  >    draft versions may not show up until that meeting week (week of July
  >    24th).
  >
  >    This is also a good point to announce that the storm WG will meet in
  >    Quebec City.
  >    I've only requested a 1-hour session, as we get most of our work
  > done
  on
  >    the mailing list.  Among the items for that meeting will be
  > figuring
  out
  >  what
  >    to do with the RDMA extensions draft (despite its name,
  >draft-ietf-storm-
  >    rdmap-ext-00, it's not currently an official work item for the
  >storm WG).
  >
  >    One thing that's missing from the consolidated iSCSI draft (and is
  >a reason
  >    why we're going to need a -03 version) is the changes that it makes
  >to the
  >    RFCs that it consolidates.  Off the top of my head, the major
  >changes
  >are:
  >         - Removal of SPKM authentication
  >         - Removal of the Marker appendix
  >         - Removal of the SHOULD requirement for SLP implementation.
  >    Have I missed anything significant?  The summary of this will need
  >to
  be
  >    added to that draft.
  >
  >    WG Last Call will be an opportunity (in fact the final opportunity)
  > to  discuss
  >    whether anything else should be removed from iSCSI, but there's no
  need
  >    to wait
  >    - I encourage people to review both drafts and post comments
  whenever
  >    they can.
  >
  >    In parallel, work will get started on any iSCSI MIB changes that
  >are needed.
  >    So far, I only see one MIB change - the iSCSIProtocolLevel from the
new
  >    features draft needs to be added to the MIB, probably with a
structure
  >    analogous to the iSCSI version support that's already in the MIB.
  >
  >    Thanks,
  >    --David (storm WG co-chair)
  >    ----------------------------------------------------
  >    David L. Black, Distinguished Engineer
  >    EMC Corporation, 176 South St., Hopkinton, MA  01748
  >    +1 (508) 293-7953             FAX: +1 (508) 293-7786
  >    david.black@emc.com        Mobile: +1 (978) 394-7754
  >    ----------------------------------------------------
  >
  >    _______________________________________________
  >    storm mailing list
  >    storm@ietf.org
  >    https://www.ietf.org/mailman/listinfo/storm
  >
  >  _______________________________________________
  >  storm mailing list
  >  storm@ietf.org
  >  https://www.ietf.org/mailman/listinfo/storm
  >
  >
  >
  >
  >
  >
  >
  >
  >
  
  _______________________________________________
  storm mailing list
  storm@ietf.org
  https://www.ietf.org/mailman/listinfo/storm
  



From nezhinsky@gmail.com  Thu Jun 30 11:07:52 2011
Return-Path: <nezhinsky@gmail.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1BA5011E823C for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 11:07:52 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.598
X-Spam-Level: 
X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8cfP4HUAAa3k for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 11:07:51 -0700 (PDT)
Received: from mail-yw0-f44.google.com (mail-yw0-f44.google.com [209.85.213.44]) by ietfa.amsl.com (Postfix) with ESMTP id C3CA111E8240 for <storm@ietf.org>; Thu, 30 Jun 2011 11:07:46 -0700 (PDT)
Received: by ywp31 with SMTP id 31so1261056ywp.31 for <storm@ietf.org>; Thu, 30 Jun 2011 11:07:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=V56kMUeXYDjUIeeM8yVTjokrNIlm4NoGQsjcH5qC6K0=; b=yEhJklZyoZ61bOiccqCPuOoncCzVVn0nWGjnBCghCg3TK0V9lnogYuMlat2eBPo+Ss S9VZyK3l+hN/rC/e9jc4AZgKG0oRqz+dNmE/0GBOYyFZDW/5oD9fvyr+ndO1dMLe+ofc 8tb0xB1MuGHIeM5P7GK/BNjCQdwIropSvbQZU=
MIME-Version: 1.0
Received: by 10.236.78.65 with SMTP id f41mr2968696yhe.233.1309457266283; Thu, 30 Jun 2011 11:07:46 -0700 (PDT)
Received: by 10.236.203.196 with HTTP; Thu, 30 Jun 2011 11:07:46 -0700 (PDT)
In-Reply-To: <971779FD6B314A40B1A3658E521C0D4A@china.huawei.com>
References: <BANLkTi=GVHUBdsZiwsRXxETd_rU=F8f85w@mail.gmail.com> <F83812DF4B59B9499C1BC978336D91745EDE207F@TK5EX14MBXC118.redmond.corp.microsoft.com> <971779FD6B314A40B1A3658E521C0D4A@china.huawei.com>
Date: Thu, 30 Jun 2011 21:07:46 +0300
Message-ID: <BANLkTik0rG2=orD0S0L3s62gBSJsLapA+w@mail.gmail.com>
From: Alexander Nezhinsky <nezhinsky@gmail.com>
To: Michael Ko <Michael@huaweisymantec.com>
Content-Type: multipart/alternative; boundary=20cf300514fa3492cb04a6f1c730
Cc: storm@ietf.org
Subject: Re: [storm] iSER draft
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 30 Jun 2011 18:07:52 -0000

--20cf300514fa3492cb04a6f1c730
Content-Type: text/plain; charset=UTF-8

On Thu, Jun 30, 2011 at 12:26 AM, Michael Ko <Michael@huaweisymantec.com>wrote:

> **
> Tom,
>
> As originally written, the iSER spec always assumes ZBVA (zero-based
> virtual address).  But when iSER is used in conjunction with transports
> other than iWARP, there is a need to include the virtual address for the
> tagged buffer.  Such is the case for Infiniband.  Accordingly, the Virtual
> Address field has been added to the iSCSI Control Type PDUs (sec. 9.2).
> Alex pointed out that the tagged buffer should now be identified by the
> 4-tuple (STag, VA, TO, and length) instead.  This should be updated
> everywhere where the term "tagged buffer" is mentioned in the spec.  For
> example, in sec. 1.1 under Advertisement, the sentence "A typical method
> would be for the iSER Layer to embed the Tagged Buffer's STag, TO, and
> buffer length in a Send Message destined for the remote iSER Layer" should
> be modified to "A typical method would be for the iSER Layer to embed the
> Tagged Buffer's STag, VA, TO, and buffer length in a Send Message destined
> for the remote iSER Layer."  BTW, this sentence also answers your question
> as to whether VA is passed on the wire.
>
>

Agree, using VA was in the draft already, I only suggested to clarify its
use (and modes of use) from the level of definitions and up,
so that it won't "suddenly" appear in the header format description section.


> The Virtual Address is literally the virtual address of the tagged buffer.
> The tagged offset is the displacement relative to the beginning of the
> buffer starting at the virtual address.
>
> We could rephrase the definition of Virtual Address in sec. 1.1 as follows:
>
> "Virtual Address (VA) - This is the virtual address of the Tagged Buffer on
> a Node.  When set to 0, it indicates that a virtual address is not needed to
> locate the Tagged Buffer."
>
>

> Read and write buffers are identified by the combination of STag and VA.
>

OK. I tried to make the same point in the following:

> Both fields should be required for any transport. This requirement does not
> preclude
> using ZBVA where possible. When VA is not used (as in iWARP) it is set to
> 0.
>
> Below is the actually used format:
>
> struct iser_hdr {
>   uint8_t   flags;
>   uint8_t   rsvd[3];
>   uint32_t  write_stag; /* write rkey in IB */
>   uint64_t  write_va;
>   uint32_t  read_stag;  /* read rkey in IB */
>   uint64_t  read_va;
> }
>
> I would like to add that InfiniBand does have ZBVA mode. It supposedly
provides less than optimal performance with IB.
And it is not exported through the current rdma-cm API, which supports
RDMA/Ethernet as well, and is the only option
to use RDMA in user space linux apps, as of today.

So you could look at the additional fields as a kind of "reserved" in case
of standard ZBVA implementation,
used to practically support certain APIs and get enhanced performance in
some cases.

Alex

--20cf300514fa3492cb04a6f1c730
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><br><div class=3D"gmail_quote">On Thu, Jun 30, 2011 at 12:=
26 AM, Michael Ko <span dir=3D"ltr">&lt;<a href=3D"mailto:Michael@huaweisym=
antec.com">Michael@huaweisymantec.com</a>&gt;</span> wrote:<br><blockquote =
class=3D"gmail_quote" style=3D"margin: 0pt 0pt 0pt 0.8ex; border-left: 1px =
solid rgb(204, 204, 204); padding-left: 1ex;">
<u></u>





<div link=3D"blue" bgcolor=3D"#ffffff" vlink=3D"purple" lang=3D"EN-US">
<div><font size=3D"2">Tom,</font></div>
<div><font size=3D"2"></font>=C2=A0</div>
<div><font size=3D"2">As originally written, the iSER spec always assumes Z=
BVA=20
(zero-based virtual address).=C2=A0 But when iSER is used in conjunction wi=
th=20
transports other than iWARP, there is a need to include the virtual address=
 for=20
the tagged buffer.=C2=A0 Such is the case for Infiniband.=C2=A0 Accordingly=
, the=20
Virtual Address field has been added to the iSCSI Control Type PDUs (sec.=
=20
9.2).=C2=A0 Alex pointed out that the tagged buffer should now be identifie=
d by=20
the 4-tuple (STag, VA, TO, and length) instead.=C2=A0 This should be update=
d=20
everywhere where the term &quot;tagged buffer&quot; is mentioned in the spe=
c.=C2=A0 For=20
example, in sec. 1.1 under Advertisement,=C2=A0the sentence &quot;A typical=
 method=20
would be for the iSER Layer to embed the Tagged Buffer&#39;s STag, TO, and =
buffer=20
length in a Send Message destined for the remote iSER Layer&quot; should be=
 modified=20
to &quot;A typical method would be for the iSER Layer to embed the Tagged B=
uffer&#39;s=20
STag, VA, TO, and buffer length in a Send Message destined for the remote i=
SER=20
Layer.&quot;=C2=A0 BTW, this sentence also answers your question as to whet=
her VA is=20
passed on the wire.</font></div>
<div><font size=3D"2"></font>=C2=A0</div></div></blockquote><div><br>Agree,=
 using VA was in the draft already, I only suggested to clarify its use (an=
d modes of use) from the level of definitions and up, <br>so that it won&#3=
9;t &quot;suddenly&quot; appear in the header format description section.<b=
r>
=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin: 0pt 0pt 0pt =
0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div =
link=3D"blue" bgcolor=3D"#ffffff" vlink=3D"purple" lang=3D"EN-US">
<div><font size=3D"2">The Virtual Address is literally the virtual address =
of the=20
tagged buffer.=C2=A0 The tagged offset is the displacement relative to the=
=20
beginning of the buffer starting at the virtual address.</font></div>
<div><font size=3D"2"></font>=C2=A0</div>
<div><font size=3D"2">We could rephrase the definition of Virtual Address i=
n sec.=20
1.1 as follows:</font></div>
<div><font size=3D"2"></font>=C2=A0</div>
<div><font size=3D"2">&quot;Virtual Address (VA) - This is the virtual addr=
ess of=20
the=C2=A0Tagged Buffer on a Node.=C2=A0 When set to 0, it indicates that a=
=20
virtual address is not needed to locate the Tagged Buffer.&quot;</font><br>=
=C2=A0</div></div></blockquote><blockquote class=3D"gmail_quote" style=3D"m=
argin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); paddin=
g-left: 1ex;">
<div link=3D"blue" bgcolor=3D"#ffffff" vlink=3D"purple" lang=3D"EN-US"><div=
>=C2=A0</div>Read and write buffers are identified by the combination=20
  of STag and VA.<br></div></blockquote><div><br>OK. I tried to make the sa=
me point in the following:<br></div><blockquote class=3D"gmail_quote" style=
=3D"margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); p=
adding-left: 1ex;">
<div link=3D"blue" bgcolor=3D"#ffffff" vlink=3D"purple" lang=3D"EN-US"><blo=
ckquote style=3D"border-left: 2px solid rgb(0, 0, 0); padding-left: 5px; pa=
dding-right: 0px; margin-left: 5px; margin-right: 0px;"><div><div class=3D"=
h5"><div>
<div><div><p class=3D"MsoNormal">Both fields should be required for any tra=
nsport. This=20
  requirement does not preclude<br>using ZBVA where possible. When VA is no=
t=20
  used (as in iWARP) it is set to 0.<br><br>Below is the actually used=20
  format:<br><br><span style=3D"font-family: &#39;Courier New&#39;;">struct=
 iser_hdr=20
  {<br>=C2=A0 uint8_t=C2=A0=C2=A0 flags;<br>=C2=A0 uint8_t=C2=A0=C2=A0=20
  rsvd[3];<br>=C2=A0 uint32_t=C2=A0 write_stag; /* write rkey in IB */<br>=
=C2=A0=20
  uint64_t=C2=A0 write_va;<br>=C2=A0 uint32_t=C2=A0 read_stag;=C2=A0 /* rea=
d=20
  rkey in IB */<br>=C2=A0 uint64_t=C2=A0 read_va;<br>}</span></p></div></di=
v></div></div></div></blockquote></div></blockquote><div>I would like to ad=
d that InfiniBand does have ZBVA mode. It supposedly provides less than opt=
imal performance with IB. <br>
And it is not exported through the current rdma-cm API, which supports RDMA=
/Ethernet as well, and is the only option <br>to use RDMA in user space lin=
ux apps, as of today. <br><br>So you could look at the additional fields as=
 a kind of &quot;reserved&quot; in case of standard ZBVA implementation, <b=
r>
used to practically support certain APIs and get enhanced performance in so=
me cases.<br></div></div><br>Alex<br></div>

--20cf300514fa3492cb04a6f1c730--

From hemal@broadcom.com  Thu Jun 30 11:10:16 2011
Return-Path: <hemal@broadcom.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7306411E822B for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 11:10:16 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.001,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id n5eOjPDDEpdn for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 11:10:14 -0700 (PDT)
Received: from mms2.broadcom.com (mms2.broadcom.com [216.31.210.18]) by ietfa.amsl.com (Postfix) with ESMTP id 54A6711E8240 for <storm@ietf.org>; Thu, 30 Jun 2011 11:10:13 -0700 (PDT)
Received: from [10.9.200.133] by mms2.broadcom.com with ESMTP (Broadcom SMTP Relay (Email Firewall v6.3.2)); Thu, 30 Jun 2011 11:14:27 -0700
X-Server-Uuid: D3C04415-6FA8-4F2C-93C1-920E106A2031
Received: from IRVEXCHCCR02.corp.ad.broadcom.com ([10.9.200.183]) by IRVEXCHHUB02.corp.ad.broadcom.com ([10.9.200.133]) with mapi; Thu, 30 Jun 2011 11:09:39 -0700
From: "Hemal Shah" <hemal@broadcom.com>
To: "Mallikarjun Chadalapaka" <cbm@chadalapaka.com>, "Pat Thaler" <pthaler@broadcom.com>, "david.black@emc.com" <david.black@emc.com>, "storm@ietf.org" <storm@ietf.org>
Date: Thu, 30 Jun 2011 11:08:43 -0700
Thread-Topic: [storm] Plan for iSCSI work
Thread-Index: AQCUfOqLFecrTE8dOe+/hpcfY+kOJQIRzTq9AgfEPyEC0X+ZOALMKXeKAZ2F7W4CMc4IEQIKcUnlAchkSwmWus8xoIAADTeg
Message-ID: <76DBE161893C324BA6D4BB507EE4C3849513371174@IRVEXCHCCR02.corp.ad.broadcom.com>
References: <7C4DFCE962635144B8FAE8CA11D0BF1E0588F426C5@MX14A.corp.emc.com> <SNT131-ds2DE0F0F2022D63A4AD747A0760@phx.gbl> <76DBE161893C324BA6D4BB507EE4C384951235CB38@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds117065D22D7BF8583485DBA0790@phx.gbl> <76DBE161893C324BA6D4BB507EE4C38495124C4828@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds219B987FD6FA0437A497FDA06D0@phx.gbl> <76DBE161893C324BA6D4BB507EE4C3849513370AB5@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds1709A999A40FA52B3C247A0590@phx.gbl> <DBFBCB90599B2C46992032279293D10020A74038C5@SJEXCHCCR01.corp.ad.broadcom.com> <SNT131-ds13E5904E7EA7050ED37F89A0580@phx.gbl>
In-Reply-To: <SNT131-ds13E5904E7EA7050ED37F89A0580@phx.gbl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
MIME-Version: 1.0
X-WSS-ID: 6212628962O16918530-01-01
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Re: [storm] Plan for iSCSI work
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 30 Jun 2011 18:10:16 -0000

Thanks Mallikarjun!

Hemal

-----Original Message-----
From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
Sent: Thursday, June 30, 2011 11:06 AM
To: Pat Thaler; Hemal Shah; david.black@emc.com; storm@ietf.org
Subject: RE: [storm] Plan for iSCSI work

Hi Pat,

Thanks for flagging the inconsistency in 10.7, that's a good catch.  Given
that and your/Hemal's feedback, I think turning the current MUST into a
SHOULD is probably the right resolution.   That syncs them both to SHOULD.
Unless I see other comments, I will get this into the next revision.

Thanks.

Mallikarjun


  -----Original Message-----
  From: Pat Thaler [mailto:pthaler@broadcom.com]
  Sent: Wednesday, June 29, 2011 4:03 PM
  To: Mallikarjun Chadalapaka; Hemal Shah; david.black@emc.com;
  storm@ietf.org
  Subject: RE: [storm] Plan for iSCSI work

  Hi Malikarjun,

  10.7 in the consolidated draft says that FastAbort is a SHOULD:
"Therefore,
  both iSCSI target and initiator implementations SHOULD support FastAbort
  multi-task abort semantics (Section 4.2.3.4)."

  When 4.2.3.5 saying "If an iSCSI target implementation is capable of
  supporting TaskReporting=3DFastAbort functionality" implies that the
  implementation was allowed to not implement FastAbort. You might
  interpret "supporting" =3D (implemented and negotiated to TaskAbort) as y=
ou
  say below, but "capable of supporting" still reads as implemented. When
  the standard puts an if in front of that, it implies that an
implementation is
  allowed to be not capable of supporting FastAbort.

  Neither of these is consistent with MUST implement for FastAbort. The
  rationale for requiring FastAbort doesn't seem strong and I'd prefer that
the
  inconsistency be resolved by not requiring its implementation.

  Regards,
  Pat

  -----Original Message-----
  From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf
  Of Mallikarjun Chadalapaka
  Sent: Wednesday, June 29, 2011 3:16 PM
  To: Hemal Shah; david.black@emc.com; storm@ietf.org
  Subject: Re: [storm] Plan for iSCSI work

  Hi Hemal,

  >1.     The implementation of TaskReporting is required and NotUnderstood
  response for TaskReporting key is not permitted

  No.  Note that TaskReporting is not defined in RFC 3720.  Patrick
MacArthur
  proposed on this list a couple of weeks ago that we should fix this
  disconnect in the consolidated draft - which I agreed with because I thin=
k
  it's a good suggestion.

   >2.    To enable FastAbort on a session, the negotiation of TaskReportin=
g
  key is required.

  Correct, specifically "TaskReporting=3DFastAbort" must be negotiated.

  >3.     Only when the TaskReporting=3DFastAbort functionality is supporte=
d,
  the protocol behavior specified in Section 4.2.3.4 is required to be
  implemented and exhibited.

  If I were to attempt re-writing your idea, I would say:

  When the protocol behavior specified in Section 4.2.3.4 is implemented an=
d
  TaskReporting=3DFastAbort is operational on a session, then the FastAbort
  functionality is supported/exhibited on that session by the
implementation.

  >a.     Section 4.2.3.5: If an iSCSI target implementation is capable of
  supporting
  >TaskReporting=3DFastAbort functionality (Section 13.23).

  "Supporting" =3D implemented and negotiated to "TaskReporting=3DFastAbort=
".
  Note that if even if an implementation has implemented this functionality
  and wants to use it, it will not be able to negotiate it on any session i=
f
it's
  only communicating with a bunch of RFC 3720 implementations.

   >b.    Section 4.2.2.3.3: Whenever the TaskReporting key (Section 12.23
  "Task Reporting") is negotiated to ResponseFence or FastAbort for an iSCS=
I
  session.
  >c.     Section 4.2.2.3.4: Whenever the TaskReporting key (Section 13.23)
is
  negotiated to ResponseFence or FastAbort for an iSCSI session.

  I don't see an issue.  Note that the discussion in these two sections is
on
  response fencing, not fast abort per se.

  >4.     ...prefixed with some text like "Only when the
  TaskReporting=3DFastAbort functionality is supported..".

  Please see the above responses.  Your wording is vague for me, as I don't
  comprehend a standalone notion of "supporting" a feature, other than
  implementing a feature and negotiating the feature usage.   Current RFC
  5048
  requirement is that implementation is MUST, and negotiation is MUST
  (may result in the feature to be used, or not).  So as such, there is
already
  ample flexibility for implementations.

  I think you are restating your earlier suggestion to loosen the "MUST
  implement" requirement in RFC5048.  And I've already shared how I feel
  about it, :)  I think it would be a step in the wrong direction, when
faster
  error recovery with multi-task aborts with large # of
sessions/LUs/initiators
  is even more critical, due to increased consolidation in data centers.

  At this point, I think it would be good to also hear from the rest of the
list.

  Thanks.

  Mallikarjun





  >Thanks Mallikarjun! I would still like to confirm my understanding of
  >what
  you describe as "MUST implement, but MUST negotiate".
  >
  >Here is my understanding:
  >
  >1.     The implementation of TaskReporting is required and NotUnderstood
  response for TaskReporting key is not permitted based on the following
text
  from RFC5048 and Section 6.2 of the consolidated draft:
  >                 All keys defined in [RFC3720] MUST be supported by all
  >            compliant implementations; a NotUnderstood answer on any of
  the
  >            [RFC3720] keys therefore MUST be considered a protocol
  > error
  and
  >            handled accordingly.
  >2.     To enable FastAbort on a session, the negotiation of TaskReportin=
g
  key is required.
  >3.     Only when the TaskReporting=3DFastAbort functionality is supporte=
d,
  the protocol behavior specified in Section 4.2.3.4 is required to be
  implemented and exhibited. There are several places in the spec that
  indicate this type of conditionality.
  >a.     Section 4.2.3.5: If an iSCSI target implementation is capable of
  supporting
  >TaskReporting=3DFastAbort functionality (Section 13.23).
  >b.     Section 4.2.2.3.3: Whenever the TaskReporting key (Section 12.23
  "Task Reporting") is negotiated to ResponseFence or FastAbort for an iSCS=
I
  session.
  >c.     Section 4.2.2.3.4: Whenever the TaskReporting key (Section 13.23)
is
  negotiated to ResponseFence or FastAbort for an iSCSI session.
  >4.     If TaskReporting=3DFastAbort functionality is not supported, then
the
  protocol behavior specified in Section 4.2.3.4 is not required to be
  implemented or exhibited. If this is true, then the first sentence in
section
  4.2.3.4 is misleading and should be removed or prefixed with some text
like
  "Only when the TaskReporting=3DFastAbort functionality is supported..".
  >
  >Hemal
  >
  >
  >-----Original Message-----
  >From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
  >Sent: Friday, June 17, 2011 3:32 PM
  >To: Hemal Shah; david.black@emc.com; storm@ietf.org
  >Subject: RE: [storm] Plan for iSCSI work
  >
  >Hi Hemal,
  >
  >During RFC 5048 effort, I recall we settled on the "MUST implement, but
  MUST
  >negotiate" formulation after some list deliberation.
  >
  >With plain RFC 3720 semantics, there are some multi-initiator scenarios
  >where multi-task aborts could essentially lead to target deadlocks,
  >waiting on initiators on third-party sessions that may never respond.
  >With the "Clarified" semantics in RFC 5048, the third-party deadlocks
  >aren't a problem anymore, but issuing initiator could still experience
  >timeouts and escalate error recovery - depending on the number of LUs,
  sessions,
  >connections and tasks affected with the issued TMF.    Escalated error
  >recovery is usually something we want to avoid as it adds more
  >delays/alerts/logs/failovers/failbacks etc.  Finally, "Updated"
  >semantics
  in
  >RFC 5048 are meant to address this timeout problem - they permit
  >targets to provide accelerated responses and allow them to deal with
  >book-keeping/quiescing operations in a lazy fashion (which are the real
  >culprits that trigger timeouts).
  >
  >Given all the above, the WG settled on FastAbort as a MUST-implement.
  >The "MUST negotiate" part then is to enable backwards-compatibility
  >with RFC
  >3720 implementations.  At this point, I'd rather not loosen the
  Consolidated
  >iSCSI draft requirement from what it is in RFC 5048.  I suggest we
  >leave it the way it is.
  >
  >Mallikarjun
  >
  >
  >
  >
  >
  >
  >From: Hemal Shah [mailto:hemal@broadcom.com]
  >Sent: Thursday, June 09, 2011 3:57 PM
  >To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org
  >Subject: RE: [storm] Plan for iSCSI work
  >
  >Thanks Mallikarjun!
  >
  >The way I read Section 4.1.3 of RFC5048 and Section 4.2.3.4 of the
  >consolidated draft, the implementation of "FastAbort" feature is a MUST.
  >From the first paragraph text from Section 4.1.3 of RFC5048, the first
  >sentence mandates that all iSCSI implementation comply to this section
  >but the second sentence say that the requirement in this section is
  >conditional to the negotiation of "FastAbort" key.
  >
  >Protocol behavior defined in this section MUST be implemented by all
  >iSCSI implementations complying with this document. Protocol behavior
  >defined in this section MUST be exhibited by iSCSI implementations on
  >an iSCSI session when they negotiate the TaskReporting (Section 9.1)
  >key to "FastAbort" on that session.
  >
  >
  >This is confusing. I think it will be better to remove the first
  >sentence from 4.2.3.4 and clarify that the requirements in this section
  >is conditional. I suggest rewording the first two sentences to below:
  >
  >       iSCSI implementations may negotiate the TaskReporting (Section
  >9.1) key to "FastAbort" on an iSCSI session. Protocol behavior defined
  >in this section MUST be exhibited by iSCSI implementations on an iSCSI
  >session when they negotiate the TaskReporting (Section 9.1) key to
  >"FastAbort" on that session.
  >
  >Hemal
  >
  >
  >-----Original Message-----
  >From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
  >Sent: Friday, May 27, 2011 5:29 PM
  >To: Hemal Shah; david.black@emc.com; storm@ietf.org
  >Subject: RE: [storm] Plan for iSCSI work
  >
  >Hi Hemal,
  >
  >The text in this draft came verbatim from section 4.1.3 of RFC 5048.
  >There have been no changes in this area.
  >
  >The new text (as well as the old text) requires the TaskReporting key
  >to be negotiated to "FastAbort" before the multi-task abort semantics
  >can be used on an iSCSI session.
  >
  >Thanks.
  >
  >Mallikarjun
  >
  >
  >
  >
  >  -----Original Message-----
  >  From: Hemal Shah [mailto:hemal@broadcom.com]
  >  Sent: Friday, May 27, 2011 4:51 PM
  >  To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org
  >  Subject: RE: [storm] Plan for iSCSI work
  >
  >  Mallikarjun and David,
  >
  >  I noticed one problematic item in the consolidated draft. This item
  > is
  the
  >  requirement to support FastAbort. This feature was already defined in
  > the  implementation guide, but it was optional. In this draft, it
  > became a  required feature MUST - see in section 4.2.3.4.
  >
  >  Do you know why the requirement was changed in the consolidated
  draft?
  >
  >  I would like to keep the requirement optional as stated in the
  > implementation guide and not break backward compatibility.
  >
  >  Hemal
  >
  >  -----Original Message-----
  >  From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On
  > Behalf  Of Mallikarjun Chadalapaka
  >  Sent: Thursday, May 26, 2011 6:50 PM
  >  To: david.black@emc.com; storm@ietf.org
  >  Subject: Re: [storm] Plan for iSCSI work
  >
  >  Hi David,
  >
  >  The list of changes you have called out are already done in the
  >latest draft.  I
  >  assume then that you are suggesting that the list itself should be
  >included
  >  in the next revision of the draft.
  >
  >  Here's what I recall we have done so far:
  >  1)  iSCSIProtocolLevel specified as "1", and added a related
  >normative
  >  reference to iSCSI-SAM draft
  >  2)  Markers and related keys were removed
  >  3)  SPKM authentication and related keys were removed
  >  4)  Added a new section on responding to obsoleted keys
  >  5)  Have explicitly allowed initiator+target implementations
  >throughout the
  >  text
  >  6)  Clarified that implementations SHOULD NOT rely on SLP-based
  >discovery
  >  7)  Added UML diagrams, and related conventions
  >
  >  The above is of course in addition to consolidating the different
  >RFCs, and
  >  making the related editorial changes.
  >
  >  Thanks.
  >
  >  Mallikarjun
  >
  >
  >
  >
  >    -----Original Message-----
  >    From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On
  Behalf
  >    Of david.black@emc.com
  >    Sent: Friday, May 20, 2011 2:49 PM
  >    To: storm@ietf.org
  >    Subject: [storm] Plan for iSCSI work
  >
  >    I thought I'd offer some advance planning/warning on this, as the
  >    consolidated iSCSI draft (draft-ietf-storm-iscsi-cons) is large
  >(over
  >300
  >    pages).  The current plan is to run a simultaneous WG Last Call on
  >both
  >  this
  >    draft and the new features draft (draft-ietf-storm-iscsi-sam)
  >starting in
  >  mid-
  >    June (probably the week of June 13, after I get back from a badly
  needed
  >    vacation).  That WG Last Call will run longer than the typical
  >2-week time
  >    period, due to the total size of the drafts, but will end by July
  >5th
  at
  >the
  >    latest so that the status of the drafts and the next steps are
  >known prior
  >  to
  >    the T10 (SCSI standards) meetings during the week of July 11.  As
  >July 11th
  >    is also the draft cutoff deadline for the Quebec City IETF
  >meetings, revised
  >    draft versions may not show up until that meeting week (week of July
  >    24th).
  >
  >    This is also a good point to announce that the storm WG will meet in
  >    Quebec City.
  >    I've only requested a 1-hour session, as we get most of our work
  > done
  on
  >    the mailing list.  Among the items for that meeting will be
  > figuring
  out
  >  what
  >    to do with the RDMA extensions draft (despite its name,
  >draft-ietf-storm-
  >    rdmap-ext-00, it's not currently an official work item for the
  >storm WG).
  >
  >    One thing that's missing from the consolidated iSCSI draft (and is
  >a reason
  >    why we're going to need a -03 version) is the changes that it makes
  >to the
  >    RFCs that it consolidates.  Off the top of my head, the major
  >changes
  >are:
  >         - Removal of SPKM authentication
  >         - Removal of the Marker appendix
  >         - Removal of the SHOULD requirement for SLP implementation.
  >    Have I missed anything significant?  The summary of this will need
  >to
  be
  >    added to that draft.
  >
  >    WG Last Call will be an opportunity (in fact the final opportunity)
  > to  discuss
  >    whether anything else should be removed from iSCSI, but there's no
  need
  >    to wait
  >    - I encourage people to review both drafts and post comments
  whenever
  >    they can.
  >
  >    In parallel, work will get started on any iSCSI MIB changes that
  >are needed.
  >    So far, I only see one MIB change - the iSCSIProtocolLevel from the
new
  >    features draft needs to be added to the MIB, probably with a
structure
  >    analogous to the iSCSI version support that's already in the MIB.
  >
  >    Thanks,
  >    --David (storm WG co-chair)
  >    ----------------------------------------------------
  >    David L. Black, Distinguished Engineer
  >    EMC Corporation, 176 South St., Hopkinton, MA  01748
  >    +1 (508) 293-7953             FAX: +1 (508) 293-7786
  >    david.black@emc.com        Mobile: +1 (978) 394-7754
  >    ----------------------------------------------------
  >
  >    _______________________________________________
  >    storm mailing list
  >    storm@ietf.org
  >    https://www.ietf.org/mailman/listinfo/storm
  >
  >  _______________________________________________
  >  storm mailing list
  >  storm@ietf.org
  >  https://www.ietf.org/mailman/listinfo/storm
  >
  >
  >
  >
  >
  >
  >
  >
  >

  _______________________________________________
  storm mailing list
  storm@ietf.org
  https://www.ietf.org/mailman/listinfo/storm






From david.black@emc.com  Thu Jun 30 13:27:04 2011
Return-Path: <david.black@emc.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6FA4321F86E1 for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 13:27:04 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -106.599
X-Spam-Level: 
X-Spam-Status: No, score=-106.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rOVzUey0C1Ll for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 13:27:03 -0700 (PDT)
Received: from mexforward.lss.emc.com (mexforward.lss.emc.com [128.222.32.20]) by ietfa.amsl.com (Postfix) with ESMTP id D291021F85B9 for <storm@ietf.org>; Thu, 30 Jun 2011 13:27:02 -0700 (PDT)
Received: from hop04-l1d11-si02.isus.emc.com (HOP04-L1D11-SI02.isus.emc.com [10.254.111.55]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id p5UKQw02012772 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 30 Jun 2011 16:26:59 -0400
Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.221.253]) by hop04-l1d11-si02.isus.emc.com (RSA Interceptor); Thu, 30 Jun 2011 16:26:46 -0400
Received: from mxhub13.corp.emc.com (mxhub13.corp.emc.com [128.221.56.102]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id p5UKMWM2006984; Thu, 30 Jun 2011 16:22:32 -0400
Received: from mx14a.corp.emc.com ([169.254.1.245]) by mxhub13.corp.emc.com ([128.221.56.102]) with mapi; Thu, 30 Jun 2011 16:22:32 -0400
From: <david.black@emc.com>
To: <hemal@broadcom.com>, <cbm@chadalapaka.com>, <pthaler@broadcom.com>, <storm@ietf.org>
Date: Thu, 30 Jun 2011 16:22:31 -0400
Thread-Topic: [storm] Plan for iSCSI work
Thread-Index: AQCUfOqLFecrTE8dOe+/hpcfY+kOJQIRzTq9AgfEPyEC0X+ZOALMKXeKAZ2F7W4CMc4IEQIKcUnlAchkSwmWus8xoIAADTeggAAlA0A=
Message-ID: <7C4DFCE962635144B8FAE8CA11D0BF1E058931036F@MX14A.corp.emc.com>
References: <7C4DFCE962635144B8FAE8CA11D0BF1E0588F426C5@MX14A.corp.emc.com> <SNT131-ds2DE0F0F2022D63A4AD747A0760@phx.gbl> <76DBE161893C324BA6D4BB507EE4C384951235CB38@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds117065D22D7BF8583485DBA0790@phx.gbl> <76DBE161893C324BA6D4BB507EE4C38495124C4828@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds219B987FD6FA0437A497FDA06D0@phx.gbl> <76DBE161893C324BA6D4BB507EE4C3849513370AB5@IRVEXCHCCR02.corp.ad.broadcom.com> <SNT131-ds1709A999A40FA52B3C247A0590@phx.gbl> <DBFBCB90599B2C46992032279293D10020A74038C5@SJEXCHCCR01.corp.ad.broadcom.com> <SNT131-ds13E5904E7EA7050ED37F89A0580@phx.gbl> <76DBE161893C324BA6D4BB507EE4C3849513371174@IRVEXCHCCR02.corp.ad.broadcom.com>
In-Reply-To: <76DBE161893C324BA6D4BB507EE4C3849513371174@IRVEXCHCCR02.corp.ad.broadcom.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-EMM-MHVC: 1
Subject: Re: [storm] Plan for iSCSI work
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 30 Jun 2011 20:27:04 -0000

<WG chair hat off> I think use of "SHOULD" is the right approach here. </WG=
 chair hat off>

<WG chair hat on> Based on list discussion, I believe "SHOULD" is the rough=
 consensus of the WG. </WG chair hat on>

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
david.black@emc.com        Mobile: +1 (978) 394-7754
----------------------------------------------------

> -----Original Message-----
> From: Hemal Shah [mailto:hemal@broadcom.com]
> Sent: Thursday, June 30, 2011 2:09 PM
> To: Mallikarjun Chadalapaka; Pat Thaler; Black, David; storm@ietf.org
> Subject: RE: [storm] Plan for iSCSI work
>
> Thanks Mallikarjun!
>
> Hemal
>
> -----Original Message-----
> From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
> Sent: Thursday, June 30, 2011 11:06 AM
> To: Pat Thaler; Hemal Shah; david.black@emc.com; storm@ietf.org
> Subject: RE: [storm] Plan for iSCSI work
>
> Hi Pat,
>
> Thanks for flagging the inconsistency in 10.7, that's a good catch.  Give=
n
> that and your/Hemal's feedback, I think turning the current MUST into a
> SHOULD is probably the right resolution.   That syncs them both to SHOULD=
.
> Unless I see other comments, I will get this into the next revision.
>
> Thanks.
>
> Mallikarjun
>
>
>   -----Original Message-----
>   From: Pat Thaler [mailto:pthaler@broadcom.com]
>   Sent: Wednesday, June 29, 2011 4:03 PM
>   To: Mallikarjun Chadalapaka; Hemal Shah; david.black@emc.com;
>   storm@ietf.org
>   Subject: RE: [storm] Plan for iSCSI work
>
>   Hi Malikarjun,
>
>   10.7 in the consolidated draft says that FastAbort is a SHOULD:
> "Therefore,
>   both iSCSI target and initiator implementations SHOULD support FastAbor=
t
>   multi-task abort semantics (Section 4.2.3.4)."
>
>   When 4.2.3.5 saying "If an iSCSI target implementation is capable of
>   supporting TaskReporting=3DFastAbort functionality" implies that the
>   implementation was allowed to not implement FastAbort. You might
>   interpret "supporting" =3D (implemented and negotiated to TaskAbort) as=
 you
>   say below, but "capable of supporting" still reads as implemented. When
>   the standard puts an if in front of that, it implies that an
> implementation is
>   allowed to be not capable of supporting FastAbort.
>
>   Neither of these is consistent with MUST implement for FastAbort. The
>   rationale for requiring FastAbort doesn't seem strong and I'd prefer th=
at
> the
>   inconsistency be resolved by not requiring its implementation.
>
>   Regards,
>   Pat
>
>   -----Original Message-----
>   From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf
>   Of Mallikarjun Chadalapaka
>   Sent: Wednesday, June 29, 2011 3:16 PM
>   To: Hemal Shah; david.black@emc.com; storm@ietf.org
>   Subject: Re: [storm] Plan for iSCSI work
>
>   Hi Hemal,
>
>   >1.     The implementation of TaskReporting is required and NotUndersto=
od
>   response for TaskReporting key is not permitted
>
>   No.  Note that TaskReporting is not defined in RFC 3720.  Patrick
> MacArthur
>   proposed on this list a couple of weeks ago that we should fix this
>   disconnect in the consolidated draft - which I agreed with because I th=
ink
>   it's a good suggestion.
>
>    >2.    To enable FastAbort on a session, the negotiation of TaskReport=
ing
>   key is required.
>
>   Correct, specifically "TaskReporting=3DFastAbort" must be negotiated.
>
>   >3.     Only when the TaskReporting=3DFastAbort functionality is suppor=
ted,
>   the protocol behavior specified in Section 4.2.3.4 is required to be
>   implemented and exhibited.
>
>   If I were to attempt re-writing your idea, I would say:
>
>   When the protocol behavior specified in Section 4.2.3.4 is implemented =
and
>   TaskReporting=3DFastAbort is operational on a session, then the FastAbo=
rt
>   functionality is supported/exhibited on that session by the
> implementation.
>
>   >a.     Section 4.2.3.5: If an iSCSI target implementation is capable o=
f
>   supporting
>   >TaskReporting=3DFastAbort functionality (Section 13.23).
>
>   "Supporting" =3D implemented and negotiated to "TaskReporting=3DFastAbo=
rt".
>   Note that if even if an implementation has implemented this functionali=
ty
>   and wants to use it, it will not be able to negotiate it on any session=
 if
> it's
>   only communicating with a bunch of RFC 3720 implementations.
>
>    >b.    Section 4.2.2.3.3: Whenever the TaskReporting key (Section 12.2=
3
>   "Task Reporting") is negotiated to ResponseFence or FastAbort for an iS=
CSI
>   session.
>   >c.     Section 4.2.2.3.4: Whenever the TaskReporting key (Section 13.2=
3)
> is
>   negotiated to ResponseFence or FastAbort for an iSCSI session.
>
>   I don't see an issue.  Note that the discussion in these two sections i=
s
> on
>   response fencing, not fast abort per se.
>
>   >4.     ...prefixed with some text like "Only when the
>   TaskReporting=3DFastAbort functionality is supported..".
>
>   Please see the above responses.  Your wording is vague for me, as I don=
't
>   comprehend a standalone notion of "supporting" a feature, other than
>   implementing a feature and negotiating the feature usage.   Current RFC
>   5048
>   requirement is that implementation is MUST, and negotiation is MUST
>   (may result in the feature to be used, or not).  So as such, there is
> already
>   ample flexibility for implementations.
>
>   I think you are restating your earlier suggestion to loosen the "MUST
>   implement" requirement in RFC5048.  And I've already shared how I feel
>   about it, :)  I think it would be a step in the wrong direction, when
> faster
>   error recovery with multi-task aborts with large # of
> sessions/LUs/initiators
>   is even more critical, due to increased consolidation in data centers.
>
>   At this point, I think it would be good to also hear from the rest of t=
he
> list.
>
>   Thanks.
>
>   Mallikarjun
>
>
>
>
>
>   >Thanks Mallikarjun! I would still like to confirm my understanding of
>   >what
>   you describe as "MUST implement, but MUST negotiate".
>   >
>   >Here is my understanding:
>   >
>   >1.     The implementation of TaskReporting is required and NotUndersto=
od
>   response for TaskReporting key is not permitted based on the following
> text
>   from RFC5048 and Section 6.2 of the consolidated draft:
>   >                 All keys defined in [RFC3720] MUST be supported by al=
l
>   >            compliant implementations; a NotUnderstood answer on any o=
f
>   the
>   >            [RFC3720] keys therefore MUST be considered a protocol
>   > error
>   and
>   >            handled accordingly.
>   >2.     To enable FastAbort on a session, the negotiation of TaskReport=
ing
>   key is required.
>   >3.     Only when the TaskReporting=3DFastAbort functionality is suppor=
ted,
>   the protocol behavior specified in Section 4.2.3.4 is required to be
>   implemented and exhibited. There are several places in the spec that
>   indicate this type of conditionality.
>   >a.     Section 4.2.3.5: If an iSCSI target implementation is capable o=
f
>   supporting
>   >TaskReporting=3DFastAbort functionality (Section 13.23).
>   >b.     Section 4.2.2.3.3: Whenever the TaskReporting key (Section 12.2=
3
>   "Task Reporting") is negotiated to ResponseFence or FastAbort for an iS=
CSI
>   session.
>   >c.     Section 4.2.2.3.4: Whenever the TaskReporting key (Section 13.2=
3)
> is
>   negotiated to ResponseFence or FastAbort for an iSCSI session.
>   >4.     If TaskReporting=3DFastAbort functionality is not supported, th=
en
> the
>   protocol behavior specified in Section 4.2.3.4 is not required to be
>   implemented or exhibited. If this is true, then the first sentence in
> section
>   4.2.3.4 is misleading and should be removed or prefixed with some text
> like
>   "Only when the TaskReporting=3DFastAbort functionality is supported..".
>   >
>   >Hemal
>   >
>   >
>   >-----Original Message-----
>   >From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
>   >Sent: Friday, June 17, 2011 3:32 PM
>   >To: Hemal Shah; david.black@emc.com; storm@ietf.org
>   >Subject: RE: [storm] Plan for iSCSI work
>   >
>   >Hi Hemal,
>   >
>   >During RFC 5048 effort, I recall we settled on the "MUST implement, bu=
t
>   MUST
>   >negotiate" formulation after some list deliberation.
>   >
>   >With plain RFC 3720 semantics, there are some multi-initiator scenario=
s
>   >where multi-task aborts could essentially lead to target deadlocks,
>   >waiting on initiators on third-party sessions that may never respond.
>   >With the "Clarified" semantics in RFC 5048, the third-party deadlocks
>   >aren't a problem anymore, but issuing initiator could still experience
>   >timeouts and escalate error recovery - depending on the number of LUs,
>   sessions,
>   >connections and tasks affected with the issued TMF.    Escalated error
>   >recovery is usually something we want to avoid as it adds more
>   >delays/alerts/logs/failovers/failbacks etc.  Finally, "Updated"
>   >semantics
>   in
>   >RFC 5048 are meant to address this timeout problem - they permit
>   >targets to provide accelerated responses and allow them to deal with
>   >book-keeping/quiescing operations in a lazy fashion (which are the rea=
l
>   >culprits that trigger timeouts).
>   >
>   >Given all the above, the WG settled on FastAbort as a MUST-implement.
>   >The "MUST negotiate" part then is to enable backwards-compatibility
>   >with RFC
>   >3720 implementations.  At this point, I'd rather not loosen the
>   Consolidated
>   >iSCSI draft requirement from what it is in RFC 5048.  I suggest we
>   >leave it the way it is.
>   >
>   >Mallikarjun
>   >
>   >
>   >
>   >
>   >
>   >
>   >From: Hemal Shah [mailto:hemal@broadcom.com]
>   >Sent: Thursday, June 09, 2011 3:57 PM
>   >To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org
>   >Subject: RE: [storm] Plan for iSCSI work
>   >
>   >Thanks Mallikarjun!
>   >
>   >The way I read Section 4.1.3 of RFC5048 and Section 4.2.3.4 of the
>   >consolidated draft, the implementation of "FastAbort" feature is a MUS=
T.
>   >From the first paragraph text from Section 4.1.3 of RFC5048, the first
>   >sentence mandates that all iSCSI implementation comply to this section
>   >but the second sentence say that the requirement in this section is
>   >conditional to the negotiation of "FastAbort" key.
>   >
>   >Protocol behavior defined in this section MUST be implemented by all
>   >iSCSI implementations complying with this document. Protocol behavior
>   >defined in this section MUST be exhibited by iSCSI implementations on
>   >an iSCSI session when they negotiate the TaskReporting (Section 9.1)
>   >key to "FastAbort" on that session.
>   >
>   >
>   >This is confusing. I think it will be better to remove the first
>   >sentence from 4.2.3.4 and clarify that the requirements in this sectio=
n
>   >is conditional. I suggest rewording the first two sentences to below:
>   >
>   >       iSCSI implementations may negotiate the TaskReporting (Section
>   >9.1) key to "FastAbort" on an iSCSI session. Protocol behavior defined
>   >in this section MUST be exhibited by iSCSI implementations on an iSCSI
>   >session when they negotiate the TaskReporting (Section 9.1) key to
>   >"FastAbort" on that session.
>   >
>   >Hemal
>   >
>   >
>   >-----Original Message-----
>   >From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
>   >Sent: Friday, May 27, 2011 5:29 PM
>   >To: Hemal Shah; david.black@emc.com; storm@ietf.org
>   >Subject: RE: [storm] Plan for iSCSI work
>   >
>   >Hi Hemal,
>   >
>   >The text in this draft came verbatim from section 4.1.3 of RFC 5048.
>   >There have been no changes in this area.
>   >
>   >The new text (as well as the old text) requires the TaskReporting key
>   >to be negotiated to "FastAbort" before the multi-task abort semantics
>   >can be used on an iSCSI session.
>   >
>   >Thanks.
>   >
>   >Mallikarjun
>   >
>   >
>   >
>   >
>   >  -----Original Message-----
>   >  From: Hemal Shah [mailto:hemal@broadcom.com]
>   >  Sent: Friday, May 27, 2011 4:51 PM
>   >  To: Mallikarjun Chadalapaka; david.black@emc.com; storm@ietf.org
>   >  Subject: RE: [storm] Plan for iSCSI work
>   >
>   >  Mallikarjun and David,
>   >
>   >  I noticed one problematic item in the consolidated draft. This item
>   > is
>   the
>   >  requirement to support FastAbort. This feature was already defined i=
n
>   > the  implementation guide, but it was optional. In this draft, it
>   > became a  required feature MUST - see in section 4.2.3.4.
>   >
>   >  Do you know why the requirement was changed in the consolidated
>   draft?
>   >
>   >  I would like to keep the requirement optional as stated in the
>   > implementation guide and not break backward compatibility.
>   >
>   >  Hemal
>   >
>   >  -----Original Message-----
>   >  From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On
>   > Behalf  Of Mallikarjun Chadalapaka
>   >  Sent: Thursday, May 26, 2011 6:50 PM
>   >  To: david.black@emc.com; storm@ietf.org
>   >  Subject: Re: [storm] Plan for iSCSI work
>   >
>   >  Hi David,
>   >
>   >  The list of changes you have called out are already done in the
>   >latest draft.  I
>   >  assume then that you are suggesting that the list itself should be
>   >included
>   >  in the next revision of the draft.
>   >
>   >  Here's what I recall we have done so far:
>   >  1)  iSCSIProtocolLevel specified as "1", and added a related
>   >normative
>   >  reference to iSCSI-SAM draft
>   >  2)  Markers and related keys were removed
>   >  3)  SPKM authentication and related keys were removed
>   >  4)  Added a new section on responding to obsoleted keys
>   >  5)  Have explicitly allowed initiator+target implementations
>   >throughout the
>   >  text
>   >  6)  Clarified that implementations SHOULD NOT rely on SLP-based
>   >discovery
>   >  7)  Added UML diagrams, and related conventions
>   >
>   >  The above is of course in addition to consolidating the different
>   >RFCs, and
>   >  making the related editorial changes.
>   >
>   >  Thanks.
>   >
>   >  Mallikarjun
>   >
>   >
>   >
>   >
>   >    -----Original Message-----
>   >    From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On
>   Behalf
>   >    Of david.black@emc.com
>   >    Sent: Friday, May 20, 2011 2:49 PM
>   >    To: storm@ietf.org
>   >    Subject: [storm] Plan for iSCSI work
>   >
>   >    I thought I'd offer some advance planning/warning on this, as the
>   >    consolidated iSCSI draft (draft-ietf-storm-iscsi-cons) is large
>   >(over
>   >300
>   >    pages).  The current plan is to run a simultaneous WG Last Call on
>   >both
>   >  this
>   >    draft and the new features draft (draft-ietf-storm-iscsi-sam)
>   >starting in
>   >  mid-
>   >    June (probably the week of June 13, after I get back from a badly
>   needed
>   >    vacation).  That WG Last Call will run longer than the typical
>   >2-week time
>   >    period, due to the total size of the drafts, but will end by July
>   >5th
>   at
>   >the
>   >    latest so that the status of the drafts and the next steps are
>   >known prior
>   >  to
>   >    the T10 (SCSI standards) meetings during the week of July 11.  As
>   >July 11th
>   >    is also the draft cutoff deadline for the Quebec City IETF
>   >meetings, revised
>   >    draft versions may not show up until that meeting week (week of Ju=
ly
>   >    24th).
>   >
>   >    This is also a good point to announce that the storm WG will meet =
in
>   >    Quebec City.
>   >    I've only requested a 1-hour session, as we get most of our work
>   > done
>   on
>   >    the mailing list.  Among the items for that meeting will be
>   > figuring
>   out
>   >  what
>   >    to do with the RDMA extensions draft (despite its name,
>   >draft-ietf-storm-
>   >    rdmap-ext-00, it's not currently an official work item for the
>   >storm WG).
>   >
>   >    One thing that's missing from the consolidated iSCSI draft (and is
>   >a reason
>   >    why we're going to need a -03 version) is the changes that it make=
s
>   >to the
>   >    RFCs that it consolidates.  Off the top of my head, the major
>   >changes
>   >are:
>   >         - Removal of SPKM authentication
>   >         - Removal of the Marker appendix
>   >         - Removal of the SHOULD requirement for SLP implementation.
>   >    Have I missed anything significant?  The summary of this will need
>   >to
>   be
>   >    added to that draft.
>   >
>   >    WG Last Call will be an opportunity (in fact the final opportunity=
)
>   > to  discuss
>   >    whether anything else should be removed from iSCSI, but there's no
>   need
>   >    to wait
>   >    - I encourage people to review both drafts and post comments
>   whenever
>   >    they can.
>   >
>   >    In parallel, work will get started on any iSCSI MIB changes that
>   >are needed.
>   >    So far, I only see one MIB change - the iSCSIProtocolLevel from th=
e
> new
>   >    features draft needs to be added to the MIB, probably with a
> structure
>   >    analogous to the iSCSI version support that's already in the MIB.
>   >
>   >    Thanks,
>   >    --David (storm WG co-chair)
>   >    ----------------------------------------------------
>   >    David L. Black, Distinguished Engineer
>   >    EMC Corporation, 176 South St., Hopkinton, MA  01748
>   >    +1 (508) 293-7953             FAX: +1 (508) 293-7786
>   >    david.black@emc.com        Mobile: +1 (978) 394-7754
>   >    ----------------------------------------------------
>   >
>   >    _______________________________________________
>   >    storm mailing list
>   >    storm@ietf.org
>   >    https://www.ietf.org/mailman/listinfo/storm
>   >
>   >  _______________________________________________
>   >  storm mailing list
>   >  storm@ietf.org
>   >  https://www.ietf.org/mailman/listinfo/storm
>   >
>   >
>   >
>   >
>   >
>   >
>   >
>   >
>   >
>
>   _______________________________________________
>   storm mailing list
>   storm@ietf.org
>   https://www.ietf.org/mailman/listinfo/storm
>
>
>
>
>


From david.black@emc.com  Thu Jun 30 13:55:17 2011
Return-Path: <david.black@emc.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 10BD911E80AA for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 13:55:17 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -106.599
X-Spam-Level: 
X-Spam-Status: No, score=-106.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pCIjVBYxSKgq for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 13:55:16 -0700 (PDT)
Received: from mexforward.lss.emc.com (mexforward.lss.emc.com [128.222.32.20]) by ietfa.amsl.com (Postfix) with ESMTP id 3ED9011E80A7 for <storm@ietf.org>; Thu, 30 Jun 2011 13:55:15 -0700 (PDT)
Received: from hop04-l1d11-si01.isus.emc.com (HOP04-L1D11-SI01.isus.emc.com [10.254.111.54]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id p5UKt1Jl019803 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 30 Jun 2011 16:55:14 -0400
Received: from mailhub.lss.emc.com (mailhubhoprd02.lss.emc.com [10.254.221.253]) by hop04-l1d11-si01.isus.emc.com (RSA Interceptor); Thu, 30 Jun 2011 16:46:30 -0400
Received: from mxhub02.corp.emc.com (mxhub02.corp.emc.com [10.254.141.104]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id p5UKUe9d014637; Thu, 30 Jun 2011 16:30:41 -0400
Received: from mx14a.corp.emc.com ([169.254.1.245]) by mxhub02.corp.emc.com ([10.254.141.104]) with mapi; Thu, 30 Jun 2011 16:30:41 -0400
From: <david.black@emc.com>
To: <cbm@chadalapaka.com>, <pmacarth@iol.unh.edu>, <storm@ietf.org>
Date: Thu, 30 Jun 2011 16:30:39 -0400
Thread-Topic: [storm] Consolidated iSCSI draft and TaskReporting
Thread-Index: AQDeOsqbPwyi4a8ddBcbQxUWgjUBFZaeCrnwgBRHMXA=
Message-ID: <7C4DFCE962635144B8FAE8CA11D0BF1E0589310372@MX14A.corp.emc.com>
References: <1308321497.3882.10.camel@chiron.ofa> <SNT131-ds212BA8CFEE311B42259821A06C0@phx.gbl>
In-Reply-To: <SNT131-ds212BA8CFEE311B42259821A06C0@phx.gbl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-EMM-MHVC: 1
Subject: Re: [storm] Consolidated iSCSI draft and TaskReporting
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 30 Jun 2011 20:55:17 -0000

<WG chair hat on>

We should do what Mallikarjun proposes.  A table will be needed that lists =
the RFCs for each value of the iSCSIProtocolLevel key.

One change - "(or would have offered)" should be removed from Mallikarjun's=
 proposed text.  Hypothetical requirements tend to be difficult to test and=
 moreover are usually pointless - the values that are actually used in nego=
tiation are what should govern protocol behavior.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA=A0 01748
+1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778=
6
david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754
----------------------------------------------------

> -----Original Message-----
> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of=
 Mallikarjun Chadalapaka
> Sent: Friday, June 17, 2011 8:24 PM
> To: 'Patrick MacArthur'; storm@ietf.org
> Subject: Re: [storm] Consolidated iSCSI draft and TaskReporting
>=20
> Hi Patrick,
>=20
> I tend to agree with you, you raise a valid point.
>=20
> The problem lies in special-casing RFC 3720.  I think the right way to
> address your comment is to require that an implementation's "NotUnderstoo=
d"
> response should correlate to the highest protocol version that it has
> offered (or would have offered) for the iSCSIProtocolLevel key in a
> negotiation.   E.g., let's say if an iSCSI implementation has *offered* a
> value of 2 in a negotiation (even though the other side has negotiated it
> down to 1), it means that the offering implementation can comprehend all
> text keys defined through the spec with iSCSIProtocolLevel=3D2.  So it mu=
st
> not respond to any of those text keys with a "NotUnderstood".
>=20
> So the formal verbiage would look something like the following:
>=20
> An iSCSI implementation MUST comprehend all text keys defined in iSCSI
> Protocol specifications from [RFC3720] through the RFC keyed to the highe=
st
> protocol version that the implementation has offered (or would have offer=
ed)
> for the iSCSIProtocolLevel key in a negotiation.  Returning a NotUndersto=
od
> response on any of these text keys therefore MUST be considered a protoco=
l
> error and handled accordingly.  For all other "later" keys, i.e. text key=
s
> defined in specifications keyed to higher values of iSCSIProtocolLevel, a
> NotUnderstood  answer concludes the negotiation for a negotiated key wher=
eas
> for a declarative key, a NotUnderstood answer simply informs the declarer=
 of
> a lack of comprehension by the receiver.
>=20
> Please let me know if you have comments or questions.  Thanks.
>=20
> Mallikarjun
>=20
>=20
>=20
>   -----Original Message-----
>   From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf
>   Of Patrick MacArthur
>   Sent: Friday, June 17, 2011 7:38 AM
>   To: storm@ietf.org
>   Subject: [storm] Consolidated iSCSI draft and TaskReporting
>=20
>   Hi,
>=20
>   I wanted to ask for clarification on an item appearing in the consolida=
ted
>   iSCSI draft.
>=20
>       "All keys defined in [RFC3720] MUST be supported by all
>       compliant implementations; a NotUnderstood answer on any of the
>       [RFC3720] keys therefore MUST be considered a protocol error and
>       handled accordingly. For all other later keys, a NotUnderstood
>       answer concludes the negotiation for a negotiated key whereas for
>       a declarative key, a NotUnderstood answer simply informs the
>       declarer of a lack of comprehension by the receiver."
>=20
>   I would like to request that the references to RFC3720 above be changed=
 to
>   RFC3720 and RFC5048.  The comments on this list seem to indicate that t=
he
>   baseline for the iSCSI work is on the combination of RFC 3720 and RFC
>   5048, which I interpret to mean that it would be applicable only to
>   implementations compliant with both RFCs.  An implementation that
>   claims to be RFC 5048-compliant and responds with
>   TaskReporting=3DNotUnderstood is broken IMHO. In that case, is there a
>   reason that we allow a NotUnderstood response on an RFC 5048 key
>   (TaskReporting)?
>=20
>   If the STORM WG is trying to ensure compatibility with the requirements=
 of
>   RFC 3720 and RFC 5048 combined, it seems strange to be allowing
>   implementers not to implement the key requirement of RFC 5048.
>=20
>   Thanks,
>=20
>   --
>   Patrick MacArthur
>   Research and Development
>   iSCSI/OFA/IPv6 Consortia
>   UNH InterOperability Laboratory
>=20
>   _______________________________________________
>   storm mailing list
>   storm@ietf.org
>   https://www.ietf.org/mailman/listinfo/storm
>=20
> _______________________________________________
> storm mailing list
> storm@ietf.org
> https://www.ietf.org/mailman/listinfo/storm


From cbm@chadalapaka.com  Thu Jun 30 15:06:44 2011
Return-Path: <cbm@chadalapaka.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0AC4811E807E for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 15:06:44 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FhnHBCnQCkXY for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 15:06:43 -0700 (PDT)
Received: from snt0-omc3-s51.snt0.hotmail.com (snt0-omc3-s51.snt0.hotmail.com [65.54.51.88]) by ietfa.amsl.com (Postfix) with ESMTP id 3C48211E809B for <storm@ietf.org>; Thu, 30 Jun 2011 15:06:43 -0700 (PDT)
Received: from SNT131-DS3 ([65.55.90.135]) by snt0-omc3-s51.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);  Thu, 30 Jun 2011 15:06:42 -0700
X-Originating-IP: [131.107.0.71]
X-Originating-Email: [cbm@chadalapaka.com]
Message-ID: <SNT131-ds3915291A564FA2E77163FA0580@phx.gbl>
From: Mallikarjun Chadalapaka <cbm@chadalapaka.com>
To: <david.black@emc.com>, <pmacarth@iol.unh.edu>, <storm@ietf.org>
References: <1308321497.3882.10.camel@chiron.ofa> <SNT131-ds212BA8CFEE311B42259821A06C0@phx.gbl> <7C4DFCE962635144B8FAE8CA11D0BF1E0589310372@MX14A.corp.emc.com>
In-Reply-To: <7C4DFCE962635144B8FAE8CA11D0BF1E0589310372@MX14A.corp.emc.com>
Date: Thu, 30 Jun 2011 15:06:42 -0700
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
X-Mailer: Microsoft Outlook 14.0
Thread-Index: AQDeOsqbPwyi4a8ddBcbQxUWgjUBFQKxRZHdAgOPlpyWjMTOgA==
Content-Language: en-us
X-OriginalArrivalTime: 30 Jun 2011 22:06:42.0810 (UTC) FILETIME=[FE889DA0:01CC3771]
Subject: Re: [storm] Consolidated iSCSI draft and TaskReporting
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 30 Jun 2011 22:06:44 -0000

I'm OK with leaving out the phrase you point out - I meant it to cover =
the
case of an acceptor in a key negotiation.  But I see your point.

I was thinking along the lines of each spec "claiming" a number for the
iSCSIProtocolLevel key, so no one spec necessarily has a table of all =
the
keys.  Currently, Consolidated draft claims  "1", and SAM draft claims =
"2".
Sounds like you're thinking a table with currently claimed
iSCSIProtocolLevel values in this spec?  I am fine with that too.

Thanks.

Mallikarjun


  -----Original Message-----
  From: david.black@emc.com [mailto:david.black@emc.com]
  Sent: Thursday, June 30, 2011 1:31 PM
  To: cbm@chadalapaka.com; pmacarth@iol.unh.edu; storm@ietf.org
  Subject: RE: [storm] Consolidated iSCSI draft and TaskReporting
 =20
  <WG chair hat on>
 =20
  We should do what Mallikarjun proposes.  A table will be needed that =
lists
  the RFCs for each value of the iSCSIProtocolLevel key.
 =20
  One change - "(or would have offered)" should be removed from
  Mallikarjun's proposed text.  Hypothetical requirements tend to be
difficult
  to test and moreover are usually pointless - the values that are =
actually
  used in negotiation are what should govern protocol behavior.
 =20
  Thanks,
  --David
  ----------------------------------------------------
  David L. Black, Distinguished Engineer
  EMC Corporation, 176 South St., Hopkinton, MA=A0 01748
  +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) =
293-7786
  david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754
  ----------------------------------------------------
 =20
  > -----Original Message-----
  > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On
  Behalf
  > Of Mallikarjun Chadalapaka
  > Sent: Friday, June 17, 2011 8:24 PM
  > To: 'Patrick MacArthur'; storm@ietf.org
  > Subject: Re: [storm] Consolidated iSCSI draft and TaskReporting
  >
  > Hi Patrick,
  >
  > I tend to agree with you, you raise a valid point.
  >
  > The problem lies in special-casing RFC 3720.  I think the right way =
to
  > address your comment is to require that an implementation's
  "NotUnderstood"
  > response should correlate to the highest protocol version that it =
has
  > offered (or would have offered) for the iSCSIProtocolLevel key in a
  > negotiation.   E.g., let's say if an iSCSI implementation has =
*offered*
a
  > value of 2 in a negotiation (even though the other side has =
negotiated
  > it down to 1), it means that the offering implementation can
  > comprehend all text keys defined through the spec with
  > iSCSIProtocolLevel=3D2.  So it must not respond to any of those text =
keys
  with a "NotUnderstood".
  >
  > So the formal verbiage would look something like the following:
  >
  > An iSCSI implementation MUST comprehend all text keys defined in =
iSCSI
  > Protocol specifications from [RFC3720] through the RFC keyed to the
  > highest protocol version that the implementation has offered (or =
would
  > have offered) for the iSCSIProtocolLevel key in a negotiation.
  > Returning a NotUnderstood response on any of these text keys =
therefore
  > MUST be considered a protocol error and handled accordingly.  For =
all
  > other "later" keys, i.e. text keys defined in specifications keyed =
to
  > higher values of iSCSIProtocolLevel, a NotUnderstood  answer =
concludes
  > the negotiation for a negotiated key whereas for a declarative key, =
a
  > NotUnderstood answer simply informs the declarer of a lack of
  comprehension by the receiver.
  >
  > Please let me know if you have comments or questions.  Thanks.
  >
  > Mallikarjun
  >
  >
  >
  >   -----Original Message-----
  >   From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On
  Behalf
  >   Of Patrick MacArthur
  >   Sent: Friday, June 17, 2011 7:38 AM
  >   To: storm@ietf.org
  >   Subject: [storm] Consolidated iSCSI draft and TaskReporting
  >
  >   Hi,
  >
  >   I wanted to ask for clarification on an item appearing in the
consolidated
  >   iSCSI draft.
  >
  >       "All keys defined in [RFC3720] MUST be supported by all
  >       compliant implementations; a NotUnderstood answer on any of =
the
  >       [RFC3720] keys therefore MUST be considered a protocol error =
and
  >       handled accordingly. For all other later keys, a NotUnderstood
  >       answer concludes the negotiation for a negotiated key whereas =
for
  >       a declarative key, a NotUnderstood answer simply informs the
  >       declarer of a lack of comprehension by the receiver."
  >
  >   I would like to request that the references to RFC3720 above be
changed
  to
  >   RFC3720 and RFC5048.  The comments on this list seem to indicate =
that
  the
  >   baseline for the iSCSI work is on the combination of RFC 3720 and =
RFC
  >   5048, which I interpret to mean that it would be applicable only =
to
  >   implementations compliant with both RFCs.  An implementation that
  >   claims to be RFC 5048-compliant and responds with
  >   TaskReporting=3DNotUnderstood is broken IMHO. In that case, is =
there a
  >   reason that we allow a NotUnderstood response on an RFC 5048 key
  >   (TaskReporting)?
  >
  >   If the STORM WG is trying to ensure compatibility with the
requirements
  of
  >   RFC 3720 and RFC 5048 combined, it seems strange to be allowing
  >   implementers not to implement the key requirement of RFC 5048.
  >
  >   Thanks,
  >
  >   --
  >   Patrick MacArthur
  >   Research and Development
  >   iSCSI/OFA/IPv6 Consortia
  >   UNH InterOperability Laboratory
  >
  >   _______________________________________________
  >   storm mailing list
  >   storm@ietf.org
  >   https://www.ietf.org/mailman/listinfo/storm
  >
  > _______________________________________________
  > storm mailing list
  > storm@ietf.org
  > https://www.ietf.org/mailman/listinfo/storm



From internet-drafts@ietf.org  Thu Jun 30 15:09:14 2011
Return-Path: <internet-drafts@ietf.org>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AEF3511E81F2; Thu, 30 Jun 2011 15:09:14 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -102.586
X-Spam-Level: 
X-Spam-Status: No, score=-102.586 tagged_above=-999 required=5 tests=[AWL=0.013, BAYES_00=-2.599, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XlsdJOGubSUz; Thu, 30 Jun 2011 15:09:14 -0700 (PDT)
Received: from ietfa.amsl.com (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 14F1111E81AC; Thu, 30 Jun 2011 15:09:14 -0700 (PDT)
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
From: internet-drafts@ietf.org
To: i-d-announce@ietf.org
X-Test-IDTracker: no
X-IETF-IDTracker: 3.55
Message-ID: <20110630220914.2457.82443.idtracker@ietfa.amsl.com>
Date: Thu, 30 Jun 2011 15:09:14 -0700
Cc: storm@ietf.org
Subject: [storm] I-D Action: draft-ietf-storm-iser-02.txt
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 30 Jun 2011 22:09:14 -0000

A New Internet-Draft is available from the on-line Internet-Drafts director=
ies. This draft is a work item of the STORage Maintenance Working Group of =
the IETF.

	Title           : iSCSI Extensions for RDMA Specification
	Author(s)       : Michael Ko
                          Alexander Nezhinsky
	Filename        : draft-ietf-storm-iser-02.txt
	Pages           : 97
	Date            : 2011-06-30

   iSCSI Extensions for RDMA provides the RDMA data transfer capability
   to iSCSI by layering iSCSI on top of an RDMA-Capable Protocol.  An
   RDMA-Capable Protocol provides RDMA Read and Write services, which
   enable data to be transferred directly into SCSI I/O Buffers without
   intermediate data copies.  This document describes the extensions to
   the iSCSI protocol to support RDMA services as provided by an RDMA-
   Capable Protocol.

   This document obsoletes RFC5046
   Table of Contents


A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-storm-iser-02.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

This Internet-Draft can be retrieved at:
ftp://ftp.ietf.org/internet-drafts/draft-ietf-storm-iser-02.txt

From Michael@huaweisymantec.com  Thu Jun 30 15:30:41 2011
Return-Path: <Michael@huaweisymantec.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0EAF011E8101 for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 15:30:41 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.598
X-Spam-Level: 
X-Spam-Status: No, score=-2.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id t8ma+WsY4sOk for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 15:30:40 -0700 (PDT)
Received: from mta2.huaweisymantec.com (mta2.huaweisymantec.com [218.17.155.15]) by ietfa.amsl.com (Postfix) with ESMTP id 2356D11E81A3 for <storm@ietf.org>; Thu, 30 Jun 2011 15:30:40 -0700 (PDT)
MIME-version: 1.0
Content-type: multipart/alternative; boundary="Boundary_(ID_Xkxnu71Pn+RJc4hwSof0BA)"
Received: from hstml02-in.huaweisymantec.com ([172.26.3.42]) by hstga02-in.huaweisymantec.com (Sun Java(tm) System Messaging Server 6.3-8.03 (built Apr 24 2009; 32bit)) with ESMTP id <0LNM005BOJV125C0@hstga02-in.huaweisymantec.com> for storm@ietf.org; Fri, 01 Jul 2011 06:30:38 +0800 (CST)
Received: from m90003900a ([69.199.248.19]) by hstml02-in.huaweisymantec.com (Sun Java(tm) System Messaging Server 6.3-8.03 (built Apr 24 2009; 32bit)) with ESMTPA id <0LNM00JASJUXAK10@hstml02-in.huaweisymantec.com> for storm@ietf.org; Fri, 01 Jul 2011 06:30:37 +0800 (CST)
Message-id: <9DCF23FF9A634AD59F511FB4F02CF2A2@china.huawei.com>
From: Michael Ko <Michael@huaweisymantec.com>
To: STORM <storm@ietf.org>
Date: Thu, 30 Jun 2011 15:30:32 -0700
X-Priority: 3
X-MSMail-priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.5931
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2900.6109
Subject: [storm] Fw:  I-D Action: draft-ietf-storm-iser-02.txt
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 30 Jun 2011 22:30:41 -0000

This is a multi-part message in MIME format.

--Boundary_(ID_Xkxnu71Pn+RJc4hwSof0BA)
Content-type: text/plain; charset=iso-8859-1
Content-transfer-encoding: 7BIT

I have uploaded the -02 version of the iSER draft since the -01 version has already expired.  The changes in this version from RFC 5046 are summarized in section 14.  It includes all the changes in the -01 version plus comments 5-8 from Alex Nezhinsky's note.  Comments 1-2 related to Virtual Address, and comments 3-4 related to iWARP-only statements have not been incorporated yet.

Mike
----- Original Message ----- 
From: internet-drafts@ietf.org 
To: i-d-announce@ietf.org 
Cc: storm@ietf.org 
Sent: Thursday, June 30, 2011 3:09 PM
Subject: [storm] I-D Action: draft-ietf-storm-iser-02.txt


A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the STORage Maintenance Working Group of the IETF.

Title           : iSCSI Extensions for RDMA Specification
Author(s)       : Michael Ko
                          Alexander Nezhinsky
Filename        : draft-ietf-storm-iser-02.txt
Pages           : 97
Date            : 2011-06-30

   iSCSI Extensions for RDMA provides the RDMA data transfer capability
   to iSCSI by layering iSCSI on top of an RDMA-Capable Protocol.  An
   RDMA-Capable Protocol provides RDMA Read and Write services, which
   enable data to be transferred directly into SCSI I/O Buffers without
   intermediate data copies.  This document describes the extensions to
   the iSCSI protocol to support RDMA services as provided by an RDMA-
   Capable Protocol.

   This document obsoletes RFC5046
   Table of Contents


A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-storm-iser-02.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

This Internet-Draft can be retrieved at:
ftp://ftp.ietf.org/internet-drafts/draft-ietf-storm-iser-02.txt
_______________________________________________
storm mailing list
storm@ietf.org
https://www.ietf.org/mailman/listinfo/storm

--Boundary_(ID_Xkxnu71Pn+RJc4hwSof0BA)
Content-type: text/html; charset=iso-8859-1
Content-transfer-encoding: 7BIT

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=iso-8859-1" http-equiv=Content-Type>
<META name=GENERATOR content="MSHTML 8.00.6001.19088">
<STYLE></STYLE>
</HEAD>
<BODY bgColor=#ffffff>
<DIV><FONT size=2>
<DIV><FONT size=2>I&nbsp;have uploaded the -02 version of the iSER draft since 
the -01 version has already expired.&nbsp; The changes in this version from RFC 
5046 are summarized in section 14.&nbsp; It includes all the changes in the -01 
version plus comments 5-8 from Alex Nezhinsky's note.&nbsp; Comments 1-2 related 
to Virtual Address, and comments 3-4 related to iWARP-only statements have not 
been incorporated yet.</FONT></DIV>
<DIV><FONT size=2></FONT>&nbsp;</DIV>
<DIV><FONT size=2>Mike</FONT></DIV></FONT></DIV>
<DIV style="FONT: 10pt arial">----- Original Message ----- 
<DIV style="BACKGROUND: #e4e4e4; font-color: black"><B>From:</B> <A 
title=internet-drafts@ietf.org 
href="mailto:internet-drafts@ietf.org">internet-drafts@ietf.org</A> </DIV>
<DIV><B>To:</B> <A title=i-d-announce@ietf.org 
href="mailto:i-d-announce@ietf.org">i-d-announce@ietf.org</A> </DIV>
<DIV><B>Cc:</B> <A title=storm@ietf.org 
href="mailto:storm@ietf.org">storm@ietf.org</A> </DIV>
<DIV><B>Sent:</B> Thursday, June 30, 2011 3:09 PM</DIV>
<DIV><B>Subject:</B> [storm] I-D Action: 
draft-ietf-storm-iser-02.txt</DIV></DIV>
<DIV><BR></DIV>A New Internet-Draft is available from the on-line 
Internet-Drafts directories. This draft is a work item of the STORage 
Maintenance Working Group of the 
IETF.<BR><BR>Title&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 
iSCSI Extensions for RDMA 
Specification<BR>Author(s)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Michael 
Ko<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Alexander Nezhinsky<BR>Filename&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 
draft-ietf-storm-iser-02.txt<BR>Pages&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
: 97<BR>Date&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 
2011-06-30<BR><BR>&nbsp;&nbsp; iSCSI Extensions for RDMA provides the RDMA data 
transfer capability<BR>&nbsp;&nbsp; to iSCSI by layering iSCSI on top of an 
RDMA-Capable Protocol.&nbsp; An<BR>&nbsp;&nbsp; RDMA-Capable Protocol provides 
RDMA Read and Write services, which<BR>&nbsp;&nbsp; enable data to be 
transferred directly into SCSI I/O Buffers without<BR>&nbsp;&nbsp; intermediate 
data copies.&nbsp; This document describes the extensions to<BR>&nbsp;&nbsp; the 
iSCSI protocol to support RDMA services as provided by an RDMA-<BR>&nbsp;&nbsp; 
Capable Protocol.<BR><BR>&nbsp;&nbsp; This document obsoletes 
RFC5046<BR>&nbsp;&nbsp; Table of Contents<BR><BR><BR>A URL for this 
Internet-Draft is:<BR><A 
href="http://www.ietf.org/internet-drafts/draft-ietf-storm-iser-02.txt">http://www.ietf.org/internet-drafts/draft-ietf-storm-iser-02.txt</A><BR><BR>Internet-Drafts 
are also available by anonymous FTP at:<BR><A 
href="ftp://ftp.ietf.org/internet-drafts/">ftp://ftp.ietf.org/internet-drafts/</A><BR><BR>This 
Internet-Draft can be retrieved at:<BR><A 
href="ftp://ftp.ietf.org/internet-drafts/draft-ietf-storm-iser-02.txt">ftp://ftp.ietf.org/internet-drafts/draft-ietf-storm-iser-02.txt</A><BR>_______________________________________________<BR>storm 
mailing list<BR><A href="mailto:storm@ietf.org">storm@ietf.org</A><BR><A 
href="https://www.ietf.org/mailman/listinfo/storm">https://www.ietf.org/mailman/listinfo/storm</A><BR></BODY></HTML>

--Boundary_(ID_Xkxnu71Pn+RJc4hwSof0BA)--

From david.black@emc.com  Thu Jun 30 16:15:43 2011
Return-Path: <david.black@emc.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9889411E80F8 for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 16:15:43 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -106.599
X-Spam-Level: 
X-Spam-Status: No, score=-106.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yOAeN4YN6s-v for <storm@ietfa.amsl.com>; Thu, 30 Jun 2011 16:15:42 -0700 (PDT)
Received: from mexforward.lss.emc.com (mexforward.lss.emc.com [128.222.32.20]) by ietfa.amsl.com (Postfix) with ESMTP id 19A0221F87F6 for <storm@ietf.org>; Thu, 30 Jun 2011 16:15:38 -0700 (PDT)
Received: from hop04-l1d11-si01.isus.emc.com (HOP04-L1D11-SI01.isus.emc.com [10.254.111.54]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id p5UNFZri024048 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 30 Jun 2011 19:15:37 -0400
Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.221.145]) by hop04-l1d11-si01.isus.emc.com (RSA Interceptor); Thu, 30 Jun 2011 19:15:17 -0400
Received: from mxhub09.corp.emc.com (mxhub09.corp.emc.com [10.254.92.104]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id p5UNF6D2024532; Thu, 30 Jun 2011 19:15:07 -0400
Received: from mx14a.corp.emc.com ([169.254.1.245]) by mxhub09.corp.emc.com ([10.254.92.104]) with mapi; Thu, 30 Jun 2011 19:15:06 -0400
From: <david.black@emc.com>
To: <cbm@chadalapaka.com>, <storm@ietf.org>
Date: Thu, 30 Jun 2011 19:15:04 -0400
Thread-Topic: [storm] Consolidated iSCSI draft and TaskReporting
Thread-Index: AQDeOsqbPwyi4a8ddBcbQxUWgjUBFQKxRZHdAgOPlpyWjMTOgIAAEfxg
Message-ID: <7C4DFCE962635144B8FAE8CA11D0BF1E05893103BF@MX14A.corp.emc.com>
References: <1308321497.3882.10.camel@chiron.ofa> <SNT131-ds212BA8CFEE311B42259821A06C0@phx.gbl> <7C4DFCE962635144B8FAE8CA11D0BF1E0589310372@MX14A.corp.emc.com> <SNT131-ds3915291A564FA2E77163FA0580@phx.gbl>
In-Reply-To: <SNT131-ds3915291A564FA2E77163FA0580@phx.gbl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-EMM-MHVC: 1
Subject: Re: [storm] Consolidated iSCSI draft and TaskReporting
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 30 Jun 2011 23:15:43 -0000

In running this down, I found something else.   Neither the consolidated dr=
aft nor the SAM draft instructs IANA to add the iSCSIProtocolLevel key to t=
he key registry - the SAM draft needs to do that (it currently does somethi=
ng close, but not quite right - see next paragraph).

In addition, the SAM draft needs to create a new "iSCSI Protocol Level" IAN=
A registry for values of the iSCSIProtocolLevel key (it currently attempts =
to add the iSCSIProtocolLevel key and the value "2" to the key registry, wh=
ich is not what should be done).  The SAM draft also needs to define the va=
lues "0" (no version claimed) and "2" for that registry.  Then the claiming=
 approach will work - each draft states the requirements for understanding =
text keys based on its own protocol level.=20

Thanks,
--David

> -----Original Message-----
> From: Mallikarjun Chadalapaka [mailto:cbm@chadalapaka.com]
> Sent: Thursday, June 30, 2011 6:07 PM
> To: Black, David; pmacarth@iol.unh.edu; storm@ietf.org
> Subject: RE: [storm] Consolidated iSCSI draft and TaskReporting
>=20
> I'm OK with leaving out the phrase you point out - I meant it to cover th=
e
> case of an acceptor in a key negotiation.  But I see your point.
>=20
> I was thinking along the lines of each spec "claiming" a number for the
> iSCSIProtocolLevel key, so no one spec necessarily has a table of all the
> keys.  Currently, Consolidated draft claims  "1", and SAM draft claims "2=
".
> Sounds like you're thinking a table with currently claimed
> iSCSIProtocolLevel values in this spec?  I am fine with that too.
>=20
> Thanks.
>=20
> Mallikarjun
>=20
>=20
>   -----Original Message-----
>   From: david.black@emc.com [mailto:david.black@emc.com]
>   Sent: Thursday, June 30, 2011 1:31 PM
>   To: cbm@chadalapaka.com; pmacarth@iol.unh.edu; storm@ietf.org
>   Subject: RE: [storm] Consolidated iSCSI draft and TaskReporting
>=20
>   <WG chair hat on>
>=20
>   We should do what Mallikarjun proposes.  A table will be needed that li=
sts
>   the RFCs for each value of the iSCSIProtocolLevel key.
>=20
>   One change - "(or would have offered)" should be removed from
>   Mallikarjun's proposed text.  Hypothetical requirements tend to be
> difficult
>   to test and moreover are usually pointless - the values that are actual=
ly
>   used in negotiation are what should govern protocol behavior.
>=20
>   Thanks,
>   --David
>   ----------------------------------------------------
>   David L. Black, Distinguished Engineer
>   EMC Corporation, 176 South St., Hopkinton, MA=A0 01748
>   +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293=
-7786
>   david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754
>   ----------------------------------------------------
>=20
>   > -----Original Message-----
>   > From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On
>   Behalf
>   > Of Mallikarjun Chadalapaka
>   > Sent: Friday, June 17, 2011 8:24 PM
>   > To: 'Patrick MacArthur'; storm@ietf.org
>   > Subject: Re: [storm] Consolidated iSCSI draft and TaskReporting
>   >
>   > Hi Patrick,
>   >
>   > I tend to agree with you, you raise a valid point.
>   >
>   > The problem lies in special-casing RFC 3720.  I think the right way t=
o
>   > address your comment is to require that an implementation's
>   "NotUnderstood"
>   > response should correlate to the highest protocol version that it has
>   > offered (or would have offered) for the iSCSIProtocolLevel key in a
>   > negotiation.   E.g., let's say if an iSCSI implementation has *offere=
d*
> a
>   > value of 2 in a negotiation (even though the other side has negotiate=
d
>   > it down to 1), it means that the offering implementation can
>   > comprehend all text keys defined through the spec with
>   > iSCSIProtocolLevel=3D2.  So it must not respond to any of those text =
keys
>   with a "NotUnderstood".
>   >
>   > So the formal verbiage would look something like the following:
>   >
>   > An iSCSI implementation MUST comprehend all text keys defined in iSCS=
I
>   > Protocol specifications from [RFC3720] through the RFC keyed to the
>   > highest protocol version that the implementation has offered (or woul=
d
>   > have offered) for the iSCSIProtocolLevel key in a negotiation.
>   > Returning a NotUnderstood response on any of these text keys therefor=
e
>   > MUST be considered a protocol error and handled accordingly.  For all
>   > other "later" keys, i.e. text keys defined in specifications keyed to
>   > higher values of iSCSIProtocolLevel, a NotUnderstood  answer conclude=
s
>   > the negotiation for a negotiated key whereas for a declarative key, a
>   > NotUnderstood answer simply informs the declarer of a lack of
>   comprehension by the receiver.
>   >
>   > Please let me know if you have comments or questions.  Thanks.
>   >
>   > Mallikarjun
>   >
>   >
>   >
>   >   -----Original Message-----
>   >   From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On
>   Behalf
>   >   Of Patrick MacArthur
>   >   Sent: Friday, June 17, 2011 7:38 AM
>   >   To: storm@ietf.org
>   >   Subject: [storm] Consolidated iSCSI draft and TaskReporting
>   >
>   >   Hi,
>   >
>   >   I wanted to ask for clarification on an item appearing in the
> consolidated
>   >   iSCSI draft.
>   >
>   >       "All keys defined in [RFC3720] MUST be supported by all
>   >       compliant implementations; a NotUnderstood answer on any of the
>   >       [RFC3720] keys therefore MUST be considered a protocol error an=
d
>   >       handled accordingly. For all other later keys, a NotUnderstood
>   >       answer concludes the negotiation for a negotiated key whereas f=
or
>   >       a declarative key, a NotUnderstood answer simply informs the
>   >       declarer of a lack of comprehension by the receiver."
>   >
>   >   I would like to request that the references to RFC3720 above be
> changed
>   to
>   >   RFC3720 and RFC5048.  The comments on this list seem to indicate th=
at
>   the
>   >   baseline for the iSCSI work is on the combination of RFC 3720 and R=
FC
>   >   5048, which I interpret to mean that it would be applicable only to
>   >   implementations compliant with both RFCs.  An implementation that
>   >   claims to be RFC 5048-compliant and responds with
>   >   TaskReporting=3DNotUnderstood is broken IMHO. In that case, is ther=
e a
>   >   reason that we allow a NotUnderstood response on an RFC 5048 key
>   >   (TaskReporting)?
>   >
>   >   If the STORM WG is trying to ensure compatibility with the
> requirements
>   of
>   >   RFC 3720 and RFC 5048 combined, it seems strange to be allowing
>   >   implementers not to implement the key requirement of RFC 5048.
>   >
>   >   Thanks,
>   >
>   >   --
>   >   Patrick MacArthur
>   >   Research and Development
>   >   iSCSI/OFA/IPv6 Consortia
>   >   UNH InterOperability Laboratory
>   >
>   >   _______________________________________________
>   >   storm mailing list
>   >   storm@ietf.org
>   >   https://www.ietf.org/mailman/listinfo/storm
>   >
>   > _______________________________________________
>   > storm mailing list
>   > storm@ietf.org
>   > https://www.ietf.org/mailman/listinfo/storm
>=20
>=20