From owner-ips@ece.cmu.edu Thu Mar 1 03:43:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA28395 for ; Thu, 1 Mar 2001 03:42:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f216J4M22161 for ips-outgoing; Thu, 1 Mar 2001 01:19:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f216IA122080 for ; Thu, 1 Mar 2001 01:18:10 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id BAA50780 for ; Thu, 1 Mar 2001 01:11:01 -0500 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id XAA75184 for ; Wed, 28 Feb 2001 23:18:08 -0700 Importance: Normal Subject: iscsi boot draft - 02 To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.5 September 22, 2000 Message-ID: From: "Prasenjit Sarkar" Date: Wed, 28 Feb 2001 22:17:56 -0800 X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.6 |December 14, 2000) at 02/28/2001 10:18:07 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk The latest version is available at: http://www.haifa.il.ibm.com/satran/ips/draft-ietf-ips-iscsi-boot-02.txt Changelog: (names of people suggesting change in quotes) 1. Tighter integration with Naming and Discovery document (Mark Bakke) 2. Addition of a protocol option in DHCP string (David Robinson) 3. Tighter specification of DHCP string (Doug Otis) 4. Security - reordering (David Robinson), PXE security (Bernard Aboba), Boot Image Verification (Glen) and Denial of Service (Costa) Comments appreciated, Prasenjit Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose From owner-ips@ece.cmu.edu Thu Mar 1 07:47:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA01692 for ; Thu, 1 Mar 2001 07:47:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f21Adwg19395 for ips-outgoing; Thu, 1 Mar 2001 05:39:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f21Ad9119347 for ; Thu, 1 Mar 2001 05:39:10 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id LAA135086 for ; Thu, 1 Mar 2001 11:38:43 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA28672 for ; Thu, 1 Mar 2001 11:37:20 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A02.003A59B0 ; Thu, 1 Mar 2001 11:37:20 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 1 Mar 2001 12:38:56 +0200 Subject: Re: iSCSI.04 Comments Mime-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=Tysg4NQkL3bd8bEwNOgrDhoCY21TL2xgUVM9OUgM35fXlkV3FYVJO9gs" Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=Tysg4NQkL3bd8bEwNOgrDhoCY21TL2xgUVM9OUgM35fXlkV3FYVJO9gs Content-type: text/plain; charset=us-ascii Content-Disposition: inline Y.P., The S bit means only that the data portion contains Sense or Response Not that it contains data - the length conveys this. It is however probably a good idea to have the F bit se to one for consistency - and I will do that. Regards, Julo "Y P Cheng" on 01/03/2001 00:49:32 Please respond to "Y P Cheng" To: "'Ips@Ece. Cmu. Edu'" cc: Subject: Re: iSCSI.04 Comments I apologize if someone has already touched this topic. I found the use of F-bit in Section 2.3 for --0__=Tysg4NQkL3bd8bEwNOgrDhoCY21TL2xgUVM9OUgM35fXlkV3FYVJO9gs Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =F4immediate data= --0__=Tysg4NQkL3bd8bEwNOgrDhoCY21TL2xgUVM9OUgM35fXlkV3FYVJO9gs Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=Tysg4NQkL3bd8bEwNOgrDhoCY21TL2xgUVM9OUgM35fXlkV3FYVJO9gs Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =F6 a little confusing. Why don= --0__=Tysg4NQkL3bd8bEwNOgrDhoCY21TL2xgUVM9OUgM35fXlkV3FYVJO9gs Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=Tysg4NQkL3bd8bEwNOgrDhoCY21TL2xgUVM9OUgM35fXlkV3FYVJO9gs Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6t we reserve bit 7 and use bit 4 instead and call it I-bit. In Section 2.4, the S-bit indicating Sense-data has similar meaning, i.e. data field after the basic header. Y.P., ConnectCom Solutions. = --0__=Tysg4NQkL3bd8bEwNOgrDhoCY21TL2xgUVM9OUgM35fXlkV3FYVJO9gs-- From owner-ips@ece.cmu.edu Thu Mar 1 09:03:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id JAA04102 for ; Thu, 1 Mar 2001 09:03:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f21BO1t22568 for ips-outgoing; Thu, 1 Mar 2001 06:24:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f21BNB122497 for ; Thu, 1 Mar 2001 06:23:11 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id MAA96372 for ; Thu, 1 Mar 2001 12:22:54 +0100 From: biran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id MAA83924 for ; Thu, 1 Mar 2001 12:22:08 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A02.003E655F ; Thu, 1 Mar 2001 12:21:31 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 1 Mar 2001 13:20:41 +0200 Subject: Re: iSCSI: Use of SRP (draft -04) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Steve, What I meant was that one "AuthMethod" will be negotiated instead of "InitAuth" and "TargetAuth", so mixed methods are not possible. The one-way / mutual will be decided in the text messages that carry the process of the selected method, in a manner specific to that method. SRP is mutual when using the last server's message by which it proves the knowledge of the password verifier (assuming the verifiers are kept confidentially by the server). From RFC2945: "To finish authentication, they must prove to each other that their keys are identical... If the server receives a correct response, it issues its own proof to the client. The client will compute the expected response using its own K to verify the authenticity of the server." We can have the initiator specifies in the first SRP message (with the U) "TargetAuth=yes" or "TargetAuth=no" which determines whether the last server message should be sent, and this will be the manner specific to SRP (while in KERB5 the initiator set the krb_ap_req mutual flag). Regards, Ofer Ofer Biran Systems and Software IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 Steve Senum on 01/03/2001 01:14:31 Please respond to Steve Senum To: Ofer Biran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: Re: iSCSI: Use of SRP (draft -04) Ofer: If SRP is mutual, then I think the draft should state that with text similiar to the Kerberos method, and also state how to handle mixed SRB and Kerberos authentication (or disallow it). Also, I am not sure I agree that SRP is entirely mutual. See draft-ietf-pppext-eap-srp-00.txt for a proposal for using SRP with PPP. Regards, Steve Senum biran@il.ibm.com wrote: > > Steve, > > You are correct, we'll change the SRP message sequence similar to telnet (U > --- N,g,s -- A -- B...). > > For simultaneous authentication processes (InitAuth, TargetAuth) it seems a > problem of over flexibility. The simpler > and reasonable way would be to negotiate one authentication method > AuthMethod and leave the one way / mutual > authentication decision to the specific method selected. In KERB5 the > client decides it by setting the krb_ap_req mutual > flag, in SRP it's actually mutual. > > Regards, > Ofer > > Ofer Biran > Systems and Software > IBM Research Lab in Haifa > biran@il.ibm.com 972-4-8296253 > > Steve Senum on 02/28/2001 01:41:01 AM > > Please respond to Steve Senum > > To: ietf-ips > cc: > Subject: iSCSI: Use of SRP (draft -04) > > Julian: > > With respect to use of the SRP protocol for authentication, > I think the current draft is incomplete. The SRP spec > requires that values for the Prime Modulus value 'N' and the > Generator value 'g' be sent by the authenticating entity > as well as 's' and 'B' (or known through some other method). > Look at RFC 2944 to see how telnet handles this. > > Also, if both Initiator and Target choose to authenticate with > SRP, or if InitAuth=KERB5 and TargetAuth=srp, the same key names > will be needed by both sides at the same time, resulting in the > same key name appearing twice in the same text message. This > will make it difficult for the receiver to know which key names > goes with which authentication process, since there can be two > going on at one time. > > Regards, > Steve Senum From owner-ips@ece.cmu.edu Thu Mar 1 10:19:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA06865 for ; Thu, 1 Mar 2001 10:19:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f21DQ3I01173 for ips-outgoing; Thu, 1 Mar 2001 08:26:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f21DPJ101099 for ; Thu, 1 Mar 2001 08:25:20 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id OAA132156 for ; Thu, 1 Mar 2001 14:25:12 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id OAA114768 for ; Thu, 1 Mar 2001 14:23:53 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A02.0049968A ; Thu, 1 Mar 2001 14:23:46 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 1 Mar 2001 15:25:23 +0200 Subject: Re: Marker Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk YP, It was in the spec. I was asked to generalize it and put the specific framing in the appendix as one possible implementation. But don't be fooled by the order - things said in an appendix can be as relevant and mandatory as in the rest (look at the security and parameters chapter). Regards, Julo "Y P Cheng" on 01/03/2001 00:58:59 Please respond to "Y P Cheng" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Marker Julo, I like the idea of inserting marker for finder the next iSCSI PDU. If we insert a marker every 2K bytes, we can derive where is the next marker from the TCP sequence number. Is there any plan to put this in the Spec., instead of keeping it in an Appendix? Thanks. Y.P. Cheng, CTO, Connectcom Solutions Corp. Tel: 408-383-5704 Fax: 408-383-9612 This email may contain confidential and privileged material for the sole use of the intended recipient. Any review or distribution by others is strictly prohibited. If you are not the intended recipient please contact the sender and delete all copies. From owner-ips@ece.cmu.edu Thu Mar 1 10:20:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA06881 for ; Thu, 1 Mar 2001 10:20:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f21Dk5H02644 for ips-outgoing; Thu, 1 Mar 2001 08:46:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sungod.ctsinc.net ([216.132.202.12]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f21Djn102628; Thu, 1 Mar 2001 08:45:49 -0500 (EST) Received: from wazoo ([216.132.202.101]) by sungod.ctsinc.net (Lotus Domino Release 5.0.2c) with ESMTP id 2001030108490410:837599 ; Thu, 1 Mar 2001 08:49:04 -0500 From: "Thomas Crowe" To: , Subject: RE: max connections exceeded (was yet another login question) Date: Thu, 1 Mar 2001 08:46:05 -0500 Message-ID: MIME-Version: 1.0 X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 X-MIMETrack: Itemize by SMTP Server on sungod/CTS(Release 5.0.2c |February 2, 2000) at 03/01/2001 08:49:04 AM, Serialize by Router on sungod/CTS(Release 5.0.2c |February 2, 2000) at 03/01/2001 08:49:06 AM Content-Type: multipart/mixed; boundary="----=_NextPart_000_0034_01C0A22C.0DC68A70" Sender: owner-ips@ece.cmu.edu Precedence: bulk ------=_NextPart_000_0034_01C0A22C.0DC68A70 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit I can understand, and support, a balance of error codes vs. usefulness. But the additional resources that additional error codes would utilize is pretty small. The only time that additional error codes would burden the environment, is when an actual error condition exists. (right??) I am not suggesting implementing a plethora of informational status', only being as specific as possible in reporting the errors that occur. ____________________________________________ Thomas Crowe Technical Director, Research and Development CTS - Atlanta Office 770-664-3900 ext. 45 ____________________________________________ -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu] Sent: Wednesday, February 28, 2001 3:15 PM To: ips@ece.cmu.edu Subject: Re: max connections exceeded (was yet another login question) Mark, Keep in mind though that you have to strike a balance between being friendly to a new implementation (initiator and target) to wasting to many resources for what will be used only during debugging. Protocol errors could be detected if you ran initiator/target again a friendly partner (with detailed error logs). You don't want to burden running things with all the clutter. Regards, Julo Mark Burton on 28/02/2001 18:12:39 Please respond to Mark Burton To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: max connections exceeded (was yet another login question) Julo, In general, I would prefer to have reasonably specific error codes rather than "catch alls". What do other people think? In this particular case it is arguable that the initiator is in error rather than the target because the maximum number of connections per session has been previously negotiated and, therefore, the initiator should not try and open a new connection. I therefore suggest: 0207 'No Free Connections' (or similar) Other login error situations not particularly well matched to the existing codes are: 1 - Connection duplicates a CID value already used in the session - I suggest: (0208 'Initiator CID Error') 2 - A mandatory parameter has not been specified but the F bit is set in a Login PDU. A helpful thing for the target to do in this situation would be to return the key(s) of the missing mandatory parameters in the data segment. This could be made an optional capability. I suggest: (0209 'Mandatory parameter missing') 3 - A value is specified for a session parameter that conflicts with a previously negotiated value (negotiated by the leading connection). A Reject response could be used in this situation (preferably with a new reject reason code). One other question is: how should a target respond if it receives a Login command on a connection that has already reach full feature phase? Rejecting the command seems obvious but with what code? Regards, Mark From: julian_satran@il.ibm.com Subject: Re: yet another login question Date: Wed, 28 Feb 2001 14:53:37 +0200 > > > Mark - you don't have to apologize. > > The 0300 (2.11.4) response is supposed to cover this type of error. > > If you and others feel that we have to be more specific - speak out loudly. > > Regards, > Julo > > > Mark Burton on 27/02/2001 22:43:39 > > Please respond to Mark Burton > > To: ips@ece.cmu.edu > cc: > Subject: yet another login question > > > > > > Hello again, > > sorry to be a pest but I have another query re the login command: > > If an initiator opens a new connection and subsequently submits a > login pdu that identifies an existing session that is already > supporting the maximum number of connections, what error code should > be returned to the initiator? Perhaps "target conflict", or maybe a > new code yet to be defined like "No free connections". > > Apologies if this is covered in the spec but I have missed it. > > Thanks, > > Mark > > > > > < ------=_NextPart_000_0034_01C0A22C.0DC68A70 Content-Type: text/x-vcard; name="Thomas Crowe.vcf" Content-Disposition: attachment; filename="Thomas Crowe.vcf" Content-Transfer-Encoding: quoted-printable BEGIN:VCARD VERSION:2.1 N:Crowe;Thomas FN:Thomas Crowe ORG:CTS TITLE:Technical Director TEL;WORK;VOICE:(770) 664-3900 TEL;WORK;FAX:(770) 664-3914 ADR;WORK:;;11600 Alpharetta Highway;Roswell;GA;30076;United States of Ameri= ca LABEL;WORK;ENCODING=3DQUOTED-PRINTABLE:11600 Alpharetta Highway=3D0D=3D0ARo= swell, GA 30076=3D0D=3D0AUnited States of Americ=3D a EMAIL;PREF;INTERNET:tcrowe@ctsinc.net REV:20010228T161349Z END:VCARD ------=_NextPart_000_0034_01C0A22C.0DC68A70-- From owner-ips@ece.cmu.edu Thu Mar 1 10:22:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA06949 for ; Thu, 1 Mar 2001 10:22:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f21DM5K00871 for ips-outgoing; Thu, 1 Mar 2001 08:22:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f21DLA100799 for ; Thu, 1 Mar 2001 08:21:11 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id OAA96918 for ; Thu, 1 Mar 2001 14:21:00 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id OAA156378 for ; Thu, 1 Mar 2001 14:20:13 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A02.0049346B ; Thu, 1 Mar 2001 14:19:35 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 1 Mar 2001 15:21:11 +0200 Subject: re: draft-ieft-ips-iSCSI-04.txt Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Martin, Thanks for the careful reading. Answers in text . Regards, Julo "Martin, Nick" on 01/03/2001 01:01:30 Please respond to "Martin, Nick" To: "'Julian_Satran@vnet.ibm.com'" cc: Subject: re: draft-ieft-ips-iSCSI-04.txt Hello Julian, I have been attempting a demonstration implementation of iSCSI in C on Linux. I started with draft 03 and have now moved to draft 04. My program is not yet very mature and has many required aspects not implemented, but using draft 03 I was able to perform some SCSI operations. +++ excellenr! +++ 05 will be here in a day or two +++ During my study of draft 04, I found what appear to be several minor errors or inconsistencies. I am happy to provide you with the following list, although it may not contain anything you do not already know about. On page 44 the field Status or Rsvd, and the S bit. It is my understanding that it is not valid to return SCSI status other than GOOD (zero) in a SCSI read data PDU. Otherwise sense data bust be sent using SCSI response PDU. If this is the case, then the S bit is the only significant field. If it is set the status must be zero, if it is not set then the status is reserved (0). +++ this is a bit murky but we (as FCP before) chose to consider over/underuns as not necessarily errors. SCSI will have to decide if its is or not based on the specific command context (e.g., it may break linked commands etc.) but not iSCSI +++ On page 46, the description of b7 is P bit, but this has been changed to F bit. +++ fixed +++ On page 54, the Op_code is 0x83 but should be 0x43. I am not clear whether the X bit should have been defined. +++ fixed +++ On page 60, the Op_code field is 0x80, but should be 0x40. No X bit is defined. On page 64, the Op_code field is 0x86, but should be 0x46. No X bit. On page 67, Op_code 0x90 should be 0x50. No X bit. On page 69, Op_code 0x91 should be 0x51. No X bit. On page 72, Op_code 0xef should be 0x6f. There is a 0 in the X bit position. +++ all fixed +++ On page 73, it states that DataPDULength and FirstBurstSize are in units of 512 bytes, however on page 110 and 111, these are stated to be in units of 4096 bytes. On page 112 there are references to DataPDULength*512 which may be related to the units of DataPDULength. +++ fixed - to 512 to be aligned with other T10 standards +++ On page 95, the polynomial for crc-31Q has some repeated terms (x**7 and x**5). For now I will presume this is intentional. I further presume the notation x**5 means x raised to the 5th power. ++++ Thanks it was a typo +++ I hope that I may send questions if I find portions of the draft which I do not understand. At the moment I am able to make sufficient progress. I would be interested in corresponding with other persons working on (or having interest in) demonstration or reference implementations of iSCSI. +++ write to the list and we all hope to get soon to a interoperability session +++ Thanks, Nick ------------------------------------------------------------------ Nick Martin M150801 Rm158A52 Systems Engineer Server Storage Products Compaq Computer Corporation P.O. Box 692000 20555 State Highway 249 Houston, TX 77269-2000 email: Nick.Martin@compaq.com voice: (281)514-2793 pager: (713)762-7153 fax: (281)514-5270 From owner-ips@ece.cmu.edu Thu Mar 1 16:16:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA23517 for ; Thu, 1 Mar 2001 16:16:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f21J2qs28066 for ips-outgoing; Thu, 1 Mar 2001 14:02:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f21J1K127949 for ; Thu, 1 Mar 2001 14:01:20 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel1.hp.com (Postfix) with ESMTP id D6BA4A60 for ; Thu, 1 Mar 2001 14:01:18 -0500 (EST) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id LAA10717 for ips@ece.cmu.edu; Thu, 1 Mar 2001 11:02:14 -0800 (PST) Message-Id: <200103011902.LAA10717@core.rose.hp.com> Subject: iSCSI: draft04 questions To: ips@ece.cmu.edu Date: Thu, 01 Mar 2001 11:02:14 PST Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, Thanks for incorporating a lot of the earlier reflector feedback into the latest draft. Some questions: 1. Section 1.2.5 states : "Unsolicited data MUST be sent on every connection in the same order in which commands were sent. If the amount of data exceeds the amount allowed for unsolicited write data, the specific connection MUST be stalled - i.e., no more unsolicited data will be sent on this connection until the specific command has finished sending all its data and has received a response." Sorry if there was a discussion on this already. Why not the iSCSI response of "Unsolicited data rejected" usable for this case? Initiator appears to be behaving erratically, and I don't see how it is expected to adhere to the stipulation that it shall not send anymore unsolicited data till this command is completed. 2. I didn't find a way for an iSCSI target to say that it does not support any unsolicited data at all. SPC-2 specifies that a zero FirstBurstSize means unlimited. 3. Section 2.14 states: "If an initiator intends to start recovery for a failing connection it MUST use the either the Logout command to "clean-up" the target end of a failing connection and enable recovery to start, or use the restart option of the Login command to the same effect." This seems to be contradicting section 2.10.1, where a prior logout of the CID is stated as a requirement for using the restart bit of the Login PDU (which I agree with). 4. Somehow, "Asynchronous Message" seems at odds with the rest of the usage in the draft in regards to PDUs (since a message is introduced as PDU in section 1.2). Should we may be just call it an AEN PDU? Section 2.14.3 calls this so. (I realize that it may not always result in a SCSI-level AER.) 5. In the same section 2.18, I dont' quite see the operational difference for an initiator, if any, between iSCSI event codes 2 & 3. Could you please comment? Also, isn't the time in seconds the maximum time than the minimum time as currently stated? 6. In section 2.20.1, reject reason code 5 is stated as a "command restart reject". Seems like the usage of "restart" in the draft elsewhere is for the connection/session login, and "retry" is for commands - except this one. Comment if I am reading too much into the usage. 7. Section 6.2 on digest errors states: "If the error is a Data-Digest-Error in a Data-PDU, the target MUST either request retransmission with a R2T or answer with a Reject iSCSI PDU and abort the task." Were you meaning the target's internal termination of the task? That could cause problems on a subsequent Data PDU reception from the initiator since there may be no state for the task in the target. Suggest dropping "and abort the task". 8. Same section states the following: "When an initiator receives an iSCSI data PDU with a Data-Digest error, it must discard the PDU and it MUST either request the missing data PDUs through SACK or terminate the command with an error." If you meant reporting a service failure to SCSI within the initiator, seems like that could cause trouble to initiator iSCSI layer since it is likely to receive the data/status PDUs for the command shortly thereafter. Suggest dropping "or terminate the command with an error", or replace with "or abort the task". 9. End of section 6.7.1 states: "An iSCSI target MAY reject a data-SACK and terminate the command with an iSCSI error response of SACK rejected." The "SACK rejected" iSCSI response code needs to be added in section 2.4.2. Thanks. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Thu Mar 1 16:16:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA23547 for ; Thu, 1 Mar 2001 16:16:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f21JnEM02254 for ips-outgoing; Thu, 1 Mar 2001 14:49:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f21Jmp102227 for ; Thu, 1 Mar 2001 14:48:51 -0500 (EST) Received: from cisco.com (dhcp-161-44-68-173.cisco.com [161.44.68.173]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id OAA04729 for ; Thu, 1 Mar 2001 14:48:45 -0500 (EST) Message-ID: <3A9EA759.EC3E8652@cisco.com> Date: Thu, 01 Mar 2001 13:47:37 -0600 From: Steve Senum X-Mailer: Mozilla 4.73 [en]C-CCK-MCD (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI: Additional Authentication Method Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit All, I would like to see an additional authentication method added (back) to the current list of Kerberos and SRP. I would propose adding something compatible with RADIUS, in particular PPP CHAP, which does not send passwords as clear text, though it does require the authenticator to have a clear text copy. I would like to see a protocol compatible with what I believe the large part of target customer base already has. Regards, Steve Senum From owner-ips@ece.cmu.edu Thu Mar 1 16:18:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA23643 for ; Thu, 1 Mar 2001 16:18:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f21JZDQ00981 for ips-outgoing; Thu, 1 Mar 2001 14:35:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f21JYU100918 for ; Thu, 1 Mar 2001 14:34:30 -0500 (EST) Received: from cisco.com (dhcp-161-44-68-173.cisco.com [161.44.68.173]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id OAA18088 for ; Thu, 1 Mar 2001 14:34:24 -0500 (EST) Message-ID: <3A9EA3FC.1E3E2DE5@cisco.com> Date: Thu, 01 Mar 2001 13:33:16 -0600 From: Steve Senum X-Mailer: Mozilla 4.73 [en]C-CCK-MCD (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: Use of SRP (draft -04) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Ofer: > We can have the initiator specifies in the first SRP message (with > the U) "TargetAuth=yes" or "TargetAuth=no" which determines whether > the last server message should be sent, and this will be the manner > specific to SRP (while in KERB5 the initiator set the krb_ap_req mutual > flag). > > Regards, > Ofer As long as the spec. makes this clear, fine with me. The present draft (-04) does not. Regards, Steve Senum From owner-ips@ece.cmu.edu Thu Mar 1 16:18:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA23654 for ; Thu, 1 Mar 2001 16:18:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f21JbEK01206 for ips-outgoing; Thu, 1 Mar 2001 14:37:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f21Jav101163 for ; Thu, 1 Mar 2001 14:36:57 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 1 Mar 2001 14:36:48 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07041015D9@corpmx9.isus.emc.com> From: Black_David@emc.com To: cbm@rose.hp.com, ips@ece.cmu.edu Subject: RE: Async Message PDU Date: Thu, 1 Mar 2001 14:36:46 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian will doubtless pick up the rest of these, but I thought I'd take this issue, since it was part of what we did in Orlando. > 4. Somehow, "Asynchronous Message" seems at odds with the rest of the > usage in the draft in regards to PDUs (since a message is introduced as > PDU in section 1.2). Should we may be just call it an AEN PDU? Section > 2.14.3 calls this so. (I realize that it may not always result in a > SCSI-level AER.) One of the things we did in Orlando was to change some of the terminology with the goal of each term and acronym being unambiguously owned by a single layer in the protocol stack, in particular making a sharp distinction between SCSI concepts and iSCSI mechanisms. All the *RN entities changed to *SN for this reason (SCSI has a CmdRN, so all *RN entities are SCSI, and all *SN entities are iSCSI). Similarly, all AE* entities are SCSI, and we went with "Asynchronous Message <*>" to name the iSCSI entities. This matters here because there are circumstances in which iSCSI will send Asynchronous Message PDUs for its own use even when SCSI has disabled AER. The usage of "AEN PDU" in 2.14.3 is probably an editing oversight. Suggestions for words to use instead of "Message" are welcome, but they must not start with the letter "E" ;-). Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Thu Mar 1 17:11:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA25913 for ; Thu, 1 Mar 2001 17:11:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f21KOGW05430 for ips-outgoing; Thu, 1 Mar 2001 15:24:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f21KNo105363 for ; Thu, 1 Mar 2001 15:23:50 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel1.hp.com (Postfix) with ESMTP id E00E84F3 for ; Thu, 1 Mar 2001 12:23:48 -0800 (PST) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id MAA26128 for ips@ece.cmu.edu; Thu, 1 Mar 2001 12:24:45 -0800 (PST) Message-Id: <200103012024.MAA26128@core.rose.hp.com> Subject: RE: iSCSI:Async Message PDU To: ips@ece.cmu.edu Date: Thu, 01 Mar 2001 12:24:44 PST Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk >Julian will doubtless pick up the rest of these, >but I thought I'd take this issue, since it was >part of what we did in Orlando. > >> 4. Somehow, "Asynchronous Message" seems at odds with the rest of the >> usage in the draft in regards to PDUs (since a message is introduced as >> PDU in section 1.2). Should we may be just call it an AEN PDU? Section >> 2.14.3 calls this so. (I realize that it may not always result in a >> SCSI-level AER.) > >One of the things we did in Orlando was to change >some of the terminology with the goal of each term and >acronym being unambiguously owned by a single layer >in the protocol stack, in particular making a sharp >distinction between SCSI concepts and iSCSI mechanisms. > >All the *RN entities changed to *SN for this reason >(SCSI has a CmdRN, so all *RN entities are SCSI, and >all *SN entities are iSCSI). Similarly, all AE* >entities are SCSI, and we went with "Asynchronous >Message <*>" to name the iSCSI entities. This matters >here because there are circumstances in which iSCSI >will send Asynchronous Message PDUs for its own use >even when SCSI has disabled AER. The usage of "AEN >PDU" in 2.14.3 is probably an editing oversight. >Suggestions for words to use instead of "Message" >are welcome, but they must not start with the letter >"E" ;-). I see what you're saying. My point was that "Asynchronous Message PDU" is redundant. Should we just call it "Async PDU"/ "Async Indication PDU"? BTW, doesn't a similar rule apply for SACK? Thanks. > >Thanks, >--David > -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Thu Mar 1 17:11:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA25926 for ; Thu, 1 Mar 2001 17:11:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f21KtJT08105 for ips-outgoing; Thu, 1 Mar 2001 15:55:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f21Kso108080 for ; Thu, 1 Mar 2001 15:54:50 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 4C5BA7C0 for ; Thu, 1 Mar 2001 13:54:49 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 61C47173 for ; Thu, 1 Mar 2001 15:54:48 -0500 (EST) Received: from matt5670 (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with SMTP id MAA15055 for ; Thu, 1 Mar 2001 12:54:46 -0800 (PST) From: "Matt Wakeley" To: "IPS Reflector" Subject: FW: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Date: Thu, 1 Mar 2001 12:57:11 -0800 Message-ID: <004001c0a292$2fb722f0$94ea8c9c@matt5670> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id f21KtJU08105 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ietf.org id RAA25926 FYI - We put this out quickly to beat the deadline - a better version will be submitted before the final cut-off... -Matt Wakeley Agilent Technologies -----Original Message----- From: nsyracus@cnri.reston.va.us [mailto:nsyracus@cnri.reston.va.us]On Behalf Of Internet-Drafts@ietf.org Sent: Thursday, March 01, 2001 4:21 AM To: IETF-Announce: Subject: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt A New Internet-Draft is available from the on-line Internet-Drafts directories. Title : iSCSI Digests û CRC or Checksum? Author(s) : V. Cavanna, M. Wakeley Filename : draft-cavanna-iscsi-crc-vs-cksum-00.txt Pages : 8 Date : 28-Feb-01 The iSCSI working group is currently considering the method of protecting iSCSI messages from errors. This I-D presents a proposal to use the Fletcher-32 checksum algorithm. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-cavanna-iscsi-crc-vs-cksum-00.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-cavanna-iscsi-crc-vs-cksum-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-cavanna-iscsi-crc-vs-cksum-00.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. From owner-ips@ece.cmu.edu Thu Mar 1 17:11:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA25946 for ; Thu, 1 Mar 2001 17:11:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f21KQDh05619 for ips-outgoing; Thu, 1 Mar 2001 15:26:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f21KPs105557 for ; Thu, 1 Mar 2001 15:25:55 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Thu, 1 Mar 2001 12:25:48 -0800 Message-ID: From: Joshua Tseng To: ips@ece.cmu.edu Subject: iSNS Draft Now Available Date: Thu, 1 Mar 2001 12:25:47 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, I have just submitted the new iSNS draft document. Until it becomes available at the IETF web site, it can be retrieved here: http://www.nishansystems.com/ietf/draft-ietf-ips-iSNS-01.txt Best regards, Josh Tseng From owner-ips@ece.cmu.edu Thu Mar 1 20:43:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA00992 for ; Thu, 1 Mar 2001 20:43:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f21NCI718778 for ips-outgoing; Thu, 1 Mar 2001 18:12:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f21NC5118737 for ; Thu, 1 Mar 2001 18:12:05 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f220LN032595; Thu, 1 Mar 2001 16:21:23 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Matt Wakeley" , "IPS Reflector" Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Date: Thu, 1 Mar 2001 15:10:31 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 In-Reply-To: <004001c0a292$2fb722f0$94ea8c9c@matt5670> Sender: owner-ips@ece.cmu.edu Precedence: bulk X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id f21NCI818778 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ietf.org id UAA00992 Matt, Interesting article, but a comment as usual. There is very little difference between Fletcher-32 and Adler-32 with respect to performance. Fletcher Checksum is divided into two (16 bit) pieces, a straight 1's complement sum and a higher order sum of the running sums. The modulo in Adler-32 is simply including a subtraction with a conditional branch. As Dafna pointed out, this test is normally done in a predictive manner by means of linear coding lessening this difference between Fletcher and Adler even further. Data access in a table lookup required for CRC represents an impediment whereas instruction cycles are easy hidden, so either Adler-32 or Fletcher-32 will likely see a much larger throughput for software implementations. As Adler-32 is already advocated for SCTP and does not represent anything substantially different with respect to performance to that of Fletcher-32, I would be interested to understand the differences in real-world situations. Perhaps there is already a history of this discussion within the sigtran archives. Doug > FYI - We put this out quickly to beat the deadline - a better version will > be submitted before the final cut-off... > > -Matt Wakeley > Agilent Technologies > > -----Original Message----- > From: nsyracus@cnri.reston.va.us [mailto:nsyracus@cnri.reston.va.us]On > Behalf Of Internet-Drafts@ietf.org > Sent: Thursday, March 01, 2001 4:21 AM > To: IETF-Announce: > Subject: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt > > > A New Internet-Draft is available from the on-line Internet-Drafts > directories. > > > Title : iSCSI Digests û CRC or Checksum? > Author(s) : V. Cavanna, M. Wakeley > Filename : draft-cavanna-iscsi-crc-vs-cksum-00.txt > Pages : 8 > Date : 28-Feb-01 > > The iSCSI working group is currently considering the method of > protecting iSCSI messages from errors. This I-D presents a proposal > to use the Fletcher-32 checksum algorithm. > > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-cavanna-iscsi-crc-vs-cks um-00.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-cavanna-iscsi-crc-vs-cksum-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-cavanna-iscsi-crc-vs-cksum-00.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. From owner-ips@ece.cmu.edu Thu Mar 1 22:32:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA04739 for ; Thu, 1 Mar 2001 22:31:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f221CK526394 for ips-outgoing; Thu, 1 Mar 2001 20:12:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.ivivity.com (user-vc8ftn3.biz.mindspring.com [216.135.246.227]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f221C6126379 for ; Thu, 1 Mar 2001 20:12:06 -0500 (EST) Received: by ATLOPS with Internet Mail Service (5.5.2448.0) id ; Thu, 1 Mar 2001 20:12:00 -0500 Message-ID: <25369470B6F0D41194820002B328BDD2026524@ATLOPS> From: Sanjay Goyal To: ips@ece.cmu.edu Subject: crc-32Q Date: Thu, 1 Mar 2001 20:11:55 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi the generator polynomial for the crc-32Q (32 31 24 22 16 14 8 7 5 3 1 0) as given in Appendix A is reducable, as I tried it on the internet on a CRC-calculator tool. The ethernet polynomial which is (32 26 23 22 16 12 11 10 8 7 5 4 2 1 0) is not reducable. Now as per my understanding reducable polynomial is not better than the irreducable one. If reducable one is not good, why did we choose crc-32Q polynomial instead of crc-32 polynomial(used for ethernet)? Can somebody please explain it to me. Regards and Thanks Sanjay Goyal From owner-ips@ece.cmu.edu Fri Mar 2 06:32:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA25464 for ; Fri, 2 Mar 2001 06:32:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f229WTp26007 for ips-outgoing; Fri, 2 Mar 2001 04:32:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f229WG125996 for ; Fri, 2 Mar 2001 04:32:16 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id KAA32440; Fri, 2 Mar 2001 10:32:08 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id KAA41358; Fri, 2 Mar 2001 10:30:45 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A03.00343EAC ; Fri, 2 Mar 2001 10:30:39 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Steve Senum cc: ips@ece.cmu.edu Message-ID: Date: Fri, 2 Mar 2001 09:57:17 +0200 Subject: Re: [Fwd: iSCSI: WN Field (draft -04)] Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Steve, There are days when I can either read and answer my mail or do some work. Yesterday was one of those - and I choose to work. Julo From owner-ips@ece.cmu.edu Fri Mar 2 06:34:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA25476 for ; Fri, 2 Mar 2001 06:34:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f229WYN26014 for ips-outgoing; Fri, 2 Mar 2001 04:32:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f229WN126001 for ; Fri, 2 Mar 2001 04:32:23 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA149754 for ; Fri, 2 Mar 2001 10:32:16 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id KAA32942 for ; Fri, 2 Mar 2001 10:30:55 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A03.00344394 ; Fri, 2 Mar 2001 10:30:51 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 2 Mar 2001 11:30:05 +0200 Subject: Re: iSCSI: draft04 questions Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun, Answers in text ... Thanks, Julo "Mallikarjun C." on 01/03/2001 21:02:14 Please respond to cbm@rose.hp.com To: ips@ece.cmu.edu cc: Subject: iSCSI: draft04 questions Julian, Thanks for incorporating a lot of the earlier reflector feedback into the latest draft. Some questions: 1. Section 1.2.5 states : "Unsolicited data MUST be sent on every connection in the same order in which commands were sent. If the amount of data exceeds the amount allowed for unsolicited write data, the specific connection MUST be stalled - i.e., no more unsolicited data will be sent on this connection until the specific command has finished sending all its data and has received a response." Sorry if there was a discussion on this already. Why not the iSCSI response of "Unsolicited data rejected" usable for this case? Initiator appears to be behaving erratically, and I don't see how it is expected to adhere to the stipulation that it shall not send anymore unsolicited data till this command is completed. ++++ This is a leftover from an older scheme (very old) thanks +++ 2. I didn't find a way for an iSCSI target to say that it does not support any unsolicited data at all. SPC-2 specifies that a zero FirstBurstSize means unlimited. +++ UseR2T=yes (default) and ImmediateData=no +++ 3. Section 2.14 states: "If an initiator intends to start recovery for a failing connection it MUST use the either the Logout command to "clean-up" the target end of a failing connection and enable recovery to start, or use the restart option of the Login command to the same effect." This seems to be contradicting section 2.10.1, where a prior logout of the CID is stated as a requirement for using the restart bit of the Login PDU (which I agree with). +++ It is a colapse of a logout-login for those cases in which a new connection is oppened _ and the target allows only one connection! It allows the target to do the logout function then follow with the login +++ 4. Somehow, "Asynchronous Message" seems at odds with the rest of the usage in the draft in regards to PDUs (since a message is introduced as PDU in section 1.2). Should we may be just call it an AEN PDU? Section 2.14.3 calls this so. (I realize that it may not always result in a SCSI-level AER.) +++ Fixed 2.14.3 - vox populi in Orlando not to create confusion -:) +++ 5. In the same section 2.18, I dont' quite see the operational difference for an initiator, if any, between iSCSI event codes 2 & 3. Could you please comment? +++ 2 is for the purists - message,logout,login, 3 indicates that the target will disconnectd the named connection +++ Also, isn't the time in seconds the maximum time than the minimum time as currently stated? +++ time is minimum - allows the target to do the cleanup before being hit with a login +++ 6. In section 2.20.1, reject reason code 5 is stated as a "command restart reject". Seems like the usage of "restart" in the draft elsewhere is for the connection/session login, and "retry" is for commands - except this one. Comment if I am reading too much into the usage. +++ i'll make it retry +++ 7. Section 6.2 on digest errors states: "If the error is a Data-Digest-Error in a Data-PDU, the target MUST either request retransmission with a R2T or answer with a Reject iSCSI PDU and abort the task." Were you meaning the target's internal termination of the task? That could cause problems on a subsequent Data PDU reception from the initiator since there may be no state for the task in the target. Suggest dropping "and abort the task". +++For this reason 2.4.2 delays the bad status and I have already added a similar statement in 6.2+++ 8. Same section states the following: "When an initiator receives an iSCSI data PDU with a Data-Digest error, it must discard the PDU and it MUST either request the missing data PDUs through SACK or terminate the command with an error." If you meant reporting a service failure to SCSI within the initiator, seems like that could cause trouble to initiator iSCSI layer since it is likely to receive the data/status PDUs for the command shortly thereafter. Suggest dropping "or terminate the command with an error", or replace with "or abort the task". +++ fixed +++ 9. End of section 6.7.1 states: "An iSCSI target MAY reject a data-SACK and terminate the command with an iSCSI error response of SACK rejected." The "SACK rejected" iSCSI response code needs to be added in section 2.4.2. +++ will fix +++ Thanks. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Fri Mar 2 06:35:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA25491 for ; Fri, 2 Mar 2001 06:35:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f229WWl26011 for ips-outgoing; Fri, 2 Mar 2001 04:32:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f229W1125989 for ; Fri, 2 Mar 2001 04:32:02 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA176996 for ; Fri, 2 Mar 2001 10:31:53 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id KAA85486 for ; Fri, 2 Mar 2001 10:31:05 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A03.00343C16 ; Fri, 2 Mar 2001 10:30:32 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 2 Mar 2001 09:20:39 +0200 Subject: Re: iSCSI: WN Field (draft -04) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk WN is fixed already in 5 (the Kerberos digest is a late addition). I am going to send-out 05 today! Julo Steve Senum on 28/02/2001 01:23:38 Please respond to Steve Senum To: ietf-ips cc: Subject: iSCSI: WN Field (draft -04) Julian: Some questions on the WN field in draft -04. wrt. the bits used to encode the digest types on the header and/or data segment: 1. Why are these needed since the use of the digests is negotiated? 2. What should they be set to if one of the Kerberos digests are used? When a proprietary digest is used? Perhaps they could be changed to two one bit fields to indicate whether the digest is simply present or not. Also, I would also suggest inverting the meaning of the next header type bit (bit 7). Regards, Steve Senum From owner-ips@ece.cmu.edu Fri Mar 2 08:22:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id IAA28493 for ; Fri, 2 Mar 2001 08:22:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f22BAWS12259 for ips-outgoing; Fri, 2 Mar 2001 06:10:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f22B9e112224 for ; Fri, 2 Mar 2001 06:09:41 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id MAA07684 for ; Fri, 2 Mar 2001 12:09:33 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id MAA99234 for ; Fri, 2 Mar 2001 12:08:44 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A03.003D2D35 ; Fri, 2 Mar 2001 12:08:12 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu cc: Dafna_Sheinwald@il.ibm.com Message-ID: Date: Fri, 2 Mar 2001 13:09:46 +0200 Subject: Re: crc-32Q Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Sanjay, Please have a look at http://www.haifa.il.ibm.com/satran/ips/draft-sheinwald-iSCSI-CRC-00.txt The reference (a paper by Jack Wolf from 1994) has this polynomial as the best performing between those explored and 4 orderds of magnitude better on bursts longer than 32 bits. Given the stature of the author and the fact the the polynomial is quoted in several papers we did not check that after division by x+1 the polynomial is irreducible. CCITT-CRC32 is x^32+x^31+x^4+1 (our previous choice) We are looking now at several other codes. IEEE802 hex 104C11DB7 - the one you quoted is among them but there are several more promising. Julo Sanjay Goyal on 02/03/2001 03:11:55 Please respond to Sanjay Goyal To: ips@ece.cmu.edu cc: Subject: crc-32Q Hi the generator polynomial for the crc-32Q (32 31 24 22 16 14 8 7 5 3 1 0) as given in Appendix A is reducable, as I tried it on the internet on a CRC-calculator tool. The ethernet polynomial which is (32 26 23 22 16 12 11 10 8 7 5 4 2 1 0) is not reducable. Now as per my understanding reducable polynomial is not better than the irreducable one. If reducable one is not good, why did we choose crc-32Q polynomial instead of crc-32 polynomial(used for ethernet)? Can somebody please explain it to me. Regards and Thanks Sanjay Goyal From owner-ips@ece.cmu.edu Fri Mar 2 08:22:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id IAA28515 for ; Fri, 2 Mar 2001 08:22:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f22C1X415176 for ips-outgoing; Fri, 2 Mar 2001 07:01:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f22C1Q115167 for ; Fri, 2 Mar 2001 07:01:26 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id NAA38518 for ; Fri, 2 Mar 2001 13:01:19 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id NAA32066 for ; Fri, 2 Mar 2001 13:00:30 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A03.0041D840 ; Fri, 2 Mar 2001 12:59:12 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 2 Mar 2001 14:00:49 +0200 Subject: iSCSI 05.txt is out Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, I've just submitted to the Internet-Drafts repository and to our list archive (at CMU) a new version of the draft. You can find the new document at: http://www.haifa.il.ibm.com/satran/ips/draft-ietf-ips-iSCSI-05.txt I intended to correct some last minute annoying errors and to add one item for which I did not have an acceptable answer at 04 time - a neat recovery for lost R2Ts (StatSN implies keeping things around and that is not worth for R2Ts). I've added them now to the DataSN sequencing. The text readability score was pretty low (even lower then the usual for a technical text - I don't know about standards) so that I handed the text to technical writer. The text is better (I am told - I can't judge) but I could not use the Acrobat to put out a version with changes since they where so many editorial changes. No major technical changes. - Fixes - Security negotiation can adopt one method for the two ends not two separate methods. - R2T as I mentioned Julo Julian Satran - IBM Research Laboratory at Haifa From owner-ips@ece.cmu.edu Fri Mar 2 10:02:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA01740 for ; Fri, 2 Mar 2001 10:02:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f22Dha620825 for ips-outgoing; Fri, 2 Mar 2001 08:43:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f22DgY120757 for ; Fri, 2 Mar 2001 08:42:34 -0500 (EST) Received: from purol.East.Sun.COM ([129.148.9.11]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id FAA17243; Fri, 2 Mar 2001 05:42:30 -0800 (PST) Received: from sun.com (speed [129.148.174.29]) by purol.East.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id IAA15242; Fri, 2 Mar 2001 08:42:29 -0500 (EST) Message-ID: <3A9FA341.6B532C52@sun.com> Date: Fri, 02 Mar 2001 08:42:25 -0500 From: John H Howard Organization: Sun Microsystems X-Mailer: Mozilla 4.75 [en] (X11; U; SunOS 5.8 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: Douglas Otis CC: IPS Reflector Subject: Re: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit CRC-32 is very expensive in software; it may require hardware assist in fast networks. Is iSCSI really intended to be a hardware only protocol? Fletcher-32 has a serious flaw: it does not distinguish between an input halfword of all ones (FFFFx) and an input halfword of all zeros (0000x). Both are equal to zero under ones' complement addition. [Stone, J., Greenwald, M., Partridge, C., & Hughes, J., "Performance of Checksums and CRC's over Real Data", IEEE/ACM Trans. on Networking, Vol., 6, No. 5, October 1988] gives several examples of situations in which this is important. Adler-32 avoids this problem by adding 8-bit inputs into its 16-bit sub-sums. It also uses a prime modulus (2**16-15) rather than ones' complement's 2**16-1. Using 8 bit rather than 16 bit inputs doubles the number of additions Adler-32 performs compared to Fletcher-32. A more subtle problem is that the high-order bits of the sums are not uniformly distributed until the block size becomes very large. I estimate that this causes Adler-32 to lose one or two bits worth of resolving power. Still, a 30 bit checksum is still quite strong. An alternative worth consideration is Fletcher-32 modified to use the prime modulus 2**16-15. It's fast, doesn't have the 0000=FFFF problem, detects permutations, and distributes all result bits uniformly. It does confuse an input halfword of 0 with 2**16-15 (and similarly for 1..14), but I think this only a minor problem because the potentially confused inputs are unlikely. By definition every checksum confuses many inputs. If you are going to argue from bad examples you really should estimate how likely your bad examples are. John Howard Sun Microsystems From owner-ips@ece.cmu.edu Fri Mar 2 10:03:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA01819 for ; Fri, 2 Mar 2001 10:03:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f22D4c718606 for ips-outgoing; Fri, 2 Mar 2001 08:04:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f22BqR114695 for ; Fri, 2 Mar 2001 06:52:27 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA25858; Fri, 2 Mar 2001 06:52:24 -0500 (EST) Message-Id: <200103021152.GAA25858@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-iscsi-boot-02.txt Date: Fri, 02 Mar 2001 06:52:24 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : A Standard for BootStrapping Clients using the iSCSI Protocol Author(s) : P. Sarkar, D. Missimer, C. Sapuntzakis Filename : draft-ietf-ips-iscsi-boot-02.txt Pages : 10 Date : 01-Mar-01 The Small Computer Systems Interface (SCSI) is a popular family of protocols for communicating with I/O devices, especially storage devices. iSCSI is a proposed transport protocol for SCSI that operates on top of TCP[12]. This memo describes a standard mechanism to enable clients to bootstrap themselves using the iSCSI protocol. The goal of this standard is to enable clients to obtain the information to open an iSCSI session with the iSCSI bootstrpping server, assuming this information is not available. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-boot-02.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-boot-02.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-boot-02.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20010301141413.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-iscsi-boot-02.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-iscsi-boot-02.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20010301141413.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Fri Mar 2 13:45:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA12025 for ; Fri, 2 Mar 2001 13:45:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f22Gfin03721 for ips-outgoing; Fri, 2 Mar 2001 11:41:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f22GfJ103701 for ; Fri, 2 Mar 2001 11:41:19 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2650.21) id ; Fri, 2 Mar 2001 11:32:45 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07041015E4@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Cc: Black_David@emc.com Subject: Announcements & Minn. Agenda Date: Fri, 2 Mar 2001 11:41:11 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk A couple of organizational announcements: - Steve Bellovin has found it necessary to step down as one of the IP Storage WG co-chairs due to other demands on his time. Steve will continue to consult with the WG on security matters. His contributions to the WG in the time he's served as co-chair have been valuable and appreciated at least by the ADs and your remaining co-chairs. Please join us in thanking Steve for his efforts. - I'm pleased to be able to announce that Franco Travostino of Nortel Networks (travos@nortelnetworks.com) has accepted the position of iFCP Technical Coordinator. Also, it's time (actually past time) to start assembling the agenda for the Minneapolis meetings. Please send requests for agenda time to me with copies to Elizabeth and the appropriate technical coordinator for items that relate to their protocol: iSCSI: John Hufferd FCIP: Murali Rajagopal iFCP: Franco Travostino Note that a submitted Internet-Draft is generally a pre-requisite to obtaining agenda time (i.e., agenda requests should indicate the draft involved), and that the primary purpose of time in the meetings is to discuss issues as opposed to giving technical presentations (i.e., assume that your audience has read your draft(s)). Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Mar 2 13:47:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA12115 for ; Fri, 2 Mar 2001 13:47:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f22Gw0704908 for ips-outgoing; Fri, 2 Mar 2001 11:58:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f22Gut104839 for ; Fri, 2 Mar 2001 11:56:55 -0500 (EST) Received: from hpbs4813.boi.hp.com (hpbs4813.boi.hp.com [15.56.8.57]) by atlrel2.hp.com (Postfix) with ESMTP id 7C604B89 for ; Fri, 2 Mar 2001 11:56:54 -0500 (EST) Received: from xboibrg1.boi.hp.com (hpbs4813.boi.hp.com [15.56.8.57]) by hpbs4813.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with ESMTP id JAA08882 for ; Fri, 2 Mar 2001 09:56:53 -0700 (MST) Received: by xboibrg1.boi.hp.com with Internet Mail Service (5.5.2653.19) id <1R04JJBR>; Fri, 2 Mar 2001 09:55:33 -0700 Message-ID: <6BD67FFB937FD411A04F00D0B74FE87802A08EB0@xrose06.rose.hp.com> From: "KRUEGER,MARJORIE (HP-Roseville,ex1)" To: "Ips Reflector (E-mail)" Subject: New iSCSI Requirements draft Date: Fri, 2 Mar 2001 09:55:32 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I've submitted the new iSCSI Requirements to the IETF and Julian has also posted it on his web site. Please get a head start on reviewing it. http://www.haifa.il.ibm.com/satran/ips/draft-ietf-ips-iscsi-rqmts-01.txt What's new? - First section is summary of requirements from all sections - security requirements - bandwidth requirements section discusses the connection model - all sections have been cleaned up Please review these requirements against the other documents. I believe this document is almost ready for submission as an informational RFC - review with that in mind. Marjorie Krueger Networked Storage Architecture Networked Storage Solutions Org. Hewlett-Packard tel: +1 916 785 2656 fax: +1 916 785 0391 email: marjorie_krueger@hp.com From owner-ips@ece.cmu.edu Fri Mar 2 16:42:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA20640 for ; Fri, 2 Mar 2001 16:42:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f22JPnH17185 for ips-outgoing; Fri, 2 Mar 2001 14:25:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f22JOn117117 for ; Fri, 2 Mar 2001 14:24:49 -0500 (EST) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id D0A1C35E; Fri, 2 Mar 2001 11:24:47 -0800 (PST) Received: from mk731913.cup.hp.com (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id LAA29506; Fri, 2 Mar 2001 11:28:00 -0800 (PST) Message-Id: <5.0.2.1.2.20010302102714.00a9f948@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Fri, 02 Mar 2001 11:08:12 -0800 To: John H Howard From: Michael Krause Subject: Re: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Cc: Douglas Otis , IPS Reflector In-Reply-To: <3A9FA341.6B532C52@sun.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 08:42 AM 3/2/2001 -0500, John H Howard wrote: >CRC-32 is very expensive in software; it may require hardware assist in >fast networks. Is iSCSI really intended to be a hardware only protocol? If one is attached to a high-speed network (Gbps and above), then hardware acceleration is a requirement to meet the customer performance objectives. If a customer wants to attach to lower-speed networks, the impacts of using a CRC instead of a checksum are relatively in the "noise" range w.r.t. system resource utilization and overall performance. As such, many of us favor using CRC-32 implementations - please see the e-mail proposal I sent out a month ago on how even a CRC-32 could be implemented to accelerate a software implementations while not penalizing hardware implementations. Mike From owner-ips@ece.cmu.edu Fri Mar 2 16:42:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA20671 for ; Fri, 2 Mar 2001 16:42:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f22JIlL16761 for ips-outgoing; Fri, 2 Mar 2001 14:18:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from crufty.research.bell-labs.com (crufty.research.bell-labs.com [204.178.16.49]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f22JHo116711 for ; Fri, 2 Mar 2001 14:17:50 -0500 (EST) Received: from grubby.research.bell-labs.com ([135.104.2.9]) by crufty; Fri Mar 2 14:14:06 EST 2001 Received: from aura.research.bell-labs.com ([135.104.46.10]) by grubby; Fri Mar 2 14:15:53 EST 2001 Received: from research.bell-labs.com (IDENT:sandeepj@sandeepj-pcmh.research.bell-labs.com [135.104.47.90]) by aura.research.bell-labs.com (8.9.1/8.9.1) with ESMTP id OAA03375 for ; Fri, 2 Mar 2001 14:15:53 -0500 (EST) Message-ID: <3A9FF169.7CE5A21@research.bell-labs.com> Date: Fri, 02 Mar 2001 14:15:53 -0500 From: Sandeep Joshi X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.2.16-3 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI 05.txt is out References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi Julian, Draft looks good. Here are a few issues i noticed. Some of these issues may have been valid with 04 too. -Sandeep Sec 2.7: Scsi Data for READ --------------------------- This is a read PDU but the fields for (O|U) do not match the corresponding positions for Read status indicators in the Scsi Response PDU. In the Scsi Response PDU, it is lowercase "o" and "u" which indicate read overflow/underflow. They are present in a different bit positions. Notice that the residual count position is consistent. So how about matching up the over/underflow bits to same positions with same lowercase syntax ? This may also change the "S" bit position. R2TDataSN ---------- Sec 6.7.1 has some new content on how to handle lost R2Ts using SACKs. But I noticed that the SACK request (Sec 2.16) has not changed to indicate whether the DataSN is a R2T DataSN or just a Read PDU DataSN (D bit) So do we demux on the read/write operation type? And how does this affect PDU loss in bidirectional commands ? Sec 2.14.1 ---------- CID: This field is valid only if the reason code is not "close connection" -- you mean "close session" right ? To further reinforce this, can we also add "CID or reserved(0)" to the PDU frame above - just as we do in other cases. Sec 2.18 Async PDU -------------------- Could you elaborate on the scenarios proposed under which a target might request a logout (event code=2)? I recall this code was added recently (in 03 or 04) but cant recall the exact semantics. For example, can this code be used if some connection does not respond to pings ? From Sec 2.14, I see that this will result in a Logout Command with reason_code=2 (target-requested logout). There is also some discussion in Sec 6.7.1.2 on this. This facility gives rise to race conditions on duplex & buffered channels - plus the Async PDU intends to preserve the statSN ordering. [Async is a misnomer :-)] Say both initiator & target decide to close a non-responding TCP connection_1. The initiator sends a Login (for conn recovery) on some other connection_2. At the same "time", the target sends an Async PDU for c_1 on connection_3. By the time Async PDU is delivered, the initiator may well have recovered the connection so its going to logout a connection which is chugging along hunky-dory. In the absence of any synchronized timestamps, it is not possible to know if one is responding to an event which has already been handled. Sec 2.10 : Version number ---------------------------- The min & max versions are just plain numbers. Any plans to support major+minor version numbers ? From owner-ips@ece.cmu.edu Fri Mar 2 16:45:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA20870 for ; Fri, 2 Mar 2001 16:45:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f22JO5u17067 for ips-outgoing; Fri, 2 Mar 2001 14:24:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com (42s3082.isus.emc.com [168.159.211.82] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f22JNI117025 for ; Fri, 2 Mar 2001 14:23:18 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 2 Mar 2001 14:24:32 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07041015E7@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: iSCSI: Next-Qualifiers and PDU diagrams Date: Fri, 2 Mar 2001 14:23:10 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk In working with some other folks, I stumbled across a format issue that's more confusing than it needs to be in the current iSCSI draft. The issue is that Next-Qualifiers (WN, etc.) and the PDUs they go with are not documented/diagrammed together, making it hard to figure out how to parse iSCSI PDUs. For an example of how this causes confusion, consider the Text, Login and NOP PDU diagrams in Sections 2.8 through 2.13. Starting from one of these diagrams, how does one figure out the size of the Text or Ping data that's appended? The answer's not in the diagrams -- there's a 4-byte Next Qualifier preceding the PDU shown in each diagram. The Text or Ping data is considered to be a data segment, and hence the format of the Next Qualifier is that given in Section 2.2.2.3, including a 3-byte length field that answers the question. That's not an obvious answer, and there are several factors that contribute to it being potentially confusing: - None of the PDU diagrams show the Next-Qualifiers - Byte 0 means different things - at the top of Section 2.2, it's the start of the Next-Qualifier, but by Section 2.2.3, it's shifted down 4 bytes to be the start of the Basic Header Segment. - Nothing obvious seems to say that the Text or the Ping Data is a "data segment". The latter might be obvious, but the former isn't. - The Next-Qualifier structure is unexpected to those familiar with other protocols. A common protocol structure is T/L/V (Type/ Length/Value). Because the BHS type and length are implicit, the iSCSI structure is actually Type N+1/Length N+1/Value N. So, much as I realize the busy work involved in revising ASCII graphics, I think all the PDU diagrams in Section 2 need to be revised to show the Next- Qualifiers. This would also take care of the second item above, because then byte 0 in all the PDU diagrams will be the first byte of the Next-Qualifier that precedes the Basic Header Segment. Addressing the third item involves a few sentences in to explain what the "data segment" indicated by the Next Qualifier before the BHS actually covers/ contains in those cases. The fourth item is a matter of taste. It would certainly be cleaner if iSCSI used a TLV structure, rather than a T+1/L+1/V structure. Much of the network community is familiar with TLV and will find the T+1/L+1/V structure confusing. Are the 4 bytes saved by not having a descriptor for the BHS really worth it? On a related topic, it would be considerably cleaner if the type (WN) field were a single byte that was completely enumerated, rather than the current bit field structure, and if the peculiar convention for the AddCDB length in 2.2.2.1 were dropped in favor of the 3-byte length field in 2.2.2.3 to remove a case from the header parse logic. Mindful of my WG co-chair role, I really think the diagram revisions and explanations of what a "data segment" is need to be done. OTOH, the above two paragraphs on whether to have a BHS descriptor and how to format the Next-Qualifiers are suggestions for further discussion. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Mar 2 18:09:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA24168 for ; Fri, 2 Mar 2001 18:09:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f22Knox23219 for ips-outgoing; Fri, 2 Mar 2001 15:49:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f22Kms123169 for ; Fri, 2 Mar 2001 15:48:54 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel1.hp.com (Postfix) with ESMTP id A40E01F6 for ; Fri, 2 Mar 2001 15:48:53 -0500 (EST) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id MAA01772 for ips@ece.cmu.edu; Fri, 2 Mar 2001 12:49:49 -0800 (PST) Message-Id: <200103022049.MAA01772@core.rose.hp.com> Subject: Re: iSCSI: draft04 questions To: ips@ece.cmu.edu Date: Fri, 02 Mar 2001 12:49:49 PST Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, Thanks. I am not clear on some. Comments. >2. I didn't find a way for an iSCSI target to say that it does not support >any unsolicited data at all. SPC-2 specifies that a zero FirstBurstSize >means unlimited. > >+++ UseR2T=yes (default) and ImmediateData=no +++ I am confused. I thought this combination disallows just the "immediate" data, and not the unsolicited data as a whole. The draft hints at this in section 1.2.5. "A target MAY separately enable immediate data without enabling the more general (separate data PDUs) form of unsolicited data." If I misunderstood, could you please comment how immediate data alone is disabled, while allowing unsolicited data of FirstBurstSize? >4. Somehow, "Asynchronous Message" seems at odds with the rest of the >usage in the draft in regards to PDUs (since a message is introduced as >PDU in section 1.2). Should we may be just call it an AEN PDU? Section >2.14.3 calls this so. (I realize that it may not always result in a >SCSI-level AER.) > >+++ Fixed 2.14.3 - vox populi in Orlando not to create confusion -:) +++ As I said in my message to David, since we said a message is a PDU in 1.2, an "Asynchronous Message PDU" would be a redundant term, unless we call it as "Asynchronous something PDU". Do you see what I was getting at? I would leave it upto you. >for an initiator, if any, between iSCSI event codes 2 & 3. Could you >please >comment? >+++ 2 is for the purists - message,logout,login, 3 indicates that the >target will disconnectd the named connection +++ >Also, isn't the time in seconds the maximum time than the minimum >time as currently stated? >+++ time is minimum - allows the target to do the cleanup before being hit >with a >login +++ Do I take it then that the draft currently doesn't specify the maximum time the targets should keep the session & connection records around hoping for a restart? I would strongly recommend adding that as an additional field in the payload, since that is a resource allocation and scalability issue. Thanks. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Fri Mar 2 18:12:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA24245 for ; Fri, 2 Mar 2001 18:12:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f22KRmF21876 for ips-outgoing; Fri, 2 Mar 2001 15:27:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f22KQk121799 for ; Fri, 2 Mar 2001 15:26:46 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2650.21) id ; Fri, 2 Mar 2001 15:18:16 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07041015EB@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: iSCSI and Mode Pages Date: Fri, 2 Mar 2001 15:26:43 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Section 3 describes several things that can be negotiated via Text parameters or mode pages. Requirements for the behavior of the interaction between these two mechanisms needs to be set out a little more precisely (e.g., what happens if MODE SELECT is used on one of these *after* login is complete), ditto implementation requirements (e.g., on what fields do MODE SENSE and/or MODE SELECT have to work). This should just be a few sentences indicating what has to be implemented and what initiators can expect when accessing these mode pages in an iSCSI target. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Sat Mar 3 12:58:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA21986 for ; Sat, 3 Mar 2001 12:58:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f23GLDs06182 for ips-outgoing; Sat, 3 Mar 2001 11:21:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f23GKv106171 for ; Sat, 3 Mar 2001 11:20:57 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA31504 for ; Sat, 3 Mar 2001 17:20:50 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA211538 for ; Sat, 3 Mar 2001 17:19:59 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A04.0059AB3F ; Sat, 3 Mar 2001 17:19:25 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 3 Mar 2001 18:21:06 +0200 Subject: Re: iSCSI and Mode Pages Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk OK - will do (probably in the mode pages chapter). The intent was to have setting through MODE or Text be equivalent (Text is only for convenience). Regards, Julo Black_David@emc.com on 02/03/2001 22:26:43 Please respond to Black_David@emc.com To: ips@ece.cmu.edu cc: Subject: iSCSI and Mode Pages Section 3 describes several things that can be negotiated via Text parameters or mode pages. Requirements for the behavior of the interaction between these two mechanisms needs to be set out a little more precisely (e.g., what happens if MODE SELECT is used on one of these *after* login is complete), ditto implementation requirements (e.g., on what fields do MODE SENSE and/or MODE SELECT have to work). This should just be a few sentences indicating what has to be implemented and what initiators can expect when accessing these mode pages in an iSCSI target. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Sat Mar 3 12:58:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA22008 for ; Sat, 3 Mar 2001 12:58:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f23FwDj04986 for ips-outgoing; Sat, 3 Mar 2001 10:58:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f23FvR104941 for ; Sat, 3 Mar 2001 10:57:28 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA144868 for ; Sat, 3 Mar 2001 16:57:21 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA79432 for ; Sat, 3 Mar 2001 16:55:56 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A04.0057851E ; Sat, 3 Mar 2001 16:55:57 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 3 Mar 2001 17:57:36 +0200 Subject: Re: Draft-05: SCSI response typo Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Thanks - Julo "Ayman Ghanem" on 02/03/2001 17:30:34 Please respond to "Ayman Ghanem" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Draft-05: SCSI response typo Julian, There is a typo in section 2.4 (SCSI RSP PDU). Bit 7 of the flags byte is set to 1 but the text says b5-7 are reserved. -Ayman From owner-ips@ece.cmu.edu Sat Mar 3 13:04:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA22068 for ; Sat, 3 Mar 2001 13:04:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f23FiDB04180 for ips-outgoing; Sat, 3 Mar 2001 10:44:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f23Fi3104170 for ; Sat, 3 Mar 2001 10:44:03 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA134644 for ; Sat, 3 Mar 2001 16:43:55 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA62782 for ; Sat, 3 Mar 2001 16:42:31 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A04.00564A62 ; Sat, 3 Mar 2001 16:42:31 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu cc: Dafna_Sheinwald@il.ibm.com Message-ID: Date: Sat, 3 Mar 2001 17:44:10 +0200 Subject: Re: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk John, We all due respect I disagree with both your statements: - CRC is not expensive in software if you are willing to spend some memory for tables and literature about how to do it is abundant. - Adler and Fletcher are weak and there is no theory behind your distribution statements, nor any simulation results as far as I know. We found that on very simple sequences the Hamming distance gets down to 2 (or lower) and the burst protection is probably not better than 16 bit. There is even a simple formula for what sequences will get you false codes (see bellow for a reference) - CRC gets you alway a 32 bit burst protection and that makes for a very low probability for undetected burst and a guaranteed Hamming distance of 3 (or higher blocks up to 2k). There seems to exist also a class of CRCs that are excellent for very long blocks with distances higher than 4. Computing the undetected error probability was never published for Adler or Flletcher and adding thhis to the lack of theoretical background make it a very poor choice for a data storage platform. If you want some theory background please read: http://www.haifa.il.ibm.com/satran/ips/draft-sheinwald-iSCSI-CRC-00.txt You will find there both theoretical references and CRC implementation references that include even code. We are planing to update it with some newer CRCs. Regards, Julo John H Howard on 02/03/2001 15:42:25 Please respond to John H Howard To: Douglas Otis cc: IPS Reflector Subject: Re: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt CRC-32 is very expensive in software; it may require hardware assist in fast networks. Is iSCSI really intended to be a hardware only protocol? Fletcher-32 has a serious flaw: it does not distinguish between an input halfword of all ones (FFFFx) and an input halfword of all zeros (0000x). Both are equal to zero under ones' complement addition. [Stone, J., Greenwald, M., Partridge, C., & Hughes, J., "Performance of Checksums and CRC's over Real Data", IEEE/ACM Trans. on Networking, Vol., 6, No. 5, October 1988] gives several examples of situations in which this is important. Adler-32 avoids this problem by adding 8-bit inputs into its 16-bit sub-sums. It also uses a prime modulus (2**16-15) rather than ones' complement's 2**16-1. Using 8 bit rather than 16 bit inputs doubles the number of additions Adler-32 performs compared to Fletcher-32. A more subtle problem is that the high-order bits of the sums are not uniformly distributed until the block size becomes very large. I estimate that this causes Adler-32 to lose one or two bits worth of resolving power. Still, a 30 bit checksum is still quite strong. An alternative worth consideration is Fletcher-32 modified to use the prime modulus 2**16-15. It's fast, doesn't have the 0000=FFFF problem, detects permutations, and distributes all result bits uniformly. It does confuse an input halfword of 0 with 2**16-15 (and similarly for 1..14), but I think this only a minor problem because the potentially confused inputs are unlikely. By definition every checksum confuses many inputs. If you are going to argue from bad examples you really should estimate how likely your bad examples are. John Howard Sun Microsystems From owner-ips@ece.cmu.edu Sat Mar 3 14:45:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA23863 for ; Sat, 3 Mar 2001 14:45:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f23HcN510270 for ips-outgoing; Sat, 3 Mar 2001 12:38:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f23Hbn110235 for ; Sat, 3 Mar 2001 12:37:49 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA138556 for ; Sat, 3 Mar 2001 18:37:43 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA103892 for ; Sat, 3 Mar 2001 18:36:17 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A04.0060B606 ; Sat, 3 Mar 2001 18:36:20 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 3 Mar 2001 19:02:58 +0200 Subject: Re: iSCSI: Next-Qualifiers and PDU diagrams Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I will try to lessen confusion somehow - I am not sure that adding the NQ everywhere is the answer. The trouble with TLV formats is that they need at least two passes. With the NQ I use the fact that the basic header is always there and the qualifier will be needed do describe the next segment (its type and length). Julo Black_David@emc.com on 02/03/2001 21:23:10 Please respond to Black_David@emc.com To: ips@ece.cmu.edu cc: Subject: iSCSI: Next-Qualifiers and PDU diagrams In working with some other folks, I stumbled across a format issue that's more confusing than it needs to be in the current iSCSI draft. The issue is that Next-Qualifiers (WN, etc.) and the PDUs they go with are not documented/diagrammed together, making it hard to figure out how to parse iSCSI PDUs. For an example of how this causes confusion, consider the Text, Login and NOP PDU diagrams in Sections 2.8 through 2.13. Starting from one of these diagrams, how does one figure out the size of the Text or Ping data that's appended? The answer's not in the diagrams -- there's a 4-byte Next Qualifier preceding the PDU shown in each diagram. The Text or Ping data is considered to be a data segment, and hence the format of the Next Qualifier is that given in Section 2.2.2.3, including a 3-byte length field that answers the question. That's not an obvious answer, and there are several factors that contribute to it being potentially confusing: - None of the PDU diagrams show the Next-Qualifiers - Byte 0 means different things - at the top of Section 2.2, it's the start of the Next-Qualifier, but by Section 2.2.3, it's shifted down 4 bytes to be the start of the Basic Header Segment. - Nothing obvious seems to say that the Text or the Ping Data is a "data segment". The latter might be obvious, but the former isn't. - The Next-Qualifier structure is unexpected to those familiar with other protocols. A common protocol structure is T/L/V (Type/ Length/Value). Because the BHS type and length are implicit, the iSCSI structure is actually Type N+1/Length N+1/Value N. So, much as I realize the busy work involved in revising ASCII graphics, I think all the PDU diagrams in Section 2 need to be revised to show the Next- Qualifiers. This would also take care of the second item above, because then byte 0 in all the PDU diagrams will be the first byte of the Next-Qualifier that precedes the Basic Header Segment. Addressing the third item involves a few sentences in to explain what the "data segment" indicated by the Next Qualifier before the BHS actually covers/ contains in those cases. The fourth item is a matter of taste. It would certainly be cleaner if iSCSI used a TLV structure, rather than a T+1/L+1/V structure. Much of the network community is familiar with TLV and will find the T+1/L+1/V structure confusing. Are the 4 bytes saved by not having a descriptor for the BHS really worth it? On a related topic, it would be considerably cleaner if the type (WN) field were a single byte that was completely enumerated, rather than the current bit field structure, and if the peculiar convention for the AddCDB length in 2.2.2.1 were dropped in favor of the 3-byte length field in 2.2.2.3 to remove a case from the header parse logic. Mindful of my WG co-chair role, I really think the diagram revisions and explanations of what a "data segment" is need to be done. OTOH, the above two paragraphs on whether to have a BHS descriptor and how to format the Next-Qualifiers are suggestions for further discussion. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Sat Mar 3 14:47:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA23933 for ; Sat, 3 Mar 2001 14:47:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f23HcLU10266 for ips-outgoing; Sat, 3 Mar 2001 12:38:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f23Hc3110245 for ; Sat, 3 Mar 2001 12:38:04 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA76526 for ; Sat, 3 Mar 2001 18:37:57 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA161988 for ; Sat, 3 Mar 2001 18:37:06 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A04.0060B8B7 ; Sat, 3 Mar 2001 18:36:27 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 3 Mar 2001 19:32:13 +0200 Subject: Re: iSCSI 05.txt is out Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Sandeep, Thanks for the carefull reading. Responses in text. Julo Sandeep Joshi on 02/03/2001 21:15:53 Please respond to Sandeep Joshi To: ips@ece.cmu.edu cc: Subject: Re: iSCSI 05.txt is out Hi Julian, Draft looks good. Here are a few issues i noticed. Some of these issues may have been valid with 04 too. -Sandeep Sec 2.7: Scsi Data for READ --------------------------- This is a read PDU but the fields for (O|U) do not match the corresponding positions for Read status indicators in the Scsi Response PDU. In the Scsi Response PDU, it is lowercase "o" and "u" which indicate read overflow/underflow. They are present in a different bit positions. Notice that the residual count position is consistent. So how about matching up the over/underflow bits to same positions with same lowercase syntax ? This may also change the "S" bit position. +++ in response we had to take care of the BiDirectional commands so there are two sets. In read data - that is only for pure Reads (not for write-then-read ++++ R2TDataSN ---------- Sec 6.7.1 has some new content on how to handle lost R2Ts using SACKs. But I noticed that the SACK request (Sec 2.16) has not changed to indicate whether the DataSN is a R2T DataSN or just a Read PDU DataSN (D bit) So do we demux on the read/write operation type? And how does this affect PDU loss in bidirectional commands ? +++ SACK is ascking for data (DataSN) the target knows Sec 2.14.1 ---------- CID: This field is valid only if the reason code is not "close connection" -- you mean "close session" right ? To further reinforce this, can we also add "CID or reserved(0)" to the PDU frame above - just as we do in other cases. +++ you are correct Sec 2.18 Async PDU -------------------- Could you elaborate on the scenarios proposed under which a target might request a logout (event code=2)? I recall this code was added recently (in 03 or 04) but cant recall the exact semantics. +++ one cause could be maintenance For example, can this code be used if some connection does not respond to pings ? From Sec 2.14, I see that this will result in a Logout Command with reason_code=2 (target-requested logout). There is also some discussion in Sec 6.7.1.2 on this. This facility gives rise to race conditions on duplex & buffered channels - plus the Async PDU intends to preserve the statSN ordering. [Async is a misnomer :-)] Say both initiator & target decide to close a non-responding TCP connection_1. The initiator sends a Login (for conn recovery) on some other connection_2. At the same "time", the target sends an Async PDU for c_1 on connection_3. +++ Login can be sent only when establishing (or restarting) a connection but --- i.e. TCP-Open followed by login - Obviously there can be races but the initiator and target can get out of them quickly +++ By the time Async PDU is delivered, the initiator may well have recovered the connection so its going to logout a connection which is chugging along hunky-dory. In the absence of any synchronized timestamps, it is not possible to know if one is responding to an event which has already been handled. +++ I & T have no synchronized clocks - and I did not see a compeling reason to add one Sec 2.10 : Version number ---------------------------- The min & max versions are just plain numbers. Any plans to support major+minor version numbers ? +++ they where in then we decided that we have enough artifacts already -:) From owner-ips@ece.cmu.edu Sat Mar 3 14:48:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA23944 for ; Sat, 3 Mar 2001 14:48:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f23HcIU10263 for ips-outgoing; Sat, 3 Mar 2001 12:38:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f23Hbo110237 for ; Sat, 3 Mar 2001 12:37:50 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA134552 for ; Sat, 3 Mar 2001 18:37:44 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA189682 for ; Sat, 3 Mar 2001 18:36:52 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A04.0060B64B ; Sat, 3 Mar 2001 18:36:21 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 3 Mar 2001 19:10:03 +0200 Subject: Re: iSCSI: draft04 questions Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk === coments in text Julo "Mallikarjun C." on 02/03/2001 22:49:49 Please respond to cbm@rose.hp.com To: ips@ece.cmu.edu cc: Subject: Re: iSCSI: draft04 questions Julian, Thanks. I am not clear on some. Comments. >2. I didn't find a way for an iSCSI target to say that it does not support >any unsolicited data at all. SPC-2 specifies that a zero FirstBurstSize >means unlimited. > >+++ UseR2T=yes (default) and ImmediateData=no +++ I am confused. I thought this combination disallows just the "immediate" data, and not the unsolicited data as a whole. The draft hints at this in section 1.2.5. "A target MAY separately enable immediate data without enabling the more general (separate data PDUs) form of unsolicited data." === for this case you have UseR2T=yes and ImmediateData=yes If I misunderstood, could you please comment how immediate data alone is disabled, while allowing unsolicited data of FirstBurstSize? === UseR2T=no ImmediateData=no >4. Somehow, "Asynchronous Message" seems at odds with the rest of the >usage in the draft in regards to PDUs (since a message is introduced as >PDU in section 1.2). Should we may be just call it an AEN PDU? Section >2.14.3 calls this so. (I realize that it may not always result in a >SCSI-level AER.) > >+++ Fixed 2.14.3 - vox populi in Orlando not to create confusion -:) +++ As I said in my message to David, since we said a message is a PDU in 1.2, an "Asynchronous Message PDU" would be a redundant term, unless we call it as "Asynchronous something PDU". Do you see what I was getting at? I would leave it upto you. >for an initiator, if any, between iSCSI event codes 2 & 3. Could you >please >comment? >+++ 2 is for the purists - message,logout,login, 3 indicates that the >target will disconnectd the named connection +++ >Also, isn't the time in seconds the maximum time than the minimum >time as currently stated? >+++ time is minimum - allows the target to do the cleanup before being hit >with a >login +++ Do I take it then that the draft currently doesn't specify the maximum time the targets should keep the session & connection records around hoping for a restart? I would strongly recommend adding that as an additional field in the payload, since that is a resource allocation and scalability issue. Thanks. === I assume that a traget can figure out by itself after a while that there is no rendezvous - but I am open to requests -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Sat Mar 3 14:49:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA23971 for ; Sat, 3 Mar 2001 14:49:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f23HcFa10257 for ips-outgoing; Sat, 3 Mar 2001 12:38:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f23Hbn110234 for ; Sat, 3 Mar 2001 12:37:49 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA138554 for ; Sat, 3 Mar 2001 18:37:43 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA67504 for ; Sat, 3 Mar 2001 18:36:51 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A04.0060B5BF ; Sat, 3 Mar 2001 18:36:19 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 3 Mar 2001 18:50:04 +0200 Subject: draft-cavanna-iscsi-crc-vs-cksum-01.txt Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, Vince Cavanna and Matt Wakeley have a new version of their draft ready. Until the IETF I_D site reopens (and long time after -:))you can find it at: http://www.haifa.il.ibm.com/satran/ips Julo From owner-ips@ece.cmu.edu Sat Mar 3 16:45:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA25891 for ; Sat, 3 Mar 2001 16:45:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f23JL5t17286 for ips-outgoing; Sat, 3 Mar 2001 14:21:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f23JK4r17193 for ; Sat, 3 Mar 2001 14:20:04 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f23KTQ037572; Sat, 3 Mar 2001 12:29:27 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: , Subject: RE: iSCSI and Mode Pages Date: Sat, 3 Mar 2001 11:18:33 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <0F31E5C394DAD311B60C00E029101A07041015EB@corpmx9.isus.emc.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, Specifying SCSI responses with respect to Mode Select or Mode Sense seems to be running into T10's domain. Perhaps this work should be placed within an appendix to indicate this information as advisory. Doug > Section 3 describes several things that can be negotiated > via Text parameters or mode pages. Requirements for > the behavior of the interaction between these two > mechanisms needs to be set out a little more precisely > (e.g., what happens if MODE SELECT is used on one > of these *after* login is complete), ditto implementation > requirements (e.g., on what fields do MODE SENSE and/or > MODE SELECT have to work). This should just be a > few sentences indicating what has to be implemented > and what initiators can expect when accessing these > mode pages in an iSCSI target. > > Thanks, > --David > > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Sun Mar 4 14:07:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA14489 for ; Sun, 4 Mar 2001 14:07:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f24H0Yi01439 for ips-outgoing; Sun, 4 Mar 2001 12:00:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from latte.2xtreme.net (latte.2xtreme.net [209.63.222.34]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f24Gxkr01405 for ; Sun, 4 Mar 2001 11:59:46 -0500 (EST) Received: (qmail 15536 invoked from network); 4 Mar 2001 16:59:45 -0000 Received: from p140.oak4.2xtreme.net (HELO ljoy) (209.63.215.140) by latte.2xtreme.net with SMTP; 4 Mar 2001 16:59:45 -0000 From: "Douglas Otis" To: Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Date: Sun, 4 Mar 2001 08:58:27 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Errors being detected are not created by a media defect or an impulse on a serial line. CRC serial burst error detection will be in place on the serial medium so focus on burst errors seems misplaced. Also, CRC requires a relatively expensive lookup compared to an algorithm that only accesses data being checked. I would not be surprised the difference running 2:1 for most architectures. The suggestion by John Howard sounds interesting for small regions of less than 512 bytes. Doug > John, > > We all due respect I disagree with both your statements: > > - CRC is not expensive in software if you are willing to spend some memory > for tables > and literature about how to do it is abundant. > > - Adler and Fletcher are weak and there is no theory behind your > distribution statements, nor any simulation results as far as I know. We > found that on very simple sequences the Hamming distance gets > down to 2 (or > lower) and the burst protection is probably not better than 16 bit. There > is even a simple formula for what sequences will get you false codes (see > bellow for a reference) > > - CRC gets you alway a 32 bit burst protection and that makes for a very > low probability for undetected burst and a guaranteed Hamming > distance of 3 > (or higher blocks up to 2k). There seems to exist also a class of > CRCs that > are excellent for very long blocks with distances higher than 4. > > Computing the undetected error probability was never published > for Adler or > Flletcher and adding thhis to the lack of theoretical background make it a > very poor choice for a data storage platform. > > > If you want some theory background please read: > > http://www.haifa.il.ibm.com/satran/ips/draft-sheinwald-iSCSI-CRC-00.txt > > > You will find there both theoretical references and CRC implementation > references that include even code. > We are planing to update it with some newer CRCs. > > Regards, > Julo > > > John H Howard on 02/03/2001 15:42:25 > > Please respond to John H Howard > > To: Douglas Otis > cc: IPS Reflector > Subject: Re: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt > > > > > > CRC-32 is very expensive in software; it may require hardware assist in > fast networks. Is iSCSI really intended to be a hardware only protocol? > > Fletcher-32 has a serious flaw: it does not distinguish between an > input halfword of all ones (FFFFx) and an input halfword of all zeros > (0000x). Both are equal to zero under ones' complement addition. > [Stone, J., Greenwald, M., Partridge, C., & Hughes, J., "Performance of > Checksums and CRC's over Real Data", IEEE/ACM Trans. on Networking, > Vol., 6, No. 5, October 1988] gives several examples of situations in > which this is important. > > Adler-32 avoids this problem by adding 8-bit inputs into its 16-bit > sub-sums. It also uses a prime modulus (2**16-15) rather than ones' > complement's 2**16-1. Using 8 bit rather than 16 bit inputs doubles the > number of additions Adler-32 performs compared to Fletcher-32. A more > subtle problem is that the high-order bits of the sums are not uniformly > distributed until the block size becomes very large. I estimate that > this causes Adler-32 to lose one or two bits worth of resolving power. > Still, a 30 bit checksum is still quite strong. > > An alternative worth consideration is Fletcher-32 modified to use the > prime modulus 2**16-15. It's fast, doesn't have the 0000=FFFF problem, > detects permutations, and distributes all result bits uniformly. It > does confuse an input halfword of 0 with 2**16-15 (and similarly for > 1..14), but I think this only a minor problem because the potentially > confused inputs are unlikely. By definition every checksum confuses > many inputs. If you are going to argue from bad examples you really > should estimate how likely your bad examples are. > > John Howard > Sun Microsystems > > > > From owner-ips@ece.cmu.edu Sun Mar 4 19:00:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA16445 for ; Sun, 4 Mar 2001 19:00:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f24MSgw19995 for ips-outgoing; Sun, 4 Mar 2001 17:28:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from crufty.research.bell-labs.com (crufty.research.bell-labs.com [204.178.16.49]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f24MRmr19955 for ; Sun, 4 Mar 2001 17:27:48 -0500 (EST) Received: from grubby.research.bell-labs.com ([135.104.2.9]) by crufty; Sun Mar 4 17:25:04 EST 2001 Received: from aura.research.bell-labs.com ([135.104.46.10]) by grubby; Sun Mar 4 17:26:50 EST 2001 Received: from research.bell-labs.com (IDENT:sandeepj@sandeepj-pcmh.research.bell-labs.com [135.104.47.90]) by aura.research.bell-labs.com (8.9.1/8.9.1) with ESMTP id RAA23209 for ; Sun, 4 Mar 2001 17:26:50 -0500 (EST) Message-ID: <3AA2C12A.1B6A12B8@research.bell-labs.com> Date: Sun, 04 Mar 2001 17:26:50 -0500 From: Sandeep Joshi X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.2.16-3 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: Next-Qualifiers and PDU diagrams References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, One suggestion. Perhaps, it may ease the confusion if the draft mentioned that iSCSI headers have been modelled analogous to the IPv6 header formats. (..confusions lessen when examples abound) The IPv6 basic header and every extension header has a "next header" segment field (T+1/L/V), and it seems to help in IPv6 routing - no varlength basic headers - no need to examine extension headers, etc On the other hand, as suggested earlier by someone, a payload length field in the BHS would ease data placement. -Sandeep julian_satran@il.ibm.com wrote: > > David, > > I will try to lessen confusion somehow - I am not sure that adding the NQ > everywhere is the answer. > > The trouble with TLV formats is that they need at least two passes. With > the NQ I use the fact that > the basic header is always there and the qualifier will be needed do > describe the next segment (its type and length). > > Julo > > Black_David@emc.com on 02/03/2001 21:23:10 > > Please respond to Black_David@emc.com > > To: ips@ece.cmu.edu > cc: > Subject: iSCSI: Next-Qualifiers and PDU diagrams > > In working with some other folks, I stumbled across > a format issue that's more confusing than it needs > to be in the current iSCSI draft. The issue is that > Next-Qualifiers (WN, etc.) and the PDUs they go with > are not documented/diagrammed together, making it > hard to figure out how to parse iSCSI PDUs. > > For an example of how this causes confusion, consider > the Text, Login and NOP PDU diagrams in Sections 2.8 > through 2.13. Starting from one of these diagrams, > how does one figure out the size of the Text or > Ping data that's appended? > > The answer's not in the diagrams -- there's a 4-byte > Next Qualifier preceding the PDU shown in each diagram. > The Text or Ping data is considered to be a data > segment, and hence the format of the Next Qualifier > is that given in Section 2.2.2.3, including a 3-byte > length field that answers the question. > > That's not an obvious answer, and there are several > factors that contribute to it being potentially confusing: > - None of the PDU diagrams show the Next-Qualifiers > - Byte 0 means different things - at the top of > Section 2.2, it's the start of the Next-Qualifier, > but by Section 2.2.3, it's shifted down 4 bytes > to be the start of the Basic Header Segment. > - Nothing obvious seems to say that the Text or the > Ping Data is a "data segment". The latter > might be obvious, but the former isn't. > - The Next-Qualifier structure is unexpected to > those familiar with other protocols. A > common protocol structure is T/L/V (Type/ > Length/Value). Because the BHS type and > length are implicit, the iSCSI structure > is actually Type N+1/Length N+1/Value N. > So, much as I realize the busy work involved in > revising ASCII graphics, I think all the PDU diagrams > in Section 2 need to be revised to show the Next- > Qualifiers. This would also take care of the second > item above, because then byte 0 in all the PDU > diagrams will be the first byte of the Next-Qualifier > that precedes the Basic Header Segment. Addressing > the third item involves a few sentences in to > explain what the "data segment" indicated by the > Next Qualifier before the BHS actually covers/ > contains in those cases. > > The fourth item is a matter of taste. It would > certainly be cleaner if iSCSI used a TLV structure, > rather than a T+1/L+1/V structure. Much of the > network community is familiar with TLV and will > find the T+1/L+1/V structure confusing. Are the > 4 bytes saved by not having a descriptor for the > BHS really worth it? > > On a related topic, it would be considerably > cleaner if the type (WN) field were a single byte > that was completely enumerated, rather than the > current bit field structure, and if the > peculiar convention for the AddCDB length in > 2.2.2.1 were dropped in favor of the 3-byte > length field in 2.2.2.3 to remove a case > from the header parse logic. > > Mindful of my WG co-chair role, I really think > the diagram revisions and explanations of what > a "data segment" is need to be done. OTOH, > the above two paragraphs on whether to have a BHS > descriptor and how to format the Next-Qualifiers > are suggestions for further discussion. > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- From owner-ips@ece.cmu.edu Sun Mar 4 19:00:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA16456 for ; Sun, 4 Mar 2001 19:00:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f24Ltgi18098 for ips-outgoing; Sun, 4 Mar 2001 16:55:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f24Lspr18037 for ; Sun, 4 Mar 2001 16:54:52 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id WAB61544 for ; Sun, 4 Mar 2001 22:54:39 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id WAA91178 for ; Sun, 4 Mar 2001 22:53:46 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A05.00783A70 ; Sun, 4 Mar 2001 22:53:12 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 4 Mar 2001 21:33:05 +0200 Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, There is no lookup in CRC computation. A table entry (indexed by the code) is used in computation. I would appreciate if you would not spread confusion. Julo "Douglas Otis" on 04/03/2001 18:58:27 Please respond to "Douglas Otis" To: ips@ece.cmu.edu cc: Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Julian, Errors being detected are not created by a media defect or an impulse on a serial line. CRC serial burst error detection will be in place on the serial medium so focus on burst errors seems misplaced. Also, CRC requires a relatively expensive lookup compared to an algorithm that only accesses data being checked. I would not be surprised the difference running 2:1 for most architectures. The suggestion by John Howard sounds interesting for small regions of less than 512 bytes. Doug > John, > > We all due respect I disagree with both your statements: > > - CRC is not expensive in software if you are willing to spend some memory > for tables > and literature about how to do it is abundant. > > - Adler and Fletcher are weak and there is no theory behind your > distribution statements, nor any simulation results as far as I know. We > found that on very simple sequences the Hamming distance gets > down to 2 (or > lower) and the burst protection is probably not better than 16 bit. There > is even a simple formula for what sequences will get you false codes (see > bellow for a reference) > > - CRC gets you alway a 32 bit burst protection and that makes for a very > low probability for undetected burst and a guaranteed Hamming > distance of 3 > (or higher blocks up to 2k). There seems to exist also a class of > CRCs that > are excellent for very long blocks with distances higher than 4. > > Computing the undetected error probability was never published > for Adler or > Flletcher and adding thhis to the lack of theoretical background make it a > very poor choice for a data storage platform. > > > If you want some theory background please read: > > http://www.haifa.il.ibm.com/satran/ips/draft-sheinwald-iSCSI-CRC-00.txt > > > You will find there both theoretical references and CRC implementation > references that include even code. > We are planing to update it with some newer CRCs. > > Regards, > Julo > > > John H Howard on 02/03/2001 15:42:25 > > Please respond to John H Howard > > To: Douglas Otis > cc: IPS Reflector > Subject: Re: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt > > > > > > CRC-32 is very expensive in software; it may require hardware assist in > fast networks. Is iSCSI really intended to be a hardware only protocol? > > Fletcher-32 has a serious flaw: it does not distinguish between an > input halfword of all ones (FFFFx) and an input halfword of all zeros > (0000x). Both are equal to zero under ones' complement addition. > [Stone, J., Greenwald, M., Partridge, C., & Hughes, J., "Performance of > Checksums and CRC's over Real Data", IEEE/ACM Trans. on Networking, > Vol., 6, No. 5, October 1988] gives several examples of situations in > which this is important. > > Adler-32 avoids this problem by adding 8-bit inputs into its 16-bit > sub-sums. It also uses a prime modulus (2**16-15) rather than ones' > complement's 2**16-1. Using 8 bit rather than 16 bit inputs doubles the > number of additions Adler-32 performs compared to Fletcher-32. A more > subtle problem is that the high-order bits of the sums are not uniformly > distributed until the block size becomes very large. I estimate that > this causes Adler-32 to lose one or two bits worth of resolving power. > Still, a 30 bit checksum is still quite strong. > > An alternative worth consideration is Fletcher-32 modified to use the > prime modulus 2**16-15. It's fast, doesn't have the 0000=FFFF problem, > detects permutations, and distributes all result bits uniformly. It > does confuse an input halfword of 0 with 2**16-15 (and similarly for > 1..14), but I think this only a minor problem because the potentially > confused inputs are unlikely. By definition every checksum confuses > many inputs. If you are going to argue from bad examples you really > should estimate how likely your bad examples are. > > John Howard > Sun Microsystems > > > > From owner-ips@ece.cmu.edu Mon Mar 5 11:30:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA20066 for ; Mon, 5 Mar 2001 11:30:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f25E3Bj17998 for ips-outgoing; Mon, 5 Mar 2001 09:03:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f25E2Jr17967 for ; Mon, 5 Mar 2001 09:02:19 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 5 Mar 2001 09:01:06 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07041015F5@corpmx9.isus.emc.com> From: Black_David@emc.com To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI: Next-Qualifiers and PDU diagrams Date: Mon, 5 Mar 2001 09:01:04 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > I will try to lessen confusion somehow - I am not sure that adding the NQ > everywhere is the answer. That'd be the most comprehensive thing to do, and I still think it's the right thing for clarity. At an absolute minimum, using one convention for which byte in a PDU is byte 0 is clearly called for. > The trouble with TLV formats is that they need at least two passes. I don't understand what you mean by "two passes", but see below because it may not be an issue. > With the NQ I use the fact that the basic header is always there > and the qualifier will be needed do describe the next segment (its type and length). Requiring that the first TLV always be the basic header segment whose size never changes is ok under a TLV approach. The differences between what we have now, and a TLV approach are: (1) The information currently in the NQ before the BHS moves to the TL after the BHS. It's still in a fixed location because the BHS is fixed size. (2) We spend another 4 bytes for the TL to describe the BHS. Now that I think about things a bit more, this initial TL could be omitted, with the TLV structure used only to describe what comes after the BHS. From a parse standpoint, the first TL after the BHS can be parsed with the BHS because that TL is always in the same place wrt the BHS. Is there a lot of value to having the TL for what comes after the BHS show up on the wire before the BHS? I'm sceptical. Is there any good reason not to merge the two formats for describing variable length segments? --David > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Saturday, March 03, 2001 12:03 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Next-Qualifiers and PDU diagrams > > > > David, > > I will try to lessen confusion somehow - I am not sure that adding the NQ > everywhere is the answer. > > The trouble with TLV formats is that they need at least two passes. With > the NQ I use the fact that > the basic header is always there and the qualifier will be needed do > describe the next segment (its type and length). > > Julo > > Black_David@emc.com on 02/03/2001 21:23:10 > > Please respond to Black_David@emc.com > > To: ips@ece.cmu.edu > cc: > Subject: iSCSI: Next-Qualifiers and PDU diagrams > > > > > In working with some other folks, I stumbled across > a format issue that's more confusing than it needs > to be in the current iSCSI draft. The issue is that > Next-Qualifiers (WN, etc.) and the PDUs they go with > are not documented/diagrammed together, making it > hard to figure out how to parse iSCSI PDUs. > > For an example of how this causes confusion, consider > the Text, Login and NOP PDU diagrams in Sections 2.8 > through 2.13. Starting from one of these diagrams, > how does one figure out the size of the Text or > Ping data that's appended? > > The answer's not in the diagrams -- there's a 4-byte > Next Qualifier preceding the PDU shown in each diagram. > The Text or Ping data is considered to be a data > segment, and hence the format of the Next Qualifier > is that given in Section 2.2.2.3, including a 3-byte > length field that answers the question. > > That's not an obvious answer, and there are several > factors that contribute to it being potentially confusing: > - None of the PDU diagrams show the Next-Qualifiers > - Byte 0 means different things - at the top of > Section 2.2, it's the start of the Next-Qualifier, > but by Section 2.2.3, it's shifted down 4 bytes > to be the start of the Basic Header Segment. > - Nothing obvious seems to say that the Text or the > Ping Data is a "data segment". The latter > might be obvious, but the former isn't. > - The Next-Qualifier structure is unexpected to > those familiar with other protocols. A > common protocol structure is T/L/V (Type/ > Length/Value). Because the BHS type and > length are implicit, the iSCSI structure > is actually Type N+1/Length N+1/Value N. > So, much as I realize the busy work involved in > revising ASCII graphics, I think all the PDU diagrams > in Section 2 need to be revised to show the Next- > Qualifiers. This would also take care of the second > item above, because then byte 0 in all the PDU > diagrams will be the first byte of the Next-Qualifier > that precedes the Basic Header Segment. Addressing > the third item involves a few sentences in to > explain what the "data segment" indicated by the > Next Qualifier before the BHS actually covers/ > contains in those cases. > > The fourth item is a matter of taste. It would > certainly be cleaner if iSCSI used a TLV structure, > rather than a T+1/L+1/V structure. Much of the > network community is familiar with TLV and will > find the T+1/L+1/V structure confusing. Are the > 4 bytes saved by not having a descriptor for the > BHS really worth it? > > On a related topic, it would be considerably > cleaner if the type (WN) field were a single byte > that was completely enumerated, rather than the > current bit field structure, and if the > peculiar convention for the AddCDB length in > 2.2.2.1 were dropped in favor of the 3-byte > length field in 2.2.2.3 to remove a case > from the header parse logic. > > Mindful of my WG co-chair role, I really think > the diagram revisions and explanations of what > a "data segment" is need to be done. OTOH, > the above two paragraphs on whether to have a BHS > descriptor and how to format the Next-Qualifiers > are suggestions for further discussion. > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > > From owner-ips@ece.cmu.edu Mon Mar 5 11:30:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA20080 for ; Mon, 5 Mar 2001 11:30:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f25E5Ae18083 for ips-outgoing; Mon, 5 Mar 2001 09:05:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f25E4Xr18064 for ; Mon, 5 Mar 2001 09:04:33 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 5 Mar 2001 09:04:27 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07041015F6@corpmx9.isus.emc.com> From: Black_David@emc.com To: dotis@sanlight.net, ips@ece.cmu.edu Subject: RE: iSCSI and Mode Pages Date: Mon, 5 Mar 2001 09:04:19 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > Specifying SCSI responses with respect to Mode Select or Mode Sense seems to > be running into T10's domain. Perhaps this work should be placed within an > appendix to indicate this information as advisory. In general, this is correct, but ... in this case all the mode pages in question are transport-specific, and hence iSCSI does need to specify how they behave, and the interaction with the non-SCSI mechanism (Text keys) that iSCSI provides as an alternative way to set the (iSCSI-specific) values they contain. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Mar 5 16:18:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA04731 for ; Mon, 5 Mar 2001 16:18:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f25IsGl08381 for ips-outgoing; Mon, 5 Mar 2001 13:54:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f25IrGr08344 for ; Mon, 5 Mar 2001 13:53:16 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Mon, 5 Mar 2001 10:53:09 -0800 Message-ID: From: Joshua Tseng To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: iSCSI Security Date: Mon, 5 Mar 2001 10:53:05 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, Why was the public key authentication method removed from version -05? Are you sure you want iSCSI to forsake the benefits of public key cryptography? I strongly suggest it be reinstated as one of the authentication methods listed in page 95. Josh From owner-ips@ece.cmu.edu Mon Mar 5 16:18:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA04743 for ; Mon, 5 Mar 2001 16:18:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f25IkJp07966 for ips-outgoing; Mon, 5 Mar 2001 13:46:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp012.mail.yahoo.com (smtp012.mail.yahoo.com [216.136.173.32]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f25IiGr07782 for ; Mon, 5 Mar 2001 13:44:17 -0500 (EST) Received: from unknown (HELO somesh) (206.112.106.119) by smtp.mail.vip.sc5.yahoo.com with SMTP; 5 Mar 2001 18:44:13 -0000 X-Apparently-From: Reply-To: From: "Somesh Gupta" To: , Subject: R2TDataSN Date: Mon, 5 Mar 2001 10:40:06 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > R2TDataSN > ---------- > Sec 6.7.1 has some new content on how to handle lost R2Ts using > SACKs. But I noticed that the SACK request (Sec 2.16) has not > changed to indicate whether the DataSN is a R2T DataSN or just > a Read PDU DataSN (D bit) > So do we demux on the read/write operation type? > And how does this affect PDU loss in bidirectional commands ? > +++ SACK is ascking for data (DataSN) the target knows > Julian, Regarding the R2TDataSN, I have a comments and a question. I think that when a PDU header fails a CRC/checksum check etc, it is a problem to depend on any information in the header (including length fields), thereby making any further processing on the connection unreliable. What scenarios do you envision where the R2TDataSN is useful. IN Orlando I think there was clear consensus that application do not try to transfer very large amounts of data using a single command. Thanks, Somesh _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Mon Mar 5 19:42:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA12190 for ; Mon, 5 Mar 2001 19:42:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f25Mcmd24249 for ips-outgoing; Mon, 5 Mar 2001 17:38:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antigonus.hosting.pacbell.net (antigonus.hosting.pacbell.net [216.100.98.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f25Mbar24207 for ; Mon, 5 Mar 2001 17:37:37 -0500 (EST) Received: from VENKAT1 (64-160-62-200.rhapsodynetworks.com [64.160.62.200] (may be forged)) by antigonus.hosting.pacbell.net id RAA26183; Mon, 5 Mar 2001 17:37:16 -0500 (EST) [ConcentricHost SMTP Relay 1.7] From: "Venkat Rangan" To: , Subject: RE: iSCSI 05.txt is out Date: Mon, 5 Mar 2001 14:40:18 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Here are a few technical and editorial comments on the latest draft. Technical: Page 12: - CmdSN - the current Command Sequence Number advanced by 1 on each command shipped Add: skipping zero, which is reserved for immediate delivery. Page 12: Remove the sentence: "A large difference between StatSN and ExpStatSN may indicate a failed connection." This may have made sense when StatSN and ExpStatSN are session-wide. Now that it is per-connection, this may not be valid. As was pointed out in the "Holes in StatSN" thread, ExpStatSN may be "stuck" at a low value when a single PDU encountered a Digest Error, followed by several well-formed PDUs. Page 24 - Section 2.1 Add: The length of the PDU SHALL include any padding bytes added. This raises a question as to whether it may be useful to include a two-bit "Fill Bytes" field in the header (BHS and AHS) which indicates the number of bytes that were added. Without this, it may be harder for the receiver to know how many bytes are part of the data. Fibre Channel has the Fill Byte specifier in its F_CTL part of the header. Page 25: This may be covered in another thread. What does the first Next-Qualifier (at byte 0) describe? Is it the BHS or the header following the BHS? If a PDU has a single BHS and a single AHS, is the sequence WN-BHS-AHS or is it WN-BHS-WN-AHS? Page 25 and 26: The text next to the bit description and the headers in the text for sections 2.2.2.1 to 2.2.2.3 appear to be inconsistent. An example is "read-data transfer" vs. "Read Data" "Long Data" vs. "Long Data Transfer" Also, 2.2.2.3 appears to have text about Long Data Header that probably belongs in section 2.2.2.2 (or I am confused about how WN is supposed to work.) Page 43 Section 2.7 The PDU diagrams should not show "Digests if any..." etc. It should be covered by NQ and corresponding AHS. Page 44 Section 2.7.2 The requirement of having to specify a valid LUN as part of WRITE Data (and NOP-Out) may pose unnecessary overhead for Target processing. The fact that Targets are now REQUIRED to reject errant initiators who may place a LUN inconsistent with the one sent in the CommandPdu means that targets should save the LUN against each context entry for the task. This was discussed earlier, and I think we said that unless the folks who originally wanted it spoke up, we will remove it. Note that Fibre Channel does not have this feature, and its original purpose can be accomplished by using Target Transfer Tag. If we still want it, targets should be given the option of negotiating this so that it can operate in a mode where a LUN specified as part of WRITE data will be ignored (as opposed to rejected). Section 2.17 The following text: "Within a connection, outstanding R2Ts MUST be fulfilled by the initiator in the order in which they were received." implies that R2Ts for the same WRITE command may be sent by the target on multiple connections? Since this is not possible, all R2Ts for a command are always on a single connection, so I am not sure how the above sentence should be interpreted. Page 75: Add: "i.e., from Most Preferred to Least Preferred" to "reverse order of preference." ------------ Editorial - typos and such 1. Page 11: Not covered are he means Change "he" to "the" 2. Page 20: Change "later" to "latter" in "the later can be used in subsequent commands." 3. Page 23: Change "isa" to "is a" Page 42: Section 2.6.1 Change to: Initiator Task Tag of the task that was not found; used in conjuction with Response value 1. It MUST be set to zero in other cases. Page 51: Change "CID does not change and this command is performs first" to "CID does not change and this command first performs" Page 55: There is nothing that indicates the status codes in the table are sent as part of Status-Detail. Page 66: Change "iSSCSI Data-out PDU" to "iSCSI Data (WRITE) PDU" The Data-out PDU is a new concept not introduced anywhere. Page 75: The term "Security Context Complete" is referred to prior to its definition. Page 81: Change "later" to "latter" ------------ Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com From owner-ips@ece.cmu.edu Mon Mar 5 19:42:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA12201 for ; Mon, 5 Mar 2001 19:42:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f25MiSr24605 for ips-outgoing; Mon, 5 Mar 2001 17:44:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hub1.san-jose.webnexus.net (hub1-20.san-jose.webnexus.net [209.140.224.20]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f25Mi9r24575 for ; Mon, 5 Mar 2001 17:44:09 -0500 (EST) Received: from rmaranon (h-209-140-231-182.webnexus.com [209.140.231.182] (may be forged)) by hub1.san-jose.webnexus.net (8.9.1a/8.9.1/WN-1.4) with SMTP id OAA18912; Mon, 5 Mar 2001 14:44:01 -0800 (PST) From: "Renato E. Maranon" To: "Kaladhar Voruganti" Cc: Subject: RE: iSCSI: Naming and Discovery Draft Date: Mon, 5 Mar 2001 14:43:01 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Some minor feedback. Appendix A is preceded by the text "8. ". "8. " Should be removed? Second. In Appendix B, B.4.5 Login fail, the description for the sample login for target removed (exception "Target removed 44") is split by the "Target Conflict 45" sample. Should probably be combined. Original text: I->Login Request InitiatorWWUI= iscsi.com.os.hostid.34567890 TargetWWUI= iscsi.com.acme.diskarray.sn.8675309 T->Login Response ("Target removed 44", F set) In this case the target has been removed. In contrast with codes 31 and 32 (in B.4.4), no redirection information is supplied. I->Login Request InitiatorWWUI= iscsi.com.os.hostid.34567890 TargetWWUI= iscsi.com.acme.diskarray.sn.8675309 T->Login Response ("Target Conflict 45", F set) Here, the target is busy with another initiator and cannot handle another one. The initiator MAY try again later. This can be the case of simple devices that can handle one device or the target has reached the limit of its initiators' capacity. In contrast to the previous examples, this rejection is temporary. I->Login Request InitiatorWWUI= iscsi.com.os.hostid.34567890 TargetWWUI= iscsi.com.acme.diskarray.sn.8675309 T->Login Response ("Target removed 44", F set) Here, the target has been removed. The initiator SHOULD terminate the login session. It MAY query the SNS for the new location of the target. (This should apply for the case when the target was not found - code 44). Suggestion: I->Login Request InitiatorWWUI= iscsi.com.os.hostid.34567890 TargetWWUI= iscsi.com.acme.diskarray.sn.8675309 T->Login Response ("Target removed 44", F set) In this case the target has been removed. In contrast with codes 31 and 32 (in B.4.4), no redirection information is supplied. Here, the target has been removed. The initiator SHOULD terminate the login session. It MAY query the SNS for the new location of the target. (This should apply for the case when the target was not found - code 44). I->Login Request InitiatorWWUI= iscsi.com.os.hostid.34567890 TargetWWUI= iscsi.com.acme.diskarray.sn.8675309 T->Login Response ("Target Conflict 45", F set) Here, the target is busy with another initiator and cannot handle another one. The initiator MAY try again later. This can be the case of simple devices that can handle one device or the target has reached the limit of its initiators' capacity. In contrast to the previous examples, this rejection is temporary. Renato Maranon Maranti Networks, Inc 920 Hillview Court Milpitas, Ca 95035 Phone: 408-719-9600 x309 Fax: 408-719-9631 email: rmaranon@marantinetworks.com home: www.marantinetworks.com -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Kaladhar Voruganti Sent: Saturday, February 24, 2001 9:51 AM To: ips@ece.cmu.edu Subject: iSCSI: Naming and Discovery Draft The following iSCSI naming and discovery draft has been submitted to IETF. This naming and discovery draft has been produced by the iSCSI naming and discovery team. The draft contains the iSCSI naming and discovery details. http://www.haifa.il.ibm.com/satran/ips/draft-ietf-ips-iSCSI-name-disc-00.txt Kaladhar Voruganti Computing Science Department IBM Almaden Research Lab San Jose, CA From owner-ips@ece.cmu.edu Mon Mar 5 19:43:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA12222 for ; Mon, 5 Mar 2001 19:43:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f25McoK24253 for ips-outgoing; Mon, 5 Mar 2001 17:38:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f25MbVr24204 for ; Mon, 5 Mar 2001 17:37:31 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f25Nji040607; Mon, 5 Mar 2001 15:45:44 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: , Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Date: Mon, 5 Mar 2001 14:34:54 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, The memory referencing technique used to compute CRC without processing each bit separately requires two accesses to memory, one for the data being processed and once again for a translation against this data, an index into a table as you have described. In the C code it looks as if there is a single memory reference and perhaps this is your confusion. As memory runs at a slower rate than instructions for any looping algorithm, such access was my concern regarding your statement about Alder-32 and CRC-32 having equivalent overhead. For any confusion that I have caused in expressing this opinion, I am sorry. Here is an example using a 256 byte table but should provide an understanding why 64k tables are desired as you indicated: unsigned char* dat_buf; unsigned long crc_syn = 0xffffffff; while(length--) crc_syn = (crc_syn >> 8) ^ crc32_table[(crc_syn & 0xff) ^ *dat_buf++]; return (crc_syn ^ 0xffffffff); Doug > Doug, > > > There is no lookup in CRC computation. > > A table entry (indexed by the code) is used in computation. > > I would appreciate if you would not spread confusion. > > Julo > > > > "Douglas Otis" on 04/03/2001 18:58:27 > > Please respond to "Douglas Otis" > > To: ips@ece.cmu.edu > cc: > Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt > > > > > Julian, > > Errors being detected are not created by a media defect or an impulse on a > serial line. CRC serial burst error detection will be in place on the > serial medium so focus on burst errors seems misplaced. Also, > CRC requires > a relatively expensive lookup compared to an algorithm that only accesses > data being checked. I would not be surprised the difference running 2:1 > for > most architectures. The suggestion by John Howard sounds interesting for > small regions of less than 512 bytes. > > Doug > > > John, > > > > We all due respect I disagree with both your statements: > > > > - CRC is not expensive in software if you are willing to spend some > memory > > for tables > > and literature about how to do it is abundant. > > > > - Adler and Fletcher are weak and there is no theory behind your > > distribution statements, nor any simulation results as far as I > know. We > > found that on very simple sequences the Hamming distance gets > > down to 2 (or > > lower) and the burst protection is probably not better than 16 bit. > There > > is even a simple formula for what sequences will get you false > codes (see > > bellow for a reference) > > > > - CRC gets you alway a 32 bit burst protection and that makes for a very > > low probability for undetected burst and a guaranteed Hamming > > distance of 3 > > (or higher blocks up to 2k). There seems to exist also a class of > > CRCs that > > are excellent for very long blocks with distances higher than 4. > > > > Computing the undetected error probability was never published > > for Adler or > > Flletcher and adding thhis to the lack of theoretical background make it > a > > very poor choice for a data storage platform. > > > > > > If you want some theory background please read: > > > > http://www.haifa.il.ibm.com/satran/ips/draft-sheinwald-iSCSI-CRC-00.txt > > > > > > You will find there both theoretical references and CRC implementation > > references that include even code. > > We are planing to update it with some newer CRCs. > > > > Regards, > > Julo > > > > > > John H Howard on 02/03/2001 15:42:25 > > > > Please respond to John H Howard > > > > To: Douglas Otis > > cc: IPS Reflector > > Subject: Re: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt > > > > > > > > > > > > CRC-32 is very expensive in software; it may require hardware assist in > > fast networks. Is iSCSI really intended to be a hardware only protocol? > > > > Fletcher-32 has a serious flaw: it does not distinguish between an > > input halfword of all ones (FFFFx) and an input halfword of all zeros > > (0000x). Both are equal to zero under ones' complement addition. > > [Stone, J., Greenwald, M., Partridge, C., & Hughes, J., "Performance of > > Checksums and CRC's over Real Data", IEEE/ACM Trans. on Networking, > > Vol., 6, No. 5, October 1988] gives several examples of situations in > > which this is important. > > > > Adler-32 avoids this problem by adding 8-bit inputs into its 16-bit > > sub-sums. It also uses a prime modulus (2**16-15) rather than ones' > > complement's 2**16-1. Using 8 bit rather than 16 bit inputs doubles the > > number of additions Adler-32 performs compared to Fletcher-32. A more > > subtle problem is that the high-order bits of the sums are not uniformly > > distributed until the block size becomes very large. I estimate that > > this causes Adler-32 to lose one or two bits worth of resolving power. > > Still, a 30 bit checksum is still quite strong. > > > > An alternative worth consideration is Fletcher-32 modified to use the > > prime modulus 2**16-15. It's fast, doesn't have the 0000=FFFF problem, > > detects permutations, and distributes all result bits uniformly. It > > does confuse an input halfword of 0 with 2**16-15 (and similarly for > > 1..14), but I think this only a minor problem because the potentially > > confused inputs are unlikely. By definition every checksum confuses > > many inputs. If you are going to argue from bad examples you really > > should estimate how likely your bad examples are. > > > > John Howard > > Sun Microsystems > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Mon Mar 5 21:03:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA13775 for ; Mon, 5 Mar 2001 21:02:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f25NxRK28603 for ips-outgoing; Mon, 5 Mar 2001 18:59:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f25NxCr28588 for ; Mon, 5 Mar 2001 18:59:12 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f2618T040676; Mon, 5 Mar 2001 17:08:30 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Douglas Otis" , , Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Date: Mon, 5 Mar 2001 15:57:38 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Sorry again, the example should have said 256 entry table as this will comprise 1024 bytes. Doug From owner-ips@ece.cmu.edu Mon Mar 5 22:13:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA16092 for ; Mon, 5 Mar 2001 22:13:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f261HRA02617 for ips-outgoing; Mon, 5 Mar 2001 20:17:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f261HFr02609 for ; Mon, 5 Mar 2001 20:17:20 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id CAA172664 for ; Tue, 6 Mar 2001 02:17:07 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id CAA136790 for ; Tue, 6 Mar 2001 02:16:12 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A07.0006E8FD ; Tue, 6 Mar 2001 02:15:28 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 6 Mar 2001 03:05:43 +0200 Subject: Re: iSCSI: Next-Qualifiers and PDU diagrams Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Sandeep, The Length field is there - it is part of the NQ. Reagrds, Julo Sandeep Joshi on 05/03/2001 00:26:50 Please respond to Sandeep Joshi To: ips@ece.cmu.edu cc: Subject: Re: iSCSI: Next-Qualifiers and PDU diagrams Julian, One suggestion. Perhaps, it may ease the confusion if the draft mentioned that iSCSI headers have been modelled analogous to the IPv6 header formats. (..confusions lessen when examples abound) The IPv6 basic header and every extension header has a "next header" segment field (T+1/L/V), and it seems to help in IPv6 routing - no varlength basic headers - no need to examine extension headers, etc On the other hand, as suggested earlier by someone, a payload length field in the BHS would ease data placement. -Sandeep julian_satran@il.ibm.com wrote: > > David, > > I will try to lessen confusion somehow - I am not sure that adding the NQ > everywhere is the answer. > > The trouble with TLV formats is that they need at least two passes. With > the NQ I use the fact that > the basic header is always there and the qualifier will be needed do > describe the next segment (its type and length). > > Julo > > Black_David@emc.com on 02/03/2001 21:23:10 > > Please respond to Black_David@emc.com > > To: ips@ece.cmu.edu > cc: > Subject: iSCSI: Next-Qualifiers and PDU diagrams > > In working with some other folks, I stumbled across > a format issue that's more confusing than it needs > to be in the current iSCSI draft. The issue is that > Next-Qualifiers (WN, etc.) and the PDUs they go with > are not documented/diagrammed together, making it > hard to figure out how to parse iSCSI PDUs. > > For an example of how this causes confusion, consider > the Text, Login and NOP PDU diagrams in Sections 2.8 > through 2.13. Starting from one of these diagrams, > how does one figure out the size of the Text or > Ping data that's appended? > > The answer's not in the diagrams -- there's a 4-byte > Next Qualifier preceding the PDU shown in each diagram. > The Text or Ping data is considered to be a data > segment, and hence the format of the Next Qualifier > is that given in Section 2.2.2.3, including a 3-byte > length field that answers the question. > > That's not an obvious answer, and there are several > factors that contribute to it being potentially confusing: > - None of the PDU diagrams show the Next-Qualifiers > - Byte 0 means different things - at the top of > Section 2.2, it's the start of the Next-Qualifier, > but by Section 2.2.3, it's shifted down 4 bytes > to be the start of the Basic Header Segment. > - Nothing obvious seems to say that the Text or the > Ping Data is a "data segment". The latter > might be obvious, but the former isn't. > - The Next-Qualifier structure is unexpected to > those familiar with other protocols. A > common protocol structure is T/L/V (Type/ > Length/Value). Because the BHS type and > length are implicit, the iSCSI structure > is actually Type N+1/Length N+1/Value N. > So, much as I realize the busy work involved in > revising ASCII graphics, I think all the PDU diagrams > in Section 2 need to be revised to show the Next- > Qualifiers. This would also take care of the second > item above, because then byte 0 in all the PDU > diagrams will be the first byte of the Next-Qualifier > that precedes the Basic Header Segment. Addressing > the third item involves a few sentences in to > explain what the "data segment" indicated by the > Next Qualifier before the BHS actually covers/ > contains in those cases. > > The fourth item is a matter of taste. It would > certainly be cleaner if iSCSI used a TLV structure, > rather than a T+1/L+1/V structure. Much of the > network community is familiar with TLV and will > find the T+1/L+1/V structure confusing. Are the > 4 bytes saved by not having a descriptor for the > BHS really worth it? > > On a related topic, it would be considerably > cleaner if the type (WN) field were a single byte > that was completely enumerated, rather than the > current bit field structure, and if the > peculiar convention for the AddCDB length in > 2.2.2.1 were dropped in favor of the 3-byte > length field in 2.2.2.3 to remove a case > from the header parse logic. > > Mindful of my WG co-chair role, I really think > the diagram revisions and explanations of what > a "data segment" is need to be done. OTOH, > the above two paragraphs on whether to have a BHS > descriptor and how to format the Next-Qualifiers > are suggestions for further discussion. > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Mar 5 22:13:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA16103 for ; Mon, 5 Mar 2001 22:13:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f261CSG02364 for ips-outgoing; Mon, 5 Mar 2001 20:12:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f261Bmr02341 for ; Mon, 5 Mar 2001 20:11:49 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id CAA163018 for ; Tue, 6 Mar 2001 02:11:40 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id CAA106352 for ; Tue, 6 Mar 2001 02:10:45 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A07.00066985 ; Tue, 6 Mar 2001 02:10:02 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 6 Mar 2001 02:34:28 +0200 Subject: RE: iSCSI: Next-Qualifiers and PDU diagrams Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, It really does not matter where the TLV is - before or after the BHS or any AHS. What matters is that you read it always with the BHS (or any subsequent AHS) and from it you derive what comes next. The simplest structure is only a BHS (no data) and even there you need the NQ to know that the data is not there. On regular TLV the sequence of operations you do is: Read TLV1 Read Variable part Read TLV2 ---- At best you can optimize it to: Read TLV1 Read Variable+TLV2 Read Variable+TLV3 ------ With our scheme - since we know that the first part BHS is always there we can read ReadBHS+TLV2 Readvariable+TLV3 ----------- There is no TLV1! I've put the NQ somewhat arbitrarily at the beginning - it could be at the end - but that is a matter of taste. I could add it to the PDU descriptions as an opaque field so that you can see your offsets at their final position (or almost final). I would like as much as you to add clarity to the document without affecting the structure of the header sequence (that is both efficient and flexible) and if adding the field to the PDU description makes it easier to understand then so be it - but let us hear some other voices too. Black_David@emc.com on 05/03/2001 16:01:04 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI: Next-Qualifiers and PDU diagrams > I will try to lessen confusion somehow - I am not sure that adding the NQ > everywhere is the answer. That'd be the most comprehensive thing to do, and I still think it's the right thing for clarity. At an absolute minimum, using one convention for which byte in a PDU is byte 0 is clearly called for. > The trouble with TLV formats is that they need at least two passes. I don't understand what you mean by "two passes", but see below because it may not be an issue. > With the NQ I use the fact that the basic header is always there > and the qualifier will be needed do describe the next segment (its type and length). Requiring that the first TLV always be the basic header segment whose size never changes is ok under a TLV approach. The differences between what we have now, and a TLV approach are: (1) The information currently in the NQ before the BHS moves to the TL after the BHS. It's still in a fixed location because the BHS is fixed size. (2) We spend another 4 bytes for the TL to describe the BHS. Now that I think about things a bit more, this initial TL could be omitted, with the TLV structure used only to describe what comes after the BHS. From a parse standpoint, the first TL after the BHS can be parsed with the BHS because that TL is always in the same place wrt the BHS. Is there a lot of value to having the TL for what comes after the BHS show up on the wire before the BHS? I'm sceptical. Is there any good reason not to merge the two formats for describing variable length segments? --David > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Saturday, March 03, 2001 12:03 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Next-Qualifiers and PDU diagrams > > > > David, > > I will try to lessen confusion somehow - I am not sure that adding the NQ > everywhere is the answer. > > The trouble with TLV formats is that they need at least two passes. With > the NQ I use the fact that > the basic header is always there and the qualifier will be needed do > describe the next segment (its type and length). > > Julo > > Black_David@emc.com on 02/03/2001 21:23:10 > > Please respond to Black_David@emc.com > > To: ips@ece.cmu.edu > cc: > Subject: iSCSI: Next-Qualifiers and PDU diagrams > > > > > In working with some other folks, I stumbled across > a format issue that's more confusing than it needs > to be in the current iSCSI draft. The issue is that > Next-Qualifiers (WN, etc.) and the PDUs they go with > are not documented/diagrammed together, making it > hard to figure out how to parse iSCSI PDUs. > > For an example of how this causes confusion, consider > the Text, Login and NOP PDU diagrams in Sections 2.8 > through 2.13. Starting from one of these diagrams, > how does one figure out the size of the Text or > Ping data that's appended? > > The answer's not in the diagrams -- there's a 4-byte > Next Qualifier preceding the PDU shown in each diagram. > The Text or Ping data is considered to be a data > segment, and hence the format of the Next Qualifier > is that given in Section 2.2.2.3, including a 3-byte > length field that answers the question. > > That's not an obvious answer, and there are several > factors that contribute to it being potentially confusing: > - None of the PDU diagrams show the Next-Qualifiers > - Byte 0 means different things - at the top of > Section 2.2, it's the start of the Next-Qualifier, > but by Section 2.2.3, it's shifted down 4 bytes > to be the start of the Basic Header Segment. > - Nothing obvious seems to say that the Text or the > Ping Data is a "data segment". The latter > might be obvious, but the former isn't. > - The Next-Qualifier structure is unexpected to > those familiar with other protocols. A > common protocol structure is T/L/V (Type/ > Length/Value). Because the BHS type and > length are implicit, the iSCSI structure > is actually Type N+1/Length N+1/Value N. > So, much as I realize the busy work involved in > revising ASCII graphics, I think all the PDU diagrams > in Section 2 need to be revised to show the Next- > Qualifiers. This would also take care of the second > item above, because then byte 0 in all the PDU > diagrams will be the first byte of the Next-Qualifier > that precedes the Basic Header Segment. Addressing > the third item involves a few sentences in to > explain what the "data segment" indicated by the > Next Qualifier before the BHS actually covers/ > contains in those cases. > > The fourth item is a matter of taste. It would > certainly be cleaner if iSCSI used a TLV structure, > rather than a T+1/L+1/V structure. Much of the > network community is familiar with TLV and will > find the T+1/L+1/V structure confusing. Are the > 4 bytes saved by not having a descriptor for the > BHS really worth it? > > On a related topic, it would be considerably > cleaner if the type (WN) field were a single byte > that was completely enumerated, rather than the > current bit field structure, and if the > peculiar convention for the AddCDB length in > 2.2.2.1 were dropped in favor of the 3-byte > length field in 2.2.2.3 to remove a case > from the header parse logic. > > Mindful of my WG co-chair role, I really think > the diagram revisions and explanations of what > a "data segment" is need to be done. OTOH, > the above two paragraphs on whether to have a BHS > descriptor and how to format the Next-Qualifiers > are suggestions for further discussion. > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > > From owner-ips@ece.cmu.edu Mon Mar 5 23:48:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA18687 for ; Mon, 5 Mar 2001 23:48:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f262fXK06672 for ips-outgoing; Mon, 5 Mar 2001 21:41:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f262ekr06646 for ; Mon, 5 Mar 2001 21:40:46 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id DAA183630 for ; Tue, 6 Mar 2001 03:40:39 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id DAA35006 for ; Tue, 6 Mar 2001 03:39:43 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A07.000E9131 ; Tue, 6 Mar 2001 03:39:06 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 6 Mar 2001 03:50:39 +0200 Subject: Re: iSCSI Security Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk ---------------------- Forwarded by Julian Satran/Haifa/IBM on 06/03/2001 03:50 --------------------------- Julian Satran 06/03/2001 03:49 To: Joshua Tseng cc: From: Julian Satran/Haifa/IBM@IBMIL Subject: Re: iSCSI Security (Document link: Julian Satran - Mail) Josh, We don't want to deal with any of the authentication schemes on which we have to keep inventing things and interfaces. Kerberos and SRP have everything needed, including being implemented on widely available platforms, and beyond them IPSec handles everything. Obviously vendors can add anything (including public key). Regards, Julo Joshua Tseng on 05/03/2001 20:53:05 Please respond to Joshua Tseng To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: iSCSI Security Julian, Why was the public key authentication method removed from version -05? Are you sure you want iSCSI to forsake the benefits of public key cryptography? I strongly suggest it be reinstated as one of the authentication methods listed in page 95. Josh From owner-ips@ece.cmu.edu Mon Mar 5 23:49:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA18698 for ; Mon, 5 Mar 2001 23:49:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f262fdk06683 for ips-outgoing; Mon, 5 Mar 2001 21:41:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f262ekr06643 for ; Mon, 5 Mar 2001 21:40:46 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id DAA66372 for ; Tue, 6 Mar 2001 03:40:39 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id DAA35010 for ; Tue, 6 Mar 2001 03:39:44 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A07.000E8E9F ; Tue, 6 Mar 2001 03:39:00 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 6 Mar 2001 03:40:57 +0200 Subject: Re: R2TDataSN Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, 1.The only consensus I heard is not to transfer a large amount of data with one PDU. 2.With DatasN and Sack we dont need any data in a bad header. 3. If an R2T is lost (received at initiator with bad digest) - the initiator will know that from the next R2T if the target has several outstanding - very likely at long distances - and will not have to way for a timeout. Other uses are marginal. Basically it is "part of a command execution" and we can painless recover from failures for this case too. Regards, Julo "Somesh Gupta" on 05/03/2001 20:40:06 Please respond to someshg@yahoo.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: R2TDataSN > R2TDataSN > ---------- > Sec 6.7.1 has some new content on how to handle lost R2Ts using > SACKs. But I noticed that the SACK request (Sec 2.16) has not > changed to indicate whether the DataSN is a R2T DataSN or just > a Read PDU DataSN (D bit) > So do we demux on the read/write operation type? > And how does this affect PDU loss in bidirectional commands ? > +++ SACK is ascking for data (DataSN) the target knows > Julian, Regarding the R2TDataSN, I have a comments and a question. I think that when a PDU header fails a CRC/checksum check etc, it is a problem to depend on any information in the header (including length fields), thereby making any further processing on the connection unreliable. What scenarios do you envision where the R2TDataSN is useful. IN Orlando I think there was clear consensus that application do not try to transfer very large amounts of data using a single command. Thanks, Somesh _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Mon Mar 5 23:50:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA18721 for ; Mon, 5 Mar 2001 23:50:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f262faY06678 for ips-outgoing; Mon, 5 Mar 2001 21:41:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f262eur06653 for ; Mon, 5 Mar 2001 21:40:56 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id DAA03388; Tue, 6 Mar 2001 03:40:48 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id DAA79380; Tue, 6 Mar 2001 03:39:15 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A07.000E9439 ; Tue, 6 Mar 2001 03:39:14 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: pat_thaler@agilent.com cc: ips@ece.cmu.edu, Dafna_Sheinwald@il.ibm.com Message-ID: Date: Tue, 6 Mar 2001 04:21:59 +0200 Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Pat, I did not run a check. That is very expensive. With CRCs there are methods that enable you to run a check on the complement code (that has only 2**32 different patterns for any block length) and derive from there the distances (Fujiwara has done this in 1989 for the IEEE CRC-32 and about some more recent experiments I'll get back to the list). And that is the trouble with this whole line of argument. There are no numbers to prove Adler32 or Fletcher32 and there are plenty for CRCs. The big question is then is there anybody out there that wants to build a modern bridge based only on its beauty? Regards, Julo pat_thaler@agilent.com on 05/03/2001 23:27:49 Please respond to pat_thaler@agilent.com To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Julian, I know that Hamming distance gets down to 2 for errors that are separated by the modulus (or a multiple of it). Is there another case? Pat > - Adler and Fletcher are weak and there is no theory behind your > distribution statements, nor any simulation results as far as I know. We > found that on very simple sequences the Hamming distance gets > down to 2 (or > lower) and the burst protection is probably not better than 16 bit. There > is even a simple formula for what sequences will get you false codes (see > bellow for a reference) > From owner-ips@ece.cmu.edu Tue Mar 6 01:05:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA19325 for ; Tue, 6 Mar 2001 01:05:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f264fWe12074 for ips-outgoing; Mon, 5 Mar 2001 23:41:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f264fQr12065 for ; Mon, 5 Mar 2001 23:41:27 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f265oj040891; Mon, 5 Mar 2001 21:50:45 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: , Cc: , Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Date: Mon, 5 Mar 2001 20:39:32 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, As there are already selectors in place for checking schemes, what reason is there for excluding Adler-32? It is not code size, speed of execution nor difficulty in hardware implementation. Excluding this alternative ensures potential loss of a competitive software solution. You wish to build a bridge to carry trucks, but there is a height barrier that prevents such trucks. Is a serial burst error high on the list of probable errors if CRC checks prevent these already? Doug > Pat, > > I did not run a check. That is very expensive. > With CRCs there are methods that enable you to run a check on the > complement code > (that has only 2**32 different patterns for any block length) and derive > from there the distances (Fujiwara has done this in 1989 for the IEEE > CRC-32 and about some more recent experiments I'll get back to the list). > > And that is the trouble with this whole line of argument. > There are no numbers to prove Adler32 or Fletcher32 and there are plenty > for CRCs. > > The big question is then is there anybody out there that wants to build a > modern bridge based only on its beauty? > > Regards, > Julo > > > pat_thaler@agilent.com on 05/03/2001 23:27:49 > > Please respond to pat_thaler@agilent.com > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt > > > > > Julian, > > I know that Hamming distance gets down to 2 for errors that are separated > by > the modulus (or a multiple of it). Is there another case? > > Pat > > > - Adler and Fletcher are weak and there is no theory behind your > > distribution statements, nor any simulation results as far as I > know. We > > found that on very simple sequences the Hamming distance gets > > down to 2 (or > > lower) and the burst protection is probably not better than 16 bit. > There > > is even a simple formula for what sequences will get you false > codes (see > > bellow for a reference) > > > > > > From owner-ips@ece.cmu.edu Tue Mar 6 01:05:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA19374 for ; Tue, 6 Mar 2001 01:05:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2644aC10490 for ips-outgoing; Mon, 5 Mar 2001 23:04:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f260QZr00012 for ; Mon, 5 Mar 2001 19:26:35 -0500 (EST) Received: from phys-ha3mpkb.Eng.Sun.COM ([129.146.44.31]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id QAA23791; Mon, 5 Mar 2001 16:26:34 -0800 (PST) Received: from sonu (sonu [129.146.48.127]) by phys-ha3mpkb.Eng.Sun.COM (8.8.8+Sun/8.8.8/ENSMAIL,v2.0) with SMTP id QAA24781; Mon, 5 Mar 2001 16:26:33 -0800 (PST) Message-Id: <200103060026.QAA24781@phys-ha3mpkb.Eng.Sun.COM> Date: Mon, 5 Mar 2001 16:27:48 -0800 (PST) From: "Tanjore K. Suresh" Reply-To: "Tanjore K. Suresh" Subject: iSCSI: Naming and Discovery Draft... To: kaladhar@us.ibm.com Cc: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: b/UMZ0+7zn3GaCfh4Jql6A== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk Voruganti, Here are some minor comments/feedback reading this document. 1. Section 5.2 Discovery Domain DD Symbolic name DD ID DD Member: WWUI DD Member: IP Address What does this DD member signify, is this one iSCSI device part of Discovery Domain or should this be a list of iSCSI devices in that Discovery Domain. I thought clarifying that would be usefull for the reader. Frankly, I didnot understand what it is? 2. Appendix A iSCSI WWUI Notes... It looks to me the examples under "Using Intitiator and Target WWUI During Login" can be clubbed with Appendix B which provides the different iSCSI Login Scenarios" It looks like some of the scenarios presented are also in the Appendix B already which seems to convey the same thing. 3. Appendix B, B.4.5, Target Conflict 45 doesnot seem to be appropriate. I have not reviewed all the documents yet to give a recommendation and hence cannot give, but feel " Target Conflict" doesnot convey the meaning of the Scenario indicating case of " simple devices that can handle one device or the target had reached the limit of its Initiators' capacity." 4. Typo in page 17 Directory Agent(DA) is an optional part of discovery. If a DA.. ......... by the user Agents, to reduce the network load of all UAs trying to discovery all SAs ^^^^^^^^ should be "discover". Thanks sureshtk From owner-ips@ece.cmu.edu Tue Mar 6 04:21:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA05712 for ; Tue, 6 Mar 2001 04:21:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f266uZv18301 for ips-outgoing; Tue, 6 Mar 2001 01:56:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f266uVr18293 for ; Tue, 6 Mar 2001 01:56:31 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Mon, 5 Mar 2001 22:56:24 -0800 Message-ID: From: Joshua Tseng To: ips@ece.cmu.edu Subject: RE: iSCSI Security Date: Mon, 5 Mar 2001 22:56:15 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, See below: > > Josh, > > We don't want to deal with any of the authentication schemes > on which we > have to keep inventing things and interfaces. The public key algorithms are already well documented. I don't think we're inventing anything new. See RFC 2437 for RSA and FIPS-186-2 for DSA. All we need are some text keys to carry the verification signatures. > > Kerberos and SRP have everything needed, including being > implemented on > widely available platforms, and beyond them IPSec handles everything. Many consider Kerberos to be less than secure. It is yesterday's technology, and it does not scale well, since it requires manual distribution and coordination of shared secrets between the server and its users. Consequently, the kerberos server is a headache to set up and maintain, especially for a large number of clients. Furthermore, it is also a single point of vulnerability, in contrast to a PKI infrastructure which can rely upon hierarchies of certificate authorities. IPSec is optimized to secure IP endpoints. It will not verify identities (i.e., WWUI's) unless you implement ISAKMP's optional features, which may be problematic if you're using an off-the-shelf ISAKMP implementation. > > Obviously vendors can add anything (including public key). If you do not add the text keys to negotiate public key authentication, then there will be no public key method. I don't think I'm asking for very much--just that you reinstate the previous public key method from the previous draft. This will make key distribution MUCH easier, safer, and scalable. Regards, Josh > > Regards, > Julo > > Joshua Tseng on 05/03/2001 20:53:05 > > Please respond to Joshua Tseng > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: iSCSI Security > > > > > Julian, > > Why was the public key authentication method removed from version -05? > Are you sure you want iSCSI to forsake the benefits of public key > cryptography? I strongly suggest it be reinstated as one of the > authentication > methods listed in page 95. > > Josh > > > > > From owner-ips@ece.cmu.edu Tue Mar 6 12:46:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA19271 for ; Tue, 6 Mar 2001 12:46:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f26FPqM21743 for ips-outgoing; Tue, 6 Mar 2001 10:25:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f26FP5r21641 for ; Tue, 6 Mar 2001 10:25:08 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAA157626 for ; Tue, 6 Mar 2001 16:24:58 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA19842 for ; Tue, 6 Mar 2001 16:23:22 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A07.005488C2 ; Tue, 6 Mar 2001 16:23:20 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 6 Mar 2001 13:00:57 +0200 Subject: RE: iSCSI Security Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Josh, We will consider it. Let's discuss this in Minneapolis. Julo Joshua Tseng on 06/03/2001 08:56:15 Please respond to Joshua Tseng To: ips@ece.cmu.edu cc: Subject: RE: iSCSI Security Julian, See below: > > Josh, > > We don't want to deal with any of the authentication schemes > on which we > have to keep inventing things and interfaces. The public key algorithms are already well documented. I don't think we're inventing anything new. See RFC 2437 for RSA and FIPS-186-2 for DSA. All we need are some text keys to carry the verification signatures. > > Kerberos and SRP have everything needed, including being > implemented on > widely available platforms, and beyond them IPSec handles everything. Many consider Kerberos to be less than secure. It is yesterday's technology, and it does not scale well, since it requires manual distribution and coordination of shared secrets between the server and its users. Consequently, the kerberos server is a headache to set up and maintain, especially for a large number of clients. Furthermore, it is also a single point of vulnerability, in contrast to a PKI infrastructure which can rely upon hierarchies of certificate authorities. IPSec is optimized to secure IP endpoints. It will not verify identities (i.e., WWUI's) unless you implement ISAKMP's optional features, which may be problematic if you're using an off-the-shelf ISAKMP implementation. > > Obviously vendors can add anything (including public key). If you do not add the text keys to negotiate public key authentication, then there will be no public key method. I don't think I'm asking for very much--just that you reinstate the previous public key method from the previous draft. This will make key distribution MUCH easier, safer, and scalable. Regards, Josh > > Regards, > Julo > > Joshua Tseng on 05/03/2001 20:53:05 > > Please respond to Joshua Tseng > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: iSCSI Security > > > > > Julian, > > Why was the public key authentication method removed from version -05? > Are you sure you want iSCSI to forsake the benefits of public key > cryptography? I strongly suggest it be reinstated as one of the > authentication > methods listed in page 95. > > Josh > > > > > From owner-ips@ece.cmu.edu Tue Mar 6 12:46:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA19283 for ; Tue, 6 Mar 2001 12:46:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f26Fl9823136 for ips-outgoing; Tue, 6 Mar 2001 10:47:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp014.mail.yahoo.com (smtp014.mail.yahoo.com [216.136.173.58]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f26FkLr23074 for ; Tue, 6 Mar 2001 10:46:21 -0500 (EST) Received: from unknown (HELO somesh) (206.112.105.84) by smtp.mail.vip.sc5.yahoo.com with SMTP; 6 Mar 2001 15:27:18 -0000 X-Apparently-From: Reply-To: From: "Somesh Gupta" To: , Subject: RE: R2TDataSN Date: Tue, 6 Mar 2001 07:23:04 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I beg to disagree. If an R2T PDU (header) has bad digest, or any other header has a bad digest - since you always need the PDU length from the header, there is some uncertainty associated with further processing. Are you proposing that the processing machine go into a "speculative" mode where the processing of the next PDU determines whether we were successfuly able to skip a bad PDU header? When there is a data digest error, further stream parsing is deterministic. But not when the PDU header digest error. Also the consensus (in my interpretation) was on applications not transfering very large amounts of data using a single command or read/write PDU. > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Monday, March 05, 2001 5:41 PM > To: ips@ece.cmu.edu > Subject: Re: R2TDataSN > > > > > Somesh, > > 1.The only consensus I heard is not to transfer a large amount of > data with > one PDU. > > 2.With DatasN and Sack we dont need any data in a bad header. > > 3. If an R2T is lost (received at initiator with bad digest) - the > initiator will know that from > the next R2T if the target has several outstanding - very likely at long > distances - and will not have to way for a timeout. Other uses are > marginal. Basically it is "part of a command execution" and we can > painless recover > from failures for this case too. > > Regards, > Julo > > "Somesh Gupta" on 05/03/2001 20:40:06 > > Please respond to someshg@yahoo.com > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: R2TDataSN > > > > > > R2TDataSN > > ---------- > > Sec 6.7.1 has some new content on how to handle lost R2Ts using > > SACKs. But I noticed that the SACK request (Sec 2.16) has not > > changed to indicate whether the DataSN is a R2T DataSN or just > > a Read PDU DataSN (D bit) > > So do we demux on the read/write operation type? > > And how does this affect PDU loss in bidirectional commands ? > > +++ SACK is ascking for data (DataSN) the target knows > > > > Julian, > > Regarding the R2TDataSN, I have a comments and a > question. > > I think that when a PDU header fails a CRC/checksum check etc, > it is a problem to depend on any information in the header (including > length fields), thereby making any further processing on > the connection unreliable. > > What scenarios do you envision where the R2TDataSN is useful. > IN Orlando I think there was clear consensus that application > do not try to transfer very large amounts of data using a > single command. > > Thanks, > Somesh > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com > > > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Tue Mar 6 12:47:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA19308 for ; Tue, 6 Mar 2001 12:47:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f26FLpg21497 for ips-outgoing; Tue, 6 Mar 2001 10:21:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f26FLfr21447 for ; Tue, 6 Mar 2001 10:21:41 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id KAA15845 for ; Tue, 6 Mar 2001 10:21:35 -0500 (EST) Message-ID: <3AA50096.8D180B12@cisco.com> Date: Tue, 06 Mar 2001 09:21:58 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: IPS Subject: iSCSI MIB Draft 02 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit A new version of the iSCSI MIB has been submitted as draft-bakke-iscsimib-02.txt. It is now available for review at on Julian's iSCSI archive site. If you are using a dial-up line, be forewarned that it is a 250 kbyte file. http://www.haifa.il.ibm.com/satran/ips/draft-bakke-iscsimib-02.txt This is the exact draft that was submitted; there are ongoing changes being made to the MIB by the team. We will submit another draft to this list in the near future. For those who would like to see the table structure, before diving into the entire MIB, a drawing of the MIB table structure is available at http://www.haifa.il.ibm.com/satran/ips/iscsi-mib-structure-02.pdf A UML model will also be available to show the relationships between the tables at a more abstract level. I will make this available shortly. -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Tue Mar 6 15:09:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA25716 for ; Tue, 6 Mar 2001 15:09:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f26Ig8K04742 for ips-outgoing; Tue, 6 Mar 2001 13:42:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f26IfOr04668 for ; Tue, 6 Mar 2001 13:41:24 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id NAA12625; Tue, 6 Mar 2001 13:40:55 -0500 (EST) Message-ID: <3AA52F4E.51D32D23@cisco.com> Date: Tue, 06 Mar 2001 12:41:18 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: "Renato E. Maranon" CC: Kaladhar Voruganti , ips@ece.cmu.edu Subject: Re: iSCSI: Naming and Discovery Draft References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Renato- You are correct; we had a duplicate example with two descriptions. We'll merge them. Thanks, -- Mark "Renato E. Maranon" wrote: > > Some minor feedback. > > Appendix A is preceded by the text "8. ". "8. " Should be removed? > > Second. > In Appendix B, B.4.5 Login fail, the description for the sample login for > target removed (exception "Target removed 44") is split by the "Target > Conflict 45" sample. Should probably be combined. > > Original text: > I->Login Request > InitiatorWWUI= iscsi.com.os.hostid.34567890 > TargetWWUI= iscsi.com.acme.diskarray.sn.8675309 > T->Login Response ("Target removed 44", F set) > > In this case the target has been removed. In contrast with codes 31 > and 32 (in B.4.4), no redirection information is supplied. > > I->Login Request > InitiatorWWUI= iscsi.com.os.hostid.34567890 > TargetWWUI= iscsi.com.acme.diskarray.sn.8675309 > T->Login Response ("Target Conflict 45", F set) > > Here, the target is busy with another initiator and cannot handle > another one. The initiator MAY try again later. This can be the case > of simple devices that can handle one device or the target has > reached the limit of its initiators' capacity. In contrast to the > previous examples, this rejection is temporary. > > I->Login Request > InitiatorWWUI= iscsi.com.os.hostid.34567890 > TargetWWUI= iscsi.com.acme.diskarray.sn.8675309 > T->Login Response ("Target removed 44", F set) > > Here, the target has been removed. The initiator SHOULD terminate the > login session. It MAY query the SNS for the new location of the > target. (This should apply for the case when the target was not found > - code 44). > > Suggestion: > > I->Login Request > InitiatorWWUI= iscsi.com.os.hostid.34567890 > TargetWWUI= iscsi.com.acme.diskarray.sn.8675309 > T->Login Response ("Target removed 44", F set) > > In this case the target has been removed. In contrast with codes 31 > and 32 (in B.4.4), no redirection information is supplied. > > Here, the target has been removed. The initiator SHOULD terminate the > login session. It MAY query the SNS for the new location of the > target. (This should apply for the case when the target was not found > - code 44). > > I->Login Request > InitiatorWWUI= iscsi.com.os.hostid.34567890 > TargetWWUI= iscsi.com.acme.diskarray.sn.8675309 > T->Login Response ("Target Conflict 45", F set) > > Here, the target is busy with another initiator and cannot handle > another one. The initiator MAY try again later. This can be the case > of simple devices that can handle one device or the target has > reached the limit of its initiators' capacity. In contrast to the > previous examples, this rejection is temporary. > > Renato Maranon > Maranti Networks, Inc > 920 Hillview Court > Milpitas, Ca 95035 > Phone: 408-719-9600 x309 > Fax: 408-719-9631 > email: rmaranon@marantinetworks.com > home: www.marantinetworks.com > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Kaladhar Voruganti > Sent: Saturday, February 24, 2001 9:51 AM > To: ips@ece.cmu.edu > Subject: iSCSI: Naming and Discovery Draft > > The following iSCSI naming and discovery draft has been submitted to IETF. > This naming and discovery draft has been produced by the iSCSI naming and > discovery team. > > The draft contains the iSCSI naming and discovery details. > > http://www.haifa.il.ibm.com/satran/ips/draft-ietf-ips-iSCSI-name-disc-00.txt > > Kaladhar Voruganti > Computing Science Department > IBM Almaden Research Lab > San Jose, CA -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Tue Mar 6 15:09:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA25748 for ; Tue, 6 Mar 2001 15:09:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f26I3rZ02175 for ips-outgoing; Tue, 6 Mar 2001 13:03:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antigonus.hosting.pacbell.net (antigonus.hosting.pacbell.net [216.100.98.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f26I3Dr02147 for ; Tue, 6 Mar 2001 13:03:14 -0500 (EST) Received: from VENKAT1 (64-160-62-200.rhapsodynetworks.com [64.160.62.200] (may be forged)) by antigonus.hosting.pacbell.net id MAA04360; Tue, 6 Mar 2001 12:57:05 -0500 (EST) [ConcentricHost SMTP Relay 1.7] From: "Venkat Rangan" To: , , Subject: RE: R2TDataSN Date: Tue, 6 Mar 2001 10:00:09 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Somesh, > When there is a data digest error, further stream parsing is > deterministic. But not when the PDU header digest error. Isn't it true that with Synch and Steering layer, you are able to skip past PDUs with header-digest errors and reach a point where you can begin deterministically processing the stream? (assuming that the headers used for Synch and Steering are in-tact). Of course, in the event that this option is "negotiated out" you don't have this ability. Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Somesh Gupta Sent: Tuesday, March 06, 2001 7:23 AM To: julian_satran@il.ibm.com; ips@ece.cmu.edu Subject: RE: R2TDataSN I beg to disagree. If an R2T PDU (header) has bad digest, or any other header has a bad digest - since you always need the PDU length from the header, there is some uncertainty associated with further processing. Are you proposing that the processing machine go into a "speculative" mode where the processing of the next PDU determines whether we were successfuly able to skip a bad PDU header? When there is a data digest error, further stream parsing is deterministic. But not when the PDU header digest error. Also the consensus (in my interpretation) was on applications not transfering very large amounts of data using a single command or read/write PDU. > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Monday, March 05, 2001 5:41 PM > To: ips@ece.cmu.edu > Subject: Re: R2TDataSN > > > > > Somesh, > > 1.The only consensus I heard is not to transfer a large amount of > data with > one PDU. > > 2.With DatasN and Sack we dont need any data in a bad header. > > 3. If an R2T is lost (received at initiator with bad digest) - the > initiator will know that from > the next R2T if the target has several outstanding - very likely at long > distances - and will not have to way for a timeout. Other uses are > marginal. Basically it is "part of a command execution" and we can > painless recover > from failures for this case too. > > Regards, > Julo > > "Somesh Gupta" on 05/03/2001 20:40:06 > > Please respond to someshg@yahoo.com > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: R2TDataSN > > > > > > R2TDataSN > > ---------- > > Sec 6.7.1 has some new content on how to handle lost R2Ts using > > SACKs. But I noticed that the SACK request (Sec 2.16) has not > > changed to indicate whether the DataSN is a R2T DataSN or just > > a Read PDU DataSN (D bit) > > So do we demux on the read/write operation type? > > And how does this affect PDU loss in bidirectional commands ? > > +++ SACK is ascking for data (DataSN) the target knows > > > > Julian, > > Regarding the R2TDataSN, I have a comments and a > question. > > I think that when a PDU header fails a CRC/checksum check etc, > it is a problem to depend on any information in the header (including > length fields), thereby making any further processing on > the connection unreliable. > > What scenarios do you envision where the R2TDataSN is useful. > IN Orlando I think there was clear consensus that application > do not try to transfer very large amounts of data using a > single command. > > Thanks, > Somesh > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com > > > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Tue Mar 6 15:09:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA25799 for ; Tue, 6 Mar 2001 15:09:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f26IStG03780 for ips-outgoing; Tue, 6 Mar 2001 13:28:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp015.mail.yahoo.com (smtp015.mail.yahoo.com [216.136.173.59]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f26IRCr03691 for ; Tue, 6 Mar 2001 13:27:12 -0500 (EST) Received: from unknown (HELO somesh) (206.112.105.186) by smtp.mail.vip.sc5.yahoo.com with SMTP; 6 Mar 2001 18:25:22 -0000 X-Apparently-From: Reply-To: From: "Somesh Gupta" To: "Venkat Rangan" , , , Subject: RE: R2TDataSN and other recovery mechanisms Date: Tue, 6 Mar 2001 10:20:56 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Venkat, You are right but this does put some additional requirements on the sync and steering layer (assuming we can agree on one and that it is not optional). Consider the marker mechanism as an example. If the marker falls within the TCP byte stream region which contains the header, and the header digest fails, then do you trust this marker or skip to the next marker. Or do we need a sum on the marker itself. We do have various recovery mechanisms in the protocol without detailed state machines or other documentation to make sure that the mechanisms are really robust and worth including. And even more important, that we will have interoperable implementations that are implemented to the standard (rather than being compliant with some defacto implementation) I am not saying that these tools do not work, and Julian (and other smart people)may even have worked them out. However, the proof is not in the document. Thanks, Somesh > -----Original Message----- > From: Venkat Rangan [mailto:venkat@rhapsodynetworks.com] > Sent: Tuesday, March 06, 2001 10:00 AM > To: someshg@yahoo.com; julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: RE: R2TDataSN > > > Somesh, > > > When there is a data digest error, further stream parsing is > > deterministic. But not when the PDU header digest error. > > Isn't it true that with Synch and Steering layer, you are able to > skip past > PDUs with header-digest errors and reach a point where you can begin > deterministically processing the stream? (assuming that the > headers used for > Synch and Steering are in-tact). > > Of course, in the event that this option is "negotiated out" you > don't have > this ability. > > Venkat Rangan > Rhapsody Networks Inc. > http://www.rhapsodynetworks.com > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Somesh Gupta > Sent: Tuesday, March 06, 2001 7:23 AM > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: RE: R2TDataSN > > > I beg to disagree. If an R2T PDU (header) has bad digest, or any other > header has a bad digest - since you always need the PDU length from > the header, there is some uncertainty associated with further processing. > > Are you proposing that the processing machine go into a "speculative" > mode where the processing of the next PDU determines whether we were > successfuly able to skip a bad PDU header? When there is a data digest > error, further stream parsing is deterministic. But not when the PDU > header digest error. > > Also the consensus (in my interpretation) was on applications > not transfering very large amounts of data using a single command or > read/write PDU. > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Monday, March 05, 2001 5:41 PM > > To: ips@ece.cmu.edu > > Subject: Re: R2TDataSN > > > > > > > > > > Somesh, > > > > 1.The only consensus I heard is not to transfer a large amount of > > data with > > one PDU. > > > > 2.With DatasN and Sack we dont need any data in a bad header. > > > > 3. If an R2T is lost (received at initiator with bad digest) - the > > initiator will know that from > > the next R2T if the target has several outstanding - very likely at long > > distances - and will not have to way for a timeout. Other uses are > > marginal. Basically it is "part of a command execution" and we can > > painless recover > > from failures for this case too. > > > > Regards, > > Julo > > > > "Somesh Gupta" on 05/03/2001 20:40:06 > > > > Please respond to someshg@yahoo.com > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: R2TDataSN > > > > > > > > > > > R2TDataSN > > > ---------- > > > Sec 6.7.1 has some new content on how to handle lost R2Ts using > > > SACKs. But I noticed that the SACK request (Sec 2.16) has not > > > changed to indicate whether the DataSN is a R2T DataSN or just > > > a Read PDU DataSN (D bit) > > > So do we demux on the read/write operation type? > > > And how does this affect PDU loss in bidirectional commands ? > > > +++ SACK is ascking for data (DataSN) the target knows > > > > > > > Julian, > > > > Regarding the R2TDataSN, I have a comments and a > > question. > > > > I think that when a PDU header fails a CRC/checksum check etc, > > it is a problem to depend on any information in the header (including > > length fields), thereby making any further processing on > > the connection unreliable. > > > > What scenarios do you envision where the R2TDataSN is useful. > > IN Orlando I think there was clear consensus that application > > do not try to transfer very large amounts of data using a > > single command. > > > > Thanks, > > Somesh > > > > _________________________________________________________ > > Do You Yahoo!? > > Get your free @yahoo.com address at http://mail.yahoo.com > > > > > > > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Tue Mar 6 15:10:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA25842 for ; Tue, 6 Mar 2001 15:10:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f26Ipuo05378 for ips-outgoing; Tue, 6 Mar 2001 13:51:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f26Ip4r05301 for ; Tue, 6 Mar 2001 13:51:05 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id NAA24762; Tue, 6 Mar 2001 13:50:56 -0500 (EST) Message-ID: <3AA531A7.E78CDDED@cisco.com> Date: Tue, 06 Mar 2001 12:51:19 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: "Tanjore K. Suresh" CC: kaladhar@us.ibm.com, ips@ece.cmu.edu Subject: Re: iSCSI: Naming and Discovery Draft... References: <200103060026.QAA24781@phys-ha3mpkb.Eng.Sun.COM> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Tanjore- Thanks for the feedback. I can comment on #3: "Tanjore K. Suresh" wrote: > 3. Appendix B, B.4.5, > Target Conflict 45 doesnot seem to be appropriate. > > I have not reviewed all the documents yet to give a > recommendation and hence cannot give, but feel > " Target Conflict" doesnot > convey the meaning of the Scenario indicating > case of " simple devices that can handle one device or > the target had reached the limit of its Initiators' capacity." Perhaps we chose the wrong term for this one. How about if call it "Target Busy", and slightly re-word it? The target is busy with another initiator and cannot handle another one. The initiator MAY try again later. This can be the case for simple devices that can handle only one initiator at a time, or for a target that has does not have the resources to support one more initiator. In contrast to the previous examples, this rejection is temporary. -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Tue Mar 6 15:11:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA25890 for ; Tue, 6 Mar 2001 15:11:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f26I10s02001 for ips-outgoing; Tue, 6 Mar 2001 13:01:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f26I0lr01968 for ; Tue, 6 Mar 2001 13:00:47 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id TAA114830 for ; Tue, 6 Mar 2001 19:00:41 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA138502 for ; Tue, 6 Mar 2001 18:59:44 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A07.0062CA77 ; Tue, 6 Mar 2001 18:59:03 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 6 Mar 2001 19:49:15 +0200 Subject: RE: iSCSI 05.txt is out Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I was afraid that that will be the answer -:). If nobody objects we will make the Target Task Tag use (alone or with LUN) an option at Login (LO) with the default being "alone". Regards, Julo "Venkat Rangan" on 06/03/2001 18:56:36 Please respond to "Venkat Rangan" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: iSCSI 05.txt is out Julian, Thanks for your responses. I've isolated a couple of your responses for clarity. +++ No need - length does not include padding +++ This is problematic because now digests will have to be run over a byte-stream and not a word-stream. My ASIC designer tells me this is a "bid deal". I assume digests cover only the length and not the padding. Again, Fibre Channel does it correctly, by having the length include the pad and the digest include the pad and the fill-bytes indicate to ULP any padding. +++We can make checking that optional although I don't understand what is all the fuss about it+++ The issue is that in some implementations, the target exchanges are maintained by context tables. Once the Command is processed, a context table entry is set up. There is no need to record the LUN in the context table. But the iSCSI requirement that the LUN in WRITE should match the LUN in Command requires the LUN also to be recorded. LUN is a large field, and in our implementation, we manage some 10,000 active exchanges. Adding 8-byte overhead just for catching an errant initiator seems wasteful. If the checking and reporting is Optional, i.e., change MUST to MAY, will ease this burden, and still keeps the mechanism available for targets that may make use of it. +++ even when received in order the initiator may decide to fulfill them out of order - this statement prevents it+++ I absolutely agree with your statement. But what I'm objecting to is the unnecessary phrase "Within a connection, " It should be removed. Regards, -venkat -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Monday, March 05, 2001 8:03 PM To: Venkat Rangan Subject: RE: iSCSI 05.txt is out Answers in text - +++ Thanks for the careful reading, Julo "Venkat Rangan" on 06/03/2001 00:40:18 Please respond to "Venkat Rangan" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI 05.txt is out Julian, Here are a few technical and editorial comments on the latest draft. Technical: Page 12: - CmdSN - the current Command Sequence Number advanced by 1 on each command shipped Add: skipping zero, which is reserved for immediate delivery. +++ will fix +++ Page 12: Remove the sentence: "A large difference between StatSN and ExpStatSN may indicate a failed connection." This may have made sense when StatSN and ExpStatSN are session-wide. Now that it is per-connection, this may not be valid. As was pointed out in the "Holes in StatSN" thread, ExpStatSN may be "stuck" at a low value when a single PDU encountered a Digest Error, followed by several well-formed PDUs. +++ if it cant't be recovered with a SACK it is stll a sign of error+++ Page 24 - Section 2.1 Add: The length of the PDU SHALL include any padding bytes added. +++ No the length does not include padding - padding is implicit +++ This raises a question as to whether it may be useful to include a two-bit "Fill Bytes" field in the header (BHS and AHS) which indicates the number of bytes that were added. Without this, it may be harder for the receiver to know how many bytes are part of the data. Fibre Channel has the Fill Byte specifier in its F_CTL part of the header. +++ No need - length does not include padding +++ Page 25: This may be covered in another thread. What does the first Next-Qualifier (at byte 0) describe? Is it the BHS or the header following the BHS? If a PDU has a single BHS and a single AHS, is the sequence WN-BHS-AHS or is it WN-BHS-WN-AHS? +++ in your example the second answer is correct and the second WN descibes the data payload +++ Page 25 and 26: The text next to the bit description and the headers in the text for sections 2.2.2.1 to 2.2.2.3 appear to be inconsistent. An example is "read-data transfer" vs. "Read Data" "Long Data" vs. "Long Data Transfer" Also, 2.2.2.3 appears to have text about Long Data Header that probably belongs in section 2.2.2.2 (or I am confused about how WN is supposed to work.) +++ will make consistent +++ Page 43 Section 2.7 The PDU diagrams should not show "Digests if any..." etc. It should be covered by NQ and corresponding AHS. +++ correct by tastes vary - and many people prefer a complete picture - I have a hard time now trying to find a way to reconcile all +++ Page 44 Section 2.7.2 The requirement of having to specify a valid LUN as part of WRITE Data (and NOP-Out) may pose unnecessary overhead for Target processing. The fact that Targets are now REQUIRED to reject errant initiators who may place a LUN inconsistent with the one sent in the CommandPdu means that targets should save the LUN against each context entry for the task. This was discussed earlier, and I think we said that unless the folks who originally wanted it spoke up, we will remove it. Note that Fibre Channel does not have this feature, and its original purpose can be accomplished by using Target Transfer Tag. If we still want it, targets should be given the option of negotiating this so that it can operate in a mode where a LUN specified as part of WRITE data will be ignored (as opposed to rejected). +++ Several people on this list reuested the freedom to build the tags at the device-server (part of the LU according to SAM) without regard to what other LUs the target has The simplest way to accomodate them is to have the initiator direct the dat through a LUN We can make checking that optional although I don't understand what is all the fuss about it+++ Section 2.17 The following text: "Within a connection, outstanding R2Ts MUST be fulfilled by the initiator in the order in which they were received." implies that R2Ts for the same WRITE command may be sent by the target on multiple connections? Since this is not possible, all R2Ts for a command are always on a single connection, so I am not sure how the above sentence should be interpreted. +++ even when received in order the initiator may decide to fulfill them out of order - this statement prevents it+++ Page 75: Add: "i.e., from Most Preferred to Least Preferred" to "reverse order of preference." +++ why? is there something ambiguous in the statement? ++ ------------ Editorial - typos and such 1. Page 11: Not covered are he means Change "he" to "the" +++ will fix ++ 2. Page 20: Change "later" to "latter" in "the later can be used in subsequent commands." +++ will fix ++ 3. Page 23: Change "isa" to "is a" +++ will fix ++ Page 42: Section 2.6.1 Change to: Initiator Task Tag of the task that was not found; used in conjuction with Response value 1. It MUST be set to zero in other cases. +++ will fix +++ Page 51: Change "CID does not change and this command is performs first" to "CID does not change and this command first performs" +++ will fix +++ Page 55: There is nothing that indicates the status codes in the table are sent as part of Status-Detail. +++ it says above the table +++ Page 66: Change "iSSCSI Data-out PDU" to "iSCSI Data (WRITE) PDU" The Data-out PDU is a new concept not introduced anywhere. +++ will fix +++ Page 75: The term "Security Context Complete" is referred to prior to its definition. Page 81: Change "later" to "latter" ++ will fix +++ ------------ Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com From owner-ips@ece.cmu.edu Tue Mar 6 15:11:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA25908 for ; Tue, 6 Mar 2001 15:11:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f26HSrO29872 for ips-outgoing; Tue, 6 Mar 2001 12:28:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f26HRpr29829 for ; Tue, 6 Mar 2001 12:27:51 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id B0F639BC; Tue, 6 Mar 2001 10:27:49 -0700 (MST) Received: from axandbh3.and.agilent.com (axandbh3.and.agilent.com [130.30.32.200]) by msgrel1.and.agilent.com (Postfix) with SMTP id ACAE41FB; Tue, 6 Mar 2001 12:24:21 -0500 (EST) Received: from 130.30.32.200 by axandbh3.and.agilent.com (InterScan E-Mail VirusWall NT); Tue, 06 Mar 2001 12:24:21 -0500 (Eastern Standard Time) Received: by axandbh3.and.agilent.com with Internet Mail Service (5.5.2653.19) id ; Tue, 6 Mar 2001 12:24:21 -0500 Message-ID: <1BEBA5E8600DD4119A50009027AF54A005165936@axcs04.cos.agilent.com> From: "THALER,PAT (A-Roseville,ex1)" To: julian_satran@il.ibm.com, "THALER,PAT (A-Roseville,ex1)" Cc: ips@ece.cmu.edu, Dafna_Sheinwald@il.ibm.com Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Date: Tue, 6 Mar 2001 12:24:13 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, It is relatively easy to investigate the Hamming distance and burst detection of Adler32 and Fletcher32 using mathematics. In the argument below, I will use "value" to identify the quantities that get summed since Adler sums bytes and Fletcher32 sums 16-bits. Both Adler and Fletcher calculate for each value: S1 = (S1 + value) mod m S2 = (S2 + S1) mod m where m is 65521 for Adler and 65535 for Fletcher. Which means that for an n value string of values V1 to Vn: S1 = (V1 + V2 + ... + Vn (+ 1 for Adler)) mod m S2 = (n * V1 + (n-1) * V2 + ... + Vn (+ n for Adler)) mod m A two bit error can only change two values. For a two bit error to be undetectable the two changed values must produce no change to S1 and S2. Let x be the change to the first errored value y be the change to the second errored value k be the separation between the first and second errored values. Then for an undetectable error: delta S1 = (x + y) mod m = 0 delta S2 = ((k + 1) * x + y) mod m = 0 = (k * x + x + y) mod m =0 = (k * x) mod m = 0 because (x + y) mod m = 0 Therefore, for an undetectable 2 bit error, k * x must be a multiple of m. For Adler, m is prime and the maximum value of x is 255 so the only way to satisfy the condition is for k to equal 65521. Then x can be any value so it could be a one bit error such as changing the lsb of the first value from 0 to 1. y would then need to be the inverse changing the lsb of the second value from 1 to 0. For Fletcher, m is factorable to 3, 5, 17, 257. For x to provide one or more of those factors, it would have to have more than a 1 bit change and y would have to have the inverse of that change which would mean an error of at least 4 bits. The only way to get a two bit undetectable error is for k to equal 65535. So, Fletcher and Adler do have 2-bit undetectable errors if the messages are longer than the modulus. What about burst protection? For Fletcher, we know that a change of a value from all 0's to all 1's produces an an undetectable error so we know that burst protection is less than 16. k = 1 for a burst spread across two consecutive values. Therefore, the only way for a burst spread across two consecutive values to produce an undetectable error is for x to equal 65535. A burst shorter than 16 can't do it. For Adler, corrupting two consecutive bytes can't produce an undetectable error because that would be the case above where k equals 1. There is no way for a burst across two consectutive values to be undetectable in Adler since x is less than the modulus. For a burst across three values: let x equal the error in the first value y equal the error in the second value z equal the error in the third value. For the burst to be undetectable Delta S1 = (x + y + z) mod m = 0 Delta S2 = (3x + 2y + z) mod m = 0 Since the maximum values of x, y, and z are 255 and m is more than 6 times that, the sums above will always be less than m and we can drop the mod m. Delta S2 = (3x + 2y + z) = 2x + y + (x + y + z) = 0 = 2x + y = 0 because x + y + z = 0 y = - 2x Delta S1 = x + y + z = 0 = x - 2x + z = 0 = -x + z z = x So an undetectable error is created when in three consecutive values the errors are x -2x x This could for instance be errors of 1, -2, and 1. Since the error to the first and third values are the same, the error must span at least 2 * the length of the value + 1. So for Adler, it must be a 17 bit burst at least. Also, note that this can be created by a 3 bit error and is equally applicable to Adler and Fletcher. Therefore, for messages shorter than the modulus + 1, the Hamming distance of Adler32 and Fletcher 32 is 3 and, for messages longer than that, the Hamming distance is 2. Regards, Pat Thaler -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Pat, I did not run a check. That is very expensive. With CRCs there are methods that enable you to run a check on the complement code (that has only 2**32 different patterns for any block length) and derive from there the distances (Fujiwara has done this in 1989 for the IEEE CRC-32 and about some more recent experiments I'll get back to the list). And that is the trouble with this whole line of argument. There are no numbers to prove Adler32 or Fletcher32 and there are plenty for CRCs. The big question is then is there anybody out there that wants to build a modern bridge based only on its beauty? Regards, Julo pat_thaler@agilent.com on 05/03/2001 23:27:49 Please respond to pat_thaler@agilent.com To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Julian, I know that Hamming distance gets down to 2 for errors that are separated by the modulus (or a multiple of it). Is there another case? Pat > - Adler and Fletcher are weak and there is no theory behind your > distribution statements, nor any simulation results as far as I know. We > found that on very simple sequences the Hamming distance gets > down to 2 (or > lower) and the burst protection is probably not better than 16 bit. There > is even a simple formula for what sequences will get you false codes (see > bellow for a reference) > From owner-ips@ece.cmu.edu Tue Mar 6 15:13:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA25993 for ; Tue, 6 Mar 2001 15:13:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f26I0vY01997 for ips-outgoing; Tue, 6 Mar 2001 13:00:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f26I0br01920 for ; Tue, 6 Mar 2001 13:00:37 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id TAA102844 for ; Tue, 6 Mar 2001 19:00:30 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA81090 for ; Tue, 6 Mar 2001 18:59:33 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A07.0062C7F8 ; Tue, 6 Mar 2001 18:58:57 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 6 Mar 2001 19:31:02 +0200 Subject: RE: R2TDataSN Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, You've lost me. I do not propose that you look at the bad R2tT but to find that you have missed one by looking at the next. This interesting for long transfers that have several outstanding R2Ts. What is speculative here? There was never a consensus that there will be no more than one outstanding R2T. Regards, Julo "Somesh Gupta" on 06/03/2001 17:23:04 Please respond to someshg@yahoo.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: R2TDataSN I beg to disagree. If an R2T PDU (header) has bad digest, or any other header has a bad digest - since you always need the PDU length from the header, there is some uncertainty associated with further processing. Are you proposing that the processing machine go into a "speculative" mode where the processing of the next PDU determines whether we were successfuly able to skip a bad PDU header? When there is a data digest error, further stream parsing is deterministic. But not when the PDU header digest error. Also the consensus (in my interpretation) was on applications not transfering very large amounts of data using a single command or read/write PDU. > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Monday, March 05, 2001 5:41 PM > To: ips@ece.cmu.edu > Subject: Re: R2TDataSN > > > > > Somesh, > > 1.The only consensus I heard is not to transfer a large amount of > data with > one PDU. > > 2.With DatasN and Sack we dont need any data in a bad header. > > 3. If an R2T is lost (received at initiator with bad digest) - the > initiator will know that from > the next R2T if the target has several outstanding - very likely at long > distances - and will not have to way for a timeout. Other uses are > marginal. Basically it is "part of a command execution" and we can > painless recover > from failures for this case too. > > Regards, > Julo > > "Somesh Gupta" on 05/03/2001 20:40:06 > > Please respond to someshg@yahoo.com > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: R2TDataSN > > > > > > R2TDataSN > > ---------- > > Sec 6.7.1 has some new content on how to handle lost R2Ts using > > SACKs. But I noticed that the SACK request (Sec 2.16) has not > > changed to indicate whether the DataSN is a R2T DataSN or just > > a Read PDU DataSN (D bit) > > So do we demux on the read/write operation type? > > And how does this affect PDU loss in bidirectional commands ? > > +++ SACK is ascking for data (DataSN) the target knows > > > > Julian, > > Regarding the R2TDataSN, I have a comments and a > question. > > I think that when a PDU header fails a CRC/checksum check etc, > it is a problem to depend on any information in the header (including > length fields), thereby making any further processing on > the connection unreliable. > > What scenarios do you envision where the R2TDataSN is useful. > IN Orlando I think there was clear consensus that application > do not try to transfer very large amounts of data using a > single command. > > Thanks, > Somesh > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com > > > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Tue Mar 6 18:21:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA02736 for ; Tue, 6 Mar 2001 18:21:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f26KJ4511612 for ips-outgoing; Tue, 6 Mar 2001 15:19:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f26KIPr11579 for ; Tue, 6 Mar 2001 15:18:25 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel2.hp.com (Postfix) with ESMTP id E8BD7540 for ; Tue, 6 Mar 2001 15:18:20 -0500 (EST) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id MAA20092 for ips@ece.cmu.edu; Tue, 6 Mar 2001 12:19:17 -0800 (PST) Message-Id: <200103062019.MAA20092@core.rose.hp.com> Subject: iSCSI: unsolicited Vs immediate; restart delay To: ips@ece.cmu.edu Date: Tue, 06 Mar 2001 12:19:17 PST Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, >To: ips@ece.cmu.edu >cc: >Subject: Re: iSCSI: draft04 questions > > > > >Julian, > >Thanks. I am not clear on some. Comments. > >>2. I didn't find a way for an iSCSI target to say that it does not support >>any unsolicited data at all. SPC-2 specifies that a zero FirstBurstSize >>means unlimited. >> >>+++ UseR2T=yes (default) and ImmediateData=no +++ > >I am confused. I thought this combination disallows just the "immediate" >data, and not the unsolicited data as a whole. The draft hints at this >in section 1.2.5. > "A target MAY separately enable immediate data without > enabling the more general (separate data PDUs) form of > unsolicited data." >=== for this case you have UseR2T=yes and ImmediateData=yes >If I misunderstood, could you please comment how immediate data alone is >disabled, while allowing unsolicited data of FirstBurstSize? >=== UseR2T=no ImmediateData=no But that puts the target in unsolicited data mode not requiring R2Ts at all! Sorry if I seem too slow. Let me try again. There are three variables - 1. solicited data mode after FirstBurstSize UseR2T=yes unsolicited data mode only UseR2T=no 2. Immediate data allowed ImmediateData=yes Immediate data disallowed ImmediateData=no 3. Unsolicited burst (immediate & separate) allowed ?? Unsolicited burst not allowed ?? FirstBurstSize can be used for (3), but a zero FirstBurstSize means "unlimited" than "not allowed". My original question was how one can distinguish the two. I would be glad to be corrected, if I am misinterpreting the usage of UseR2T. >Do I take it then that the draft currently doesn't specify the maximum time >the targets should keep the session & connection records around hoping >for a restart? I would strongly recommend adding that as an additional >field in the payload, since that is a resource allocation and scalability >issue. > >=== I assume that a traget can figure out by itself after a while that >there is no rendezvous - but I am open to requests There's no architected mechanism to figure out on its own! Either it has to keep the (session, connection, task) states around for a long time, OR it can clean up the these states and trigger an unnecessary ULP recovery on the initiator which is surprised at an unexpected restart failure (keep in mind that initiator may not be able to restart a login precisely after the "minimum time" specified, typically there's an aggregation of multiple O/S timer requests into one master handler). Providing an upper limit is a cleaner, safer design for both initiator and target. Thanks. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Tue Mar 6 19:47:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA05356 for ; Tue, 6 Mar 2001 19:46:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f26MUxF20672 for ips-outgoing; Tue, 6 Mar 2001 17:30:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp012.mail.yahoo.com (smtp012.mail.yahoo.com [216.136.173.32]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f26MUbr20608 for ; Tue, 6 Mar 2001 17:30:37 -0500 (EST) Received: from unknown (HELO somesh) (206.112.110.135) by smtp.mail.vip.sc5.yahoo.com with SMTP; 6 Mar 2001 22:30:30 -0000 X-Apparently-From: Reply-To: From: "Somesh Gupta" To: , Subject: RE: R2TDataSN Date: Tue, 6 Mar 2001 14:26:10 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Tuesday, March 06, 2001 9:31 AM > To: ips@ece.cmu.edu > Subject: RE: R2TDataSN > > > > > Somesh, > > You've lost me. I do not propose that you look at the bad R2tT but to find > that you have missed one > by looking at the next. Since iSCSI PDUs define how long they are, you have to look at one PDU to determine where the next PDU is. (unless ofcourse the sync and steering layer is doing the work - see my exchange with Venkat on that). On the long transfer etc., I was not sure what scenarios an R2TDataSN was providing recovery from. Since you clarified that it is to recover from a header digest error, we can focus on that scenario. This interesting for long transfers that have > several outstanding R2Ts. > What is speculative here? There was never a consensus that there > will be no > more than one outstanding R2T. > > Regards, > Julo > > "Somesh Gupta" on 06/03/2001 17:23:04 > > Please respond to someshg@yahoo.com > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: R2TDataSN > > > > > I beg to disagree. If an R2T PDU (header) has bad digest, or any other > header has a bad digest - since you always need the PDU length from > the header, there is some uncertainty associated with further processing. > > Are you proposing that the processing machine go into a "speculative" > mode where the processing of the next PDU determines whether we were > successfuly able to skip a bad PDU header? When there is a data digest > error, further stream parsing is deterministic. But not when the PDU > header digest error. > > Also the consensus (in my interpretation) was on applications > not transfering very large amounts of data using a single command or > read/write PDU. > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Monday, March 05, 2001 5:41 PM > > To: ips@ece.cmu.edu > > Subject: Re: R2TDataSN > > > > > > > > > > Somesh, > > > > 1.The only consensus I heard is not to transfer a large amount of > > data with > > one PDU. > > > > 2.With DatasN and Sack we dont need any data in a bad header. > > > > 3. If an R2T is lost (received at initiator with bad digest) - the > > initiator will know that from > > the next R2T if the target has several outstanding - very likely at long > > distances - and will not have to way for a timeout. Other uses are > > marginal. Basically it is "part of a command execution" and we can > > painless recover > > from failures for this case too. > > > > Regards, > > Julo > > > > "Somesh Gupta" on 05/03/2001 20:40:06 > > > > Please respond to someshg@yahoo.com > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: R2TDataSN > > > > > > > > > > > R2TDataSN > > > ---------- > > > Sec 6.7.1 has some new content on how to handle lost R2Ts using > > > SACKs. But I noticed that the SACK request (Sec 2.16) has not > > > changed to indicate whether the DataSN is a R2T DataSN or just > > > a Read PDU DataSN (D bit) > > > So do we demux on the read/write operation type? > > > And how does this affect PDU loss in bidirectional commands ? > > > +++ SACK is ascking for data (DataSN) the target knows > > > > > > > Julian, > > > > Regarding the R2TDataSN, I have a comments and a > > question. > > > > I think that when a PDU header fails a CRC/checksum check etc, > > it is a problem to depend on any information in the header (including > > length fields), thereby making any further processing on > > the connection unreliable. > > > > What scenarios do you envision where the R2TDataSN is useful. > > IN Orlando I think there was clear consensus that application > > do not try to transfer very large amounts of data using a > > single command. > > > > Thanks, > > Somesh > > Thanks, Somesh Gupta _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Tue Mar 6 20:21:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA05964 for ; Tue, 6 Mar 2001 20:21:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f26NL6w23874 for ips-outgoing; Tue, 6 Mar 2001 18:21:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f26NKdr23806 for ; Tue, 6 Mar 2001 18:20:39 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id AAA16730 for ; Wed, 7 Mar 2001 00:20:31 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id AAA23688 for ; Wed, 7 Mar 2001 00:19:34 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A07.008013E2 ; Wed, 7 Mar 2001 00:18:56 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu cc: "THALER,PAT (A-Roseville,ex1)" , ips@ece.cmu.edu, Dafna_Sheinwald@il.ibm.com Message-ID: Date: Tue, 6 Mar 2001 20:39:57 +0200 Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Pat, Thanks. I am not going to check every line for awhile and I assume it is correct. For the benefit of the list the summary of the statement is that: -the burst protection is for 16 bit (I recall the text saying 32) -the numbers to be used for low noise symmetric channel protection are: - dmin= 3 for codes shorter than m (approx 8kbytes) - dmin= 2 for codes longer than m Do you have a formula to evaluate Pud for low noise channels with BER = e? Any formula to evaluate the conditional probability of an undetected error on bursts of 17, 18, 19 bits? Is there any reason to believe hat we can approximate them to ones used for CRC? I would like to have all the codes readily comparable. Regards, Julo "THALER,PAT (A-Roseville,ex1)" on 06/03/2001 19:24:13 Please respond to "THALER,PAT (A-Roseville,ex1)" To: Julian Satran/Haifa/IBM@IBMIL, "THALER,PAT (A-Roseville,ex1)" cc: ips@ece.cmu.edu, Dafna_Sheinwald@il.ibm.com Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Julian, It is relatively easy to investigate the Hamming distance and burst detection of Adler32 and Fletcher32 using mathematics. In the argument below, I will use "value" to identify the quantities that get summed since Adler sums bytes and Fletcher32 sums 16-bits. Both Adler and Fletcher calculate for each value: S1 = (S1 + value) mod m S2 = (S2 + S1) mod m where m is 65521 for Adler and 65535 for Fletcher. Which means that for an n value string of values V1 to Vn: S1 = (V1 + V2 + ... + Vn (+ 1 for Adler)) mod m S2 = (n * V1 + (n-1) * V2 + ... + Vn (+ n for Adler)) mod m A two bit error can only change two values. For a two bit error to be undetectable the two changed values must produce no change to S1 and S2. Let x be the change to the first errored value y be the change to the second errored value k be the separation between the first and second errored values. Then for an undetectable error: delta S1 = (x + y) mod m = 0 delta S2 = ((k + 1) * x + y) mod m = 0 = (k * x + x + y) mod m =0 = (k * x) mod m = 0 because (x + y) mod m = 0 Therefore, for an undetectable 2 bit error, k * x must be a multiple of m. For Adler, m is prime and the maximum value of x is 255 so the only way to satisfy the condition is for k to equal 65521. Then x can be any value so it could be a one bit error such as changing the lsb of the first value from 0 to 1. y would then need to be the inverse changing the lsb of the second value from 1 to 0. For Fletcher, m is factorable to 3, 5, 17, 257. For x to provide one or more of those factors, it would have to have more than a 1 bit change and y would have to have the inverse of that change which would mean an error of at least 4 bits. The only way to get a two bit undetectable error is for k to equal 65535. So, Fletcher and Adler do have 2-bit undetectable errors if the messages are longer than the modulus. What about burst protection? For Fletcher, we know that a change of a value from all 0's to all 1's produces an an undetectable error so we know that burst protection is less than 16. k = 1 for a burst spread across two consecutive values. Therefore, the only way for a burst spread across two consecutive values to produce an undetectable error is for x to equal 65535. A burst shorter than 16 can't do it. For Adler, corrupting two consecutive bytes can't produce an undetectable error because that would be the case above where k equals 1. There is no way for a burst across two consectutive values to be undetectable in Adler since x is less than the modulus. For a burst across three values: let x equal the error in the first value y equal the error in the second value z equal the error in the third value. For the burst to be undetectable Delta S1 = (x + y + z) mod m = 0 Delta S2 = (3x + 2y + z) mod m = 0 Since the maximum values of x, y, and z are 255 and m is more than 6 times that, the sums above will always be less than m and we can drop the mod m. Delta S2 = (3x + 2y + z) = 2x + y + (x + y + z) = 0 = 2x + y = 0 because x + y + z = 0 y = - 2x Delta S1 = x + y + z = 0 = x - 2x + z = 0 = -x + z z = x So an undetectable error is created when in three consecutive values the errors are x -2x x This could for instance be errors of 1, -2, and 1. Since the error to the first and third values are the same, the error must span at least 2 * the length of the value + 1. So for Adler, it must be a 17 bit burst at least. Also, note that this can be created by a 3 bit error and is equally applicable to Adler and Fletcher. Therefore, for messages shorter than the modulus + 1, the Hamming distance of Adler32 and Fletcher 32 is 3 and, for messages longer than that, the Hamming distance is 2. Regards, Pat Thaler -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Pat, I did not run a check. That is very expensive. With CRCs there are methods that enable you to run a check on the complement code (that has only 2**32 different patterns for any block length) and derive from there the distances (Fujiwara has done this in 1989 for the IEEE CRC-32 and about some more recent experiments I'll get back to the list). And that is the trouble with this whole line of argument. There are no numbers to prove Adler32 or Fletcher32 and there are plenty for CRCs. The big question is then is there anybody out there that wants to build a modern bridge based only on its beauty? Regards, Julo pat_thaler@agilent.com on 05/03/2001 23:27:49 Please respond to pat_thaler@agilent.com To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Julian, I know that Hamming distance gets down to 2 for errors that are separated by the modulus (or a multiple of it). Is there another case? Pat > - Adler and Fletcher are weak and there is no theory behind your > distribution statements, nor any simulation results as far as I know. We > found that on very simple sequences the Hamming distance gets > down to 2 (or > lower) and the burst protection is probably not better than 16 bit. There > is even a simple formula for what sequences will get you false codes (see > bellow for a reference) > From owner-ips@ece.cmu.edu Tue Mar 6 20:21:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA05975 for ; Tue, 6 Mar 2001 20:21:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f26NL4c23868 for ips-outgoing; Tue, 6 Mar 2001 18:21:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f26NKer23811 for ; Tue, 6 Mar 2001 18:20:40 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id AAA94040; Wed, 7 Mar 2001 00:20:33 +0100 From: julian_satran@il.ibm.com Received: from d12mta05.de.ibm.com (d12mta06_cs0 [9.165.222.255]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id AAA195334; Wed, 7 Mar 2001 00:19:36 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A07.00803913 ; Wed, 7 Mar 2001 00:20:32 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu cc: "THALER,PAT (A-Roseville,ex1)" , ips@ece.cmu.edu, Dafna_Sheinwald@il.ibm.com Message-ID: Date: Tue, 6 Mar 2001 20:39:57 +0200 Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Pat, Thanks. I am not going to check every line for awhile and I assume it is correct. For the benefit of the list the summary of the statement is that: -the burst protection is for 16 bit (I recall the text saying 32) -the numbers to be used for low noise symmetric channel protection are: - dmin= 3 for codes shorter than m (approx 8kbytes) - dmin= 2 for codes longer than m Do you have a formula to evaluate Pud for low noise channels with BER = e? Any formula to evaluate the conditional probability of an undetected error on bursts of 17, 18, 19 bits? Is there any reason to believe hat we can approximate them to ones used for CRC? I would like to have all the codes readily comparable. Regards, Julo "THALER,PAT (A-Roseville,ex1)" on 06/03/2001 19:24:13 Please respond to "THALER,PAT (A-Roseville,ex1)" To: Julian Satran/Haifa/IBM@IBMIL, "THALER,PAT (A-Roseville,ex1)" cc: ips@ece.cmu.edu, Dafna_Sheinwald@il.ibm.com Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Julian, It is relatively easy to investigate the Hamming distance and burst detection of Adler32 and Fletcher32 using mathematics. In the argument below, I will use "value" to identify the quantities that get summed since Adler sums bytes and Fletcher32 sums 16-bits. Both Adler and Fletcher calculate for each value: S1 = (S1 + value) mod m S2 = (S2 + S1) mod m where m is 65521 for Adler and 65535 for Fletcher. Which means that for an n value string of values V1 to Vn: S1 = (V1 + V2 + ... + Vn (+ 1 for Adler)) mod m S2 = (n * V1 + (n-1) * V2 + ... + Vn (+ n for Adler)) mod m A two bit error can only change two values. For a two bit error to be undetectable the two changed values must produce no change to S1 and S2. Let x be the change to the first errored value y be the change to the second errored value k be the separation between the first and second errored values. Then for an undetectable error: delta S1 = (x + y) mod m = 0 delta S2 = ((k + 1) * x + y) mod m = 0 = (k * x + x + y) mod m =0 = (k * x) mod m = 0 because (x + y) mod m = 0 Therefore, for an undetectable 2 bit error, k * x must be a multiple of m. For Adler, m is prime and the maximum value of x is 255 so the only way to satisfy the condition is for k to equal 65521. Then x can be any value so it could be a one bit error such as changing the lsb of the first value from 0 to 1. y would then need to be the inverse changing the lsb of the second value from 1 to 0. For Fletcher, m is factorable to 3, 5, 17, 257. For x to provide one or more of those factors, it would have to have more than a 1 bit change and y would have to have the inverse of that change which would mean an error of at least 4 bits. The only way to get a two bit undetectable error is for k to equal 65535. So, Fletcher and Adler do have 2-bit undetectable errors if the messages are longer than the modulus. What about burst protection? For Fletcher, we know that a change of a value from all 0's to all 1's produces an an undetectable error so we know that burst protection is less than 16. k = 1 for a burst spread across two consecutive values. Therefore, the only way for a burst spread across two consecutive values to produce an undetectable error is for x to equal 65535. A burst shorter than 16 can't do it. For Adler, corrupting two consecutive bytes can't produce an undetectable error because that would be the case above where k equals 1. There is no way for a burst across two consectutive values to be undetectable in Adler since x is less than the modulus. For a burst across three values: let x equal the error in the first value y equal the error in the second value z equal the error in the third value. For the burst to be undetectable Delta S1 = (x + y + z) mod m = 0 Delta S2 = (3x + 2y + z) mod m = 0 Since the maximum values of x, y, and z are 255 and m is more than 6 times that, the sums above will always be less than m and we can drop the mod m. Delta S2 = (3x + 2y + z) = 2x + y + (x + y + z) = 0 = 2x + y = 0 because x + y + z = 0 y = - 2x Delta S1 = x + y + z = 0 = x - 2x + z = 0 = -x + z z = x So an undetectable error is created when in three consecutive values the errors are x -2x x This could for instance be errors of 1, -2, and 1. Since the error to the first and third values are the same, the error must span at least 2 * the length of the value + 1. So for Adler, it must be a 17 bit burst at least. Also, note that this can be created by a 3 bit error and is equally applicable to Adler and Fletcher. Therefore, for messages shorter than the modulus + 1, the Hamming distance of Adler32 and Fletcher 32 is 3 and, for messages longer than that, the Hamming distance is 2. Regards, Pat Thaler -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Pat, I did not run a check. That is very expensive. With CRCs there are methods that enable you to run a check on the complement code (that has only 2**32 different patterns for any block length) and derive from there the distances (Fujiwara has done this in 1989 for the IEEE CRC-32 and about some more recent experiments I'll get back to the list). And that is the trouble with this whole line of argument. There are no numbers to prove Adler32 or Fletcher32 and there are plenty for CRCs. The big question is then is there anybody out there that wants to build a modern bridge based only on its beauty? Regards, Julo pat_thaler@agilent.com on 05/03/2001 23:27:49 Please respond to pat_thaler@agilent.com To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Julian, I know that Hamming distance gets down to 2 for errors that are separated by the modulus (or a multiple of it). Is there another case? Pat > - Adler and Fletcher are weak and there is no theory behind your > distribution statements, nor any simulation results as far as I know. We > found that on very simple sequences the Hamming distance gets > down to 2 (or > lower) and the burst protection is probably not better than 16 bit. There > is even a simple formula for what sequences will get you false codes (see > bellow for a reference) > From owner-ips@ece.cmu.edu Tue Mar 6 21:54:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA08420 for ; Tue, 6 Mar 2001 21:54:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f27163O00049 for ips-outgoing; Tue, 6 Mar 2001 20:06:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antipater.hosting.pacbell.net (antipater.hosting.pacbell.net [216.100.99.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f27151r29949 for ; Tue, 6 Mar 2001 20:05:01 -0500 (EST) Received: from VENKAT1 (64-160-62-200.rhapsodynetworks.com [64.160.62.200] (may be forged)) by antipater.hosting.pacbell.net id UAA11612; Tue, 6 Mar 2001 20:04:53 -0500 (EST) [ConcentricHost SMTP Relay 1.7] From: "Venkat Rangan" To: , , Subject: RE: iSCSI: Reliability of markers (was Re: R2TDataSN and other recovery mechanisms) Date: Tue, 6 Mar 2001 17:07:57 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Somesh, Thanks for your insights. > Consider the marker mechanism as an example. If the marker > falls within the TCP byte stream region which contains the > header, and the header digest fails, then do you trust this > marker or skip to the next marker. Or do we need a sum on the > marker itself. You point out one of the disadvantages of the current marker approach. In this respect, the word-stuffing proposal as described in draft-weber-fcip-encaps-00.txt is a better message boundary identification approach. One key point you bring up is if there is just a two-bit burst error with one bit on the Header and the other bit on the Synch-and-Steering header, you've lost a reliable way of synching to any part of the stream. This is possible because the Synch-and-steering header can be right smack in the middle of an iSCSI header. You may also have cases where the two copies of the Next-iSCSI-PDU-start pointer disagree on where the next pdu starts. What then? With constant-overhead-word-stuffing, all is not lost if the synch words get corrupted - you skip over enough words in the stream until you find one to synch with. Also, you do not have nearly the level of with UFL mentioned in section 1.2.8.3 of the draft. I don't remember all the technical discussion that went into the selection of current Synch-and-steering, but it does seem to have some weaknesses. Regards, Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com From owner-ips@ece.cmu.edu Tue Mar 6 21:54:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA08416 for ; Tue, 6 Mar 2001 21:54:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f270m2R28995 for ips-outgoing; Tue, 6 Mar 2001 19:48:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f270l7r28954 for ; Tue, 6 Mar 2001 19:47:07 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id E700D405; Tue, 6 Mar 2001 17:47:06 -0700 (MST) Received: from axcsbh2.cos.agilent.com (axcsbh2.cos.agilent.com [130.29.152.144]) by msgrel1.cos.agilent.com (Postfix) with SMTP id C7DD924; Tue, 6 Mar 2001 17:47:06 -0700 (MST) Received: from 130.29.152.144 by axcsbh2.cos.agilent.com (InterScan E-Mail VirusWall NT); Tue, 06 Mar 2001 17:47:06 -0700 (Mountain Standard Time) Received: by axcsbh2.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Tue, 6 Mar 2001 17:47:06 -0700 Message-ID: <1BEBA5E8600DD4119A50009027AF54A00530E7DA@axcs04.cos.agilent.com> From: pat_thaler@agilent.com To: julian_satran@il.ibm.com, ips@ece.cmu.edu Cc: pat_thaler@agilent.com, ips@ece.cmu.edu, Dafna_Sheinwald@il.ibm.com, vince_cavanna@agilent.com Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Date: Tue, 6 Mar 2001 17:47:03 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, Good summary. One correction. "codes shorter than m" is shorter than m values not m bits and m is about 64 K. Therefore, it is shorter than about 63 kbytes for Adler and shorter than 127 kbytes for Fletcher. I'm thinking about your question about Pud but it will take me a bit to come up with an answer. For the three bit error case, I think the formulas used for CRC bit errors will be approximately correct. For the burst errors and the errors where two (Fletcher only) or three non-contiguous bytes are corrupted, I'm not sure what to use for the probability that a byte gets corrupted. Regards, Pat -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Pat, Thanks. I am not going to check every line for awhile and I assume it is correct. For the benefit of the list the summary of the statement is that: -the burst protection is for 16 bit (I recall the text saying 32) -the numbers to be used for low noise symmetric channel protection are: - dmin= 3 for codes shorter than m (approx 8kbytes) - dmin= 2 for codes longer than m Do you have a formula to evaluate Pud for low noise channels with BER = e? Any formula to evaluate the conditional probability of an undetected error on bursts of 17, 18, 19 bits? Is there any reason to believe hat we can approximate them to ones used for CRC? I would like to have all the codes readily comparable. Regards, Julo "THALER,PAT (A-Roseville,ex1)" on 06/03/2001 19:24:13 Please respond to "THALER,PAT (A-Roseville,ex1)" To: Julian Satran/Haifa/IBM@IBMIL, "THALER,PAT (A-Roseville,ex1)" cc: ips@ece.cmu.edu, Dafna_Sheinwald@il.ibm.com Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Julian, It is relatively easy to investigate the Hamming distance and burst detection of Adler32 and Fletcher32 using mathematics. In the argument below, I will use "value" to identify the quantities that get summed since Adler sums bytes and Fletcher32 sums 16-bits. Both Adler and Fletcher calculate for each value: S1 = (S1 + value) mod m S2 = (S2 + S1) mod m where m is 65521 for Adler and 65535 for Fletcher. Which means that for an n value string of values V1 to Vn: S1 = (V1 + V2 + ... + Vn (+ 1 for Adler)) mod m S2 = (n * V1 + (n-1) * V2 + ... + Vn (+ n for Adler)) mod m A two bit error can only change two values. For a two bit error to be undetectable the two changed values must produce no change to S1 and S2. Let x be the change to the first errored value y be the change to the second errored value k be the separation between the first and second errored values. Then for an undetectable error: delta S1 = (x + y) mod m = 0 delta S2 = ((k + 1) * x + y) mod m = 0 = (k * x + x + y) mod m =0 = (k * x) mod m = 0 because (x + y) mod m = 0 Therefore, for an undetectable 2 bit error, k * x must be a multiple of m. For Adler, m is prime and the maximum value of x is 255 so the only way to satisfy the condition is for k to equal 65521. Then x can be any value so it could be a one bit error such as changing the lsb of the first value from 0 to 1. y would then need to be the inverse changing the lsb of the second value from 1 to 0. For Fletcher, m is factorable to 3, 5, 17, 257. For x to provide one or more of those factors, it would have to have more than a 1 bit change and y would have to have the inverse of that change which would mean an error of at least 4 bits. The only way to get a two bit undetectable error is for k to equal 65535. So, Fletcher and Adler do have 2-bit undetectable errors if the messages are longer than the modulus. What about burst protection? For Fletcher, we know that a change of a value from all 0's to all 1's produces an an undetectable error so we know that burst protection is less than 16. k = 1 for a burst spread across two consecutive values. Therefore, the only way for a burst spread across two consecutive values to produce an undetectable error is for x to equal 65535. A burst shorter than 16 can't do it. For Adler, corrupting two consecutive bytes can't produce an undetectable error because that would be the case above where k equals 1. There is no way for a burst across two consectutive values to be undetectable in Adler since x is less than the modulus. For a burst across three values: let x equal the error in the first value y equal the error in the second value z equal the error in the third value. For the burst to be undetectable Delta S1 = (x + y + z) mod m = 0 Delta S2 = (3x + 2y + z) mod m = 0 Since the maximum values of x, y, and z are 255 and m is more than 6 times that, the sums above will always be less than m and we can drop the mod m. Delta S2 = (3x + 2y + z) = 2x + y + (x + y + z) = 0 = 2x + y = 0 because x + y + z = 0 y = - 2x Delta S1 = x + y + z = 0 = x - 2x + z = 0 = -x + z z = x So an undetectable error is created when in three consecutive values the errors are x -2x x This could for instance be errors of 1, -2, and 1. Since the error to the first and third values are the same, the error must span at least 2 * the length of the value + 1. So for Adler, it must be a 17 bit burst at least. Also, note that this can be created by a 3 bit error and is equally applicable to Adler and Fletcher. Therefore, for messages shorter than the modulus + 1, the Hamming distance of Adler32 and Fletcher 32 is 3 and, for messages longer than that, the Hamming distance is 2. Regards, Pat Thaler -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Pat, I did not run a check. That is very expensive. With CRCs there are methods that enable you to run a check on the complement code (that has only 2**32 different patterns for any block length) and derive from there the distances (Fujiwara has done this in 1989 for the IEEE CRC-32 and about some more recent experiments I'll get back to the list). And that is the trouble with this whole line of argument. There are no numbers to prove Adler32 or Fletcher32 and there are plenty for CRCs. The big question is then is there anybody out there that wants to build a modern bridge based only on its beauty? Regards, Julo pat_thaler@agilent.com on 05/03/2001 23:27:49 Please respond to pat_thaler@agilent.com To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Julian, I know that Hamming distance gets down to 2 for errors that are separated by the modulus (or a multiple of it). Is there another case? Pat > - Adler and Fletcher are weak and there is no theory behind your > distribution statements, nor any simulation results as far as I know. We > found that on very simple sequences the Hamming distance gets > down to 2 (or > lower) and the burst protection is probably not better than 16 bit. There > is even a simple formula for what sequences will get you false codes (see > bellow for a reference) > From owner-ips@ece.cmu.edu Tue Mar 6 21:54:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA08418 for ; Tue, 6 Mar 2001 21:54:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f27194i00209 for ips-outgoing; Tue, 6 Mar 2001 20:09:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f27187r00173 for ; Tue, 6 Mar 2001 20:08:07 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id CAA13474 for ; Wed, 7 Mar 2001 02:07:59 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id CAA159216 for ; Wed, 7 Mar 2001 02:07:02 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A08.0006110C ; Wed, 7 Mar 2001 02:06:15 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 7 Mar 2001 02:01:24 +0200 Subject: RE: R2TDataSN Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, I am still lost. R2T is fixed length and you know that you have a bad digest after reading it. No length involved. ??? Julo "Somesh Gupta" on 07/03/2001 00:26:10 Please respond to someshg@yahoo.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: R2TDataSN Julian, > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Tuesday, March 06, 2001 9:31 AM > To: ips@ece.cmu.edu > Subject: RE: R2TDataSN > > > > > Somesh, > > You've lost me. I do not propose that you look at the bad R2tT but to find > that you have missed one > by looking at the next. Since iSCSI PDUs define how long they are, you have to look at one PDU to determine where the next PDU is. (unless ofcourse the sync and steering layer is doing the work - see my exchange with Venkat on that). On the long transfer etc., I was not sure what scenarios an R2TDataSN was providing recovery from. Since you clarified that it is to recover from a header digest error, we can focus on that scenario. This interesting for long transfers that have > several outstanding R2Ts. > What is speculative here? There was never a consensus that there > will be no > more than one outstanding R2T. > > Regards, > Julo > > "Somesh Gupta" on 06/03/2001 17:23:04 > > Please respond to someshg@yahoo.com > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: R2TDataSN > > > > > I beg to disagree. If an R2T PDU (header) has bad digest, or any other > header has a bad digest - since you always need the PDU length from > the header, there is some uncertainty associated with further processing. > > Are you proposing that the processing machine go into a "speculative" > mode where the processing of the next PDU determines whether we were > successfuly able to skip a bad PDU header? When there is a data digest > error, further stream parsing is deterministic. But not when the PDU > header digest error. > > Also the consensus (in my interpretation) was on applications > not transfering very large amounts of data using a single command or > read/write PDU. > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Monday, March 05, 2001 5:41 PM > > To: ips@ece.cmu.edu > > Subject: Re: R2TDataSN > > > > > > > > > > Somesh, > > > > 1.The only consensus I heard is not to transfer a large amount of > > data with > > one PDU. > > > > 2.With DatasN and Sack we dont need any data in a bad header. > > > > 3. If an R2T is lost (received at initiator with bad digest) - the > > initiator will know that from > > the next R2T if the target has several outstanding - very likely at long > > distances - and will not have to way for a timeout. Other uses are > > marginal. Basically it is "part of a command execution" and we can > > painless recover > > from failures for this case too. > > > > Regards, > > Julo > > > > "Somesh Gupta" on 05/03/2001 20:40:06 > > > > Please respond to someshg@yahoo.com > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: R2TDataSN > > > > > > > > > > > R2TDataSN > > > ---------- > > > Sec 6.7.1 has some new content on how to handle lost R2Ts using > > > SACKs. But I noticed that the SACK request (Sec 2.16) has not > > > changed to indicate whether the DataSN is a R2T DataSN or just > > > a Read PDU DataSN (D bit) > > > So do we demux on the read/write operation type? > > > And how does this affect PDU loss in bidirectional commands ? > > > +++ SACK is ascking for data (DataSN) the target knows > > > > > > > Julian, > > > > Regarding the R2TDataSN, I have a comments and a > > question. > > > > I think that when a PDU header fails a CRC/checksum check etc, > > it is a problem to depend on any information in the header (including > > length fields), thereby making any further processing on > > the connection unreliable. > > > > What scenarios do you envision where the R2TDataSN is useful. > > IN Orlando I think there was clear consensus that application > > do not try to transfer very large amounts of data using a > > single command. > > > > Thanks, > > Somesh > > Thanks, Somesh Gupta _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Tue Mar 6 21:56:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA08478 for ; Tue, 6 Mar 2001 21:56:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f27185q00170 for ips-outgoing; Tue, 6 Mar 2001 20:08:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gordan.pl.gadzoox.com (i248.gadzoox.com [216.52.31.248]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2717jr00145 for ; Tue, 6 Mar 2001 20:07:45 -0500 (EST) Received: by gordan with Internet Mail Service (5.5.2650.21) id ; Tue, 6 Mar 2001 17:08:47 -0800 Message-ID: <312419998E3CD211A52900A0C991A47A81AE67@gordan> From: Vi Chau To: ips@ece.cmu.edu Subject: FCIP iFCP encapsulation proposal Date: Tue, 6 Mar 2001 17:08:43 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear Colleagues, I sent out a draft to propose a common encapsulation scheme for FCIP and iFCP last week; I am afraid I have missed the cut-off time for new drafts. Here is a summary of the draft to help stimulate discussion at the upcoming IETF meeting: The proposed encapsulation scheme allows easy integration of FCIP and iFCP headers; it provides header integrity which was lacking in the current FCIP draft. This scheme also provides integrity to the extended headr and FC SOF and EOF codes in the payload (as the FC CRC does not cover SOF and EOF.) A time stamp helps solve FC timeout issues. Fianlly, the header CRC can be used to help with framing. The draft proposes that FCIP/iFCP encapsulation include a header, header CRC, extended header(s) for new functions, the payload (which is an entire FC frame), and a 32-bit CRC that covers the entire FCIP/iFCP frame. The FCIP/iFCP frame is shown in the diagram below. +---------------------------------+ | Header (12 bytes) | +---------------------------------+ | Header CRC (4 Bytes) | +---------------------------------+ | Extended Header (N Bytes) | +---------------------------------+ | FC SOF (4 Bytes) (FCIP Only) | +---------------------------------+ | FC Frame Header (24 bytes) | +---------------------------------+ | FC Frame Payload (M Bytes) | +---------------------------------+ | FC Frame CRC (4 Bytes) | +---------------------------------+ | FC EOF (4 Bytes) (FCIP Only) | +---------------------------------+ | Frame CRC (4 Bytes) | +---------------------------------+ The header has the following format: |3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 | |1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 | +-------+-------+---------------+--------------------------------+ | P. N. | VER. | Flags | Frame length | +-------+-------+---------------+--------------------------------+ | Time Stamp (High order) | +----------------------------------------------------------------+ | Time Stamp (Low order) | +----------------------------------------------------------------+ | Header CRC | +----------------------------------------------------------------+ | Extended Header (optional) | +----------------------------------------------------------------+ Protocol Number field differentiates FCIP from iFCP. Each encapsulated protocol may have its own version number. There is currently only one flag defined: if the flag is set to 1, an extended header is present. The Frame Length field contains the number of 32-bit words in the frame including the header, payload, and trailer. The time stamp follows the format defined by SNTP-2 (RFC 2030). The first 3 words of this frame structure is covered by a 32-bit CRC placed immediately after the time stamp. The extended header has the following format: |3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 | |1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 | +---------------+---------------+--------------------------------+ | Type | Flags | Extended Header length | +---------------+---------------+--------------------------------+ | Extended Header Content | +----------------------------------------------------------------+ The Type field indicates the kind of information carried in this etended header. There is currently only one flag defined; if the flag is set to one, another extended header follows this one. The extended header length field carries the number of 32-bit words in this extended header, including the first word. The header CRC not only provides assurance of header integrity, but also allows an efficient method to re-synchronize to the data stream when frame synchronization is lost due to errors or other events. In the event of loss of synchronization, a state machine which normally checks the FCIP/iFCP header CRC can be continuously enabled on the data stream which should provide for a "good CRC" compare when an FCIP/iFCP header arrives. There is a remote possibility that a false compare could occur from other data in the stream so it is necessary to continue to check the "tentative" FCIP/iFCP payload and CRC also before assuming a correct synchronization. If both CRC checks are good, this certification should be at least as good as the data integrity provided by the CRCs in a synchronized stream. The CRC in the FCIP/iFCP header is important to this method as it allows a fast and hardware efficient check for a "tentative" header. The CRCs take less space than delimiter schemes, allow synchronization using existing hardware state machines, and provide for addition integrity. Regards, Vi Chau From owner-ips@ece.cmu.edu Tue Mar 6 23:28:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA10871 for ; Tue, 6 Mar 2001 23:28:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f271q6H02701 for ips-outgoing; Tue, 6 Mar 2001 20:52:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f271por02680 for ; Tue, 6 Mar 2001 20:51:50 -0500 (EST) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by atlrel2.hp.com (Postfix) with ESMTP id AC30D13E0; Tue, 6 Mar 2001 20:51:49 -0500 (EST) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA08853; Tue, 6 Mar 2001 18:51:48 -0700 (MST) Received: by xboibrg2.boi.hp.com with Internet Mail Service (5.5.2653.19) id ; Tue, 6 Mar 2001 18:51:48 -0700 Message-ID: <6BD67FFB937FD411A04F00D0B74FE87802A08ECF@xrose06.rose.hp.com> From: "KRUEGER,MARJORIE (HP-Roseville,ex1)" To: "'Venkat Rangan'" , ips@ece.cmu.edu Subject: RE: iSCSI: Reliability of markers (was Re: R2TDataSN and other re covery mechanisms) Date: Tue, 6 Mar 2001 18:49:10 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Venkat, > > Consider the marker mechanism as an example. If the marker > > falls within the TCP byte stream region which contains the > > header, and the header digest fails, then do you trust this > > marker or skip to the next marker. Or do we need a sum on the > > marker itself. > > You point out one of the disadvantages of the current marker > approach. Not sure what you mean "current marker approach"? The appendix in iSCSI 05.txt lists two approaches, clearly the second marker approach (fixed intervals) doesn't suffer from this disadvantage. In any case, a marker scheme would be roughly between the iSCSI layer and the TCP layer, so it's not appropriate to include in the iSCSI header digest. Marjorie Krueger Networked Storage Architecture Networked Storage Solutions Org. Hewlett-Packard tel: +1 916 785 2656 fax: +1 916 785 0391 email: marjorie_krueger@hp.com From owner-ips@ece.cmu.edu Tue Mar 6 23:29:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA10882 for ; Tue, 6 Mar 2001 23:29:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f27203u03118 for ips-outgoing; Tue, 6 Mar 2001 21:00:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f271xjr03103 for ; Tue, 6 Mar 2001 20:59:45 -0500 (EST) Received: from omgw5.rsvl.itc.hp.com (omgw5.rsvl.itc.hp.com [15.34.240.65]) by palrel1.hp.com (Postfix) with ESMTP id 06BC98EE; Tue, 6 Mar 2001 17:59:43 -0800 (PST) Received: from xboibrg1.boi.hp.com (xboibrg1.boi.hp.com [15.56.8.167]) by omgw5.rsvl.itc.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA03099; Tue, 6 Mar 2001 18:59:42 -0700 (MST) Received: by xboibrg1.boi.hp.com with Internet Mail Service (5.5.2653.19) id ; Tue, 6 Mar 2001 18:59:42 -0700 Message-ID: <6BD67FFB937FD411A04F00D0B74FE87802A08ED0@xrose06.rose.hp.com> From: "KRUEGER,MARJORIE (HP-Roseville,ex1)" To: "'Venkat Rangan'" , "'ips@ece.cmu.edu'" Subject: RE: iSCSI: Reliability of markers (was Re: R2TDataSN and other re covery mechanisms) Date: Tue, 6 Mar 2001 18:59:41 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > Not sure what you mean "current marker approach"? The > appendix in iSCSI 05.txt lists two approaches, clearly the > second marker approach (fixed intervals) doesn't suffer from > this disadvantage. In any case, a marker scheme would be > roughly between the iSCSI layer and the TCP layer, so it's > not appropriate to include in the iSCSI header digest. > Oops, I again re-read appendix C, it appears to have changed from my original understanding of "markers at fixed intervals" and as specified it would suffer from the same undetected corruption if not protected. However, this approach would not suffer greatly from unprotected marker information, since an invalid marker could just cause an implementation to skip ahead to the next marker and statistically, one should eventually be right ;-) Cheers, Marj From owner-ips@ece.cmu.edu Wed Mar 7 02:18:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA27638 for ; Wed, 7 Mar 2001 02:18:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f274p7t12211 for ips-outgoing; Tue, 6 Mar 2001 23:51:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp018.mail.yahoo.com (smtp018.mail.yahoo.com [216.136.174.115]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f274oUr12139 for ; Tue, 6 Mar 2001 23:50:31 -0500 (EST) Received: from unknown (HELO somesh) (206.112.105.116) by smtp.mail.vip.sc5.yahoo.com with SMTP; 7 Mar 2001 04:50:25 -0000 X-Apparently-From: Reply-To: From: "Somesh Gupta" To: , Subject: RE: R2TDataSN Date: Tue, 6 Mar 2001 20:46:07 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, How do I know it is an R2T if the header digest failed? I am sorry if I am missing something very obvious here. Let us say that a header digest failed. What information in the header am I able to trust at this point? I don't know if it is an R2T or not. I don't know if the length is ok or not. Thanks, Somesh > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Tuesday, March 06, 2001 4:01 PM > To: ips@ece.cmu.edu > Subject: RE: R2TDataSN > > > > > Somesh, > > I am still lost. R2T is fixed length and you know that you have a bad > digest after reading it. > No length involved. ??? > > Julo > > "Somesh Gupta" on 07/03/2001 00:26:10 > > Please respond to someshg@yahoo.com > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: R2TDataSN > > > > > Julian, > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Tuesday, March 06, 2001 9:31 AM > > To: ips@ece.cmu.edu > > Subject: RE: R2TDataSN > > > > > > > > > > Somesh, > > > > You've lost me. I do not propose that you look at the bad R2tT but to > find > > that you have missed one > > by looking at the next. > > Since iSCSI PDUs define how long they are, you have to look at > one PDU to determine where the next PDU is. (unless ofcourse > the sync and steering layer is doing the work - see my exchange > with Venkat on that). > > On the long transfer etc., I was not sure what scenarios > an R2TDataSN was providing recovery from. Since you clarified > that it is to recover from a header digest error, we can > focus on that scenario. > > > This interesting for long transfers that have > > several outstanding R2Ts. > > What is speculative here? There was never a consensus that there > > will be no > > more than one outstanding R2T. > > > > Regards, > > Julo > > > > "Somesh Gupta" on 06/03/2001 17:23:04 > > > > Please respond to someshg@yahoo.com > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: R2TDataSN > > > > > > > > > > I beg to disagree. If an R2T PDU (header) has bad digest, or any other > > header has a bad digest - since you always need the PDU length from > > the header, there is some uncertainty associated with further > processing. > > > > Are you proposing that the processing machine go into a "speculative" > > mode where the processing of the next PDU determines whether we were > > successfuly able to skip a bad PDU header? When there is a data digest > > error, further stream parsing is deterministic. But not when the PDU > > header digest error. > > > > Also the consensus (in my interpretation) was on applications > > not transfering very large amounts of data using a single command or > > read/write PDU. > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Monday, March 05, 2001 5:41 PM > > > To: ips@ece.cmu.edu > > > Subject: Re: R2TDataSN > > > > > > > > > > > > > > > Somesh, > > > > > > 1.The only consensus I heard is not to transfer a large amount of > > > data with > > > one PDU. > > > > > > 2.With DatasN and Sack we dont need any data in a bad header. > > > > > > 3. If an R2T is lost (received at initiator with bad digest) - the > > > initiator will know that from > > > the next R2T if the target has several outstanding - very likely at > long > > > distances - and will not have to way for a timeout. Other uses are > > > marginal. Basically it is "part of a command execution" and we can > > > painless recover > > > from failures for this case too. > > > > > > Regards, > > > Julo > > > > > > "Somesh Gupta" on 05/03/2001 20:40:06 > > > > > > Please respond to someshg@yahoo.com > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: R2TDataSN > > > > > > > > > > > > > > > > R2TDataSN > > > > ---------- > > > > Sec 6.7.1 has some new content on how to handle lost R2Ts using > > > > SACKs. But I noticed that the SACK request (Sec 2.16) has not > > > > changed to indicate whether the DataSN is a R2T DataSN or just > > > > a Read PDU DataSN (D bit) > > > > So do we demux on the read/write operation type? > > > > And how does this affect PDU loss in bidirectional commands ? > > > > +++ SACK is ascking for data (DataSN) the target knows > > > > > > > > > > Julian, > > > > > > Regarding the R2TDataSN, I have a comments and a > > > question. > > > > > > I think that when a PDU header fails a CRC/checksum check etc, > > > it is a problem to depend on any information in the header (including > > > length fields), thereby making any further processing on > > > the connection unreliable. > > > > > > What scenarios do you envision where the R2TDataSN is useful. > > > IN Orlando I think there was clear consensus that application > > > do not try to transfer very large amounts of data using a > > > single command. > > > > > > Thanks, > > > Somesh > > > > > Thanks, > Somesh Gupta > > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com > > > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Wed Mar 7 06:58:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA29972 for ; Wed, 7 Mar 2001 06:58:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f279cD527163 for ips-outgoing; Wed, 7 Mar 2001 04:38:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from anti_vir1.rad.co.il ([192.116.244.108]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f279bor27153 for ; Wed, 7 Mar 2001 04:37:50 -0500 (EST) Received: from anti_vir1.rad.co.il (localhost [127.0.0.1]) by anti_vir1.rad.co.il (8.9.3+Sun/8.9.3) with ESMTP id LAA21180 for ; Wed, 7 Mar 2001 11:36:37 +0200 (IST) Received: from radmail.rad.co.il (radmail [192.114.24.30]) by anti_vir1.rad.co.il (8.9.3+Sun/8.9.3) with ESMTP id LAA21173; Wed, 7 Mar 2001 11:36:35 +0200 (IST) Received: from yaron ([172.17.200.73]) by radmail.rad.co.il (Post.Office MTA v3.5.3 release 223 ID# 0-69802U1300L1200S0V35) with SMTP id il; Wed, 7 Mar 2001 10:40:34 +0200 Reply-To: From: "Yaron Klein" To: "'Tanjore K. Suresh'" Cc: Subject: RE: iSCSI: Naming and Discovery Draft... Date: Wed, 7 Mar 2001 09:38:07 -0000 Message-ID: <001501c0a6ea$52017840$49c811ac@sanrad.co.il> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0) In-Reply-To: <3AA531A7.E78CDDED@cisco.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Tanjore, Some more comments: The error statuses codes on Appendix B are not synchronized with the main draft. We will fix it. The term "target conflict" was borrowed from HTTP. Mark clarified this scenario well. I would like to add that this status enables better resolution and knowledge to the target. That is, in those cases the target can just not open the connection or just reject it like server error. However, this will not give indication of the situation as described by Mark. Regards, Yaron -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Mark Bakke Sent: Tuesday, March 06, 2001 6:51 PM To: Tanjore K. Suresh Cc: kaladhar@us.ibm.com; ips@ece.cmu.edu Subject: Re: iSCSI: Naming and Discovery Draft... Tanjore- Thanks for the feedback. I can comment on #3: "Tanjore K. Suresh" wrote: > 3. Appendix B, B.4.5, > Target Conflict 45 doesnot seem to be appropriate. > > I have not reviewed all the documents yet to give a > recommendation and hence cannot give, but feel > " Target Conflict" doesnot > convey the meaning of the Scenario indicating > case of " simple devices that can handle one device or > the target had reached the limit of its Initiators' capacity." Perhaps we chose the wrong term for this one. How about if call it "Target Busy", and slightly re-word it? The target is busy with another initiator and cannot handle another one. The initiator MAY try again later. This can be the case for simple devices that can handle only one initiator at a time, or for a target that has does not have the resources to support one more initiator. In contrast to the previous examples, this rejection is temporary. -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Wed Mar 7 08:14:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id IAA02659 for ; Wed, 7 Mar 2001 08:14:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f27AxGA09944 for ips-outgoing; Wed, 7 Mar 2001 05:59:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f27Awpr09925 for ; Wed, 7 Mar 2001 05:58:51 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id LAA127654 for ; Wed, 7 Mar 2001 11:58:38 +0100 From: julian_satran@il.ibm.com Received: from d12mta05.de.ibm.com (d12mta06_cs0 [9.165.222.255]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA124646 for ; Wed, 7 Mar 2001 11:57:41 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A08.003C4B50 ; Wed, 7 Mar 2001 11:58:34 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: someshg@yahoo.com cc: ips@ece.cmu.edu Message-ID: Date: Wed, 7 Mar 2001 12:57:15 +0200 Subject: RE: R2TDataSN Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk You don't. the next R2T will tell you you have missed one. Julo "Somesh Gupta" on 07/03/2001 06:46:07 Please respond to someshg@yahoo.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: R2TDataSN Julian, How do I know it is an R2T if the header digest failed? I am sorry if I am missing something very obvious here. Let us say that a header digest failed. What information in the header am I able to trust at this point? I don't know if it is an R2T or not. I don't know if the length is ok or not. Thanks, Somesh > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Tuesday, March 06, 2001 4:01 PM > To: ips@ece.cmu.edu > Subject: RE: R2TDataSN > > > > > Somesh, > > I am still lost. R2T is fixed length and you know that you have a bad > digest after reading it. > No length involved. ??? > > Julo > > "Somesh Gupta" on 07/03/2001 00:26:10 > > Please respond to someshg@yahoo.com > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: R2TDataSN > > > > > Julian, > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Tuesday, March 06, 2001 9:31 AM > > To: ips@ece.cmu.edu > > Subject: RE: R2TDataSN > > > > > > > > > > Somesh, > > > > You've lost me. I do not propose that you look at the bad R2tT but to > find > > that you have missed one > > by looking at the next. > > Since iSCSI PDUs define how long they are, you have to look at > one PDU to determine where the next PDU is. (unless ofcourse > the sync and steering layer is doing the work - see my exchange > with Venkat on that). > > On the long transfer etc., I was not sure what scenarios > an R2TDataSN was providing recovery from. Since you clarified > that it is to recover from a header digest error, we can > focus on that scenario. > > > This interesting for long transfers that have > > several outstanding R2Ts. > > What is speculative here? There was never a consensus that there > > will be no > > more than one outstanding R2T. > > > > Regards, > > Julo > > > > "Somesh Gupta" on 06/03/2001 17:23:04 > > > > Please respond to someshg@yahoo.com > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: R2TDataSN > > > > > > > > > > I beg to disagree. If an R2T PDU (header) has bad digest, or any other > > header has a bad digest - since you always need the PDU length from > > the header, there is some uncertainty associated with further > processing. > > > > Are you proposing that the processing machine go into a "speculative" > > mode where the processing of the next PDU determines whether we were > > successfuly able to skip a bad PDU header? When there is a data digest > > error, further stream parsing is deterministic. But not when the PDU > > header digest error. > > > > Also the consensus (in my interpretation) was on applications > > not transfering very large amounts of data using a single command or > > read/write PDU. > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Monday, March 05, 2001 5:41 PM > > > To: ips@ece.cmu.edu > > > Subject: Re: R2TDataSN > > > > > > > > > > > > > > > Somesh, > > > > > > 1.The only consensus I heard is not to transfer a large amount of > > > data with > > > one PDU. > > > > > > 2.With DatasN and Sack we dont need any data in a bad header. > > > > > > 3. If an R2T is lost (received at initiator with bad digest) - the > > > initiator will know that from > > > the next R2T if the target has several outstanding - very likely at > long > > > distances - and will not have to way for a timeout. Other uses are > > > marginal. Basically it is "part of a command execution" and we can > > > painless recover > > > from failures for this case too. > > > > > > Regards, > > > Julo > > > > > > "Somesh Gupta" on 05/03/2001 20:40:06 > > > > > > Please respond to someshg@yahoo.com > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: R2TDataSN > > > > > > > > > > > > > > > > R2TDataSN > > > > ---------- > > > > Sec 6.7.1 has some new content on how to handle lost R2Ts using > > > > SACKs. But I noticed that the SACK request (Sec 2.16) has not > > > > changed to indicate whether the DataSN is a R2T DataSN or just > > > > a Read PDU DataSN (D bit) > > > > So do we demux on the read/write operation type? > > > > And how does this affect PDU loss in bidirectional commands ? > > > > +++ SACK is ascking for data (DataSN) the target knows > > > > > > > > > > Julian, > > > > > > Regarding the R2TDataSN, I have a comments and a > > > question. > > > > > > I think that when a PDU header fails a CRC/checksum check etc, > > > it is a problem to depend on any information in the header (including > > > length fields), thereby making any further processing on > > > the connection unreliable. > > > > > > What scenarios do you envision where the R2TDataSN is useful. > > > IN Orlando I think there was clear consensus that application > > > do not try to transfer very large amounts of data using a > > > single command. > > > > > > Thanks, > > > Somesh > > > > > Thanks, > Somesh Gupta > > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com > > > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Wed Mar 7 11:11:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA09097 for ; Wed, 7 Mar 2001 11:11:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f27DbJm17801 for ips-outgoing; Wed, 7 Mar 2001 08:37:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f27DaLr17729 for ; Wed, 7 Mar 2001 08:36:22 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id OAA146522 for ; Wed, 7 Mar 2001 14:36:13 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id OAA39928 for ; Wed, 7 Mar 2001 14:34:35 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A08.004A91B4 ; Wed, 7 Mar 2001 14:34:29 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 7 Mar 2001 15:20:46 +0200 Subject: RE: R2TDataSN and other recovery mechanisms Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, OK - so what? You will get another digest error (if choose a good digest -:)) and look for the next marker. Regards, Julo "Somesh Gupta" on 07/03/2001 04:09:31 Please respond to someshg@yahoo.com To: Julian Satran/Haifa/IBM@IBMIL, someshg@yahoo.com cc: Subject: RE: R2TDataSN and other recovery mechanisms Julian, The point was not about whether the markers were covered by digests. The point is about the fact that errors occured in that range on the TCP stream that were caught (header digest error) - it does seem likely that the marker might have an error too? Somesh > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Tuesday, March 06, 2001 3:52 PM > To: someshg@yahoo.com > Subject: RE: R2TDataSN and other recovery mechanisms > > > > > Markers are not covered by digests (they are not supposed to be seen at > digest formation. > The layer model appears in the draft. > > Regards, > Julo > > "Somesh Gupta" on 06/03/2001 20:20:56 > > Please respond to someshg@yahoo.com > > To: "Venkat Rangan" , someshg@yahoo.com, > Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: R2TDataSN and other recovery mechanisms > > > > > Venkat, > > You are right but this does put some additional requirements > on the sync and steering layer (assuming we can agree on one > and that it is not optional). > > Consider the marker mechanism as an example. If the marker > falls within the TCP byte stream region which contains the > header, and the header digest fails, then do you trust this > marker or skip to the next marker. Or do we need a sum on the > marker itself. > > We do have various recovery mechanisms in the protocol without > detailed state machines or other documentation to make sure > that the mechanisms are really robust and worth including. And > even more important, that we will have interoperable > implementations that are implemented to the standard (rather > than being compliant with some defacto implementation) > > I am not saying that these tools do not work, and Julian > (and other smart people)may even have worked them out. > However, the proof is not in the document. > > Thanks, > Somesh > > > -----Original Message----- > > From: Venkat Rangan [mailto:venkat@rhapsodynetworks.com] > > Sent: Tuesday, March 06, 2001 10:00 AM > > To: someshg@yahoo.com; julian_satran@il.ibm.com; ips@ece.cmu.edu > > Subject: RE: R2TDataSN > > > > > > Somesh, > > > > > When there is a data digest error, further stream parsing is > > > deterministic. But not when the PDU header digest error. > > > > Isn't it true that with Synch and Steering layer, you are able to > > skip past > > PDUs with header-digest errors and reach a point where you can begin > > deterministically processing the stream? (assuming that the > > headers used for > > Synch and Steering are in-tact). > > > > Of course, in the event that this option is "negotiated out" you > > don't have > > this ability. > > > > Venkat Rangan > > Rhapsody Networks Inc. > > http://www.rhapsodynetworks.com > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Somesh Gupta > > Sent: Tuesday, March 06, 2001 7:23 AM > > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > > Subject: RE: R2TDataSN > > > > > > I beg to disagree. If an R2T PDU (header) has bad digest, or any other > > header has a bad digest - since you always need the PDU length from > > the header, there is some uncertainty associated with further > processing. > > > > Are you proposing that the processing machine go into a "speculative" > > mode where the processing of the next PDU determines whether we were > > successfuly able to skip a bad PDU header? When there is a data digest > > error, further stream parsing is deterministic. But not when the PDU > > header digest error. > > > > Also the consensus (in my interpretation) was on applications > > not transfering very large amounts of data using a single command or > > read/write PDU. > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Monday, March 05, 2001 5:41 PM > > > To: ips@ece.cmu.edu > > > Subject: Re: R2TDataSN > > > > > > > > > > > > > > > Somesh, > > > > > > 1.The only consensus I heard is not to transfer a large amount of > > > data with > > > one PDU. > > > > > > 2.With DatasN and Sack we dont need any data in a bad header. > > > > > > 3. If an R2T is lost (received at initiator with bad digest) - the > > > initiator will know that from > > > the next R2T if the target has several outstanding - very likely at > long > > > distances - and will not have to way for a timeout. Other uses are > > > marginal. Basically it is "part of a command execution" and we can > > > painless recover > > > from failures for this case too. > > > > > > Regards, > > > Julo > > > > > > "Somesh Gupta" on 05/03/2001 20:40:06 > > > > > > Please respond to someshg@yahoo.com > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: R2TDataSN > > > > > > > > > > > > > > > > R2TDataSN > > > > ---------- > > > > Sec 6.7.1 has some new content on how to handle lost R2Ts using > > > > SACKs. But I noticed that the SACK request (Sec 2.16) has not > > > > changed to indicate whether the DataSN is a R2T DataSN or just > > > > a Read PDU DataSN (D bit) > > > > So do we demux on the read/write operation type? > > > > And how does this affect PDU loss in bidirectional commands ? > > > > +++ SACK is ascking for data (DataSN) the target knows > > > > > > > > > > Julian, > > > > > > Regarding the R2TDataSN, I have a comments and a > > > question. > > > > > > I think that when a PDU header fails a CRC/checksum check etc, > > > it is a problem to depend on any information in the header (including > > > length fields), thereby making any further processing on > > > the connection unreliable. > > > > > > What scenarios do you envision where the R2TDataSN is useful. > > > IN Orlando I think there was clear consensus that application > > > do not try to transfer very large amounts of data using a > > > single command. > > > > > > Thanks, > > > Somesh > > > > > > _________________________________________________________ > > > Do You Yahoo!? > > > Get your free @yahoo.com address at http://mail.yahoo.com > > > > > > > > > > > > > _________________________________________________________ > > Do You Yahoo!? > > Get your free @yahoo.com address at http://mail.yahoo.com > > > > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com > > > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Wed Mar 7 15:19:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA21242 for ; Wed, 7 Mar 2001 15:18:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f27HEUS01853 for ips-outgoing; Wed, 7 Mar 2001 12:14:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hub1.san-jose.webnexus.net (hub1-20.san-jose.webnexus.net [209.140.224.20]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f27HE4r01828 for ; Wed, 7 Mar 2001 12:14:04 -0500 (EST) Received: from rmaranon (h-209-140-231-182.webnexus.com [209.140.231.182] (may be forged)) by hub1.san-jose.webnexus.net (8.9.1a/8.9.1/WN-1.4) with SMTP id JAA09486; Wed, 7 Mar 2001 09:13:26 -0800 (PST) From: "Renato E. Maranon" To: , "'Tanjore K. Suresh'" Cc: Subject: RE: iSCSI: Naming and Discovery Draft... Date: Wed, 7 Mar 2001 09:12:24 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: <001501c0a6ea$52017840$49c811ac@sanrad.co.il> Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit My two cents. I like to suggest "Target Reserved", "Target Reservation Conflict" or "Target Committed". For this condition to occur, the target may not be necessarily busy, but just out of resources, or can only handle one. "Target Busy" seems to imply busy. Building on Mark's desciption below, something like: The target has committed resources to one or more initiators and cannot handle another one. The initiator MAY try again later. This can be the case for simple devices that can handle only one initiator at a time, or for a target that has does not have the resources to support one more initiator. In contrast to the previous examples, this rejection is temporary. Renato Maranon Maranti Networks, Inc 920 Hillview Court Milpitas, Ca 95035 Phone: 408-719-9600 x309 Fax: 408-719-9631 email: rmaranon@marantinetworks.com home: www.marantinetworks.com -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Yaron Klein Sent: Wednesday, March 07, 2001 1:38 AM To: 'Tanjore K. Suresh' Cc: ips@ece.cmu.edu Subject: RE: iSCSI: Naming and Discovery Draft... Tanjore, Some more comments: The error statuses codes on Appendix B are not synchronized with the main draft. We will fix it. The term "target conflict" was borrowed from HTTP. Mark clarified this scenario well. I would like to add that this status enables better resolution and knowledge to the target. That is, in those cases the target can just not open the connection or just reject it like server error. However, this will not give indication of the situation as described by Mark. Regards, Yaron -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Mark Bakke Sent: Tuesday, March 06, 2001 6:51 PM To: Tanjore K. Suresh Cc: kaladhar@us.ibm.com; ips@ece.cmu.edu Subject: Re: iSCSI: Naming and Discovery Draft... Tanjore- Thanks for the feedback. I can comment on #3: "Tanjore K. Suresh" wrote: > 3. Appendix B, B.4.5, > Target Conflict 45 doesnot seem to be appropriate. > > I have not reviewed all the documents yet to give a > recommendation and hence cannot give, but feel > " Target Conflict" doesnot > convey the meaning of the Scenario indicating > case of " simple devices that can handle one device or > the target had reached the limit of its Initiators' capacity." Perhaps we chose the wrong term for this one. How about if call it "Target Busy", and slightly re-word it? The target is busy with another initiator and cannot handle another one. The initiator MAY try again later. This can be the case for simple devices that can handle only one initiator at a time, or for a target that has does not have the resources to support one more initiator. In contrast to the previous examples, this rejection is temporary. -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Wed Mar 7 15:22:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA21459 for ; Wed, 7 Mar 2001 15:22:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f27IHQ706203 for ips-outgoing; Wed, 7 Mar 2001 13:17:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f27IHBr06186 for ; Wed, 7 Mar 2001 13:17:11 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id A3955A07; Wed, 7 Mar 2001 11:17:10 -0700 (MST) Received: from axcsbh2.cos.agilent.com (axcsbh2.cos.agilent.com [130.29.152.144]) by msgrel1.cos.agilent.com (Postfix) with SMTP id 362DF22; Wed, 7 Mar 2001 11:17:10 -0700 (MST) Received: from 130.29.152.144 by axcsbh2.cos.agilent.com (InterScan E-Mail VirusWall NT); Wed, 07 Mar 2001 11:17:08 -0700 (Mountain Standard Time) Received: by axcsbh2.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Wed, 7 Mar 2001 11:17:08 -0700 Message-ID: From: vince_cavanna@agilent.com To: julian_satran@il.ibm.com Cc: pat_thaler@agilent.com, ips@ece.cmu.edu, Dafna_Sheinwald@il.ibm.com, vince_cavanna@agilent.com Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Date: Wed, 7 Mar 2001 11:17:05 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Julian, I will attempt to answer your questions about the probability of undetected error since your question strikes to the heart of what I was trying to point out in my I-D on CRCs vs. Checksums, i.e. where CRCs shine and where they do are not as good as many people think. For a link where random noise with Gaussian probability distribution and White spectral density is the source of errors, it is easy to determine the probability, Pu, of undetected error. This type of link is not only amenable to analysis but is a good model for many well designed links. For this type of link the BER (bit error ratio) is determined by the probability that the noise amplitude will exceed the signal amplitude and can be easily shown to be BER(z) = 1/2 erfc(z/sqroot(2)) where erfc is the complementary error function and z is the SNR (signal to noise ratio), that is, the signal amplitude as a multiple of the standard deviation [note1] of the noise. For example, at a SNR of 7, the BER is 10^(-12). With this noise mechanism and small BER, patterns with a small number of erroneous bits are far more probable than those with large numbers or errors and the probability of getting a certain number of errors, x, decreases monotonically and rapidly with increasing x. I have made a mathcad worksheet showing this dramatically in graphical form. Ask me if you want it. The number of errors in a pattern is a random variable with binomial distribution. The probability of x errors in n bits, where the probability of each bit being in error is BER, may be gotten from the binomial distribution with parameters n and p. The number of independent trials, n, is the number of bits; the probability of success in each trial, p, is the BER. The binomial distribution allows us to calculate the probability of a pattern with a certain number of errors and we can use it to find a bound for the probability of x errors at a given BER [note 2]. Let's take an ethernet-like link using some kind of group encoding such as 8B:10B. It is trivial to design a CRC polynomial that will catch all two bit errors. Two or less bits in error result in no more than 2 bursts of size 8 (due to 8B:10B encoding). If the CRC is designed to catch all such double bursts (easy to do and the CRC-32 used in ethernet does) all such errors are caught. An undetected error thus occurs only if there are 3 or more bits in error on the link within a frame - this is conservative since most such errors will also be caught but I will assume none are. I am also neglecting the fact that it is easy to design a CRC to catch all 3 bit errors. I don't want to take advantage of it because 3 bit errors may (again due to group encoding) result in 3 bursts which are not so easy to detect. With this error mechanism large bursts happen because of the group encoding. Without group encoding it would be extremely unlikely to get a burst of size 3 or larger. The probability of 3 or more bits is approximately equal to the probability of exactly 3 bits since as long as p << 0.5 the probability of a pattern having x errors decreases rapidly as x increases. For a BER of 10^(-12) the prob of 3 or more errors in 1500*8 bits (ethernet frame) is approximately the probability of exactly 3 errors in 1500*8 bits which is 5.62 x 10^(-25). At 10 gig this represents a mean time between undetected errors of 5.6 x 10^6 years. This assumes we catch all 2 bit errors. If we could only catch single bit errors the mean time between undetected errors would be 10 days! What a difference a bit makes!! Also the probability of exactly 4 errors is 2.1 x10^(-33). Again, what a difference a bit makes! I have a mathcad spreadsheet where I have done this analysis for 10 gig ethernet. Let me know if you are interested in a copy. This is why CRC is so nice for a link with this error source. We can claim practically complete protection. Ther are no holes. For cases, such as iSCSI protection environment, where the BER is not determined by the SNR then we may not be able to claim that a small number of erroneous bits are far more probable than a large number of them and the analysis is not as simple and in such cases CRCs are not necessarily better than checksums - it depends on how probable the patterns of errors that are undetected are. Vince notes: [1] strictly speaking I should have said rms value instead of standard deviation but for a random process for which the time statistics are the same as the ensemble statistics (an ergodic process) the two are equivalent. [2] Binomial distribution wiht parameters n and p: P(x,n,p) = (n!/(x!*(n-x))*(p^x(1-p)^(n-x)) |-----Original Message----- |From: THALER,PAT (A-Roseville,ex1) |Sent: Tuesday, March 06, 2001 4:47 PM |To: julian_satran@il.ibm.com; ips@ece.cmu.edu |Cc: THALER,PAT (A-Roseville,ex1); ips@ece.cmu.edu; |Dafna_Sheinwald@il.ibm.com; CAVANNA, VICENTE |Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt | | |Julian, | |Good summary. One correction. "codes shorter than m" is |shorter than m values not m bits and m is about 64 K. |Therefore, it is shorter than about 63 kbytes for Adler and |shorter than 127 kbytes for Fletcher. | |I'm thinking about your question about Pud but it will take me |a bit to come up with an answer. For the three bit error case, |I think the formulas used for CRC bit errors will be |approximately correct. For the burst errors and the errors |where two (Fletcher only) or three non-contiguous bytes are |corrupted, I'm not sure what to use for the probability that a |byte gets corrupted. | |Regards, |Pat | |-----Original Message----- |From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] | |Pat, | |Thanks. I am not going to check every line for awhile and I |assume it is |correct. |For the benefit of the list the summary of the statement is that: | |-the burst protection is for 16 bit (I recall the text saying 32) |-the numbers to be used for low noise symmetric channel protection are: | - dmin= 3 for codes shorter than m (approx 8kbytes) | - dmin= 2 for codes longer than m | |Do you have a formula to evaluate Pud for low noise channels |with BER = e? |Any formula to evaluate the conditional probability of an |undetected error |on bursts of 17, 18, 19 bits? | |Is there any reason to believe hat we can approximate them to |ones used for |CRC? | |I would like to have all the codes readily comparable. | |Regards, |Julo | |"THALER,PAT (A-Roseville,ex1)" on 06/03/2001 |19:24:13 | |Please respond to "THALER,PAT (A-Roseville,ex1)" | | |To: Julian Satran/Haifa/IBM@IBMIL, "THALER,PAT (A-Roseville,ex1)" | |cc: ips@ece.cmu.edu, Dafna_Sheinwald@il.ibm.com |Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt | | | | |Julian, | |It is relatively easy to investigate the Hamming distance and burst |detection of Adler32 and Fletcher32 using mathematics. In the argument |below, I will use "value" to identify the quantities that get |summed since |Adler sums bytes and Fletcher32 sums 16-bits. | |Both Adler and Fletcher calculate for each value: | | S1 = (S1 + value) mod m | S2 = (S2 + S1) mod m | |where m is 65521 for Adler and 65535 for Fletcher. | |Which means that for an n value string of values V1 to Vn: | S1 = (V1 + V2 + ... + Vn (+ 1 for Adler)) mod m | S2 = (n * V1 + (n-1) * V2 + ... + Vn (+ n for Adler)) mod m | |A two bit error can only change two values. For a two bit error to be |undetectable the two changed values must produce no change to |S1 and S2. | |Let | x be the change to the first errored value | y be the change to the second errored value | k be the separation between the first and second errored values. | |Then for an undetectable error: | | delta S1 = (x + y) mod m = 0 | delta S2 = ((k + 1) * x + y) mod m = 0 | = (k * x + x + y) mod m =0 | = (k * x) mod m = 0 because (x + y) mod m = 0 | |Therefore, for an undetectable 2 bit error, k * x must be a |multiple of m. | |For Adler, m is prime and the maximum value of x is 255 so the |only way to |satisfy the condition is for k to equal 65521. Then x can be |any value so |it |could be a one bit error such as changing the lsb of the first |value from 0 |to 1. y would then need to be the inverse changing the lsb of |the second |value from 1 to 0. | |For Fletcher, m is factorable to 3, 5, 17, 257. For x to provide one or |more |of those factors, it would have to have more than a 1 bit change and y |would |have to have the inverse of that change which would mean an error of at |least 4 bits. The only way to get a two bit undetectable error |is for k to |equal 65535. | |So, Fletcher and Adler do have 2-bit undetectable errors if |the messages |are |longer than the modulus. | |What about burst protection? For Fletcher, we know that a |change of a value |from all 0's to all 1's produces an an undetectable error so |we know that |burst protection is less than 16. k = 1 for a burst spread across two |consecutive values. Therefore, the only way for a burst spread |across two |consecutive values to produce an undetectable error is for x to equal |65535. |A burst shorter than 16 can't do it. | |For Adler, corrupting two consecutive bytes can't produce an |undetectable |error because that would be the case above where k equals 1. |There is no |way |for a burst across two consectutive values to be undetectable in Adler |since |x is less than the modulus. | |For a burst across three values: | |let x equal the error in the first value | y equal the error in the second value | z equal the error in the third value. | |For the burst to be undetectable | | Delta S1 = (x + y + z) mod m = 0 | Delta S2 = (3x + 2y + z) mod m = 0 | |Since the maximum values of x, y, and z are 255 and m is more |than 6 times |that, the sums above will always be less than m and we can |drop the mod m. | | Delta S2 = (3x + 2y + z) = 2x + y + (x + y + z) = 0 | = 2x + y = 0 because x + y + z = 0 | | y = - 2x | | Delta S1 = x + y + z = 0 | = x - 2x + z = 0 | = -x + z | z = x | |So an undetectable error is created when in three consecutive |values the |errors are | x | -2x | x | |This could for instance be errors of 1, -2, and 1. Since the |error to the |first and third values are the same, the error must span at |least 2 * the |length of the value + 1. So for Adler, it must be a 17 bit |burst at least. | |Also, note that this can be created by a 3 bit error and is equally |applicable to Adler and Fletcher. | |Therefore, for messages shorter than the modulus + 1, the |Hamming distance |of Adler32 and Fletcher 32 is 3 and, for messages longer than that, the |Hamming distance is 2. | |Regards, |Pat Thaler | | | | | |-----Original Message----- |From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] | |Pat, | |I did not run a check. That is very expensive. |With CRCs there are methods that enable you to run a check on the |complement code |(that has only 2**32 different patterns for any block length) |and derive |from there the distances (Fujiwara has done this in 1989 for the IEEE |CRC-32 and about some more recent experiments I'll get back to |the list). | |And that is the trouble with this whole line of argument. |There are no numbers to prove Adler32 or Fletcher32 and there |are plenty |for CRCs. | |The big question is then is there anybody out there that wants |to build a |modern bridge based only on its beauty? | |Regards, |Julo | | |pat_thaler@agilent.com on 05/03/2001 23:27:49 | |Please respond to pat_thaler@agilent.com | |To: Julian Satran/Haifa/IBM@IBMIL |cc: |Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt | | | | |Julian, | |I know that Hamming distance gets down to 2 for errors that |are separated |by |the modulus (or a multiple of it). Is there another case? | |Pat | |> - Adler and Fletcher are weak and there is no theory behind your |> distribution statements, nor any simulation results as far |as I know. We |> found that on very simple sequences the Hamming distance gets |> down to 2 (or |> lower) and the burst protection is probably not better than 16 bit. |There |> is even a simple formula for what sequences will get you |false codes (see |> bellow for a reference) |> | | | | | From owner-ips@ece.cmu.edu Wed Mar 7 15:28:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA21647 for ; Wed, 7 Mar 2001 15:28:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f27JaR911698 for ips-outgoing; Wed, 7 Mar 2001 14:36:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f27JY3r11524 for ; Wed, 7 Mar 2001 14:34:04 -0500 (EST) Received: from ypportable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id 11CPR157; Wed, 7 Mar 2001 11:34:01 -0800 From: "Y P Cheng" To: "'Ips@Ece. Cmu. Edu'" Cc: Subject: FW: PKI tech talk continued Date: Wed, 7 Mar 2001 11:30:03 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I received the followin email from Cylink, a company does IPSec. It makes sense to me. Y.P. , Connectcom Solutions. -----Original Message----- From: Carlin Covey [mailto:ccovey@cylink.com] Sent: Wednesday, March 07, 2001 8:07 AM It sounds like their should be some "cross-pollination" between iSCSI and TCPA (Trusted Computer Platform Alliance). The TCPA was founded by Compaq, HP, IBM, Intel and Microsoft to promote new standards for next generation computers. They have just released the first version of their specification. It provides support for PKI (secure storage for private keys and trusted roots) and makes use of PKI to authenticate various computer components. It seems to me that the same authentication methods would work regardless of whether those components communicate over a bus or over a TCP/IP connection. I think NetAuthority would be willing to provide PKI support. - Carlin From owner-ips@ece.cmu.edu Wed Mar 7 15:30:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA21693 for ; Wed, 7 Mar 2001 15:30:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f27J4RM09345 for ips-outgoing; Wed, 7 Mar 2001 14:04:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([208.50.99.84]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f27J49r09326 for ; Wed, 7 Mar 2001 14:04:10 -0500 (EST) Received: from muralipc ([192.168.2.51]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id LAA17291; Wed, 7 Mar 2001 11:03:57 -0800 (PST) From: "Murali Rajagopal" To: Cc: "Murali Rajagopal" , "Black_David@emc. com" , "Elizabeth G Rodriguez \(Elizabeth\)" Subject: FCIP Draft and Author Discussions Summary Date: Wed, 7 Mar 2001 11:05:10 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 In-Reply-To: <0F31E5C394DAD311B60C00E029101A07041015E4@corpmx9.isus.emc.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit A summary of the recent discussions amongst the FCIP authors appears below. 1) FCIP Models: Port modeling discussion centers on how the TCP/IP port fits into the FC network and how to coordinate the working and inter-working of the FC and TCP worlds. The main area of discussion has been the lack of an interface between an FCIP device and the FC fabric. The discussion has been led by FC experts from T11. < A more detailed summary of the proposals that reached some consensus appaers later in this email> 2) FCIP discovery (Facing Internet)to possibly adopt iSCSI methods if suitable. 3) Common encapsulation for FCIP and iFCP being floated as seperate efforts. Outcome of Minneapolis meeting will tell what formats to use. A new draft incorporating *some* of above items will be released in the near future including: 1) change the stack diagram (fig. 2) to reflect interface to FC-SW 2) section 5.2: add paragraph to explain TCP-FC frame mapping. 3) fig. 4 modified to show a non-one-to-one mapping of FC frame inside a TCP segment as suggested by R. Weber. 4) section 9: replace a paragraph discussing fragmentation and performance. 5) annex A: pare down list of byte-codes that FCIP support to those supported by FC-BB as discussed in Orlando. *****************FCIP Models Deatiled Summary *************************** ========================================================= +-----+ +-----+ | SW1 | | SW3 | +-----+ +-----+ | +-----+ +-----+ | | | | | | | +-----+E B| BBW | /\/\/\/\/\/\ | BBW |E E+-----+ | SW2 |----| 1 |---\ \---| 2 |----| SW4 | +-----+ | | / / | | +-----+ +-----+ \ \ +-----+ / / <========= Virtual ISL =======> (SW-2 to BBW 2) <========= Virtual ISL =======> (SW-2 to BBW 4) <== Virtual ISL =======> (BBW 3 to BBW 2) \ \ / IP / +-----+ \ Network \ +-----+ | | / / | | +-----+E E| BBW | \ \ | BBW |B E+-----+ | SW6 |----| 3 |---/ /---| 4 |----| SW7 | +-----+ | | \/\/\/\/\/\/ | | +-----+ | +-----+ +-----+ | | | +-----+ +-----+ | SW5 | | SW8 | +-----+ +-----+ ======================================================== The Fibre Channel model for an FCIP device is proposed to be an extension of the Backbone WAN (BBW) device currently defined for ATM (BBW_ATM) and SONET (BBW_SONET) in FC-BB. The planned extensions to the current BBW model include: 1) Defining the Fibre Channel operational details of a BBW_TCP/IP device 2) Providing for three Fibre Channel interconnection configurations transported by BBW_TCP/IP devices: a) [B_Port]..BBW_TCP/IP<---->BBW_TCP/IP..[B_Port] b) [E_Port]..BBW_TCP/IP<---->BBW_TCP/IP..[E_Port] c) [B_Port]..BBW_TCP/IP<---->BBW_TCP/IP..[E_Port] Note: The mirror case of c) is operationally identical to c). 3) Defining Virtual inter-switch links (ISL's) between the E_ports on BBW_TCP/IP devices. Because substantial Fibre Channel fluency is required for the BBW_TCP/IP definition, it is anticipated that FCIP will contain only a high level overview of the model, with the details appearing in FC-BB-2 (an existing T11 project created specifically to extend the Backbone concept to TCP/IP). Regards, Murali Rajagopal IPS WG FCIP TC From owner-ips@ece.cmu.edu Wed Mar 7 16:06:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA22665 for ; Wed, 7 Mar 2001 16:06:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f27JBYb09820 for ips-outgoing; Wed, 7 Mar 2001 14:11:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f27JBLr09807 for ; Wed, 7 Mar 2001 14:11:21 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Wed, 7 Mar 2001 11:11:15 -0800 Message-ID: From: Joshua Tseng To: "Tanjore K. Suresh" , kaladhar@us.ibm.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI: Naming and Discovery Draft... Date: Wed, 7 Mar 2001 11:11:13 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Tanjore, See below: > > Voruganti, > > > > Here are some minor comments/feedback reading this document. > > 1. Section 5.2 Discovery Domain > > DD Symbolic name > DD ID > > DD Member: WWUI > DD Member: IP Address > > > What does this DD member signify, is this one iSCSI device > part of Discovery Domain or should this be a list of > iSCSI devices > in that Discovery Domain. I thought clarifying that would be > usefull for the reader. Frankly, I didnot understand what it is? There is an error in this model which will be corrected in the next revision of this document and the iSNS document. DD_MEMBER should be another object in the model which is contained by DISCOVERY_DOMAIN. DD_Member is an attribute which identifies the WWUI or IP address that is a member of the Discovery Domain. DD_ID is another attribute of the DD_MEMBER object that identifies the DISCOVERY_DOMAIN of the member. Josh > > > 2. Appendix A iSCSI WWUI Notes... > > It looks to me the examples under > > "Using Intitiator and Target WWUI During Login" can be > clubbed with Appendix B > which provides the different iSCSI Login Scenarios" It looks > like some > of the scenarios presented are also in the Appendix B already > which seems > to convey the same thing. > > > 3. Appendix B, B.4.5, > Target Conflict 45 doesnot seem to be appropriate. > > I have not reviewed all the documents yet to give a > recommendation and hence cannot give, but feel > " Target Conflict" doesnot > convey the meaning of the Scenario indicating > case of " simple devices that can handle one device or > the target had reached the limit of its > Initiators' capacity." > > > > 4. Typo in page 17 > > Directory Agent(DA) is an optional part of discovery. If a DA.. > > ......... by the user Agents, to reduce the network > load of all UAs > trying to discovery all SAs > ^^^^^^^^ should be "discover". > > Thanks > sureshtk > > > > From owner-ips@ece.cmu.edu Wed Mar 7 17:48:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA25516 for ; Wed, 7 Mar 2001 17:48:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f27KOVQ15019 for ips-outgoing; Wed, 7 Mar 2001 15:24:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from crufty.research.bell-labs.com (crufty.research.bell-labs.com [204.178.16.49]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f27KNor14974 for ; Wed, 7 Mar 2001 15:23:50 -0500 (EST) Received: from scummy.research.bell-labs.com ([135.104.2.10]) by crufty; Wed Mar 7 15:20:45 EST 2001 Received: from aura.research.bell-labs.com ([135.104.46.10]) by scummy; Wed Mar 7 15:22:32 EST 2001 Received: from research.bell-labs.com (IDENT:sandeepj@sandeepj-pcmh.research.bell-labs.com [135.104.47.90]) by aura.research.bell-labs.com (8.9.1/8.9.1) with ESMTP id PAA24580 for ; Wed, 7 Mar 2001 15:22:32 -0500 (EST) Message-ID: <3AA69888.51C3DF15@research.bell-labs.com> Date: Wed, 07 Mar 2001 15:22:32 -0500 From: Sandeep Joshi X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.2.16-3 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSNS comments Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit some comments on the new iSNS draft 1) it lacks a Table of contents, unlike iSCSI. please add one. (given that, i am liable to have missed some section which might have answered the questions raised below!) 2) What happens if an iSNS client tries to update its TCP/UDP port or IP address (for Portal/ESI/etc) ? So the entry exists, and now the client sends a RegDevAttr request with the update flag set for changing the above. If the port(s) is going to be well-known, the questions below may not arise. If not,... a) Will the old TCP connection be broken by the iSNS server ? b) Would the RegDevResponse be sent to the old/new port ? c) Who initiates the new connection (client or server) ? d) How would the client know the request succeeded ? 3) Is there a requirement to provide (keyword=MUST) a secondary iSNS server ? If I am not mistaken, DNS does mandate a secondary server for every zone to avoid single-pt-of-failure. -Sandeep From owner-ips@ece.cmu.edu Wed Mar 7 21:32:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA00157 for ; Wed, 7 Mar 2001 21:32:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f280XZG00938 for ips-outgoing; Wed, 7 Mar 2001 19:33:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f280Wkr00908 for ; Wed, 7 Mar 2001 19:32:46 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Wed, 7 Mar 2001 19:32:39 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101634@corpmx9.isus.emc.com> From: Black_David@emc.com To: vchau@gadzoox.com, ips@ece.cmu.edu Subject: RE: FCIP iFCP encapsulation proposal Date: Wed, 7 Mar 2001 19:32:38 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, > I sent out a draft to propose a common encapsulation scheme for > FCIP and iFCP last week; I am afraid I have missed the cut-off time > for new drafts. Here is a summary of the draft to help stimulate > discussion at the upcoming IETF meeting: Due to the importance of making progress on this topic in Minneapolis, this will be an exception to the usual rule of not discussing drafts that miss the draft cutoff. Please don't assume that exceptions of this sort will be readily available in the future. Also, everyone should remember that all IETF deadlines are in Eastern Time (contrary to popular rumor, Silicon Valley is not the center of the known universe) and hence should allow for time zone differences in figuring out when to get drafts submitted. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Wed Mar 7 22:25:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA01614 for ; Wed, 7 Mar 2001 22:25:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f281faE04575 for ips-outgoing; Wed, 7 Mar 2001 20:41:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f281ecr04478 for ; Wed, 7 Mar 2001 20:40:38 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 9236A8EA for ; Wed, 7 Mar 2001 18:40:37 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 757C4143 for ; Wed, 7 Mar 2001 20:40:35 -0500 (EST) Received: from agilent.com (cos1nai133249.cos.agilent.com [141.184.133.249]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id RAA29774 for ; Wed, 7 Mar 2001 17:40:31 -0800 (PST) Message-ID: <3AA6B8FD.DD83C75D@agilent.com> Date: Wed, 07 Mar 2001 14:41:01 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI error recovery References: <0F31E5C394DAD311B60C00E029101A07041014D0@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Black_David@emc.com wrote: > - Given some of the CRC discussion, I think the conclusion > in Orlando to have separate header and data CRCs is > NOT the rough consensus of the WG. We need to go > back to a requirements discussion on this rather than > debating exactly where to put the CRCs. Would those > envisioning middleboxes/gateways/etc. that would > benefit from this sort of CRC separation please post > short use cases/descriptions indicating the basic > functionality of the box and which fields it needs > to change (let's do this with reference to the header > layout in -03 rather than subsequent changes)? As > part of the use case/description, please explain > how/why Fibre Channel's single CRC covering both > the frame header and data causes problems/difficulties. David, The use of a separate header/data CRC is required in order to perform efficient data steering. If the iSCSI header and data are both protected by the same CRC (which for efficiencies in transmission would be after the data), then an iSCSI adapter would have to buffer up an entire iSCSI PDU before it could begin DMAing the PDU payload to host memory (because it could not guarantee that the information in the iSCSI header was correct to begin DMAing it earlier). Since Julian insists that PDUs could be in lengths requiring 32 bit length fields, this is a lot of unnecessary storage. -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Wed Mar 7 22:25:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA01624 for ; Wed, 7 Mar 2001 22:25:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f281fXr04570 for ips-outgoing; Wed, 7 Mar 2001 20:41:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f281ewr04543 for ; Wed, 7 Mar 2001 20:40:58 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id F005955A; Wed, 7 Mar 2001 18:40:57 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 2ACC8141; Wed, 7 Mar 2001 20:40:56 -0500 (EST) Received: from agilent.com (cos1nai133249.cos.agilent.com [141.184.133.249]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id RAA29818; Wed, 7 Mar 2001 17:40:52 -0800 (PST) Message-ID: <3AA6C842.A41FE75E@agilent.com> Date: Wed, 07 Mar 2001 15:46:10 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: R2TDataSN References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > 1.The only consensus I heard is not to transfer a large amount of data with > one PDU. Ok, then why do you have "Long Data Header" in the document? -Matt From owner-ips@ece.cmu.edu Wed Mar 7 22:26:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA01648 for ; Wed, 7 Mar 2001 22:26:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f281egF04487 for ips-outgoing; Wed, 7 Mar 2001 20:40:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f281dUr04425 for ; Wed, 7 Mar 2001 20:39:30 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id CAA59978; Thu, 8 Mar 2001 02:39:15 +0100 From: julian_satran@il.ibm.com Received: from d12mta05.de.ibm.com (d12mta06_cs0 [9.165.222.255]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id CAA110228; Thu, 8 Mar 2001 02:38:16 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A09.0009130C ; Thu, 8 Mar 2001 02:39:07 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: vince_cavanna@agilent.com cc: pat_thaler@agilent.com, ips@ece.cmu.edu, Dafna_Sheinwald@il.ibm.com, vince_cavanna@agilent.com Message-ID: Date: Thu, 8 Mar 2001 03:38:44 +0200 Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Vince, We have now a good handle and plethora of references for CRC behavior in low noise symmetric channels (the ones you consider) and on bursts. To complete the comparison we will need formulas for the checksums you propose (Adler and Fletcher 32). We will corroborate it with information from other sources wherever available. Regards, Julo vince_cavanna@agilent.com on 07/03/2001 20:17:05 Please respond to vince_cavanna@agilent.com To: Julian Satran/Haifa/IBM@IBMIL cc: pat_thaler@agilent.com, ips@ece.cmu.edu, Dafna_Sheinwald@il.ibm.com, vince_cavanna@agilent.com Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Hi Julian, I will attempt to answer your questions about the probability of undetected error since your question strikes to the heart of what I was trying to point out in my I-D on CRCs vs. Checksums, i.e. where CRCs shine and where they do are not as good as many people think. For a link where random noise with Gaussian probability distribution and White spectral density is the source of errors, it is easy to determine the probability, Pu, of undetected error. This type of link is not only amenable to analysis but is a good model for many well designed links. For this type of link the BER (bit error ratio) is determined by the probability that the noise amplitude will exceed the signal amplitude and can be easily shown to be BER(z) = 1/2 erfc(z/sqroot(2)) where erfc is the complementary error function and z is the SNR (signal to noise ratio), that is, the signal amplitude as a multiple of the standard deviation [note1] of the noise. For example, at a SNR of 7, the BER is 10^(-12). With this noise mechanism and small BER, patterns with a small number of erroneous bits are far more probable than those with large numbers or errors and the probability of getting a certain number of errors, x, decreases monotonically and rapidly with increasing x. I have made a mathcad worksheet showing this dramatically in graphical form. Ask me if you want it. The number of errors in a pattern is a random variable with binomial distribution. The probability of x errors in n bits, where the probability of each bit being in error is BER, may be gotten from the binomial distribution with parameters n and p. The number of independent trials, n, is the number of bits; the probability of success in each trial, p, is the BER. The binomial distribution allows us to calculate the probability of a pattern with a certain number of errors and we can use it to find a bound for the probability of x errors at a given BER [note 2]. Let's take an ethernet-like link using some kind of group encoding such as 8B:10B. It is trivial to design a CRC polynomial that will catch all two bit errors. Two or less bits in error result in no more than 2 bursts of size 8 (due to 8B:10B encoding). If the CRC is designed to catch all such double bursts (easy to do and the CRC-32 used in ethernet does) all such errors are caught. An undetected error thus occurs only if there are 3 or more bits in error on the link within a frame - this is conservative since most such errors will also be caught but I will assume none are. I am also neglecting the fact that it is easy to design a CRC to catch all 3 bit errors. I don't want to take advantage of it because 3 bit errors may (again due to group encoding) result in 3 bursts which are not so easy to detect. With this error mechanism large bursts happen because of the group encoding. Without group encoding it would be extremely unlikely to get a burst of size 3 or larger. The probability of 3 or more bits is approximately equal to the probability of exactly 3 bits since as long as p << 0.5 the probability of a pattern having x errors decreases rapidly as x increases. For a BER of 10^(-12) the prob of 3 or more errors in 1500*8 bits (ethernet frame) is approximately the probability of exactly 3 errors in 1500*8 bits which is 5.62 x 10^(-25). At 10 gig this represents a mean time between undetected errors of 5.6 x 10^6 years. This assumes we catch all 2 bit errors. If we could only catch single bit errors the mean time between undetected errors would be 10 days! What a difference a bit makes!! Also the probability of exactly 4 errors is 2.1 x10^(-33). Again, what a difference a bit makes! I have a mathcad spreadsheet where I have done this analysis for 10 gig ethernet. Let me know if you are interested in a copy. This is why CRC is so nice for a link with this error source. We can claim practically complete protection. Ther are no holes. For cases, such as iSCSI protection environment, where the BER is not determined by the SNR then we may not be able to claim that a small number of erroneous bits are far more probable than a large number of them and the analysis is not as simple and in such cases CRCs are not necessarily better than checksums - it depends on how probable the patterns of errors that are undetected are. Vince notes: [1] strictly speaking I should have said rms value instead of standard deviation but for a random process for which the time statistics are the same as the ensemble statistics (an ergodic process) the two are equivalent. [2] Binomial distribution wiht parameters n and p: P(x,n,p) = (n!/(x!*(n-x))*(p^x(1-p)^(n-x)) |-----Original Message----- |From: THALER,PAT (A-Roseville,ex1) |Sent: Tuesday, March 06, 2001 4:47 PM |To: julian_satran@il.ibm.com; ips@ece.cmu.edu |Cc: THALER,PAT (A-Roseville,ex1); ips@ece.cmu.edu; |Dafna_Sheinwald@il.ibm.com; CAVANNA, VICENTE |Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt | | |Julian, | |Good summary. One correction. "codes shorter than m" is |shorter than m values not m bits and m is about 64 K. |Therefore, it is shorter than about 63 kbytes for Adler and |shorter than 127 kbytes for Fletcher. | |I'm thinking about your question about Pud but it will take me |a bit to come up with an answer. For the three bit error case, |I think the formulas used for CRC bit errors will be |approximately correct. For the burst errors and the errors |where two (Fletcher only) or three non-contiguous bytes are |corrupted, I'm not sure what to use for the probability that a |byte gets corrupted. | |Regards, |Pat | |-----Original Message----- |From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] | |Pat, | |Thanks. I am not going to check every line for awhile and I |assume it is |correct. |For the benefit of the list the summary of the statement is that: | |-the burst protection is for 16 bit (I recall the text saying 32) |-the numbers to be used for low noise symmetric channel protection are: | - dmin= 3 for codes shorter than m (approx 8kbytes) | - dmin= 2 for codes longer than m | |Do you have a formula to evaluate Pud for low noise channels |with BER = e? |Any formula to evaluate the conditional probability of an |undetected error |on bursts of 17, 18, 19 bits? | |Is there any reason to believe hat we can approximate them to |ones used for |CRC? | |I would like to have all the codes readily comparable. | |Regards, |Julo | |"THALER,PAT (A-Roseville,ex1)" on 06/03/2001 |19:24:13 | |Please respond to "THALER,PAT (A-Roseville,ex1)" | | |To: Julian Satran/Haifa/IBM@IBMIL, "THALER,PAT (A-Roseville,ex1)" | |cc: ips@ece.cmu.edu, Dafna_Sheinwald@il.ibm.com |Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt | | | | |Julian, | |It is relatively easy to investigate the Hamming distance and burst |detection of Adler32 and Fletcher32 using mathematics. In the argument |below, I will use "value" to identify the quantities that get |summed since |Adler sums bytes and Fletcher32 sums 16-bits. | |Both Adler and Fletcher calculate for each value: | | S1 = (S1 + value) mod m | S2 = (S2 + S1) mod m | |where m is 65521 for Adler and 65535 for Fletcher. | |Which means that for an n value string of values V1 to Vn: | S1 = (V1 + V2 + ... + Vn (+ 1 for Adler)) mod m | S2 = (n * V1 + (n-1) * V2 + ... + Vn (+ n for Adler)) mod m | |A two bit error can only change two values. For a two bit error to be |undetectable the two changed values must produce no change to |S1 and S2. | |Let | x be the change to the first errored value | y be the change to the second errored value | k be the separation between the first and second errored values. | |Then for an undetectable error: | | delta S1 = (x + y) mod m = 0 | delta S2 = ((k + 1) * x + y) mod m = 0 | = (k * x + x + y) mod m =0 | = (k * x) mod m = 0 because (x + y) mod m = 0 | |Therefore, for an undetectable 2 bit error, k * x must be a |multiple of m. | |For Adler, m is prime and the maximum value of x is 255 so the |only way to |satisfy the condition is for k to equal 65521. Then x can be |any value so |it |could be a one bit error such as changing the lsb of the first |value from 0 |to 1. y would then need to be the inverse changing the lsb of |the second |value from 1 to 0. | |For Fletcher, m is factorable to 3, 5, 17, 257. For x to provide one or |more |of those factors, it would have to have more than a 1 bit change and y |would |have to have the inverse of that change which would mean an error of at |least 4 bits. The only way to get a two bit undetectable error |is for k to |equal 65535. | |So, Fletcher and Adler do have 2-bit undetectable errors if |the messages |are |longer than the modulus. | |What about burst protection? For Fletcher, we know that a |change of a value |from all 0's to all 1's produces an an undetectable error so |we know that |burst protection is less than 16. k = 1 for a burst spread across two |consecutive values. Therefore, the only way for a burst spread |across two |consecutive values to produce an undetectable error is for x to equal |65535. |A burst shorter than 16 can't do it. | |For Adler, corrupting two consecutive bytes can't produce an |undetectable |error because that would be the case above where k equals 1. |There is no |way |for a burst across two consectutive values to be undetectable in Adler |since |x is less than the modulus. | |For a burst across three values: | |let x equal the error in the first value | y equal the error in the second value | z equal the error in the third value. | |For the burst to be undetectable | | Delta S1 = (x + y + z) mod m = 0 | Delta S2 = (3x + 2y + z) mod m = 0 | |Since the maximum values of x, y, and z are 255 and m is more |than 6 times |that, the sums above will always be less than m and we can |drop the mod m. | | Delta S2 = (3x + 2y + z) = 2x + y + (x + y + z) = 0 | = 2x + y = 0 because x + y + z = 0 | | y = - 2x | | Delta S1 = x + y + z = 0 | = x - 2x + z = 0 | = -x + z | z = x | |So an undetectable error is created when in three consecutive |values the |errors are | x | -2x | x | |This could for instance be errors of 1, -2, and 1. Since the |error to the |first and third values are the same, the error must span at |least 2 * the |length of the value + 1. So for Adler, it must be a 17 bit |burst at least. | |Also, note that this can be created by a 3 bit error and is equally |applicable to Adler and Fletcher. | |Therefore, for messages shorter than the modulus + 1, the |Hamming distance |of Adler32 and Fletcher 32 is 3 and, for messages longer than that, the |Hamming distance is 2. | |Regards, |Pat Thaler | | | | | |-----Original Message----- |From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] | |Pat, | |I did not run a check. That is very expensive. |With CRCs there are methods that enable you to run a check on the |complement code |(that has only 2**32 different patterns for any block length) |and derive |from there the distances (Fujiwara has done this in 1989 for the IEEE |CRC-32 and about some more recent experiments I'll get back to |the list). | |And that is the trouble with this whole line of argument. |There are no numbers to prove Adler32 or Fletcher32 and there |are plenty |for CRCs. | |The big question is then is there anybody out there that wants |to build a |modern bridge based only on its beauty? | |Regards, |Julo | | |pat_thaler@agilent.com on 05/03/2001 23:27:49 | |Please respond to pat_thaler@agilent.com | |To: Julian Satran/Haifa/IBM@IBMIL |cc: |Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt | | | | |Julian, | |I know that Hamming distance gets down to 2 for errors that |are separated |by |the modulus (or a multiple of it). Is there another case? | |Pat | |> - Adler and Fletcher are weak and there is no theory behind your |> distribution statements, nor any simulation results as far |as I know. We |> found that on very simple sequences the Hamming distance gets |> down to 2 (or |> lower) and the burst protection is probably not better than 16 bit. |There |> is even a simple formula for what sequences will get you |false codes (see |> bellow for a reference) |> | | | | | From owner-ips@ece.cmu.edu Wed Mar 7 22:28:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA01675 for ; Wed, 7 Mar 2001 22:28:36 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f281fdP04579 for ips-outgoing; Wed, 7 Mar 2001 20:41:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f281f5r04550 for ; Wed, 7 Mar 2001 20:41:05 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 3A28623C for ; Wed, 7 Mar 2001 18:41:04 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id BA5A7141 for ; Wed, 7 Mar 2001 20:41:02 -0500 (EST) Received: from agilent.com (cos1nai133249.cos.agilent.com [141.184.133.249]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id RAA29826 for ; Wed, 7 Mar 2001 17:40:57 -0800 (PST) Message-ID: <3AA6CA22.1A853368@agilent.com> Date: Wed, 07 Mar 2001 15:54:10 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI: Header Digest Failure (was R2TDataSN) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, What Somesh is talking about has nothing to do with R2T specifically. His point is, the length field in the PDU header determines how long *this* PDU is, and where to find the *next* PDU. If *this* PDU fails its header digest check, you cannot trust the length field, therefore, you cannot know where the *next* PDU is in the byte stream. Given that, is seems like whenever there is a header digest error, you have no choice but to terminate the TCP connection and start a new one. -Matt Wakeley Agilent Technologies Somesh Gupta wrote: > > Julian, > > How do I know it is an R2T if the header digest failed? > I am sorry if I am missing something very obvious here. > > Let us say that a header digest failed. What information > in the header am I able to trust at this point? I don't > know if it is an R2T or not. I don't know if the length > is ok or not. > > Thanks, > Somesh > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Tuesday, March 06, 2001 4:01 PM > > To: ips@ece.cmu.edu > > Subject: RE: R2TDataSN > > > > > > > > > > Somesh, > > > > I am still lost. R2T is fixed length and you know that you have a bad > > digest after reading it. > > No length involved. ??? > > > > Julo > > > > "Somesh Gupta" on 07/03/2001 00:26:10 > > > > Please respond to someshg@yahoo.com > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: R2TDataSN > > > > > > > > > > Julian, > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Tuesday, March 06, 2001 9:31 AM > > > To: ips@ece.cmu.edu > > > Subject: RE: R2TDataSN > > > > > > > > > > > > > > > Somesh, > > > > > > You've lost me. I do not propose that you look at the bad R2tT but to > > find > > > that you have missed one > > > by looking at the next. > > > > Since iSCSI PDUs define how long they are, you have to look at > > one PDU to determine where the next PDU is. (unless ofcourse > > the sync and steering layer is doing the work - see my exchange > > with Venkat on that). > > > > On the long transfer etc., I was not sure what scenarios > > an R2TDataSN was providing recovery from. Since you clarified > > that it is to recover from a header digest error, we can > > focus on that scenario. > > > > > > This interesting for long transfers that have > > > several outstanding R2Ts. > > > What is speculative here? There was never a consensus that there > > > will be no > > > more than one outstanding R2T. > > > > > > Regards, > > > Julo > > > > > > "Somesh Gupta" on 06/03/2001 17:23:04 > > > > > > Please respond to someshg@yahoo.com > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: RE: R2TDataSN > > > > > > > > > > > > > > > I beg to disagree. If an R2T PDU (header) has bad digest, or any other > > > header has a bad digest - since you always need the PDU length from > > > the header, there is some uncertainty associated with further > > processing. > > > > > > Are you proposing that the processing machine go into a "speculative" > > > mode where the processing of the next PDU determines whether we were > > > successfuly able to skip a bad PDU header? When there is a data digest > > > error, further stream parsing is deterministic. But not when the PDU > > > header digest error. > > > > > > Also the consensus (in my interpretation) was on applications > > > not transfering very large amounts of data using a single command or > > > read/write PDU. > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > julian_satran@il.ibm.com > > > > Sent: Monday, March 05, 2001 5:41 PM > > > > To: ips@ece.cmu.edu > > > > Subject: Re: R2TDataSN > > > > > > > > > > > > > > > > > > > > Somesh, > > > > > > > > 1.The only consensus I heard is not to transfer a large amount of > > > > data with > > > > one PDU. > > > > > > > > 2.With DatasN and Sack we dont need any data in a bad header. > > > > > > > > 3. If an R2T is lost (received at initiator with bad digest) - the > > > > initiator will know that from > > > > the next R2T if the target has several outstanding - very likely at > > long > > > > distances - and will not have to way for a timeout. Other uses are > > > > marginal. Basically it is "part of a command execution" and we can > > > > painless recover > > > > from failures for this case too. > > > > > > > > Regards, > > > > Julo > > > > > > > > "Somesh Gupta" on 05/03/2001 20:40:06 > > > > > > > > Please respond to someshg@yahoo.com > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: R2TDataSN > > > > > > > > > > > > > > > > > > > > > R2TDataSN > > > > > ---------- > > > > > Sec 6.7.1 has some new content on how to handle lost R2Ts using > > > > > SACKs. But I noticed that the SACK request (Sec 2.16) has not > > > > > changed to indicate whether the DataSN is a R2T DataSN or just > > > > > a Read PDU DataSN (D bit) > > > > > So do we demux on the read/write operation type? > > > > > And how does this affect PDU loss in bidirectional commands ? > > > > > +++ SACK is ascking for data (DataSN) the target knows > > > > > > > > > > > > > Julian, > > > > > > > > Regarding the R2TDataSN, I have a comments and a > > > > question. > > > > > > > > I think that when a PDU header fails a CRC/checksum check etc, > > > > it is a problem to depend on any information in the header (including > > > > length fields), thereby making any further processing on > > > > the connection unreliable. > > > > > > > > What scenarios do you envision where the R2TDataSN is useful. > > > > IN Orlando I think there was clear consensus that application > > > > do not try to transfer very large amounts of data using a > > > > single command. > > > > > > > > Thanks, > > > > Somesh > > > > > > > > Thanks, > > Somesh Gupta > > > > > > _________________________________________________________ > > Do You Yahoo!? > > Get your free @yahoo.com address at http://mail.yahoo.com > > > > > > > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Wed Mar 7 23:38:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA03723 for ; Wed, 7 Mar 2001 23:38:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f282XZk07511 for ips-outgoing; Wed, 7 Mar 2001 21:33:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zmamail05.zma.compaq.com (zmamail05.zma.compaq.com [161.114.64.105]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f282WYr07474 for ; Wed, 7 Mar 2001 21:32:34 -0500 (EST) Received: by zmamail05.zma.compaq.com (Postfix, from userid 12345) id 98C3331FB; Wed, 7 Mar 2001 21:32:32 -0500 (EST) Received: from exchou-gh02.cca.cpqcorp.net (exchou-gh02.cca.cpqcorp.net [16.110.248.202]) by zmamail05.zma.compaq.com (Postfix) with ESMTP id 75073410C; Wed, 7 Mar 2001 21:32:31 -0500 (EST) Received: by exchou-gh02.cca.cpqcorp.net with Internet Mail Service (5.5.2652.78) id ; Wed, 7 Mar 2001 20:32:30 -0600 Message-ID: <78AF3C342AEAEF4BA33B35A8A15668C659C0F8@cceexc17.americas.cpqcorp.net> From: "Elliott, Robert" To: ips@ece.cmu.edu, "'Renato E. Maranon'" , klein@sanrad.com, "'Tanjore K. Suresh'" Subject: RE: iSCSI: Naming and Discovery Draft... Date: Wed, 7 Mar 2001 20:32:24 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2652.78) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk "Busy" and "Reservation Conflict" are SCSI Status code names defined in SAM-2; I'd avoid using them for this different purpose. --- Robert.Elliott@compaq.com Compaq Server Storage > -----Original Message----- > From: Renato E. Maranon [mailto:rmaranon@marantinetworks.com] > Sent: Wednesday, March 07, 2001 11:12 AM > To: klein@sanrad.com; 'Tanjore K. Suresh' > Cc: ips@ece.cmu.edu > Subject: RE: iSCSI: Naming and Discovery Draft... > > > My two cents. I like to suggest "Target Reserved", "Target > Reservation > Conflict" or "Target Committed". For this condition to > occur, the target > may not be necessarily busy, but just out of resources, or > can only handle > one. "Target Busy" seems to imply busy. Building on Mark's > desciption > below, something like: > > The target has committed resources to one or more > initiators and cannot > handle > another one. The initiator MAY try again later. This can > be the case > for simple devices that can handle only one initiator at a time, or > for a target that has does not have the resources to > support one more > initiator. In contrast to the previous examples, this > rejection is > temporary. > > > > Renato Maranon > Maranti Networks, Inc > 920 Hillview Court > Milpitas, Ca 95035 > Phone: 408-719-9600 x309 > Fax: 408-719-9631 > email: rmaranon@marantinetworks.com > home: www.marantinetworks.com > > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Yaron Klein > Sent: Wednesday, March 07, 2001 1:38 AM > To: 'Tanjore K. Suresh' > Cc: ips@ece.cmu.edu > Subject: RE: iSCSI: Naming and Discovery Draft... > > > Tanjore, > > Some more comments: > > The error statuses codes on Appendix B are not synchronized > with the main > draft. We will fix it. > > The term "target conflict" was borrowed from HTTP. Mark clarified this > scenario well. I would like to add that this status enables better > resolution and knowledge to the target. That is, in those > cases the target > can just not open the connection or just reject it like server error. > However, this will not give indication of the situation as > described by > Mark. > > Regards, > > Yaron > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > Behalf Of Mark > Bakke > Sent: Tuesday, March 06, 2001 6:51 PM > To: Tanjore K. Suresh > Cc: kaladhar@us.ibm.com; ips@ece.cmu.edu > Subject: Re: iSCSI: Naming and Discovery Draft... > > > Tanjore- > > Thanks for the feedback. I can comment on #3: > > "Tanjore K. Suresh" wrote: > > 3. Appendix B, B.4.5, > > Target Conflict 45 doesnot seem to be appropriate. > > > > I have not reviewed all the documents yet to give a > > recommendation and hence cannot give, but feel > > " Target Conflict" doesnot > > convey the meaning of the Scenario indicating > > case of " simple devices that can handle > one device or > > the target had reached the limit of its Initiators' > capacity." > > Perhaps we chose the wrong term for this one. How about if call it > "Target Busy", and slightly re-word it? > > The target is busy with another initiator and cannot handle > another one. The initiator MAY try again later. This can > be the case > for simple devices that can handle only one initiator at a time, or > for a target that has does not have the resources to > support one more > initiator. In contrast to the previous examples, this > rejection is > temporary. > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 > From owner-ips@ece.cmu.edu Thu Mar 8 00:22:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA04082 for ; Thu, 8 Mar 2001 00:22:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f283FaP09870 for ips-outgoing; Wed, 7 Mar 2001 22:15:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp014.mail.yahoo.com (smtp014.mail.yahoo.com [216.136.173.58]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f283Eqr09834 for ; Wed, 7 Mar 2001 22:14:52 -0500 (EST) Received: from unknown (HELO somesh) (206.112.109.244) by smtp.mail.vip.sc5.yahoo.com with SMTP; 8 Mar 2001 03:14:44 -0000 X-Apparently-From: Reply-To: From: "Somesh Gupta" To: , Subject: RE: R2TDataSN and other recovery mechanisms Date: Wed, 7 Mar 2001 19:10:24 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Well it does seem that you are proposing that the processing machine go into a "speculative" mode where the processing of the next PDU determines whether we were successfuly able to skip a bad PDU header? I think it is time to start another seperate thread on requirement for more details and rigorous/perhaps formal description of various recovery tools that we have and determine how to use them, and identify any possible holes in them. Somesh > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Wednesday, March 07, 2001 5:21 AM > To: ips@ece.cmu.edu > Subject: RE: R2TDataSN and other recovery mechanisms > > > > > Somesh, > > OK - so what? You will get another digest error (if choose a good digest > -:)) and look for the next marker. > > Regards, > Julo > > "Somesh Gupta" on 07/03/2001 04:09:31 > > Please respond to someshg@yahoo.com > > To: Julian Satran/Haifa/IBM@IBMIL, someshg@yahoo.com > cc: > Subject: RE: R2TDataSN and other recovery mechanisms > > > > > Julian, > > The point was not about whether the markers were covered > by digests. The point is about the fact that errors > occured in that range on the TCP stream that were > caught (header digest error) - it does seem likely that > the marker might have an error too? > > Somesh > > > -----Original Message----- > > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > > Sent: Tuesday, March 06, 2001 3:52 PM > > To: someshg@yahoo.com > > Subject: RE: R2TDataSN and other recovery mechanisms > > > > > > > > > > Markers are not covered by digests (they are not supposed to be seen at > > digest formation. > > The layer model appears in the draft. > > > > Regards, > > Julo > > > > "Somesh Gupta" on 06/03/2001 20:20:56 > > > > Please respond to someshg@yahoo.com > > > > To: "Venkat Rangan" , someshg@yahoo.com, > > Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: R2TDataSN and other recovery mechanisms > > > > > > > > > > Venkat, > > > > You are right but this does put some additional requirements > > on the sync and steering layer (assuming we can agree on one > > and that it is not optional). > > > > Consider the marker mechanism as an example. If the marker > > falls within the TCP byte stream region which contains the > > header, and the header digest fails, then do you trust this > > marker or skip to the next marker. Or do we need a sum on the > > marker itself. > > > > We do have various recovery mechanisms in the protocol without > > detailed state machines or other documentation to make sure > > that the mechanisms are really robust and worth including. And > > even more important, that we will have interoperable > > implementations that are implemented to the standard (rather > > than being compliant with some defacto implementation) > > > > I am not saying that these tools do not work, and Julian > > (and other smart people)may even have worked them out. > > However, the proof is not in the document. > > > > Thanks, > > Somesh > > > > > -----Original Message----- > > > From: Venkat Rangan [mailto:venkat@rhapsodynetworks.com] > > > Sent: Tuesday, March 06, 2001 10:00 AM > > > To: someshg@yahoo.com; julian_satran@il.ibm.com; ips@ece.cmu.edu > > > Subject: RE: R2TDataSN > > > > > > > > > Somesh, > > > > > > > When there is a data digest error, further stream parsing is > > > > deterministic. But not when the PDU header digest error. > > > > > > Isn't it true that with Synch and Steering layer, you are able to > > > skip past > > > PDUs with header-digest errors and reach a point where you can begin > > > deterministically processing the stream? (assuming that the > > > headers used for > > > Synch and Steering are in-tact). > > > > > > Of course, in the event that this option is "negotiated out" you > > > don't have > > > this ability. > > > > > > Venkat Rangan > > > Rhapsody Networks Inc. > > > http://www.rhapsodynetworks.com > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Somesh Gupta > > > Sent: Tuesday, March 06, 2001 7:23 AM > > > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > > > Subject: RE: R2TDataSN > > > > > > > > > I beg to disagree. If an R2T PDU (header) has bad digest, or any other > > > header has a bad digest - since you always need the PDU length from > > > the header, there is some uncertainty associated with further > > processing. > > > > > > Are you proposing that the processing machine go into a "speculative" > > > mode where the processing of the next PDU determines whether we were > > > successfuly able to skip a bad PDU header? When there is a data digest > > > error, further stream parsing is deterministic. But not when the PDU > > > header digest error. > > > > > > Also the consensus (in my interpretation) was on applications > > > not transfering very large amounts of data using a single command or > > > read/write PDU. > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > > > > julian_satran@il.ibm.com > > > > Sent: Monday, March 05, 2001 5:41 PM > > > > To: ips@ece.cmu.edu > > > > Subject: Re: R2TDataSN > > > > > > > > > > > > > > > > > > > > Somesh, > > > > > > > > 1.The only consensus I heard is not to transfer a large amount of > > > > data with > > > > one PDU. > > > > > > > > 2.With DatasN and Sack we dont need any data in a bad header. > > > > > > > > 3. If an R2T is lost (received at initiator with bad digest) - the > > > > initiator will know that from > > > > the next R2T if the target has several outstanding - very likely at > > long > > > > distances - and will not have to way for a timeout. Other uses are > > > > marginal. Basically it is "part of a command execution" and we can > > > > painless recover > > > > from failures for this case too. > > > > > > > > Regards, > > > > Julo > > > > > > > > "Somesh Gupta" on 05/03/2001 20:40:06 > > > > > > > > Please respond to someshg@yahoo.com > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: R2TDataSN > > > > > > > > > > > > > > > > > > > > > R2TDataSN > > > > > ---------- > > > > > Sec 6.7.1 has some new content on how to handle lost R2Ts using > > > > > SACKs. But I noticed that the SACK request (Sec 2.16) has not > > > > > changed to indicate whether the DataSN is a R2T DataSN or just > > > > > a Read PDU DataSN (D bit) > > > > > So do we demux on the read/write operation type? > > > > > And how does this affect PDU loss in bidirectional commands ? > > > > > +++ SACK is ascking for data (DataSN) the target knows > > > > > > > > > > > > > Julian, > > > > > > > > Regarding the R2TDataSN, I have a comments and a > > > > question. > > > > > > > > I think that when a PDU header fails a CRC/checksum check etc, > > > > it is a problem to depend on any information in the header > (including > > > > length fields), thereby making any further processing on > > > > the connection unreliable. > > > > > > > > What scenarios do you envision where the R2TDataSN is useful. > > > > IN Orlando I think there was clear consensus that application > > > > do not try to transfer very large amounts of data using a > > > > single command. > > > > > > > > Thanks, > > > > Somesh > > > > > > > > _________________________________________________________ > > > > Do You Yahoo!? > > > > Get your free @yahoo.com address at http://mail.yahoo.com > > > > > > > > > > > > > > > > > > _________________________________________________________ > > > Do You Yahoo!? > > > Get your free @yahoo.com address at http://mail.yahoo.com > > > > > > > > > _________________________________________________________ > > Do You Yahoo!? > > Get your free @yahoo.com address at http://mail.yahoo.com > > > > > > > > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com > > > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Thu Mar 8 00:24:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA04110 for ; Thu, 8 Mar 2001 00:24:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f283SZ510618 for ips-outgoing; Wed, 7 Mar 2001 22:28:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp013.mail.yahoo.com (smtp013.mail.yahoo.com [216.136.173.57]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f283SFr10598 for ; Wed, 7 Mar 2001 22:28:15 -0500 (EST) Received: from unknown (HELO somesh) (206.112.109.244) by smtp.mail.vip.sc5.yahoo.com with SMTP; 8 Mar 2001 03:28:13 -0000 X-Apparently-From: Reply-To: From: "Somesh Gupta" To: Subject: description of recovery mechanisms Date: Wed, 7 Mar 2001 19:23:53 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I hope David and Julian will excuse me for using the following sentences from the Orlando minutes. ----------- start of quote ------------------------------- - There will be a significant connection recovery write-up, including details, procedures and examples added to the draft. ----------- end of quote --------------------------------- As an engineer, I believe that we do need detailed and thorough description of the usage of all the recovery tools in the protocol. This ensures 1. Determination that there are no holes. Presence of "holes" will lead to the mechanisms not being used (but implemented) or determine that there are no holes which will lead to testing nightmares. 2. Ensure interoperability among implementations I hope this does not sound like I am asking Julian to do my work for me. But it is better hashed out and debated in one place. If the rest of the working group does feel that the level of detail in the spec is enough and the rest is an "exercise for the reader", I will accept that. Thanks, Somesh _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Thu Mar 8 08:40:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id IAA25963 for ; Thu, 8 Mar 2001 08:40:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28BVh319417 for ips-outgoing; Thu, 8 Mar 2001 06:31:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f28BVQr19403 for ; Thu, 8 Mar 2001 06:31:26 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id MAA200494 for ; Thu, 8 Mar 2001 12:31:19 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id MAA156984 for ; Thu, 8 Mar 2001 12:30:19 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A09.003F2268 ; Thu, 8 Mar 2001 12:29:35 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 8 Mar 2001 13:24:08 +0200 Subject: Re: iSCSI: Header Digest Failure (was R2TDataSN) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I understood that. But markers are meant to solve this too aren't they? Julo Matt Wakeley on 08/03/2001 01:54:10 Please respond to Matt Wakeley To: ips@ece.cmu.edu cc: Subject: iSCSI: Header Digest Failure (was R2TDataSN) Julian, What Somesh is talking about has nothing to do with R2T specifically. His point is, the length field in the PDU header determines how long *this* PDU is, and where to find the *next* PDU. If *this* PDU fails its header digest check, you cannot trust the length field, therefore, you cannot know where the *next* PDU is in the byte stream. Given that, is seems like whenever there is a header digest error, you have no choice but to terminate the TCP connection and start a new one. -Matt Wakeley Agilent Technologies Somesh Gupta wrote: > > Julian, > > How do I know it is an R2T if the header digest failed? > I am sorry if I am missing something very obvious here. > > Let us say that a header digest failed. What information > in the header am I able to trust at this point? I don't > know if it is an R2T or not. I don't know if the length > is ok or not. > > Thanks, > Somesh > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Tuesday, March 06, 2001 4:01 PM > > To: ips@ece.cmu.edu > > Subject: RE: R2TDataSN > > > > > > > > > > Somesh, > > > > I am still lost. R2T is fixed length and you know that you have a bad > > digest after reading it. > > No length involved. ??? > > > > Julo > > > > "Somesh Gupta" on 07/03/2001 00:26:10 > > > > Please respond to someshg@yahoo.com > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: R2TDataSN > > > > > > > > > > Julian, > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Tuesday, March 06, 2001 9:31 AM > > > To: ips@ece.cmu.edu > > > Subject: RE: R2TDataSN > > > > > > > > > > > > > > > Somesh, > > > > > > You've lost me. I do not propose that you look at the bad R2tT but to > > find > > > that you have missed one > > > by looking at the next. > > > > Since iSCSI PDUs define how long they are, you have to look at > > one PDU to determine where the next PDU is. (unless ofcourse > > the sync and steering layer is doing the work - see my exchange > > with Venkat on that). > > > > On the long transfer etc., I was not sure what scenarios > > an R2TDataSN was providing recovery from. Since you clarified > > that it is to recover from a header digest error, we can > > focus on that scenario. > > > > > > This interesting for long transfers that have > > > several outstanding R2Ts. > > > What is speculative here? There was never a consensus that there > > > will be no > > > more than one outstanding R2T. > > > > > > Regards, > > > Julo > > > > > > "Somesh Gupta" on 06/03/2001 17:23:04 > > > > > > Please respond to someshg@yahoo.com > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: RE: R2TDataSN > > > > > > > > > > > > > > > I beg to disagree. If an R2T PDU (header) has bad digest, or any other > > > header has a bad digest - since you always need the PDU length from > > > the header, there is some uncertainty associated with further > > processing. > > > > > > Are you proposing that the processing machine go into a "speculative" > > > mode where the processing of the next PDU determines whether we were > > > successfuly able to skip a bad PDU header? When there is a data digest > > > error, further stream parsing is deterministic. But not when the PDU > > > header digest error. > > > > > > Also the consensus (in my interpretation) was on applications > > > not transfering very large amounts of data using a single command or > > > read/write PDU. > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > julian_satran@il.ibm.com > > > > Sent: Monday, March 05, 2001 5:41 PM > > > > To: ips@ece.cmu.edu > > > > Subject: Re: R2TDataSN > > > > > > > > > > > > > > > > > > > > Somesh, > > > > > > > > 1.The only consensus I heard is not to transfer a large amount of > > > > data with > > > > one PDU. > > > > > > > > 2.With DatasN and Sack we dont need any data in a bad header. > > > > > > > > 3. If an R2T is lost (received at initiator with bad digest) - the > > > > initiator will know that from > > > > the next R2T if the target has several outstanding - very likely at > > long > > > > distances - and will not have to way for a timeout. Other uses are > > > > marginal. Basically it is "part of a command execution" and we can > > > > painless recover > > > > from failures for this case too. > > > > > > > > Regards, > > > > Julo > > > > > > > > "Somesh Gupta" on 05/03/2001 20:40:06 > > > > > > > > Please respond to someshg@yahoo.com > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: R2TDataSN > > > > > > > > > > > > > > > > > > > > > R2TDataSN > > > > > ---------- > > > > > Sec 6.7.1 has some new content on how to handle lost R2Ts using > > > > > SACKs. But I noticed that the SACK request (Sec 2.16) has not > > > > > changed to indicate whether the DataSN is a R2T DataSN or just > > > > > a Read PDU DataSN (D bit) > > > > > So do we demux on the read/write operation type? > > > > > And how does this affect PDU loss in bidirectional commands ? > > > > > +++ SACK is ascking for data (DataSN) the target knows > > > > > > > > > > > > > Julian, > > > > > > > > Regarding the R2TDataSN, I have a comments and a > > > > question. > > > > > > > > I think that when a PDU header fails a CRC/checksum check etc, > > > > it is a problem to depend on any information in the header (including > > > > length fields), thereby making any further processing on > > > > the connection unreliable. > > > > > > > > What scenarios do you envision where the R2TDataSN is useful. > > > > IN Orlando I think there was clear consensus that application > > > > do not try to transfer very large amounts of data using a > > > > single command. > > > > > > > > Thanks, > > > > Somesh > > > > > > > > Thanks, > > Somesh Gupta > > > > > > _________________________________________________________ > > Do You Yahoo!? > > Get your free @yahoo.com address at http://mail.yahoo.com > > > > > > > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Thu Mar 8 08:41:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id IAA25984 for ; Thu, 8 Mar 2001 08:41:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28BVl619438 for ips-outgoing; Thu, 8 Mar 2001 06:31:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f28BVSr19404 for ; Thu, 8 Mar 2001 06:31:28 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id MAA81052 for ; Thu, 8 Mar 2001 12:31:20 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id MAA159124 for ; Thu, 8 Mar 2001 12:30:20 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A09.003F24C9 ; Thu, 8 Mar 2001 12:29:41 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 8 Mar 2001 13:25:32 +0200 Subject: RE: R2TDataSN and other recovery mechanisms Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk No I am proposing we use markers. Julo "Somesh Gupta" on 08/03/2001 05:10:24 Please respond to someshg@yahoo.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: R2TDataSN and other recovery mechanisms Well it does seem that you are proposing that the processing machine go into a "speculative" mode where the processing of the next PDU determines whether we were successfuly able to skip a bad PDU header? I think it is time to start another seperate thread on requirement for more details and rigorous/perhaps formal description of various recovery tools that we have and determine how to use them, and identify any possible holes in them. Somesh > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Wednesday, March 07, 2001 5:21 AM > To: ips@ece.cmu.edu > Subject: RE: R2TDataSN and other recovery mechanisms > > > > > Somesh, > > OK - so what? You will get another digest error (if choose a good digest > -:)) and look for the next marker. > > Regards, > Julo > > "Somesh Gupta" on 07/03/2001 04:09:31 > > Please respond to someshg@yahoo.com > > To: Julian Satran/Haifa/IBM@IBMIL, someshg@yahoo.com > cc: > Subject: RE: R2TDataSN and other recovery mechanisms > > > > > Julian, > > The point was not about whether the markers were covered > by digests. The point is about the fact that errors > occured in that range on the TCP stream that were > caught (header digest error) - it does seem likely that > the marker might have an error too? > > Somesh > > > -----Original Message----- > > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > > Sent: Tuesday, March 06, 2001 3:52 PM > > To: someshg@yahoo.com > > Subject: RE: R2TDataSN and other recovery mechanisms > > > > > > > > > > Markers are not covered by digests (they are not supposed to be seen at > > digest formation. > > The layer model appears in the draft. > > > > Regards, > > Julo > > > > "Somesh Gupta" on 06/03/2001 20:20:56 > > > > Please respond to someshg@yahoo.com > > > > To: "Venkat Rangan" , someshg@yahoo.com, > > Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: R2TDataSN and other recovery mechanisms > > > > > > > > > > Venkat, > > > > You are right but this does put some additional requirements > > on the sync and steering layer (assuming we can agree on one > > and that it is not optional). > > > > Consider the marker mechanism as an example. If the marker > > falls within the TCP byte stream region which contains the > > header, and the header digest fails, then do you trust this > > marker or skip to the next marker. Or do we need a sum on the > > marker itself. > > > > We do have various recovery mechanisms in the protocol without > > detailed state machines or other documentation to make sure > > that the mechanisms are really robust and worth including. And > > even more important, that we will have interoperable > > implementations that are implemented to the standard (rather > > than being compliant with some defacto implementation) > > > > I am not saying that these tools do not work, and Julian > > (and other smart people)may even have worked them out. > > However, the proof is not in the document. > > > > Thanks, > > Somesh > > > > > -----Original Message----- > > > From: Venkat Rangan [mailto:venkat@rhapsodynetworks.com] > > > Sent: Tuesday, March 06, 2001 10:00 AM > > > To: someshg@yahoo.com; julian_satran@il.ibm.com; ips@ece.cmu.edu > > > Subject: RE: R2TDataSN > > > > > > > > > Somesh, > > > > > > > When there is a data digest error, further stream parsing is > > > > deterministic. But not when the PDU header digest error. > > > > > > Isn't it true that with Synch and Steering layer, you are able to > > > skip past > > > PDUs with header-digest errors and reach a point where you can begin > > > deterministically processing the stream? (assuming that the > > > headers used for > > > Synch and Steering are in-tact). > > > > > > Of course, in the event that this option is "negotiated out" you > > > don't have > > > this ability. > > > > > > Venkat Rangan > > > Rhapsody Networks Inc. > > > http://www.rhapsodynetworks.com > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Somesh Gupta > > > Sent: Tuesday, March 06, 2001 7:23 AM > > > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > > > Subject: RE: R2TDataSN > > > > > > > > > I beg to disagree. If an R2T PDU (header) has bad digest, or any other > > > header has a bad digest - since you always need the PDU length from > > > the header, there is some uncertainty associated with further > > processing. > > > > > > Are you proposing that the processing machine go into a "speculative" > > > mode where the processing of the next PDU determines whether we were > > > successfuly able to skip a bad PDU header? When there is a data digest > > > error, further stream parsing is deterministic. But not when the PDU > > > header digest error. > > > > > > Also the consensus (in my interpretation) was on applications > > > not transfering very large amounts of data using a single command or > > > read/write PDU. > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > > > > julian_satran@il.ibm.com > > > > Sent: Monday, March 05, 2001 5:41 PM > > > > To: ips@ece.cmu.edu > > > > Subject: Re: R2TDataSN > > > > > > > > > > > > > > > > > > > > Somesh, > > > > > > > > 1.The only consensus I heard is not to transfer a large amount of > > > > data with > > > > one PDU. > > > > > > > > 2.With DatasN and Sack we dont need any data in a bad header. > > > > > > > > 3. If an R2T is lost (received at initiator with bad digest) - the > > > > initiator will know that from > > > > the next R2T if the target has several outstanding - very likely at > > long > > > > distances - and will not have to way for a timeout. Other uses are > > > > marginal. Basically it is "part of a command execution" and we can > > > > painless recover > > > > from failures for this case too. > > > > > > > > Regards, > > > > Julo > > > > > > > > "Somesh Gupta" on 05/03/2001 20:40:06 > > > > > > > > Please respond to someshg@yahoo.com > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: R2TDataSN > > > > > > > > > > > > > > > > > > > > > R2TDataSN > > > > > ---------- > > > > > Sec 6.7.1 has some new content on how to handle lost R2Ts using > > > > > SACKs. But I noticed that the SACK request (Sec 2.16) has not > > > > > changed to indicate whether the DataSN is a R2T DataSN or just > > > > > a Read PDU DataSN (D bit) > > > > > So do we demux on the read/write operation type? > > > > > And how does this affect PDU loss in bidirectional commands ? > > > > > +++ SACK is ascking for data (DataSN) the target knows > > > > > > > > > > > > > Julian, > > > > > > > > Regarding the R2TDataSN, I have a comments and a > > > > question. > > > > > > > > I think that when a PDU header fails a CRC/checksum check etc, > > > > it is a problem to depend on any information in the header > (including > > > > length fields), thereby making any further processing on > > > > the connection unreliable. > > > > > > > > What scenarios do you envision where the R2TDataSN is useful. > > > > IN Orlando I think there was clear consensus that application > > > > do not try to transfer very large amounts of data using a > > > > single command. > > > > > > > > Thanks, > > > > Somesh > > > > > > > > _________________________________________________________ > > > > Do You Yahoo!? > > > > Get your free @yahoo.com address at http://mail.yahoo.com > > > > > > > > > > > > > > > > > > _________________________________________________________ > > > Do You Yahoo!? > > > Get your free @yahoo.com address at http://mail.yahoo.com > > > > > > > > > _________________________________________________________ > > Do You Yahoo!? > > Get your free @yahoo.com address at http://mail.yahoo.com > > > > > > > > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com > > > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Thu Mar 8 10:32:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA29533 for ; Thu, 8 Mar 2001 10:32:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28DSj426098 for ips-outgoing; Thu, 8 Mar 2001 08:28:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f28DRmr26018 for ; Thu, 8 Mar 2001 08:27:48 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id IAA22697; Thu, 8 Mar 2001 08:27:41 -0500 (EST) Message-ID: <3AA788DF.53349242@cisco.com> Date: Thu, 08 Mar 2001 07:27:59 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Matt Wakeley CC: ips@ece.cmu.edu Subject: Re: iSCSI error recovery References: <0F31E5C394DAD311B60C00E029101A07041014D0@corpmx9.isus.emc.com> <3AA6B8FD.DD83C75D@agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Quite right. Not only that, but the other reason for separating the header and data CRCs is to make at least the data CRC end-to-end when an iSCSI proxy or gateway is in the middle. -- Mark Matt Wakeley wrote: > > Black_David@emc.com wrote: > > > - Given some of the CRC discussion, I think the conclusion > > in Orlando to have separate header and data CRCs is > > NOT the rough consensus of the WG. We need to go > > back to a requirements discussion on this rather than > > debating exactly where to put the CRCs. Would those > > envisioning middleboxes/gateways/etc. that would > > benefit from this sort of CRC separation please post > > short use cases/descriptions indicating the basic > > functionality of the box and which fields it needs > > to change (let's do this with reference to the header > > layout in -03 rather than subsequent changes)? As > > part of the use case/description, please explain > > how/why Fibre Channel's single CRC covering both > > the frame header and data causes problems/difficulties. > > David, > > The use of a separate header/data CRC is required in order to perform > efficient data steering. If the iSCSI header and data are both protected by > the same CRC (which for efficiencies in transmission would be after the data), > then an iSCSI adapter would have to buffer up an entire iSCSI PDU before it > could begin DMAing the PDU payload to host memory (because it could not > guarantee that the information in the iSCSI header was correct to begin DMAing > it earlier). Since Julian insists that PDUs could be in lengths requiring 32 > bit length fields, this is a lot of unnecessary storage. > > -Matt Wakeley > Agilent Technologies -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Thu Mar 8 10:33:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA29550 for ; Thu, 8 Mar 2001 10:33:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28E6lZ28531 for ips-outgoing; Thu, 8 Mar 2001 09:06:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f28E62r28453 for ; Thu, 8 Mar 2001 09:06:02 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id JAA28701 for ; Thu, 8 Mar 2001 09:05:56 -0500 (EST) Message-ID: <3AA791D6.416FFEBB@cisco.com> Date: Thu, 08 Mar 2001 08:06:14 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: IPS Subject: Fwd: I-D ACTION:draft-bakke-iscsimib-02.txt Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit FYI. For a pictorial view of this MIB, please see http://www.haifa.il.ibm.com/satran/ips/iscsi-mib-structure-02.pdf -- Mark > A New Internet-Draft is available from the on-line Internet-Drafts directories. > > > Title : Definitions of Managed Objects for SCSI over TCP > Author(s) : M. Bakke, J. Muchow, M. Krueger > Filename : draft-bakke-iscsimib-02.txt > Pages : 141 > Date : 06-Mar-01 > > This memo defines a portion of the Management Information Base (MIB) > for use with network management protocols in TCP/IP based internets. > In particular it defines objects for managing a client using the > iSCSI (SCSI over TCP) protocol. It is meant to match the latest > version of iSCSI defined in [ISCSI]. > > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-bakke-iscsimib-02.txt > > Internet-Drafts are also available by anonymous FTP. Login with the username > "anonymous" and a password of your e-mail address. After logging in, > type "cd internet-drafts" and then > "get draft-bakke-iscsimib-02.txt". > > A list of Internet-Drafts directories can be found in > http://www.ietf.org/shadow.html > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > > > Internet-Drafts can also be obtained by e-mail. > > Send a message to: > mailserv@ietf.org. > In the body type: > "FILE /internet-drafts/draft-bakke-iscsimib-02.txt". > > NOTE: The mail server at ietf.org can return the document in > MIME-encoded form by using the "mpack" utility. To use this > feature, insert the command "ENCODING mime" before the "FILE" > command. To decode the response(s), you will need "munpack" or > a MIME-compliant mail reader. Different MIME-compliant mail readers > exhibit different behavior, especially when dealing with > "multipart" MIME messages (i.e. documents which have been split > up into multiple messages), so check your local documentation on > how to manipulate these messages. > > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Thu Mar 8 10:34:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA29586 for ; Thu, 8 Mar 2001 10:34:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28E1l028190 for ips-outgoing; Thu, 8 Mar 2001 09:01:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f28E1Fr28158 for ; Thu, 8 Mar 2001 09:01:15 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id JAA23738; Thu, 8 Mar 2001 09:01:08 -0500 (EST) Message-ID: <3AA790B7.3EADFCB9@cisco.com> Date: Thu, 08 Mar 2001 08:01:27 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: "Elliott, Robert" CC: ips@ece.cmu.edu, "'Renato E. Maranon'" , klein@sanrad.com, "'Tanjore K. Suresh'" Subject: Re: iSCSI: Naming and Discovery Draft... References: <78AF3C342AEAEF4BA33B35A8A15668C659C0F8@cceexc17.americas.cpqcorp.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Sounds like we are left with "Target Committed", which Renato's new description seems to fit. Any other ideas, or does that seem to work? -- Mark "Elliott, Robert" wrote: > > "Busy" and "Reservation Conflict" are SCSI Status code names defined in > SAM-2; I'd avoid using them for this different purpose. > --- > Robert.Elliott@compaq.com > Compaq Server Storage > > > -----Original Message----- > > From: Renato E. Maranon [mailto:rmaranon@marantinetworks.com] > > Sent: Wednesday, March 07, 2001 11:12 AM > > To: klein@sanrad.com; 'Tanjore K. Suresh' > > Cc: ips@ece.cmu.edu > > Subject: RE: iSCSI: Naming and Discovery Draft... > > > > > > My two cents. I like to suggest "Target Reserved", "Target > > Reservation > > Conflict" or "Target Committed". For this condition to > > occur, the target > > may not be necessarily busy, but just out of resources, or > > can only handle > > one. "Target Busy" seems to imply busy. Building on Mark's > > desciption > > below, something like: > > > > The target has committed resources to one or more > > initiators and cannot > > handle > > another one. The initiator MAY try again later. This can > > be the case > > for simple devices that can handle only one initiator at a time, or > > for a target that has does not have the resources to > > support one more > > initiator. In contrast to the previous examples, this > > rejection is > > temporary. > > > > > > > > Renato Maranon > > Maranti Networks, Inc > > 920 Hillview Court > > Milpitas, Ca 95035 > > Phone: 408-719-9600 x309 > > Fax: 408-719-9631 > > email: rmaranon@marantinetworks.com > > home: www.marantinetworks.com > > > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Yaron Klein > > Sent: Wednesday, March 07, 2001 1:38 AM > > To: 'Tanjore K. Suresh' > > Cc: ips@ece.cmu.edu > > Subject: RE: iSCSI: Naming and Discovery Draft... > > > > > > Tanjore, > > > > Some more comments: > > > > The error statuses codes on Appendix B are not synchronized > > with the main > > draft. We will fix it. > > > > The term "target conflict" was borrowed from HTTP. Mark clarified this > > scenario well. I would like to add that this status enables better > > resolution and knowledge to the target. That is, in those > > cases the target > > can just not open the connection or just reject it like server error. > > However, this will not give indication of the situation as > > described by > > Mark. > > > > Regards, > > > > Yaron > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > > Behalf Of Mark > > Bakke > > Sent: Tuesday, March 06, 2001 6:51 PM > > To: Tanjore K. Suresh > > Cc: kaladhar@us.ibm.com; ips@ece.cmu.edu > > Subject: Re: iSCSI: Naming and Discovery Draft... > > > > > > Tanjore- > > > > Thanks for the feedback. I can comment on #3: > > > > "Tanjore K. Suresh" wrote: > > > 3. Appendix B, B.4.5, > > > Target Conflict 45 doesnot seem to be appropriate. > > > > > > I have not reviewed all the documents yet to give a > > > recommendation and hence cannot give, but feel > > > " Target Conflict" doesnot > > > convey the meaning of the Scenario indicating > > > case of " simple devices that can handle > > one device or > > > the target had reached the limit of its Initiators' > > capacity." > > > > Perhaps we chose the wrong term for this one. How about if call it > > "Target Busy", and slightly re-word it? > > > > The target is busy with another initiator and cannot handle > > another one. The initiator MAY try again later. This can > > be the case > > for simple devices that can handle only one initiator at a time, or > > for a target that has does not have the resources to > > support one more > > initiator. In contrast to the previous examples, this > > rejection is > > temporary. > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 > > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Thu Mar 8 13:21:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA07171 for ; Thu, 8 Mar 2001 13:21:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28GKuI08271 for ips-outgoing; Thu, 8 Mar 2001 11:20:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hub1.san-jose.webnexus.net (hub1-20.san-jose.webnexus.net [209.140.224.20]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f28GJrr08131 for ; Thu, 8 Mar 2001 11:19:54 -0500 (EST) Received: from rmaranon (h-209-140-231-182.webnexus.com [209.140.231.182] (may be forged)) by hub1.san-jose.webnexus.net (8.9.1a/8.9.1/WN-1.4) with SMTP id IAA11304; Thu, 8 Mar 2001 08:19:44 -0800 (PST) From: "Renato E. Maranon" To: , , Subject: RE: FCIP iFCP encapsulation proposal Date: Thu, 8 Mar 2001 08:18:38 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704101634@corpmx9.isus.emc.com> Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit This is a procedural/process question. Who determines/decides when to make exceptions to drafts missing the draft cutoff, per the message below? Renato Maranon Maranti Networks, Inc 920 Hillview Court Milpitas, Ca 95035 Phone: 408-719-9600 x309 Fax: 408-719-9631 email: rmaranon@marantinetworks.com home: www.marantinetworks.com -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Black_David@emc.com Sent: Wednesday, March 07, 2001 4:33 PM To: vchau@gadzoox.com; ips@ece.cmu.edu Subject: RE: FCIP iFCP encapsulation proposal Folks, > I sent out a draft to propose a common encapsulation scheme for > FCIP and iFCP last week; I am afraid I have missed the cut-off time > for new drafts. Here is a summary of the draft to help stimulate > discussion at the upcoming IETF meeting: Due to the importance of making progress on this topic in Minneapolis, this will be an exception to the usual rule of not discussing drafts that miss the draft cutoff. Please don't assume that exceptions of this sort will be readily available in the future. Also, everyone should remember that all IETF deadlines are in Eastern Time (contrary to popular rumor, Silicon Valley is not the center of the known universe) and hence should allow for time zone differences in figuring out when to get drafts submitted. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Thu Mar 8 13:26:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA07333 for ; Thu, 8 Mar 2001 13:26:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28Famg04786 for ips-outgoing; Thu, 8 Mar 2001 10:36:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from crufty.research.bell-labs.com (crufty.research.bell-labs.com [204.178.16.49]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f28FZsr04701 for ; Thu, 8 Mar 2001 10:35:54 -0500 (EST) Received: from scummy.research.bell-labs.com ([135.104.2.10]) by crufty; Thu Mar 8 10:33:39 EST 2001 Received: from aura.research.bell-labs.com ([135.104.46.10]) by scummy; Thu Mar 8 10:35:30 EST 2001 Received: (from sandeepj@localhost) by aura.research.bell-labs.com (8.9.1/8.9.1) id KAA15922 for ips@ece.cmu.edu; Thu, 8 Mar 2001 10:35:29 -0500 (EST) Date: Thu, 8 Mar 2001 10:35:29 -0500 (EST) Message-Id: <200103081535.KAA15922@aura.research.bell-labs.com> From: sandeepj@research.bell-labs.com (Sandeep Joshi) To: ips@ece.cmu.edu Subject: Re: iSNS comments Sender: owner-ips@ece.cmu.edu Precedence: bulk > some comments on the new iSNS draft > > 1) it lacks a Table of contents, unlike iSCSI. please add one. > (given that, i am liable to have missed some section which > might have answered the questions raised below!) > > 2) What happens if an iSNS client tries to update its > TCP/UDP port or IP address (for Portal/ESI/etc) ? > > So the entry exists, and now the client sends a RegDevAttr > request with the update flag set for changing the above. > > If the port(s) is going to be well-known, the questions below > may not arise. If not,... > > a) Will the old TCP connection be broken by the iSNS server ? > b) Would the RegDevResponse be sent to the old/new port ? > c) Who initiates the new connection (client or server) ? > d) How would the client know the request succeeded ? i see what i missed here.. SLP, which is orthogonally managing the "control" plane. the questions above dont arise. > > 3) Is there a requirement to provide (keyword=MUST) a secondary > iSNS server ? If I am not mistaken, DNS does mandate a > secondary server for every zone to avoid single-pt-of-failure. > > > -Sandeep From owner-ips@ece.cmu.edu Thu Mar 8 14:51:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA10636 for ; Thu, 8 Mar 2001 14:51:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28I6pQ15638 for ips-outgoing; Thu, 8 Mar 2001 13:06:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com (42s3082.isus.emc.com [168.159.211.82] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f28I6Dr15570 for ; Thu, 8 Mar 2001 13:06:13 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 8 Mar 2001 13:07:28 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101639@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: Agenda and draft cutoff Date: Thu, 8 Mar 2001 13:06:06 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > This is a procedural/process question. Who determines/decides when > to make exceptions to drafts missing the draft cutoff, per the message > below? The WG co-chairs assemble the agenda and make this sort of decision in consultation with and subject to review by the Area Directors. If these sort of exceptions happen on a regular basis, the Area Directors will get seriously annoyed, and it's not a good idea to annoy the Area Directors ;-). In this case, the draft missed the cutoff by a few hours, and relates to the most important technical issue in front of FCIP and iFCP, so holding up technical progress to make this procedural point doesn't seem like a wise move ... but if this sort of deadline gaming starts to become a regular event, it may become necessary to enforce the deadlines without exception. Please try to get drafts in several days ahead of the deadline to avoid this possibility. With luck, the agenda will appear tomorrow - the snowstorm in New England didn't help :-( . Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Thu Mar 8 14:54:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA10784 for ; Thu, 8 Mar 2001 14:54:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28HWtM13304 for ips-outgoing; Thu, 8 Mar 2001 12:32:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dns.LampreyNetworks.com (IDENT:root@mail.lampreynetworks.com [64.31.72.26]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f28EfTr00833 for ; Thu, 8 Mar 2001 09:41:29 -0500 (EST) Received: from breinhold (p-24.ts-3.qnc.ma.ttlc.net [140.186.40.233]) by dns.LampreyNetworks.com (8.9.3/8.8.7) with SMTP id JAA07712 for ; Thu, 8 Mar 2001 09:38:15 -0500 From: "Barry Reinhold - Lamprey POP3" To: "ISCSI" Subject: Group Test Period for iSCSI interoperability - call for interest Date: Thu, 8 Mar 2001 09:40:08 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit To anyone who has an interest: A number of people have expressed some interest in getting together in a technical debugging environment to try and see what types of issues come up with early iSCSI prototypes. This is usually a great way to see how different people have read and understood a draft, and is also a big help in identifying interoperability issues before they get ingrained into implementatons. So, as a "toe in the water" step, I would like to propose a group test period, as outlined below, and get some feedback as to who would like to participate in such an event. Group Test Period Overview: A three day test period that would run Tuesday - Thursday. Monday and Friday would be open for people to make adhoc arrangements if they so desired. Very informal "pair up with someone else and try" process for the first two days followed by an attempt at putting together groups of devices on the third day. There would be no real expectation that anything has to "work right" and people could drop out, fix things, and rejoin on the first two days. This type of early testing is most effective when development engineers attend and bring the toys they need to fix bugs. The goal would be to work based on the current draft but implementations that were at previous draft levels could test against other implentations that were at that level. ISCSI would be the focus but if there are groups of people who want to get together and do other storage over IP protocols, and there is someone willing to coordinate part of the effort, it should be easy to accomodate. Time wise -- mid April (17-19)? This would be turned to accomodate the interested parties. A fee would be charged to cover whatever costs we incurred for food and facilities. That being said, if your organization has an interest, send me an email or post to the reflector. If there are six + implementations ready to try we'll give it a go as an "event". If there is a larger group (15 + players) the date may have to be pushed back due to organizational issues. The following information would be helpful: 1. Date desired 2. Protocol(s) you want to do 3. Roles your stuff can play (iSCSI initiator, target, bridge, gateway) 4. Draft level you want to test at 5. Contact email and phone. Barry Reinhold 603-868-5144 From owner-ips@ece.cmu.edu Thu Mar 8 15:51:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA12743 for ; Thu, 8 Mar 2001 15:51:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28JQvc20871 for ips-outgoing; Thu, 8 Mar 2001 14:26:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f28JQWr20812 for ; Thu, 8 Mar 2001 14:26:32 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id AD62B353 for ; Thu, 8 Mar 2001 12:26:31 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id C0586138 for ; Thu, 8 Mar 2001 14:26:30 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id LAA27003 for ; Thu, 8 Mar 2001 11:26:29 -0800 (PST) Message-ID: <3AA7DDAF.12A5E7B7@agilent.com> Date: Thu, 08 Mar 2001 11:29:51 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: [Fwd: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-01.txt] Content-Type: multipart/mixed; boundary="------------71B5C3B4C5C5F29505BF733A" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------71B5C3B4C5C5F29505BF733A Content-Type: text/plain; charset=iso-8859-1 X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id f28JQvd20871 Content-Transfer-Encoding: quoted-printable -------- Original Message -------- From: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-01.txt To: IETF-Announce: ; A New Internet-Draft is available from the on-line Internet-Drafts directories. Title : iSCSI Digests =FB CRC or Checksum? Author(s) : V. Cavanna, M. Wakeley Filename : draft-cavanna-iscsi-crc-vs-cksum-01.txt Pages : 12 Date : 07-Mar-01 =09 The iSCSI working group is currently considering the method of=20 protecting iSCSI messages from errors. This I-D presents a proposal=20 to use the Fletcher-32 checksum algorithm or some variant of it=20 rather than Adler-32; if we must use CRC, to make it CRC32 (or=20 CRC32Q) rather than CRC64. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-cavanna-iscsi-crc-vs-cksum-01.t= xt Internet-Drafts are also available by anonymous FTP. Login with the usern= ame "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-cavanna-iscsi-crc-vs-cksum-01.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html=20 or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-cavanna-iscsi-crc-vs-cksum-01.txt". =09 NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. =09 =09 Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --------------71B5C3B4C5C5F29505BF733A Content-Type: Message/External-body; name="draft-cavanna-iscsi-crc-vs-cksum-01.txt" Content-Disposition: inline; filename="draft-cavanna-iscsi-crc-vs-cksum-01.txt" Content-Transfer-Encoding: 7bit Content-Type: text/plain Content-ID: <20010307142049.I-D@ietf.org> --------------71B5C3B4C5C5F29505BF733A-- From owner-ips@ece.cmu.edu Thu Mar 8 15:51:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA12783 for ; Thu, 8 Mar 2001 15:51:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28Igob17999 for ips-outgoing; Thu, 8 Mar 2001 13:42:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f28Ie9r17742 for ; Thu, 8 Mar 2001 13:40:09 -0500 (EST) Received: from phys-ha3mpkb.Eng.Sun.COM ([129.146.43.31]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id KAA09214; Thu, 8 Mar 2001 10:40:08 -0800 (PST) Received: from sonu (sonu [129.146.48.127]) by phys-ha3mpkb.Eng.Sun.COM (8.8.8+Sun/8.8.8/ENSMAIL,v2.0) with SMTP id KAA26071; Thu, 8 Mar 2001 10:40:05 -0800 (PST) Message-Id: <200103081840.KAA26071@phys-ha3mpkb.Eng.Sun.COM> Date: Thu, 8 Mar 2001 10:41:20 -0800 (PST) From: "Tanjore K. Suresh" Reply-To: "Tanjore K. Suresh" Subject: RE: iSCSI: Naming and Discovery Draft... To: Tanjore.Suresh@sun.com, kaladhar@us.ibm.com, jtseng@NishanSystems.com Cc: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: NAtqYHTEpE2oGCosXZ2CzA== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk Josh, > There is an error in this model which will be corrected in > the next revision of this document and the iSNS document. > DD_MEMBER should be another object in the model which is > contained by DISCOVERY_DOMAIN. DD_Member is an attribute > which identifies the WWUI or IP address that is a member > of the Discovery Domain. DD_ID is another attribute of > the DD_MEMBER object that identifies the DISCOVERY_DOMAIN > of the member. Thanks for the clarification, will stay tuned for reviewing the next revision. Thanks sureshtk From owner-ips@ece.cmu.edu Thu Mar 8 15:51:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA12784 for ; Thu, 8 Mar 2001 15:51:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28JYoK21396 for ips-outgoing; Thu, 8 Mar 2001 14:34:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f28JYDr21369 for ; Thu, 8 Mar 2001 14:34:13 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Thu, 8 Mar 2001 11:34:01 -0800 Message-ID: From: Kevin Gibbons To: "'sandeepj@research.bell-labs.com'" Cc: "'ips@ece.cmu.edu'" Subject: RE: iSNS comments Date: Thu, 8 Mar 2001 11:34:00 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Sandeep, thanks for the iSNS feedback. Please see responses below. Regards, Kevin -----Original Message----- From: sandeepj@research.bell-labs.com [mailto:sandeepj@research.bell-labs.com] Sent: Thursday, March 08, 2001 7:35 AM To: ips@ece.cmu.edu Subject: Re: iSNS comments > some comments on the new iSNS draft > > 1) it lacks a Table of contents, unlike iSCSI. please add one. > (given that, i am liable to have missed some section which > might have answered the questions raised below!) kg> we can add a TOC to the next update of the document. > > 2) What happens if an iSNS client tries to update its > TCP/UDP port or IP address (for Portal/ESI/etc) ? > > So the entry exists, and now the client sends a RegDevAttr > request with the update flag set for changing the above. > > If the port(s) is going to be well-known, the questions below > may not arise. If not,... > > a) Will the old TCP connection be broken by the iSNS server ? kg> Who breaks the former connection first is implementation dependent. However, the iSNS server initiates the TCP connection to the new port for the next ESI message. The ip address and port used for ESI messages does not have to be the same as portal addresses that an entity has registered. > b) Would the RegDevResponse be sent to the old/new port ? kg> as described in section 7.2, the response will be sent to the same IP Address and Port that the registration message originated from. > c) Who initiates the new connection (client or server) ? kg> the iSNS server will initiate the connection for the next ESI. There was concern during the design that the server, rather then the client, be in control of the ESI period and connection process to manage the ESI handling overhead. > d) How would the client know the request succeeded ? > i see what i missed here.. SLP, which is orthogonally managing > the "control" plane. the questions above dont arise. kg> The client will know the request succeeded from the RegDevResponse message status field, sent by the iSNS is response to the request. > > 3) Is there a requirement to provide (keyword=MUST) a secondary > iSNS server ? If I am not mistaken, DNS does mandate a > secondary server for every zone to avoid single-pt-of-failure. > kg> There was concern that the design not mandate a requirement for a secondary server for situations where High Availability is not a concern. If desired, additional text can be added to recommend secondary server in HA configurations. > > -Sandeep From owner-ips@ece.cmu.edu Thu Mar 8 15:57:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA13063 for ; Thu, 8 Mar 2001 15:57:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28IfpP17914 for ips-outgoing; Thu, 8 Mar 2001 13:41:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f28Ibnr17648 for ; Thu, 8 Mar 2001 13:37:49 -0500 (EST) Received: from phys-ha3mpkb.Eng.Sun.COM ([129.146.43.31]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id KAA21381; Thu, 8 Mar 2001 10:37:47 -0800 (PST) Received: from sonu (sonu [129.146.48.127]) by phys-ha3mpkb.Eng.Sun.COM (8.8.8+Sun/8.8.8/ENSMAIL,v2.0) with SMTP id KAA24876; Thu, 8 Mar 2001 10:37:46 -0800 (PST) Message-Id: <200103081837.KAA24876@phys-ha3mpkb.Eng.Sun.COM> Date: Thu, 8 Mar 2001 10:39:01 -0800 (PST) From: "Tanjore K. Suresh" Reply-To: "Tanjore K. Suresh" Subject: RE: iSCSI: Naming and Discovery Draft... To: Tanjore.Suresh@sun.com, klein@sanrad.com Cc: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: 73bQmI1agV2dKXzGHhiLiA== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk Yaron, > Some more comments: > > The error statuses codes on Appendix B are not synchronized with the main > draft. We will fix it. > > The term "target conflict" was borrowed from HTTP. Mark clarified this > scenario well. I would like to add that this status enables better > resolution and knowledge to the target. That is, in those cases the target > can just not open the connection or just reject it like server error. I donot understand this part of your statement. " or just reject it like server error" I thought reject (0x6f) is used only when the target receives - a message with format error - a digest error. Am i missing something here or have i misunderstood this. > However, this will not give indication of the situation as described by > Mark. > I aggree with Yaron to give a better resolution error message depending on the scenerio. Today according to iSCSI revision 05 draft from Julian, it looks like two status codes are looking appropriate in the login response(0x43) 0x205 which means " Target Conflict" ==> this means target is in use by another Initiator and doesnot support Multiple initiators". 0x300 which means " Target Error" == > this means error occurred in iSCSI target( out of resources... etc). I think first one corresponds to the scenario " The Target is busy with another Initiator and cannot handle another one" The second one corresponds to " Case of simple device and target has reached the limit of its initiator capacity (out of resources)." IMO, May be splitting into two examples is better here. Thanks sureshtk > Regards, > > Yaron > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Mark > Bakke > Sent: Tuesday, March 06, 2001 6:51 PM > To: Tanjore K. Suresh > Cc: kaladhar@us.ibm.com; ips@ece.cmu.edu > Subject: Re: iSCSI: Naming and Discovery Draft... > > > Tanjore- > > Thanks for the feedback. I can comment on #3: > > "Tanjore K. Suresh" wrote: > > 3. Appendix B, B.4.5, > > Target Conflict 45 doesnot seem to be appropriate. > > > > I have not reviewed all the documents yet to give a > > recommendation and hence cannot give, but feel > > " Target Conflict" doesnot > > convey the meaning of the Scenario indicating > > case of " simple devices that can handle one device or > > the target had reached the limit of its Initiators' > capacity." > > Perhaps we chose the wrong term for this one. How about if call it > "Target Busy", and slightly re-word it? > > The target is busy with another initiator and cannot handle > another one. The initiator MAY try again later. This can be the case > for simple devices that can handle only one initiator at a time, or > for a target that has does not have the resources to support one more > initiator. In contrast to the previous examples, this rejection is > temporary. > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 > From owner-ips@ece.cmu.edu Thu Mar 8 18:28:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA17100 for ; Thu, 8 Mar 2001 18:28:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28L64X27634 for ips-outgoing; Thu, 8 Mar 2001 16:06:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f28L5Mr27539 for ; Thu, 8 Mar 2001 16:05:22 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0G9W0053ZD7R3X@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Thu, 8 Mar 2001 13:04:41 -0800 (PST) Date: Thu, 08 Mar 2001 13:03:02 -0800 From: Douglas Otis Subject: RE: iSCSI: Header Digest Failure (was R2TDataSN) In-reply-to: To: julian_satran@il.ibm.com, ips@ece.cmu.edu Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, It should follow that frame markers are not an option if header digests are used otherwise dropping the connection becomes required in the event of an error. If header digests are used, there should also be an advisory recommending checking both values of the frame marker PDU pointer to detect a potential error at this level, as well as, a recommended null stuffing of the connection to allow resynchronization between framing markers. Here in the draft, you have overlooked a concern regarding a digest error as it pertains to framing. Page 20: "In situations where IP packets are delivered in order from the network, iSCSI message framing is not an issue; messages are processed one after the other. In the presence of IP packet reordering (e.g., frames being dropped), legacy TCP implementations store the "out of order" TCP segments in temporary buffers until the missing TCP segments arrive, upon which the data must be copied to the application buffers. In iSCSI it is desirable to steer the SCSI data within these out of order TCP segments into the pre-allocated SCSI buffers rather than store them in temporary buffers. This decreases the need for dedicated reassembly buffers as well as the latency and bandwidth related to extra copies." Page 12: "The iSCSI target layer MUST deliver the commands to the SCSI target layer in the order specified by CmdSN." What happens if there is an unrecovered gap in the CmdSN due to a failed NIC? Why are these 31 bits? (2**31-1) is the limit set for several values. This is 2,147,483,647. In other words, this is a 31 bit limitation. If the MS bit is not to be used, perhaps this bit could be used to signal Immediate without disrupting the command sequence. I doubt this is intended however as there are examples that indicate a count at 0xffffffff. If the MS bit was not used, then the maximum count would have been 0x7fffffff. I hope I am not repeating a prior notice. Page 18: - IPv6 address, in dotted decimal notation. Assumed if the name contains more than four, but at most 16 numbers, separated by dots (.), where each number is in the range 0..255. Staying with decimal byte notation for IPv6, define how a value with fewer than 16 digits is interpreted by means of padding. Do you zero pad MS positions? This will require scanning all values with reversed order placement. I doubt there is a more expensive scheme. On page 25, perhaps a better choice of words would be 'section' rather than 'segment' to describe portions of the various structures that comprise the PDU layers. On page 83, you go to some length explaining R2T recovery based on a timeout but fail to provide any indication of a suitable timeout value. How are these timeouts negotiated? Can equipment be expected to interoperate without timeouts specified? Doug > I understood that. But markers are meant to solve this too aren't they? > > Julo > > Matt Wakeley on 08/03/2001 01:54:10 > > Please respond to Matt Wakeley > > To: ips@ece.cmu.edu > cc: > Subject: iSCSI: Header Digest Failure (was R2TDataSN) > > > > > Julian, > > What Somesh is talking about has nothing to do with R2T specifically. His > point is, the length field in the PDU header determines how long > *this* PDU > is, and where to find the *next* PDU. If *this* PDU fails its header > digest > check, you cannot trust the length field, therefore, you cannot know where > the > *next* PDU is in the byte stream. > > Given that, is seems like whenever there is a header digest > error, you have > no > choice but to terminate the TCP connection and start a new one. > > -Matt Wakeley > Agilent Technologies > > > Somesh Gupta wrote: > > > > Julian, > > > > How do I know it is an R2T if the header digest failed? > > I am sorry if I am missing something very obvious here. > > > > Let us say that a header digest failed. What information > > in the header am I able to trust at this point? I don't > > know if it is an R2T or not. I don't know if the length > > is ok or not. > > > > Thanks, > > Somesh > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Tuesday, March 06, 2001 4:01 PM > > > To: ips@ece.cmu.edu > > > Subject: RE: R2TDataSN > > > > > > > > > > > > > > > Somesh, > > > > > > I am still lost. R2T is fixed length and you know that you have a bad > > > digest after reading it. > > > No length involved. ??? > > > > > > Julo > > > > > > "Somesh Gupta" on 07/03/2001 00:26:10 > > > > > > Please respond to someshg@yahoo.com > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: RE: R2TDataSN > > > > > > > > > > > > > > > Julian, > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > > > > julian_satran@il.ibm.com > > > > Sent: Tuesday, March 06, 2001 9:31 AM > > > > To: ips@ece.cmu.edu > > > > Subject: RE: R2TDataSN > > > > > > > > > > > > > > > > > > > > Somesh, > > > > > > > > You've lost me. I do not propose that you look at the bad > R2tT but to > > > find > > > > that you have missed one > > > > by looking at the next. > > > > > > Since iSCSI PDUs define how long they are, you have to look at > > > one PDU to determine where the next PDU is. (unless ofcourse > > > the sync and steering layer is doing the work - see my exchange > > > with Venkat on that). > > > > > > On the long transfer etc., I was not sure what scenarios > > > an R2TDataSN was providing recovery from. Since you clarified > > > that it is to recover from a header digest error, we can > > > focus on that scenario. > > > > > > > > > This interesting for long transfers that have > > > > several outstanding R2Ts. > > > > What is speculative here? There was never a consensus that there > > > > will be no > > > > more than one outstanding R2T. > > > > > > > > Regards, > > > > Julo > > > > > > > > "Somesh Gupta" on 06/03/2001 17:23:04 > > > > > > > > Please respond to someshg@yahoo.com > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: RE: R2TDataSN > > > > > > > > > > > > > > > > > > > > I beg to disagree. If an R2T PDU (header) has bad digest, or any > other > > > > header has a bad digest - since you always need the PDU length from > > > > the header, there is some uncertainty associated with further > > > processing. > > > > > > > > Are you proposing that the processing machine go into a > "speculative" > > > > mode where the processing of the next PDU determines whether we were > > > > successfuly able to skip a bad PDU header? When there is a data > digest > > > > error, further stream parsing is deterministic. But not when the PDU > > > > header digest error. > > > > > > > > Also the consensus (in my interpretation) was on applications > > > > not transfering very large amounts of data using a single command or > > > > read/write PDU. > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > > > > > julian_satran@il.ibm.com > > > > > Sent: Monday, March 05, 2001 5:41 PM > > > > > To: ips@ece.cmu.edu > > > > > Subject: Re: R2TDataSN > > > > > > > > > > > > > > > > > > > > > > > > > Somesh, > > > > > > > > > > 1.The only consensus I heard is not to transfer a large amount of > > > > > data with > > > > > one PDU. > > > > > > > > > > 2.With DatasN and Sack we dont need any data in a bad header. > > > > > > > > > > 3. If an R2T is lost (received at initiator with bad digest) - the > > > > > initiator will know that from > > > > > the next R2T if the target has several outstanding - very > likely at > > > long > > > > > distances - and will not have to way for a timeout. Other uses > are > > > > > marginal. Basically it is "part of a command execution" > and we can > > > > > painless recover > > > > > from failures for this case too. > > > > > > > > > > Regards, > > > > > Julo > > > > > > > > > > "Somesh Gupta" on 05/03/2001 20:40:06 > > > > > > > > > > Please respond to someshg@yahoo.com > > > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > > cc: > > > > > Subject: R2TDataSN > > > > > > > > > > > > > > > > > > > > > > > > > > R2TDataSN > > > > > > ---------- > > > > > > Sec 6.7.1 has some new content on how to handle lost R2Ts using > > > > > > SACKs. But I noticed that the SACK request (Sec 2.16) has not > > > > > > changed to indicate whether the DataSN is a R2T DataSN or just > > > > > > a Read PDU DataSN (D bit) > > > > > > So do we demux on the read/write operation type? > > > > > > And how does this affect PDU loss in bidirectional commands ? > > > > > > +++ SACK is ascking for data (DataSN) the target knows > > > > > > > > > > > > > > > > Julian, > > > > > > > > > > Regarding the R2TDataSN, I have a comments and a > > > > > question. > > > > > > > > > > I think that when a PDU header fails a CRC/checksum check etc, > > > > > it is a problem to depend on any information in the header > (including > > > > > length fields), thereby making any further processing on > > > > > the connection unreliable. > > > > > > > > > > What scenarios do you envision where the R2TDataSN is useful. > > > > > IN Orlando I think there was clear consensus that application > > > > > do not try to transfer very large amounts of data using a > > > > > single command. > > > > > > > > > > Thanks, > > > > > Somesh > > > > > > > > > > > Thanks, > > > Somesh Gupta > > > > > > > > > _________________________________________________________ > > > Do You Yahoo!? > > > Get your free @yahoo.com address at http://mail.yahoo.com > > > > > > > > > > > > > _________________________________________________________ > > Do You Yahoo!? > > Get your free @yahoo.com address at http://mail.yahoo.com > > > > > From owner-ips@ece.cmu.edu Thu Mar 8 20:06:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA19138 for ; Thu, 8 Mar 2001 20:06:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28NEtm06004 for ips-outgoing; Thu, 8 Mar 2001 18:14:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta6.snfc21.pbi.net (mta6.snfc21.pbi.net [206.13.28.240]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f28NElr05996 for ; Thu, 8 Mar 2001 18:14:48 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta6.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0G9W00ERHJ7CCH@mta6.snfc21.pbi.net> for ips@ece.cmu.edu; Thu, 8 Mar 2001 15:14:01 -0800 (PST) Date: Thu, 08 Mar 2001 15:12:18 -0800 From: Douglas Otis Subject: RE: [Fwd: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-01.txt] In-reply-to: <3AA7DDAF.12A5E7B7@agilent.com> To: Matt Wakeley , IPS Reflector Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset=iso-8859-1 Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id f28NEtn06004 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ietf.org id UAA19138 Matt, Adler-32 was designed to handle any byte length which is not something required for iSCSI if padding is included in the digest. Modification to allow two bytes at a time should not be a problem as you would reduce the number of accesses in half. (One quarter the accesses required for CRC-32!) If a 16-bit add is used, then a subtraction with a carry tests for a modulo wrap will occur for each location. This is not a substantial difference and will still be faster than normal Adler-32 as you have cut the number of iterations in half. It should be likely Adler-16:32 vs. Adler-8:32 is more robust if burst error detection is required. I doubt Fletcher-32 is as robust as Adler-xx as DMA or FIFO memory errors are likely to induce the same modulo used in Fletcher. Doug > -------- Original Message -------- > From: Internet-Drafts@ietf.org > Subject: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-01.txt > To: IETF-Announce: ; > > A New Internet-Draft is available from the on-line Internet-Drafts > directories. > > > Title : iSCSI Digests û CRC or Checksum? > Author(s) : V. Cavanna, M. Wakeley > Filename : draft-cavanna-iscsi-crc-vs-cksum-01.txt > Pages : 12 > Date : 07-Mar-01 > > The iSCSI working group is currently considering the method of > protecting iSCSI messages from errors. This I-D presents a proposal > to use the Fletcher-32 checksum algorithm or some variant of it > rather than Adler-32; if we must use CRC, to make it CRC32 (or > CRC32Q) rather than CRC64. > > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-cavanna-iscsi-crc-vs-cks um-01.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-cavanna-iscsi-crc-vs-cksum-01.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-cavanna-iscsi-crc-vs-cksum-01.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. From owner-ips@ece.cmu.edu Thu Mar 8 20:07:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA19216 for ; Thu, 8 Mar 2001 20:07:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f28NNrv06652 for ips-outgoing; Thu, 8 Mar 2001 18:23:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f28NNlr06644 for ; Thu, 8 Mar 2001 18:23:48 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 8 Mar 2001 18:23:42 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0708015254@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: New milestones Date: Thu, 8 Mar 2001 18:23:39 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk The following new set of anticipated milestones has just been submitted to the Secretariat to replace the current set of yet to be completed milestones in the charter. It is perfectly acceptable (and encouraged) to accomplish things earlier than indicated by the milestones, but the WG co-chairs and Technical Coordinators believe that these are reasonable estimates at the moment: Mar 01 Discuss framework, specification and related drafts (e.g., MIBs, discovery) for the protocol encapsulations at IETF meeting in Minneapolis. Apr 01 Submit final version of iSCSI requirements draft to the IESG for consideration as Informational RFC. May 01 Submit initial Internet-Draft of FCIP/iFCP common encapsulation format Jun 01 Begin revision of WG charter in consultation with the Area Directors. Aug 01 Meet at IETF meeting in London to discuss specification and related drafts (e.g., MIBs, discovery) for the protocol encapsulations. Sep 01 Submit protocol specification drafts (iSCSI, FCIP, iFCP, FCIP/iFCP common encapsulation format) to the IESG for consideration as Proposed Standard RFCs. Oct 01 Submit iSNS and MIB drafts to the IESG for consideration appropriate to each draft. Activities that may complete later than October do not have milestones, and will be picked up as part of the next major change to milestones -- part of the charter revision that will begin in June. Also, those with keen eyes will have noticed that there are two versions of the iSCSI Naming and Discovery Requirements draft linked to the IPS WG charter page at IETF. The longer one, draft-ietf-ips-iscsi-name-disc-00.txt is the current version. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Thu Mar 8 21:00:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA19656 for ; Thu, 8 Mar 2001 21:00:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f290Bta09494 for ips-outgoing; Thu, 8 Mar 2001 19:11:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lysimachus.hosting.pacbell.net (lysimachus.hosting.pacbell.net [216.100.98.17]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f290Awr09415 for ; Thu, 8 Mar 2001 19:10:58 -0500 (EST) Received: from VENKAT1 (64-160-62-200.rhapsodynetworks.com [64.160.62.200] (may be forged)) by lysimachus.hosting.pacbell.net id TAA25162; Thu, 8 Mar 2001 19:10:39 -0500 (EST) [ConcentricHost SMTP Relay 1.7] From: "Venkat Rangan" To: Cc: "IP Storage Working Group" Subject: iSCSI: Markers Date: Thu, 8 Mar 2001 16:13:46 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Immediately following the initial marker-less interval, what appears on the stream? Is it the first marker or is it the first PDU? Also, in heading 19 (page 110), "Default is 4096" would imply the default initial markerless-interval is 16K, but in earlier section (page 105), it states the default is 4K. The marker offsets are in bytes, and the left most two bits are reserved. So, one would be unable to store a marker for a LongDataWord PDU. We don't think it's a good idea to send such a long PDU, but providing smaller number of bits in marker offsets appears to be a sneaky way to impose a limit. Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com From owner-ips@ece.cmu.edu Thu Mar 8 21:01:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA19682 for ; Thu, 8 Mar 2001 21:01:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f290Ytl10773 for ips-outgoing; Thu, 8 Mar 2001 19:34:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f290Y0r10729 for ; Thu, 8 Mar 2001 19:34:00 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 8 Mar 2001 19:33:53 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0708015255@corpmx9.isus.emc.com> From: Black_David@emc.com To: someshg@yahoo.com, ips@ece.cmu.edu Subject: RE: description of recovery mechanisms Date: Thu, 8 Mar 2001 19:33:52 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk What Somesh is asking for is necessary. We can either put it in now, or discover it the hard way later in interoperability testing with considerably greater expenditure of time and effort. --David > -----Original Message----- > From: Somesh Gupta [SMTP:someshg@yahoo.com] > Sent: Wednesday, March 07, 2001 10:24 PM > To: ips@ece.cmu.edu > Subject: description of recovery mechanisms > > I hope David and Julian will excuse me for using the following > sentences from the Orlando minutes. > > ----------- start of quote ------------------------------- > > - There will be a significant connection recovery write-up, > including details, procedures and examples added to the draft. > > ----------- end of quote --------------------------------- > > > As an engineer, I believe that we do need detailed and thorough > description of the usage of all the recovery tools in the > protocol. This ensures > > 1. Determination that there are no holes. Presence of "holes" > will lead to the mechanisms not being used (but implemented) > > or determine that there are no holes which will lead > to testing nightmares. > > 2. Ensure interoperability among implementations > > I hope this does not sound like I am asking Julian to do > my work for me. But it is better hashed out and debated > in one place. > From owner-ips@ece.cmu.edu Thu Mar 8 21:49:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA21090 for ; Thu, 8 Mar 2001 21:49:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29124712417 for ips-outgoing; Thu, 8 Mar 2001 20:02:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2911Gr12347 for ; Thu, 8 Mar 2001 20:01:16 -0500 (EST) Received: from omgw1.boi.hp.com (omgw1.boi.hp.com [15.56.8.101]) by palrel1.hp.com (Postfix) with ESMTP id 17D0A746 for ; Thu, 8 Mar 2001 17:01:15 -0800 (PST) Received: from xpabh2.boi.hp.com (xpabh2.boi.hp.com [15.56.8.28]) by omgw1.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with ESMTP id SAA15345 for ; Thu, 8 Mar 2001 18:01:14 -0700 (MST) Received: by xpabh2.boi.hp.com with Internet Mail Service (5.5.2653.19) id ; Thu, 8 Mar 2001 17:01:13 -0800 Message-ID: <6BD67FFB937FD411A04F00D0B74FE87802A08EEA@xrose06.rose.hp.com> From: "KRUEGER,MARJORIE (HP-Roseville,ex1)" To: ips@ece.cmu.edu Subject: RE: description of recovery mechanisms Date: Thu, 8 Mar 2001 17:01:11 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk More than just a description is necessary. I believe the proliferation of options regarding recovery will make for an interoperability nightmare similar to early iterations of Fibre Channel protocols. Too many options and too much ambiguity result in too many conflicting interpretations. We need to simplify, remove options, over specify with examples. Marjorie Krueger Networked Storage Architecture Networked Storage Solutions Org. Hewlett-Packard tel: +1 916 785 2656 fax: +1 916 785 0391 email: marjorie_krueger@hp.com > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Thursday, March 08, 2001 4:34 PM > To: someshg@yahoo.com; ips@ece.cmu.edu > Subject: RE: description of recovery mechanisms > > > What Somesh is asking for is necessary. We can > either put it in now, or discover it the hard > way later in interoperability testing with > considerably greater expenditure of time and > effort. > > --David > > > -----Original Message----- > > From: Somesh Gupta [SMTP:someshg@yahoo.com] > > Sent: Wednesday, March 07, 2001 10:24 PM > > To: ips@ece.cmu.edu > > Subject: description of recovery mechanisms > > > > I hope David and Julian will excuse me for using the following > > sentences from the Orlando minutes. > > > > ----------- start of quote ------------------------------- > > > > - There will be a significant connection recovery write-up, > > including details, procedures and examples added to the draft. > > > > ----------- end of quote --------------------------------- > > > > > > As an engineer, I believe that we do need detailed and thorough > > description of the usage of all the recovery tools in the > > protocol. This ensures > > > > 1. Determination that there are no holes. Presence of "holes" > > will lead to the mechanisms not being used (but implemented) > > > > or determine that there are no holes which will lead > > to testing nightmares. > > > > 2. Ensure interoperability among implementations > > > > I hope this does not sound like I am asking Julian to do > > my work for me. But it is better hashed out and debated > > in one place. > > > From owner-ips@ece.cmu.edu Thu Mar 8 21:51:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA21139 for ; Thu, 8 Mar 2001 21:51:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f290r3711878 for ips-outgoing; Thu, 8 Mar 2001 19:53:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2906nr09184 for ; Thu, 8 Mar 2001 19:06:49 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id C6CAD877 for ; Thu, 8 Mar 2001 17:06:48 -0700 (MST) Received: from axandbh3.and.agilent.com (axandbh3.and.agilent.com [130.30.32.200]) by msgrel1.and.agilent.com (Postfix) with SMTP id 35EA22CB for ; Thu, 8 Mar 2001 19:06:47 -0500 (EST) Received: from 130.30.32.200 by axandbh3.and.agilent.com (InterScan E-Mail VirusWall NT); Thu, 08 Mar 2001 19:06:47 -0500 (Eastern Standard Time) Received: by axandbh3.and.agilent.com with Internet Mail Service (5.5.2653.19) id ; Thu, 8 Mar 2001 19:06:47 -0500 Message-ID: <9F8400020EC5D311AF62009027C3961602D6A7EC@axcs09.cos.agilent.com> From: "LEMAY,KEVIN (A-Roseville,ex1)" To: "'iSCSI Mail List'" Subject: StatSn Questions Date: Thu, 8 Mar 2001 19:06:46 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk The iSCSI spec implies that the StatSn handshake is supposed to be for command responses, but all target PDUs except Login response, logout response and reject contain StatSn. The asynchronous message has a comment that it is a acknowledgeable event. For what purpose is StatSn on the other PDUs and how is it used? Julian answered a question earlier about how to handle holes in the StatSn, but what about the case where the header is corrupted. It would be impossible to determine which command to retry. Additionally, any responses that may be discarded while we attempt to resync the iSCSI headers via markers would require to be retied also. Will the spec be changed to address the appropriate way to handle these cases? The cases shown in appendix B the command complete with the reception of the command response from the target. In reality, the command is not complete until the incremented ExpStatSn is received at the target. Kevin Lemay From owner-ips@ece.cmu.edu Thu Mar 8 21:52:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA21167 for ; Thu, 8 Mar 2001 21:52:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f291SuL13949 for ips-outgoing; Thu, 8 Mar 2001 20:28:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f291Rsr13900 for ; Thu, 8 Mar 2001 20:27:54 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 82ED29A4 for ; Thu, 8 Mar 2001 18:27:53 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 8B8C6124 for ; Thu, 8 Mar 2001 20:27:52 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id RAA08345 for ; Thu, 8 Mar 2001 17:27:51 -0800 (PST) Message-ID: <3AA8325C.16101ED0@agilent.com> Date: Thu, 08 Mar 2001 17:31:08 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: Header Digest Failure (was R2TDataSN) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit What you say is that some method of framing is *required* to perform recovery after a header digest failure. So, is it then to be inferred that framing in iSCSI is mandatory? -Matt Wakeley julian_satran@il.ibm.com wrote: > > I understood that. But markers are meant to solve this too aren't they? > > Julo > > Matt Wakeley on 08/03/2001 01:54:10 > > Please respond to Matt Wakeley > > To: ips@ece.cmu.edu > cc: > Subject: iSCSI: Header Digest Failure (was R2TDataSN) > > Julian, > > What Somesh is talking about has nothing to do with R2T specifically. His > point is, the length field in the PDU header determines how long *this* PDU > is, and where to find the *next* PDU. If *this* PDU fails its header > digest > check, you cannot trust the length field, therefore, you cannot know where > the > *next* PDU is in the byte stream. > > Given that, is seems like whenever there is a header digest error, you have > no > choice but to terminate the TCP connection and start a new one. > > -Matt Wakeley > Agilent Technologies > > Somesh Gupta wrote: > > > > Julian, > > > > How do I know it is an R2T if the header digest failed? > > I am sorry if I am missing something very obvious here. > > > > Let us say that a header digest failed. What information > > in the header am I able to trust at this point? I don't > > know if it is an R2T or not. I don't know if the length > > is ok or not. > > > > Thanks, > > Somesh > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Tuesday, March 06, 2001 4:01 PM > > > To: ips@ece.cmu.edu > > > Subject: RE: R2TDataSN > > > > > > > > > > > > > > > Somesh, > > > > > > I am still lost. R2T is fixed length and you know that you have a bad > > > digest after reading it. > > > No length involved. ??? > > > > > > Julo > > > > > > "Somesh Gupta" on 07/03/2001 00:26:10 > > > > > > Please respond to someshg@yahoo.com > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: RE: R2TDataSN > > > > > > > > > > > > > > > Julian, > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > > > > julian_satran@il.ibm.com > > > > Sent: Tuesday, March 06, 2001 9:31 AM > > > > To: ips@ece.cmu.edu > > > > Subject: RE: R2TDataSN > > > > > > > > > > > > > > > > > > > > Somesh, > > > > > > > > You've lost me. I do not propose that you look at the bad R2tT but to > > > find > > > > that you have missed one > > > > by looking at the next. > > > > > > Since iSCSI PDUs define how long they are, you have to look at > > > one PDU to determine where the next PDU is. (unless ofcourse > > > the sync and steering layer is doing the work - see my exchange > > > with Venkat on that). > > > > > > On the long transfer etc., I was not sure what scenarios > > > an R2TDataSN was providing recovery from. Since you clarified > > > that it is to recover from a header digest error, we can > > > focus on that scenario. > > > > > > > > > This interesting for long transfers that have > > > > several outstanding R2Ts. > > > > What is speculative here? There was never a consensus that there > > > > will be no > > > > more than one outstanding R2T. > > > > > > > > Regards, > > > > Julo > > > > > > > > "Somesh Gupta" on 06/03/2001 17:23:04 > > > > > > > > Please respond to someshg@yahoo.com > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: RE: R2TDataSN > > > > > > > > > > > > > > > > > > > > I beg to disagree. If an R2T PDU (header) has bad digest, or any > other > > > > header has a bad digest - since you always need the PDU length from > > > > the header, there is some uncertainty associated with further > > > processing. > > > > > > > > Are you proposing that the processing machine go into a "speculative" > > > > mode where the processing of the next PDU determines whether we were > > > > successfuly able to skip a bad PDU header? When there is a data > digest > > > > error, further stream parsing is deterministic. But not when the PDU > > > > header digest error. > > > > > > > > Also the consensus (in my interpretation) was on applications > > > > not transfering very large amounts of data using a single command or > > > > read/write PDU. > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > > > > > julian_satran@il.ibm.com > > > > > Sent: Monday, March 05, 2001 5:41 PM > > > > > To: ips@ece.cmu.edu > > > > > Subject: Re: R2TDataSN > > > > > > > > > > > > > > > > > > > > > > > > > Somesh, > > > > > > > > > > 1.The only consensus I heard is not to transfer a large amount of > > > > > data with > > > > > one PDU. > > > > > > > > > > 2.With DatasN and Sack we dont need any data in a bad header. > > > > > > > > > > 3. If an R2T is lost (received at initiator with bad digest) - the > > > > > initiator will know that from > > > > > the next R2T if the target has several outstanding - very likely at > > > long > > > > > distances - and will not have to way for a timeout. Other uses > are > > > > > marginal. Basically it is "part of a command execution" and we can > > > > > painless recover > > > > > from failures for this case too. > > > > > > > > > > Regards, > > > > > Julo > > > > > > > > > > "Somesh Gupta" on 05/03/2001 20:40:06 > > > > > > > > > > Please respond to someshg@yahoo.com > > > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > > cc: > > > > > Subject: R2TDataSN > > > > > > > > > > > > > > > > > > > > > > > > > > R2TDataSN > > > > > > ---------- > > > > > > Sec 6.7.1 has some new content on how to handle lost R2Ts using > > > > > > SACKs. But I noticed that the SACK request (Sec 2.16) has not > > > > > > changed to indicate whether the DataSN is a R2T DataSN or just > > > > > > a Read PDU DataSN (D bit) > > > > > > So do we demux on the read/write operation type? > > > > > > And how does this affect PDU loss in bidirectional commands ? > > > > > > +++ SACK is ascking for data (DataSN) the target knows > > > > > > > > > > > > > > > > Julian, > > > > > > > > > > Regarding the R2TDataSN, I have a comments and a > > > > > question. > > > > > > > > > > I think that when a PDU header fails a CRC/checksum check etc, > > > > > it is a problem to depend on any information in the header > (including > > > > > length fields), thereby making any further processing on > > > > > the connection unreliable. > > > > > > > > > > What scenarios do you envision where the R2TDataSN is useful. > > > > > IN Orlando I think there was clear consensus that application > > > > > do not try to transfer very large amounts of data using a > > > > > single command. > > > > > > > > > > Thanks, > > > > > Somesh > > > > > > > > > > > Thanks, > > > Somesh Gupta > > > > > > > > > _________________________________________________________ > > > Do You Yahoo!? > > > Get your free @yahoo.com address at http://mail.yahoo.com > > > > > > > > > > > > > _________________________________________________________ > > Do You Yahoo!? > > Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Fri Mar 9 05:25:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA10575 for ; Fri, 9 Mar 2001 05:25:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f298c1A07689 for ips-outgoing; Fri, 9 Mar 2001 03:38:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f298b8r07650 for ; Fri, 9 Mar 2001 03:37:09 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 48B4960E for ; Fri, 9 Mar 2001 01:37:08 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 935EADB for ; Fri, 9 Mar 2001 03:37:07 -0500 (EST) Received: from agilent.com (cos1nai132071.cos.agilent.com [141.184.132.71]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id AAA12765 for ; Fri, 9 Mar 2001 00:37:05 -0800 (PST) Message-ID: <3AA8962F.50767617@agilent.com> Date: Fri, 09 Mar 2001 00:37:03 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IP Storage Working Group Subject: Re: iSCSI: Markers References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I believe that the "initial marker-less interval" is unnecessary. At connection time, there are no "markers". When both sides agree on markers, they should also agree on when to start them. The time might well be before or after the default "initial marker-less interval" defined in the standard. -Matt Wakeley Agilent Technologies Venkat Rangan wrote: > > Julian, > > Immediately following the initial marker-less interval, what appears on the > stream? Is it the first marker or is it the first PDU? > > Also, in heading 19 (page 110), "Default is 4096" would imply the default > initial markerless-interval is 16K, but in earlier section (page 105), it > states the default is 4K. > > The marker offsets are in bytes, and the left most two bits are reserved. > So, one would be unable to store a marker for a LongDataWord PDU. We don't > think it's a good idea to send such a long PDU, but providing smaller number > of bits in marker offsets appears to be a sneaky way to impose a limit. > > Venkat Rangan > Rhapsody Networks Inc. > http://www.rhapsodynetworks.com From owner-ips@ece.cmu.edu Fri Mar 9 05:27:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA10591 for ; Fri, 9 Mar 2001 05:27:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f298p1I08335 for ips-outgoing; Fri, 9 Mar 2001 03:51:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f298o1r08245 for ; Fri, 9 Mar 2001 03:50:02 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 1D51B4AD for ; Fri, 9 Mar 2001 01:50:01 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 68274DB for ; Fri, 9 Mar 2001 03:50:00 -0500 (EST) Received: from agilent.com (cos1nai132071.cos.agilent.com [141.184.132.71]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id AAA12808; Fri, 9 Mar 2001 00:49:57 -0800 (PST) Message-ID: <3AA89933.B76BFD13@agilent.com> Date: Fri, 09 Mar 2001 00:49:55 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: "LEMAY,KEVIN (A-Roseville,ex1)" , IPS Reflector Subject: Re: StatSn Questions References: <9F8400020EC5D311AF62009027C3961602D6A7EC@axcs09.cos.agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi Kevin, The intent (I believe) is that all target-to-initiator messages will convey the current StatSN. That way, if the "status" message that contained the StatSN was discarded due to header digest error, the initiator would know about it when the next PDU was received, and could then perform error recovery (if possible). In the SCSI Task Response, Text Response and NOP-In PDUs,- StatSN has the same meaning as SCSI Response (StatSN is incremented before PDU is sent). In Data and R2T, it just transfers the previous value (except if Data is the last Data and conveys status). -Matt "LEMAY,KEVIN (A-Roseville,ex1)" wrote: > > The iSCSI spec implies that the StatSn handshake is supposed to be for > command responses, but all target PDUs except Login response, logout > response and reject contain StatSn. The asynchronous message has a comment > that it is a acknowledgeable event. For what purpose is StatSn on the other > PDUs and how is it used? > > Julian answered a question earlier about how to handle holes in the StatSn, > but what about the case where the header is corrupted. It would be > impossible to determine which command to retry. > > Additionally, any responses that may be discarded while we attempt to resync > the iSCSI headers via markers would require to be retied also. Will the spec > be changed to address the appropriate way to handle these cases? > > The cases shown in appendix B the command complete with the reception of the > command response from the target. In reality, the command is not complete > until the incremented ExpStatSn is received at the target. > > Kevin Lemay > From owner-ips@ece.cmu.edu Fri Mar 9 08:04:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id IAA12288 for ; Fri, 9 Mar 2001 08:04:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29Ap2k24613 for ips-outgoing; Fri, 9 Mar 2001 05:51:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29Aodr24550 for ; Fri, 9 Mar 2001 05:50:40 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 2365E46E for ; Fri, 9 Mar 2001 03:50:39 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 680D9184 for ; Fri, 9 Mar 2001 05:50:38 -0500 (EST) Received: from agilent.com (cos1nai132071.cos.agilent.com [141.184.132.71]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id CAA13091 for ; Fri, 9 Mar 2001 02:50:36 -0800 (PST) Message-ID: <3AA8B57A.5AF5F26A@agilent.com> Date: Fri, 09 Mar 2001 02:50:34 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iscsi: editorial comments to iSCSI-05 Content-Type: multipart/mixed; boundary="------------60B69447C0DF4510D63DA3A7" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------60B69447C0DF4510D63DA3A7 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit --------------60B69447C0DF4510D63DA3A7 Content-Type: text/plain; charset=us-ascii; name="iscs4-cmnts-e.txt" Content-Disposition: inline; filename="iscs4-cmnts-e.txt" Content-Transfer-Encoding: 7bit editorial comments to iSCSI 05: Global: In 1.2 the iSCSI PDU is defined, yet the phrase "message" is used interchangeably with PDU. Change all occurences of "message" to the correct term "PDU", especially fix instances of blahblah message. Also, change "packets" to PDUs. Global: "MUST", "SHOULD", etc are sometimes all upper case, sometimes lowercase. Please make them consistent. Abstract, page 2; 1.2 page 10: change: "on top of TCP" to: "over TCP". 1.1, middle page 9: change: "SCSI is client-server..." to: "SCSI is a client-server..." 1.1, bottom page 9: change: "Command Descriptor Blocks (CDB) is ..." to: "The Command Descriptor Block (CDB) is ..." 1.2.2.1, last paragraph page 12: "iSCSI initiators and target ..." add "s" to target. 1.2.3, page 14, last paragraph: change: "Any message except login and text reaching a target on a TCP connection before the full feature phase MUST be ..." to: "... full feature phase is a protocol error (see 6.1) and MUST be..." 2.3.2, page 33: Include a reference to SAM-2. 2.3.6, page 34: Indicate that the "Extended CDB" what's next field must be used. Change "48-byte header" to "BHS". 2.4.5, page 37: Indicate that sense or response data is "data" and not part of the header. 2.7, page 43 and 44: Change "SCSI Data packet" to "iSCSI Data PDU" 2.8.3, page 48: move 'If the Text Response does not contain a key that was requested, the initiator must assume that the key was not understood by the target or, whenever appropriate, that the response was "none".' to after "Any other key not understood by the target may be ignored without affecting basic function." 2.10.1, page 51: change "command is performs" to "command performs" change "logout is not" to "logout was not" 2.10.6, page 52: change "This an" to "This is an". 2.11.3, page 54: The sentance "If the login phase involves two login responses then each of them will hold for the subsequent responses." does not make sense to me. 2.12.2, page 58: change "only if it issued" to "only if it is issued" 2.14.1, page 61: do you mean "close session" instead of "close connection"? 2.15, page 63: change "for the failed connection" to simply "for the connection". 2.16.4, page 65: change "additional missed" to "additional sequential missed". 2.17, page 66: "The last PDU should have the F bit set to 1". Shouldn't this be "must have the F bit set to 1"? 2.17, page 66: change: "The target may send several R2T PDUs and..." to to: The target may send several R2T PDUs (if negotiated) and..." 2.18, page 68: change: "the target specifies the status for..." to: "the target specifies the status and reason for..." 2.18.1, page 69: change: "the codes returned for..." to: "the codes sent for..." (and event is not "returned") 2.20, page 71: change "inexistent" to "non-existent" 6.1, page 80: change "format errors" to "protocol errors". 7.2, page 86: change "ACA helps preserving" to "ACA helps preserve" 8, page 87: The first paragraph says "iSCSI implementations MUST provide..." and the second says "should be provided". Make these consistent. --------------60B69447C0DF4510D63DA3A7-- From owner-ips@ece.cmu.edu Fri Mar 9 08:05:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id IAA12316 for ; Fri, 9 Mar 2001 08:05:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29Aq2F24684 for ips-outgoing; Fri, 9 Mar 2001 05:52:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29ApAr24617 for ; Fri, 9 Mar 2001 05:51:10 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 72FDE52F for ; Fri, 9 Mar 2001 03:51:09 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id B8E47182 for ; Fri, 9 Mar 2001 05:51:08 -0500 (EST) Received: from agilent.com (cos1nai132071.cos.agilent.com [141.184.132.71]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id CAA13096 for ; Fri, 9 Mar 2001 02:51:06 -0800 (PST) Message-ID: <3AA8B599.77189BC5@agilent.com> Date: Fri, 09 Mar 2001 02:51:05 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iscsi: technical comments to iSCSI 05: Content-Type: multipart/mixed; boundary="------------8CC682ACDC7D021A2F6FD764" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------8CC682ACDC7D021A2F6FD764 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit --------------8CC682ACDC7D021A2F6FD764 Content-Type: text/plain; charset=us-ascii; name="iscs4-cmnts-t.txt" Content-Disposition: inline; filename="iscs4-cmnts-t.txt" Content-Transfer-Encoding: 7bit technical comments to iSCSI 05: Global: (especially in section 6) timeouts are not defined. We went through a lot of work defining time-out values for FCP-2. I think that time-out values need to be specified for interoperability. 1.2, page 10: "The iSCSI protocol is a mapping of the SCSI remote procedure invocation model on top of the TCP protocol." is iSCSI limited to TCP? Seems like it could be implemented over other transports, such as SCTP with no modification... 1.2.3, page 14: After the sentance "Otherwise, it sends a response with a "login reject" ..." add that the connection is terminated. 1.2.5, page 16 (middle) and 6.7.1, page 84: The initiator should not be required ("MUST") to honor R2Ts. Conditions may have occurred that might prevent the initiator from processing the R2T, in which case appropriate error recovery should be performed. I suggest "SHOULD if at all possible...". 1.2.5, page 16 (bottom): "Unsolicited data MUST be sent on every connection in the same order in which commands were sent." - must the data be sent immediately after the command, or can other commands be sent before the data? 1.2.7, page 19, middle of page must "iSCSI://..." be caps? is "iscsi://..." allowed? 2.2.3, page 28: Are "what's next" fields included in the header digest? It's not stated that they are. 2.7.2, page 45: Why must a LUN be associated with a Target Transfer Tag? 2.7.4, page 45: It is not clear if the DataSN restarts from zero for each R2T. 2.7.6, page 46: It is not stated, but seems implied that a bi-di commands must send status in a separate response PDU. If this is so, please state it. 2.10.9, page 52: What is the default maximum length for login parameters? 2.15, page 63: What is the initiator supposed to do when it receives a code of "cleanup failed"? It seems like it's the target's problem and should not be reported to the initiator. 2.17, page 66: *if* as you state in 2.7.2 a LUN must be sent when a target transfer tag is specified, shouldn't the LUN be also specified in the R2T? 2.20, page 71: remove "reserved fields not 0". This makes it sound like reserved fields must be checked, which they should not must be checked. 2.20, page 71: add "data digest error" to the list of reject reasons. 3.1.2 and 3.1.3, page 72: I don't care what SCSI has defined for it's mode pages, for iSCSI, these fields should be byte values, not multiples of 512. 4.3, page 78: It may not be clear to everyone that a target can send a text or login response to a login or text command. That is, a target could respond to a text command with a login response. This should be made more clear. 6.2, page 81, last paragraph: Can a "restart" also be issued in this case? Appendix A: 01, page 94: I don't think that crc-64 "MUST" be implemented. I also don't agree with the phrase "Cyclic codes are particularly well suited for hardware implementations." This is not true if iSCSI PDUs span TCP segments and arrive out of order. Appendix A: 01, page 94: "Implementations MAY also negotiate digests with security significance for data authentication and integrity as detailed in the following table:" needs a lot more description. Appendix C: page 104: change: "will leave at least" to "should leave at least". Appendix C: 06, page 105: change: "When reduced to iSCSI terms markers MUST point to..." to: "When reduced to iSCSI terms markers MUST indicate the offset to..." Appendix C: 07, page 105: "The marker-less interval is not less than 4 kbytes and the default is 4 kbytes." It should be whatever the the receiver needs it to be (not min 4K). Appendix C: 17,18 page 109: The sender should send markers at whatever interval the reciever requires them. Not the "larger" of the sender and receiver. Otherwise, the sender could say 0xffffffff and it would be of no use to the receiver. Also, the default should be "none", not 2050. Appendix C: 22 page 111: Indicate what "no,no" and "yes,no" means. Appendix C: 23 page 111: This parameter should be in bytes, not multiples of 512, especially since framing mechanisms may require that a PDU not be larger than MSS. (perhaps -1 can be used to indicate MSS) Appendix C: 26 page 112: This should be deleted, since CmdSN is now mandatory. Appendix C: 27 page 112: PingMaxReplyLength is both a target and initiator negotiated value. Therefore, text saying "the target MUST..." should be change to indicate both target and initiator. --------------8CC682ACDC7D021A2F6FD764-- From owner-ips@ece.cmu.edu Fri Mar 9 10:50:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA18682 for ; Fri, 9 Mar 2001 10:50:56 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29DjAP03824 for ips-outgoing; Fri, 9 Mar 2001 08:45:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29Digr03793 for ; Fri, 9 Mar 2001 08:44:42 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id OAA158906 for ; Fri, 9 Mar 2001 14:44:35 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id OAA214082 for ; Fri, 9 Mar 2001 14:43:32 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0A.004B560F ; Fri, 9 Mar 2001 14:42:52 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 9 Mar 2001 11:11:08 +0200 Subject: Re: iSCSI: Naming and Discovery Draft... Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Strange English. In computerese committed is associated usually with: - written successfully to stable media (like in committed to disk) - transaction unit completed (transaction commit) Even in informal English it is used as a negation only in statements like "sorry, I had other (previous) commitments". When left alone it has "positive connotation" (like I am committed to do it). You fellows don't make learning American an easy exercise for us poor non-native speakers -:) On a more serious tone: with controlers and devices busy is usually meant to convey 2 things: - I am busy for a short interval - I will tell you when I get free If you don't intend to tell when you become free don't use busy. You can use occupied (like in "lavatories occupied" - I am in an airplane now -:)) It conveys exactly what you want. Don't take me too serious - I am not good at naming but don't over-commit committed. Julo Mark Bakke on 08/03/2001 16:01:27 Please respond to Mark Bakke To: "Elliott, Robert" cc: ips@ece.cmu.edu, "'Renato E. Maranon'" , klein@sanrad.com, "'Tanjore K. Suresh'" Subject: Re: iSCSI: Naming and Discovery Draft... Sounds like we are left with "Target Committed", which Renato's new description seems to fit. Any other ideas, or does that seem to work? -- Mark "Elliott, Robert" wrote: > > "Busy" and "Reservation Conflict" are SCSI Status code names defined in > SAM-2; I'd avoid using them for this different purpose. > --- > Robert.Elliott@compaq.com > Compaq Server Storage > > > -----Original Message----- > > From: Renato E. Maranon [mailto:rmaranon@marantinetworks.com] > > Sent: Wednesday, March 07, 2001 11:12 AM > > To: klein@sanrad.com; 'Tanjore K. Suresh' > > Cc: ips@ece.cmu.edu > > Subject: RE: iSCSI: Naming and Discovery Draft... > > > > > > My two cents. I like to suggest "Target Reserved", "Target > > Reservation > > Conflict" or "Target Committed". For this condition to > > occur, the target > > may not be necessarily busy, but just out of resources, or > > can only handle > > one. "Target Busy" seems to imply busy. Building on Mark's > > desciption > > below, something like: > > > > The target has committed resources to one or more > > initiators and cannot > > handle > > another one. The initiator MAY try again later. This can > > be the case > > for simple devices that can handle only one initiator at a time, or > > for a target that has does not have the resources to > > support one more > > initiator. In contrast to the previous examples, this > > rejection is > > temporary. > > > > > > > > Renato Maranon > > Maranti Networks, Inc > > 920 Hillview Court > > Milpitas, Ca 95035 > > Phone: 408-719-9600 x309 > > Fax: 408-719-9631 > > email: rmaranon@marantinetworks.com > > home: www.marantinetworks.com > > > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Yaron Klein > > Sent: Wednesday, March 07, 2001 1:38 AM > > To: 'Tanjore K. Suresh' > > Cc: ips@ece.cmu.edu > > Subject: RE: iSCSI: Naming and Discovery Draft... > > > > > > Tanjore, > > > > Some more comments: > > > > The error statuses codes on Appendix B are not synchronized > > with the main > > draft. We will fix it. > > > > The term "target conflict" was borrowed from HTTP. Mark clarified this > > scenario well. I would like to add that this status enables better > > resolution and knowledge to the target. That is, in those > > cases the target > > can just not open the connection or just reject it like server error. > > However, this will not give indication of the situation as > > described by > > Mark. > > > > Regards, > > > > Yaron > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > > Behalf Of Mark > > Bakke > > Sent: Tuesday, March 06, 2001 6:51 PM > > To: Tanjore K. Suresh > > Cc: kaladhar@us.ibm.com; ips@ece.cmu.edu > > Subject: Re: iSCSI: Naming and Discovery Draft... > > > > > > Tanjore- > > > > Thanks for the feedback. I can comment on #3: > > > > "Tanjore K. Suresh" wrote: > > > 3. Appendix B, B.4.5, > > > Target Conflict 45 doesnot seem to be appropriate. > > > > > > I have not reviewed all the documents yet to give a > > > recommendation and hence cannot give, but feel > > > " Target Conflict" doesnot > > > convey the meaning of the Scenario indicating > > > case of " simple devices that can handle > > one device or > > > the target had reached the limit of its Initiators' > > capacity." > > > > Perhaps we chose the wrong term for this one. How about if call it > > "Target Busy", and slightly re-word it? > > > > The target is busy with another initiator and cannot handle > > another one. The initiator MAY try again later. This can > > be the case > > for simple devices that can handle only one initiator at a time, or > > for a target that has does not have the resources to > > support one more > > initiator. In contrast to the previous examples, this > > rejection is > > temporary. > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 > > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Fri Mar 9 10:51:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA18700 for ; Fri, 9 Mar 2001 10:51:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29Dj4O03814 for ips-outgoing; Fri, 9 Mar 2001 08:45:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29Dirr03802 for ; Fri, 9 Mar 2001 08:44:53 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id OAA128338 for ; Fri, 9 Mar 2001 14:44:46 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id OAA72828 for ; Fri, 9 Mar 2001 14:43:44 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0A.004B5947 ; Fri, 9 Mar 2001 14:43:00 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 9 Mar 2001 11:53:12 +0200 Subject: Re: R2TDataSN Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Only to allow for growth -:) Large PDUs will be possible with IPSec and with stronger digests. Julo Matt Wakeley on 08/03/2001 01:46:10 Please respond to Matt Wakeley To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: R2TDataSN julian_satran@il.ibm.com wrote: > 1.The only consensus I heard is not to transfer a large amount of data with > one PDU. Ok, then why do you have "Long Data Header" in the document? -Matt From owner-ips@ece.cmu.edu Fri Mar 9 10:56:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA18927 for ; Fri, 9 Mar 2001 10:56:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29Dj7m03821 for ips-outgoing; Fri, 9 Mar 2001 08:45:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29Dihr03794 for ; Fri, 9 Mar 2001 08:44:43 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id OAA165924 for ; Fri, 9 Mar 2001 14:44:35 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id OAA72812 for ; Fri, 9 Mar 2001 14:43:33 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0A.004B55CA ; Fri, 9 Mar 2001 14:42:51 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 9 Mar 2001 10:44:45 +0200 Subject: Re: iSCSI: unsolicited Vs immediate; restart delay (fwd) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun, Sorry - I did not see it. Comments in text. Julo "Mallikarjun C." on 08/03/2001 20:26:21 Please respond to cbm@rose.hp.com To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: iSCSI: unsolicited Vs immediate; restart delay (fwd) Julian, Did you by any chance respond to this, and I did not get it (I had noticed instances of this)? Or, do I take it that you're going to define a mechanism for the missing functionality in the next rev of the draft? Regards. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com Forwarded message: X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Message-Id: <200103062019.MAA20092@core.rose.hp.com> Subject: iSCSI: unsolicited Vs immediate; restart delay To: ips@ece.cmu.edu Date: Tue, 06 Mar 2001 12:19:17 PST Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Status: RO Julian, >To: ips@ece.cmu.edu >cc: >Subject: Re: iSCSI: draft04 questions > > > > >Julian, > >Thanks. I am not clear on some. Comments. > >>2. I didn't find a way for an iSCSI target to say that it does not support >>any unsolicited data at all. SPC-2 specifies that a zero FirstBurstSize >>means unlimited. >> >>+++ UseR2T=yes (default) and ImmediateData=no +++ > >I am confused. I thought this combination disallows just the "immediate" >data, and not the unsolicited data as a whole. The draft hints at this >in section 1.2.5. > "A target MAY separately enable immediate data without > enabling the more general (separate data PDUs) form of > unsolicited data." >=== for this case you have UseR2T=yes and ImmediateData=yes >If I misunderstood, could you please comment how immediate data alone is >disabled, while allowing unsolicited data of FirstBurstSize? >=== UseR2T=no ImmediateData=no But that puts the target in unsolicited data mode not requiring R2Ts at all! Sorry if I seem too slow. Let me try again. There are three variables - 1. solicited data mode after FirstBurstSize UseR2T=yes unsolicited data mode only UseR2T=no 2. Immediate data allowed ImmediateData=yes Immediate data disallowed ImmediateData=no 3. Unsolicited burst (immediate & separate) allowed ?? Unsolicited burst not allowed ?? FirstBurstSize can be used for (3), but a zero FirstBurstSize means "unlimited" than "not allowed". My original question was how one can distinguish the two. I would be glad to be corrected, if I am misinterpreting the usage of UseR2T. +++ the confusion stems from the role of UseR2T. UseR2T is all about allowing an unsolicited first burst. After the first burst you need always R2T. You are left now with only 2 parameters controlled by the two variables. The only think still open is that for comletness it is probably advisable to have for Immediate also a bit in the corresponding mode page bit+++ +++++++++++++++++++++++++++++++++++++++ >Do I take it then that the draft currently doesn't specify the maximum time >the targets should keep the session & connection records around hoping >for a restart? I would strongly recommend adding that as an additional >field in the payload, since that is a resource allocation and scalability >issue. > >=== I assume that a traget can figure out by itself after a while that >there is no rendezvous - but I am open to requests There's no architected mechanism to figure out on its own! Either it has to keep the (session, connection, task) states around for a long time, OR it can clean up the these states and trigger an unnecessary ULP recovery on the initiator which is surprised at an unexpected restart failure (keep in mind that initiator may not be able to restart a login precisely after the "minimum time" specified, typically there's an aggregation of multiple O/S timer requests into one master handler). Providing an upper limit is a cleaner, safer design for both initiator and target. +++ I will consider it for 06 - no promise though -:) the problem with limits of this kind is who's time counts +++ Thanks. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Fri Mar 9 12:13:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA23467 for ; Fri, 9 Mar 2001 12:13:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29Fn7j11747 for ips-outgoing; Fri, 9 Mar 2001 10:49:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sandmail.sandburst.com (sandburst-gw.bstn-gw01.ma.us.intelilink.net [216.57.129.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29FmEr11692 for ; Fri, 9 Mar 2001 10:48:14 -0500 (EST) Received: from cs.uchicago.edu (dynamite-38.sandburst.com [172.16.5.38]) by sandmail.sandburst.com (Postfix) with ESMTP id 2BA9D94009 for ; Fri, 9 Mar 2001 10:48:14 -0500 (EST) To: ips@ece.cmu.edu Subject: Re: description of recovery mechanisms In-Reply-To: Message from "KRUEGER,MARJORIE (HP-Roseville,ex1)" of "Thu, 08 Mar 2001 17:01:11 PST." <6BD67FFB937FD411A04F00D0B74FE87802A08EEA@xrose06.rose.hp.com> References: <6BD67FFB937FD411A04F00D0B74FE87802A08EEA@xrose06.rose.hp.com> Date: Fri, 09 Mar 2001 10:47:16 -0500 From: Stephen Bailey Message-Id: <20010309154814.2BA9D94009@sandmail.sandburst.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > > > What Somesh is asking for is necessary. We can > > either put it in now, or discover it the hard > > way later in interoperability testing with > > considerably greater expenditure of time and > > effort. To be completely clear (as David know), we don't need to `discover' this the hard way, because it's ALREADY been discovered in FC and before that in ||SCSI. We KNOW this needs to be specified (OR, we won't interoperate---your choice). > More than just a description is necessary. I believe the proliferation of > options regarding recovery will make for an interoperability nightmare > similar to early iterations of Fibre Channel protocols. Too many options > and too much ambiguity result in too many conflicting interpretations. We > need to simplify, remove options, over specify with examples. What she said. Steph From owner-ips@ece.cmu.edu Fri Mar 9 12:14:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA23504 for ; Fri, 9 Mar 2001 12:14:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29Fk9a11525 for ips-outgoing; Fri, 9 Mar 2001 10:46:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29FiSr11389 for ; Fri, 9 Mar 2001 10:44:29 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA67902 for ; Fri, 9 Mar 2001 16:44:21 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA67562 for ; Fri, 9 Mar 2001 16:43:19 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0A.00564C54 ; Fri, 9 Mar 2001 16:42:36 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 9 Mar 2001 17:44:43 +0200 Subject: RE: description of recovery mechanisms Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, If you mean a complete formal description (including state charts) the last complete description I've seen was the SNA manual and that was many years ago (and included hundreds of pages on recovery alone). I've tried to go into enough detail so that an implementation should be possible and given the amount of options possible I am afraid that either we give them all and that is an inordinate (and wasteful) amount of work. None of the major TCP/IP protocols have it and they have all added details in a pragmatic fashion. The same thing can be claimed for the whole protocol not only for recovery. If anybody has a specific suggestion about something that is missing or a another structure for the recovery chapter (in addition or instead the two-dimensional approach that I have attempted) and would not require me to write detailed recovery pseudocode (I could be amenable to write a rough pseudocode skeleton for the recovery) I am ready to listen. Regards, Julo Black_David@emc.com on 09/03/2001 02:33:52 Please respond to Black_David@emc.com To: someshg@yahoo.com, ips@ece.cmu.edu cc: Subject: RE: description of recovery mechanisms What Somesh is asking for is necessary. We can either put it in now, or discover it the hard way later in interoperability testing with considerably greater expenditure of time and effort. --David > -----Original Message----- > From: Somesh Gupta [SMTP:someshg@yahoo.com] > Sent: Wednesday, March 07, 2001 10:24 PM > To: ips@ece.cmu.edu > Subject: description of recovery mechanisms > > I hope David and Julian will excuse me for using the following > sentences from the Orlando minutes. > > ----------- start of quote ------------------------------- > > - There will be a significant connection recovery write-up, > including details, procedures and examples added to the draft. > > ----------- end of quote --------------------------------- > > > As an engineer, I believe that we do need detailed and thorough > description of the usage of all the recovery tools in the > protocol. This ensures > > 1. Determination that there are no holes. Presence of "holes" > will lead to the mechanisms not being used (but implemented) > > or determine that there are no holes which will lead > to testing nightmares. > > 2. Ensure interoperability among implementations > > I hope this does not sound like I am asking Julian to do > my work for me. But it is better hashed out and debated > in one place. > From owner-ips@ece.cmu.edu Fri Mar 9 12:14:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA23538 for ; Fri, 9 Mar 2001 12:14:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29FQ9n10164 for ips-outgoing; Fri, 9 Mar 2001 10:26:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29FP5r10052 for ; Fri, 9 Mar 2001 10:25:06 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA200682 for ; Fri, 9 Mar 2001 16:24:58 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA113794 for ; Fri, 9 Mar 2001 16:23:53 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0A.00548293 ; Fri, 9 Mar 2001 16:23:04 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 9 Mar 2001 17:25:08 +0200 Subject: Re: iSCSI: Markers Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk ... oops - the first question - whatever comes first. The whole idea of the markerless interval is to allow negotiation. I assume that implementers will want to insert markers after negotiating the relevant parameters - but some will certainly take the easy way out - i.e., leave enough room so that the first marker will appear well after login is finished. Julo "Venkat Rangan" on 09/03/2001 02:13:46 Please respond to "Venkat Rangan" To: Julian Satran/Haifa/IBM@IBMIL cc: "IP Storage Working Group" Subject: iSCSI: Markers Julian, Immediately following the initial marker-less interval, what appears on the stream? Is it the first marker or is it the first PDU? Also, in heading 19 (page 110), "Default is 4096" would imply the default initial markerless-interval is 16K, but in earlier section (page 105), it states the default is 4K. The marker offsets are in bytes, and the left most two bits are reserved. So, one would be unable to store a marker for a LongDataWord PDU. We don't think it's a good idea to send such a long PDU, but providing smaller number of bits in marker offsets appears to be a sneaky way to impose a limit. Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com From owner-ips@ece.cmu.edu Fri Mar 9 12:14:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA23558 for ; Fri, 9 Mar 2001 12:14:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29FL6E09799 for ips-outgoing; Fri, 9 Mar 2001 10:21:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29FKer09749 for ; Fri, 9 Mar 2001 10:20:40 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA200612 for ; Fri, 9 Mar 2001 16:20:33 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA54322 for ; Fri, 9 Mar 2001 16:18:49 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0A.00541DB6 ; Fri, 9 Mar 2001 16:18:46 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 9 Mar 2001 17:20:50 +0200 Subject: Re: iSCSI: Markers Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Venkat, You have a good point. I think I'll give up on the "long data" (I'll remove it) unless somebody gets really upset that it can't transfer more than 16Mbytes in a PDU. I will also fix the default to read 1024. Julo "Venkat Rangan" on 09/03/2001 02:13:46 Please respond to "Venkat Rangan" To: Julian Satran/Haifa/IBM@IBMIL cc: "IP Storage Working Group" Subject: iSCSI: Markers Julian, Immediately following the initial marker-less interval, what appears on the stream? Is it the first marker or is it the first PDU? Also, in heading 19 (page 110), "Default is 4096" would imply the default initial markerless-interval is 16K, but in earlier section (page 105), it states the default is 4K. The marker offsets are in bytes, and the left most two bits are reserved. So, one would be unable to store a marker for a LongDataWord PDU. We don't think it's a good idea to send such a long PDU, but providing smaller number of bits in marker offsets appears to be a sneaky way to impose a limit. Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com From owner-ips@ece.cmu.edu Fri Mar 9 12:15:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA23607 for ; Fri, 9 Mar 2001 12:15:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29Fk7911522 for ips-outgoing; Fri, 9 Mar 2001 10:46:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sandmail.sandburst.com (sandburst-gw.bstn-gw01.ma.us.intelilink.net [216.57.129.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29FjSr11431 for ; Fri, 9 Mar 2001 10:45:28 -0500 (EST) Received: from cs.uchicago.edu (dynamite-38.sandburst.com [172.16.5.38]) by sandmail.sandburst.com (Postfix) with ESMTP id C708894006 for ; Fri, 9 Mar 2001 10:45:27 -0500 (EST) To: ips@ece.cmu.edu Subject: Re: iSCSI: unsolicited Vs immediate; restart delay In-Reply-To: Message from "Mallikarjun C." of "Tue, 06 Mar 2001 12:19:17 PST." <200103062019.MAA20092@core.rose.hp.com> References: <200103062019.MAA20092@core.rose.hp.com> Date: Fri, 09 Mar 2001 10:44:30 -0500 From: Stephen Bailey Message-Id: <20010309154527.C708894006@sandmail.sandburst.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk > There's no architected mechanism to figure out on its own! Either it > has to keep the (session, connection, task) states around for a long time, > OR it can clean up the these states and trigger an unnecessary ULP > recovery on the initiator which is surprised at an unexpected restart failure > > (keep in mind that initiator may not be able to restart a login precisely > after the "minimum time" specified, typically there's an aggregation > of multiple O/S timer requests into one master handler). Providing an > upper limit is a cleaner, safer design for both initiator and target. I agree with Mallikarjun. This is a big problem already faced by increasingly general FCP configurations. The idea that RR_TOV (the resource recovery timer, which is equivalent to the timer being discussed here) could be fixedly specified (as opposed to communicated in the protocol) only really works with a narrow set of assumptions about the application configurations. With many order of magnitude range in delays, as happens with `internet' protocols, we really can't specify a fixed delay. Furthermore, in this particular case, we can't really punt it to the target to select. The target has no idea. The sad fact is that the initiator may not have much of an idea either, but if we don't provide the knob, a target choice is guaranteed to be wrong in some cases. Of course, you probably remember that I'm not a fan of recovery in any case, so my solution would be to punt recovery altogether. Clearly that's not going to happen, so the next choice is to make it work :^) Steph From owner-ips@ece.cmu.edu Fri Mar 9 12:16:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA23688 for ; Fri, 9 Mar 2001 12:16:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29FoA411843 for ips-outgoing; Fri, 9 Mar 2001 10:50:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29FnXr11773 for ; Fri, 9 Mar 2001 10:49:33 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAB199310; Fri, 9 Mar 2001 16:49:27 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA162036; Fri, 9 Mar 2001 16:48:24 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0A.0056C431 ; Fri, 9 Mar 2001 16:47:43 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: someshg@yahoo.com cc: ips@ece.cmu.edu Message-ID: Date: Fri, 9 Mar 2001 17:49:46 +0200 Subject: RE: description of recovery mechanisms Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, I would appreciate if your suggestions could be more concrete. In general terms I feel that I have enough details to proceed to implementation. If you feel otherwise please tell me what exactly you are missing. As I said before - If you expect an SNA manual you will probably not get one. Regards, Julo "Somesh Gupta" on 09/03/2001 03:19:23 Please respond to someshg@yahoo.com To: Julian Satran/Haifa/IBM@IBMIL, someshg@yahoo.com cc: Subject: RE: description of recovery mechanisms Julian, My message is in no way an attack - above or below the belt. I am taking an implementor's view of things which by necessity is detail focused, and likes to avoid interoperability issues, testing uncertainty and speculative processing among others. It is a different view than others (you may or may not fall in the category) who are more on the visionary side and less detail focused (not that they cannot do as good or better detail work). I don't think the level of detail in the document is enough at present. Regards, Somesh > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Thursday, March 08, 2001 3:28 AM > To: someshg@yahoo.com > Subject: Re: description of recovery mechanisms > > > > > That was sort of under the belt. There will always be peple out there that > any amount of descrption is not enough. > > Julo > > "Somesh Gupta" on 08/03/2001 05:23:53 > > Please respond to someshg@yahoo.com > > To: ips@ece.cmu.edu > cc: > Subject: description of recovery mechanisms > > > > > I hope David and Julian will excuse me for using the following > sentences from the Orlando minutes. > > ----------- start of quote ------------------------------- > > - There will be a significant connection recovery write-up, > including details, procedures and examples added to the draft. > > ----------- end of quote --------------------------------- > > > As an engineer, I believe that we do need detailed and thorough > description of the usage of all the recovery tools in the > protocol. This ensures > > 1. Determination that there are no holes. Presence of "holes" > will lead to the mechanisms not being used (but implemented) > > or determine that there are no holes which will lead > to testing nightmares. > > 2. Ensure interoperability among implementations > > I hope this does not sound like I am asking Julian to do > my work for me. But it is better hashed out and debated > in one place. > > If the rest of the working group does feel that the level of > detail in the spec is enough and the rest is an "exercise > for the reader", I will accept that. > > Thanks, > Somesh > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com > > > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Fri Mar 9 12:23:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA24145 for ; Fri, 9 Mar 2001 12:23:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29Fi8611375 for ips-outgoing; Fri, 9 Mar 2001 10:44:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sandmail.sandburst.com (sandburst-gw.bstn-gw01.ma.us.intelilink.net [216.57.129.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29FhQr11325 for ; Fri, 9 Mar 2001 10:43:26 -0500 (EST) Received: from cs.uchicago.edu (dynamite-38.sandburst.com [172.16.5.38]) by sandmail.sandburst.com (Postfix) with ESMTP id 117FF94006 for ; Fri, 9 Mar 2001 10:43:07 -0500 (EST) To: ips@ece.cmu.edu Subject: Re: FCIP iFCP encapsulation proposal In-Reply-To: Message from Vi Chau of "Tue, 06 Mar 2001 17:08:43 PST." <312419998E3CD211A52900A0C991A47A81AE67@gordan> References: <312419998E3CD211A52900A0C991A47A81AE67@gordan> Date: Fri, 09 Mar 2001 10:42:09 -0500 From: Stephen Bailey Message-Id: <20010309154307.117FF94006@sandmail.sandburst.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk > There is a remote possibility that a false compare could occur > from other data in the stream so it is necessary to continue to > check the "tentative" FCIP/iFCP payload and CRC also before > assuming a correct synchronization. If both CRC checks are good, > this certification should be at least as good as the data > integrity provided by the CRCs in a synchronized stream. Nonsense. The original FCIP proposal that the receiver scan for an EOF/HDR/SOF sandwich to recover synchronization has the same problem. This technique is fundamentally bogus. Any time you're scanning user-controllable data (as in from the completely uncontrolled, end application), you MUST assume absolute worst case (to fool the sync algorithm) data. In the worst case, the probability of faking out your resynchronization heuristic is much (!!!) greater than the probability of an integrity CRC giving you a false positive acceptance check. In other words, the user can pack the data stream with `cooked' data patterns that are just waiting for a dropped frame and an attempt to resynchronize. If you assume that the shortest pattern required to fool the resynchronization algorithm is n words (no longer than the minimum well-formed PDU size), and TCP segmentation is effectively random (*), the chance of your resynch algorithm getting spoofed (resulting in the user getting some control of trusted `machinery') is simply 1 in n. You can fix this problem by using a shared `secret' (from the client) between the two endpoints. For example, if you put a 64-bit random number that both endpoints know in the end/begin sandwich, THEN you can talk about vanishingly small probabilities of the sandwich discovery going wrong. Personally, I think this running sync rediscovery is all makework because it's still a far distant second (or third, if you like periodic markers) to ensuring that each segment is sufficiently self-describing (AKA framing). A framing solves several other problems, such as the digest failure problem currently being discussed by Julian & Somesh. Steph * It isn't but, AGAIN, you must make WORST CASE assumptions, because, with properly periodic or aperiodic patterns, the user can actually encourage the segmentation algorithm to work in her favor. From owner-ips@ece.cmu.edu Fri Mar 9 14:19:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA00784 for ; Fri, 9 Mar 2001 14:19:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29H3Dn17126 for ips-outgoing; Fri, 9 Mar 2001 12:03:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lysimachus.hosting.pacbell.net (lysimachus.hosting.pacbell.net [216.100.98.17]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29H27r17066 for ; Fri, 9 Mar 2001 12:02:07 -0500 (EST) Received: from VENKAT1 (64-160-62-200.rhapsodynetworks.com [64.160.62.200] (may be forged)) by lysimachus.hosting.pacbell.net id MAA28873; Fri, 9 Mar 2001 12:01:53 -0500 (EST) [ConcentricHost SMTP Relay 1.7] From: "Venkat Rangan" To: , Subject: RE: iSCSI: Next-Qualifiers and PDU diagrams Date: Fri, 9 Mar 2001 09:04:59 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Since you asked for other opinions... The parsing of the following PDU structure: WN-AHS1 BHS WN-AHS2 AHS1 WN-AHS3 AHS2 WN-DATA AHS3 Header-Digest DATA Data-Digest requires the parse engine to "remember" the WN until the corresponding header is parsed. This adds additional complexity to the state machine. The alternative: BHS WN-AHS1 AHS1 WN-AHS2 AHS2 WN-AHS3 AHS3 WN-DATA Header-Digest DATA Data-Digest allows the parser to read the WN and the value and not have to carry around the last-seen-WN while it reads the current value. There is also some value in: WN-DATA BHS WN-AHS1 AHS1 WN-AHS2 AHS2 WN-AHS3 AHS3 Header-Digest DATA Data-Digest because the WN-DATA tells us whether there are digests. If you can also add the overall length of all headers in WN-DATA, you can prepare and launch the header digest while the header is being read. In parallel, you also know where the Data is and whether a data digest is needed, so that can be launched as well. Regards, Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Monday, March 05, 2001 4:34 PM To: ips@ece.cmu.edu Subject: RE: iSCSI: Next-Qualifiers and PDU diagrams David, It really does not matter where the TLV is - before or after the BHS or any AHS. What matters is that you read it always with the BHS (or any subsequent AHS) and from it you derive what comes next. The simplest structure is only a BHS (no data) and even there you need the NQ to know that the data is not there. On regular TLV the sequence of operations you do is: Read TLV1 Read Variable part Read TLV2 ---- At best you can optimize it to: Read TLV1 Read Variable+TLV2 Read Variable+TLV3 ------ With our scheme - since we know that the first part BHS is always there we can read ReadBHS+TLV2 Readvariable+TLV3 ----------- There is no TLV1! I've put the NQ somewhat arbitrarily at the beginning - it could be at the end - but that is a matter of taste. I could add it to the PDU descriptions as an opaque field so that you can see your offsets at their final position (or almost final). I would like as much as you to add clarity to the document without affecting the structure of the header sequence (that is both efficient and flexible) and if adding the field to the PDU description makes it easier to understand then so be it - but let us hear some other voices too. Black_David@emc.com on 05/03/2001 16:01:04 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI: Next-Qualifiers and PDU diagrams > I will try to lessen confusion somehow - I am not sure that adding the NQ > everywhere is the answer. That'd be the most comprehensive thing to do, and I still think it's the right thing for clarity. At an absolute minimum, using one convention for which byte in a PDU is byte 0 is clearly called for. > The trouble with TLV formats is that they need at least two passes. I don't understand what you mean by "two passes", but see below because it may not be an issue. > With the NQ I use the fact that the basic header is always there > and the qualifier will be needed do describe the next segment (its type and length). Requiring that the first TLV always be the basic header segment whose size never changes is ok under a TLV approach. The differences between what we have now, and a TLV approach are: (1) The information currently in the NQ before the BHS moves to the TL after the BHS. It's still in a fixed location because the BHS is fixed size. (2) We spend another 4 bytes for the TL to describe the BHS. Now that I think about things a bit more, this initial TL could be omitted, with the TLV structure used only to describe what comes after the BHS. >From a parse standpoint, the first TL after the BHS can be parsed with the BHS because that TL is always in the same place wrt the BHS. Is there a lot of value to having the TL for what comes after the BHS show up on the wire before the BHS? I'm sceptical. Is there any good reason not to merge the two formats for describing variable length segments? --David > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Saturday, March 03, 2001 12:03 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Next-Qualifiers and PDU diagrams > > > > David, > > I will try to lessen confusion somehow - I am not sure that adding the NQ > everywhere is the answer. > > The trouble with TLV formats is that they need at least two passes. With > the NQ I use the fact that > the basic header is always there and the qualifier will be needed do > describe the next segment (its type and length). > > Julo > > Black_David@emc.com on 02/03/2001 21:23:10 > > Please respond to Black_David@emc.com > > To: ips@ece.cmu.edu > cc: > Subject: iSCSI: Next-Qualifiers and PDU diagrams > > > > > In working with some other folks, I stumbled across > a format issue that's more confusing than it needs > to be in the current iSCSI draft. The issue is that > Next-Qualifiers (WN, etc.) and the PDUs they go with > are not documented/diagrammed together, making it > hard to figure out how to parse iSCSI PDUs. > > For an example of how this causes confusion, consider > the Text, Login and NOP PDU diagrams in Sections 2.8 > through 2.13. Starting from one of these diagrams, > how does one figure out the size of the Text or > Ping data that's appended? > > The answer's not in the diagrams -- there's a 4-byte > Next Qualifier preceding the PDU shown in each diagram. > The Text or Ping data is considered to be a data > segment, and hence the format of the Next Qualifier > is that given in Section 2.2.2.3, including a 3-byte > length field that answers the question. > > That's not an obvious answer, and there are several > factors that contribute to it being potentially confusing: > - None of the PDU diagrams show the Next-Qualifiers > - Byte 0 means different things - at the top of > Section 2.2, it's the start of the Next-Qualifier, > but by Section 2.2.3, it's shifted down 4 bytes > to be the start of the Basic Header Segment. > - Nothing obvious seems to say that the Text or the > Ping Data is a "data segment". The latter > might be obvious, but the former isn't. > - The Next-Qualifier structure is unexpected to > those familiar with other protocols. A > common protocol structure is T/L/V (Type/ > Length/Value). Because the BHS type and > length are implicit, the iSCSI structure > is actually Type N+1/Length N+1/Value N. > So, much as I realize the busy work involved in > revising ASCII graphics, I think all the PDU diagrams > in Section 2 need to be revised to show the Next- > Qualifiers. This would also take care of the second > item above, because then byte 0 in all the PDU > diagrams will be the first byte of the Next-Qualifier > that precedes the Basic Header Segment. Addressing > the third item involves a few sentences in to > explain what the "data segment" indicated by the > Next Qualifier before the BHS actually covers/ > contains in those cases. > > The fourth item is a matter of taste. It would > certainly be cleaner if iSCSI used a TLV structure, > rather than a T+1/L+1/V structure. Much of the > network community is familiar with TLV and will > find the T+1/L+1/V structure confusing. Are the > 4 bytes saved by not having a descriptor for the > BHS really worth it? > > On a related topic, it would be considerably > cleaner if the type (WN) field were a single byte > that was completely enumerated, rather than the > current bit field structure, and if the > peculiar convention for the AddCDB length in > 2.2.2.1 were dropped in favor of the 3-byte > length field in 2.2.2.3 to remove a case > from the header parse logic. > > Mindful of my WG co-chair role, I really think > the diagram revisions and explanations of what > a "data segment" is need to be done. OTOH, > the above two paragraphs on whether to have a BHS > descriptor and how to format the Next-Qualifiers > are suggestions for further discussion. > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > > From owner-ips@ece.cmu.edu Fri Mar 9 16:59:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA06617 for ; Fri, 9 Mar 2001 16:59:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29J2CX24967 for ips-outgoing; Fri, 9 Mar 2001 14:02:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29J23r24955 for ; Fri, 9 Mar 2001 14:02:03 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel2.hp.com (Postfix) with ESMTP id F0FCBAF6 for ; Fri, 9 Mar 2001 14:01:53 -0500 (EST) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id LAA23020 for ips@ece.cmu.edu; Fri, 9 Mar 2001 11:02:50 -0800 (PST) Message-Id: <200103091902.LAA23020@core.rose.hp.com> Subject: Re: StatSn Questions To: ips@ece.cmu.edu Date: Fri, 09 Mar 2001 11:02:50 PST Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt and Julian, >Hi Kevin, > >The intent (I believe) is that all target-to-initiator messages will convey >the current StatSN. That way, if the "status" message that contained the >StatSN was discarded due to header digest error, the initiator would know >about it when the next PDU was received, and could then perform error recovery >(if possible). > >In the SCSI Task Response, Text Response and NOP-In PDUs,- StatSN has the same >meaning as SCSI Response (StatSN is incremented before PDU is sent). OK. >In Data >and R2T, it just transfers the previous value (except if Data is the last Data >and conveys status). StatSN in a read data PDU is *not* the previous value. Instead, it is a new (incremented) value and the field is valid only if the S-bit is set. I hope Matt's interpretation of StatSN in R2T is correct - Julian, could you please add text explaining it? Also, please rename it as "Current StatSN"/"Running DataSN" (along those lines, since it is quoting a previous value) than a simple "StatSN" (which to me implies a unique StatSN assigned to this PDU). From the current names, I was inclined to think that the R2T PDU has two sequence numbers! -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com > >-Matt > > > >"LEMAY,KEVIN (A-Roseville,ex1)" wrote: >> >> The iSCSI spec implies that the StatSn handshake is supposed to be for >> command responses, but all target PDUs except Login response, logout >> response and reject contain StatSn. The asynchronous message has a comment >> that it is a acknowledgeable event. For what purpose is StatSn on the other >> PDUs and how is it used? >> >> Julian answered a question earlier about how to handle holes in the StatSn, >> but what about the case where the header is corrupted. It would be >> impossible to determine which command to retry. >> >> Additionally, any responses that may be discarded while we attempt to resync >> the iSCSI headers via markers would require to be retied also. Will the spec >> be changed to address the appropriate way to handle these cases? >> >> The cases shown in appendix B the command complete with the reception of the >> command response from the target. In reality, the command is not complete >> until the incremented ExpStatSn is received at the target. >> >> Kevin Lemay >> > > From owner-ips@ece.cmu.edu Fri Mar 9 17:00:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA06681 for ; Fri, 9 Mar 2001 17:00:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29JsIc28741 for ips-outgoing; Fri, 9 Mar 2001 14:54:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gordan.pl.gadzoox.com (i248.gadzoox.com [216.52.31.248]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29Jrar28708 for ; Fri, 9 Mar 2001 14:53:37 -0500 (EST) Received: from gadzoox.com (tickle.pl.gadzoox.com [209.95.220.48]) by gordan.pl.gadzoox.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id GSMZHJQC; Fri, 9 Mar 2001 11:54:41 -0800 Message-ID: <3AA934D7.9030807@gadzoox.com> Date: Fri, 09 Mar 2001 11:53:59 -0800 From: lani@gadzoox.com User-Agent: Mozilla/5.0 (Windows; U; WinNT4.0; en-US; m18) Gecko/20010131 Netscape6/6.01 X-Accept-Language: en MIME-Version: 1.0 To: Stephen Bailey CC: ips@ece.cmu.edu Subject: Re: FCIP iFCP encapsulation proposal References: <312419998E3CD211A52900A0C991A47A81AE67@gordan> <20010309154307.117FF94006@sandmail.sandburst.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Steph I will not argue with your opinion of the value of running synch rediscovery, but I disagree with the observance that the chance of spoofing the resynch mechanism is a linear "one in n". Although not directly mentioned in the document, it is implied that the qualification for a CRC acceptance on the payload portion be restrained to assume only a payload length which exactly matches the length as defined in the length field of the tentative header. Certain reality checks on the other header fields would not be out of order either. Lani Stephen Bailey wrote: >> There is a remote possibility that a false compare could occur >> from other data in the stream so it is necessary to continue to >> check the "tentative" FCIP/iFCP payload and CRC also before >> assuming a correct synchronization. If both CRC checks are good, >> this certification should be at least as good as the data >> integrity provided by the CRCs in a synchronized stream. > > > Nonsense. > > The original FCIP proposal that the receiver scan for an EOF/HDR/SOF > sandwich to recover synchronization has the same problem. This > technique is fundamentally bogus. > > Any time you're scanning user-controllable data (as in from the > completely uncontrolled, end application), you MUST assume absolute > worst case (to fool the sync algorithm) data. In the worst case, the > probability of faking out your resynchronization heuristic is much > (!!!) greater than the probability of an integrity CRC giving you a > false positive acceptance check. > > In other words, the user can pack the data stream with `cooked' data > patterns that are just waiting for a dropped frame and an attempt to > resynchronize. If you assume that the shortest pattern required to > fool the resynchronization algorithm is n words (no longer than the > minimum well-formed PDU size), and TCP segmentation is effectively > random (*), the chance of your resynch algorithm getting spoofed > (resulting in the user getting some control of trusted `machinery') is > simply 1 in n. > > You can fix this problem by using a shared `secret' (from the client) > between the two endpoints. For example, if you put a 64-bit random > number that both endpoints know in the end/begin sandwich, THEN you > can talk about vanishingly small probabilities of the sandwich > discovery going wrong. > > Personally, I think this running sync rediscovery is all makework > because it's still a far distant second (or third, if you like > periodic markers) to ensuring that each segment is sufficiently > self-describing (AKA framing). A framing solves several other > problems, such as the digest failure problem currently being discussed > by Julian & Somesh. > > Steph > > * It isn't but, AGAIN, you must make WORST CASE assumptions, because, > with properly periodic or aperiodic patterns, the user can actually > encourage the segmentation algorithm to work in her favor. From owner-ips@ece.cmu.edu Fri Mar 9 17:02:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA06741 for ; Fri, 9 Mar 2001 17:02:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29JTAZ27035 for ips-outgoing; Fri, 9 Mar 2001 14:29:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hub1.san-jose.webnexus.net (hub1-20.san-jose.webnexus.net [209.140.224.20]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29JS8r26988 for ; Fri, 9 Mar 2001 14:28:08 -0500 (EST) Received: from rmaranon (h-209-140-231-182.webnexus.com [209.140.231.182] (may be forged)) by hub1.san-jose.webnexus.net (8.9.1a/8.9.1/WN-1.4) with SMTP id LAA23276 for ; Fri, 9 Mar 2001 11:28:02 -0800 (PST) From: "Renato E. Maranon" To: Subject: iSCSI:draft05 SCSI Response Sense/Response Data Date: Fri, 9 Mar 2001 11:26:52 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <20010309154307.117FF94006@sandmail.sandburst.com> Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I don't see a length field for the Sense Data or Response Data. Would Sense or Response Data be another segment (AHS)as part of the same PDU? If it is an AHS, would that be defined as a data segment (WN byte 0, Bit 7 = 0, WN byte 1-3 = length of sense/response data)? This question applies to Text Command and Text Response also. Thanks, Renato Maranon Maranti Networks, Inc 920 Hillview Court Milpitas, Ca 95035 Phone: 408-719-9600 x309 Fax: 408-719-9631 email: rmaranon@marantinetworks.com home: www.marantinetworks.com From owner-ips@ece.cmu.edu Fri Mar 9 17:44:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA08567 for ; Fri, 9 Mar 2001 17:44:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29KtCg02996 for ips-outgoing; Fri, 9 Mar 2001 15:55:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com (mxic1.isus.emc.com [168.159.211.82]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29Kscr02962 for ; Fri, 9 Mar 2001 15:54:38 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 9 Mar 2001 15:55:54 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0708015269@corpmx9.isus.emc.com> From: Black_David@emc.com To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: description of recovery mechanisms Date: Fri, 9 Mar 2001 15:54:31 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, > If you mean a complete formal description (including state charts) the last > complete description I've seen was the SNA manual and that was many years > ago (and included hundreds of pages on recovery alone). I've tried to go > into enough detail so that an implementation should be possible and given > the amount of options possible I am afraid that either we give them all and > that is an inordinate (and wasteful) amount of work. None of the major > TCP/IP protocols have it and they have all added details in a pragmatic > fashion. The underlying issue here is that "given the number of options possible" it's almost certainly possible for implementers to choose different sets of options in a way that the implementations don't interoperate. If it takes state charts to prevent this, so be it, and if that's "an inordinate (and wasteful) amount of work", then the option space needs to be dramatically reduced in size. The requirement is not for state charts per se, but for a sufficient level of clarity and explanation of important details that independent implementers applying the same text to different implementations wind up with results that interoperate (or that fail to follow the explicit instructions given in the text). TCP error recovery is specified in excruciating detail down to minor tweaks in the retransmit algorithms. I hope this is clear ... because if this isn't done right, it will have to be done over courtesy of the draft not making it through WG Last Call. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Mar 9 17:44:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA08580 for ; Fri, 9 Mar 2001 17:44:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29KZCp01608 for ips-outgoing; Fri, 9 Mar 2001 15:35:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29KYRr01567 for ; Fri, 9 Mar 2001 15:34:27 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 9 Mar 2001 15:34:21 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0708015268@corpmx9.isus.emc.com> From: Black_David@emc.com To: matt_wakeley@agilent.com, ips@ece.cmu.edu Subject: RE: iscsi: editorial comments to iSCSI-05 Date: Fri, 9 Mar 2001 15:34:19 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > Global: "MUST", "SHOULD", etc are sometimes all upper case, > sometimes lowercase. Please make them consistent. Actually, that's deliberate. Upper case versions of these and other terms express requirements, recommendations, and options for interoperability and the like and are defined in RFC 2119 (see statement on p.3 of the iSCSI draft). Lower case versions of these terms may be used, but do not have the RFC 2119 meanings and implications of the upper case terms. The decision to use upper case vs. lower case is a deliberate decision about whether this is an interoperability issue or one of similar import/impact - in particular RFC 2119 advises that these upper case terms "be used with care and sparingly". --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Mar 9 18:36:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA10510 for ; Fri, 9 Mar 2001 18:36:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29LFC504361 for ips-outgoing; Fri, 9 Mar 2001 16:15:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com (mxic1.isus.emc.com [168.159.211.82]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29LEOr04328 for ; Fri, 9 Mar 2001 16:14:24 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 9 Mar 2001 16:15:40 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A070801526C@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: Minneapolis Agenda Date: Fri, 9 Mar 2001 16:14:17 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Here's the current IP Storage WG agenda for Minneapolis. It's subject to minor changes and tweaks (and Agenda Bashing at the start of the session). IETF IP Storage (IPS) Working Group Minneapolis IETF Meeting, March 18-23, 2001 ------------------------------------------- CHAIRS: David Black Elizabeth Rodriguez AGENDA: SUBJECT TO CHANGE NOTE: TCP Framing issues are in the domain of the Transport Area Working Group (tsvwg), and will not be discussed in these IPS working group meetings. ---- Monday, March 19, 0900-1130 ---- - Agenda Bashing and Administrivia (10 min) This will include a brief announcement of a possible T10 SAM-3 SCSI Architecture Model project to solicit interest. The project would be pursued in T10, not IETF. - iSCSI Requirements and Design Considerations (15 min) draft-ietf-ips-iscsi-reqmts-01.txt NOTE: The WG co-chairs intend to issue a Last Call on this draft in the near future. - iSCSI (75 min) draft-ietf-ips-iscsi-05.txt Topics to be covered include: - Checksum/CRC draft-cavanna-iscsi-crc-vs-cksum-01.txt - Security - Error Recovery - iSCSI Bootstrapping (5 min) draft-ietf-ips-iscsi-boot-02.txt - iSCSI MIB (15 min) draft-bakke-iscsimib-02.txt Structure diagram available at: http://www.haifa.il.ibm.com/satran/ips/iscsi-mib-structure-02.pdf NOTE: The authors would like the WG to adopt this as an official work item. - iSCSI Naming and Discovery Requirements (20 min) draft-ietf-ips-iscsi-name-disc-00.txt NOTE: draft-ietf-ips-iscsi-disc-reqts-00.txt is an older version of this draft, and should be ignored in favor of draft-ietf-ips-iscsi-name-disc-00.txt - URN Namespace for iSCSI WWUIs (10 min) draft-bakke-iscsi-wwui-urn-00 NOTE: The authors would like the WG to adopt this as an official work item. ---- Wednesday, March 21, 0900-1130 ---- - iSNS (20 min) draft-ietf-ips-isns-01.txt - Finding iSCSI Targets and Nameservers using SLP (15 min) draft-bakke-iscsi-slp-00 NOTE: The authors would like the WG to adopt this as an official work item. - FCIP/iFCP Common encapsulation discussion (70 min) draft-monia-ips-ifcpenc-00.txt draft-weber-fcip-encaps-00.txt Also, please see Vi Chau's email to the list on this topic: http://ips.pdl.cs.cmu.edu/mail/msg03565.html The goal of this discussion is to establish a rough consensus on direction for a common encapsulation of FC-2 Fibre Channel frames in TCP/IP to be used by both FCIP and iFCP. A design team will be formed to produce an initial proposal in line with this direction. - FCIP (5 min) draft-ietf-ips-fcovertcpip-01.txt A brief update, as this draft has not been changed since the San Diego meeting - FCIP Model (15 min) A discussion of formalizing the interface between FCIP and the Fibre Channel Fabric, including which Fibre Channel port type(s) are involved. The goal is to crisply separate IETF specification of the FCIP protocol from T11.3 specification of how it is used by a Fibre Channel Fabric. - iFCP (25 min) draft-ietf-ips-ifcp-00.txt DESCRIPTION: The IP Storage WG works on using IP-based networks to transport block storage traffic. See http://www.ietf.org/html.charters/ips-charter.html From owner-ips@ece.cmu.edu Fri Mar 9 20:06:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA12608 for ; Fri, 9 Mar 2001 20:06:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29NCTL11803 for ips-outgoing; Fri, 9 Mar 2001 18:12:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29NBpr11781 for ; Fri, 9 Mar 2001 18:11:51 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 47493780 for ; Fri, 9 Mar 2001 15:11:31 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id PAA28122; Fri, 9 Mar 2001 15:11:24 -0800 (PST) From: Santosh Rao Message-Id: <200103092311.PAA28122@hpcuhe.cup.hp.com> Subject: iSCSI : SCSI Response PDU To: ips@ece.cmu.edu (ips) Date: Fri, 9 Mar 2001 15:11:23 -0800 (PST) Cc: santoshr@hpcuhe.cup.hp.com (Santosh Rao) X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian/All, Some comments on recent changes to the SCSI Response PDU in Rev 04/05 : 1) Sense Length has been removed. This field is required and should be re-introduced. (Can anyone comment on the reasons behind its removal(?) ). 2) There is a reference to Response Data [without a corresponding response length. Once again, response length is required, if a response data is used.]. Response Data is modelled on the FC response data and is useless in the current context. (The FC response data only contained a response code). Any target that wishes to send vendor unique information could do so through vendor unique sense data. All references to Response Data should be removed. 3) A range of iSCSI Response codes should be assigned for vendor-unique responses. 4) The spec should state explicitly whether implementors are to use the REJECT PDU or a Scsi Response PDU with iSCSI Response code to indicate failure to process a command due to invalid command pdu fields. If the SCSI Response PDU is to be used to convey this, an additional iSCSI response code must be added for "Invalid Command PDU". 5) In section 2.5.1, pg 40, the spec states : "For all these functions, the SCSI Task Management Response MUST be returned using the Initiator Task Tag to identify the operation for which it is responding." The above MUST should be changed to SHOULD since a task mgmt command can be responded to with a REJECT response on a format error. Comments ? Regards, Santosh Rao -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Fri Mar 9 20:07:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA12629 for ; Fri, 9 Mar 2001 20:07:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f29NKE712254 for ips-outgoing; Fri, 9 Mar 2001 18:20:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gordan.pl.gadzoox.com (i248.gadzoox.com [216.52.31.248]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f29NJhr12235 for ; Fri, 9 Mar 2001 18:19:44 -0500 (EST) Received: by gordan with Internet Mail Service (5.5.2650.21) id ; Fri, 9 Mar 2001 15:20:48 -0800 Message-ID: <312419998E3CD211A52900A0C991A47A81AE84@gordan> From: Vi Chau To: ips@ece.cmu.edu Subject: RE: FCIP iFCP encapsulation proposal Date: Fri, 9 Mar 2001 15:20:48 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Stephen, I am confused by your argument. The re-synchronization that we proposed is between two FCIP devices/ports; why would one side want to confuse the other side by sending patterns that cause false re-synchronization? FCIP devices are defined to be the providers of tunnels for FC data to pass through. So the users of the FCIP devices are the FC end nodes that want to communicate with each other so they do not have a tendency to intentionally send confusing or "cooked" data patterns. Vi > -----Original Message----- > From: Stephen Bailey [mailto:steph@cs.uchicago.edu] > Sent: Friday, March 09, 2001 7:42 AM > To: ips@ece.cmu.edu > Subject: Re: FCIP iFCP encapsulation proposal > > > > There is a remote possibility that a false compare could occur > > from other data in the stream so it is necessary to continue to > > check the "tentative" FCIP/iFCP payload and CRC also before > > assuming a correct synchronization. If both CRC checks are good, > > this certification should be at least as good as the data > > integrity provided by the CRCs in a synchronized stream. > > Nonsense. > > The original FCIP proposal that the receiver scan for an EOF/HDR/SOF > sandwich to recover synchronization has the same problem. This > technique is fundamentally bogus. > > Any time you're scanning user-controllable data (as in from the > completely uncontrolled, end application), you MUST assume absolute > worst case (to fool the sync algorithm) data. In the worst case, the > probability of faking out your resynchronization heuristic is much > (!!!) greater than the probability of an integrity CRC giving you a > false positive acceptance check. > > In other words, the user can pack the data stream with `cooked' data > patterns that are just waiting for a dropped frame and an attempt to > resynchronize. If you assume that the shortest pattern required to > fool the resynchronization algorithm is n words (no longer than the > minimum well-formed PDU size), and TCP segmentation is effectively > random (*), the chance of your resynch algorithm getting spoofed > (resulting in the user getting some control of trusted `machinery') is > simply 1 in n. > > You can fix this problem by using a shared `secret' (from the client) > between the two endpoints. For example, if you put a 64-bit random > number that both endpoints know in the end/begin sandwich, THEN you > can talk about vanishingly small probabilities of the sandwich > discovery going wrong. > > Personally, I think this running sync rediscovery is all makework > because it's still a far distant second (or third, if you like > periodic markers) to ensuring that each segment is sufficiently > self-describing (AKA framing). A framing solves several other > problems, such as the digest failure problem currently being discussed > by Julian & Somesh. > > Steph > > * It isn't but, AGAIN, you must make WORST CASE assumptions, because, > with properly periodic or aperiodic patterns, the user can actually > encourage the segmentation algorithm to work in her favor. > From owner-ips@ece.cmu.edu Fri Mar 9 21:42:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA15351 for ; Fri, 9 Mar 2001 21:42:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2A0jGY17308 for ips-outgoing; Fri, 9 Mar 2001 19:45:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2A0iwr17293 for ; Fri, 9 Mar 2001 19:44:58 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 9FB4418F5 for ; Fri, 9 Mar 2001 16:44:57 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id QAA09530 for ips@ece.cmu.edu; Fri, 9 Mar 2001 16:44:53 -0800 (PST) From: Santosh Rao Message-Id: <200103100044.QAA09530@hpcuhe.cup.hp.com> Subject: iSCSI : EndDataSN missing in READ Data PDU To: ips@ece.cmu.edu (ips) Date: Fri, 9 Mar 2001 16:44:52 -0800 (PST) X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian & All, The EndDataSN is missing from the READ Data PDU. (Perhaps, this can replace the "Residual Count" field, since any READ that encountered an underrun would not result in Status being sent with the Data PDU.) Regards, Santosh -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Fri Mar 9 23:01:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA17749 for ; Fri, 9 Mar 2001 23:01:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2A1wHM21984 for ips-outgoing; Fri, 9 Mar 2001 20:58:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2A1vHr21949 for ; Fri, 9 Mar 2001 20:57:17 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Fri, 9 Mar 2001 17:57:12 -0800 Message-ID: From: Charles Monia To: Vi Chau Cc: "Ips (E-mail)" Subject: RE: FCIP iFCP encapsulation proposal Date: Fri, 9 Mar 2001 17:57:04 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Vi: My .02: I assume the argument is that a malicious end user could generate an FC payload stream filled with a bogus synchonization signature. One way to generate such a stream is to create a large file filled with such data. The malicious user then does a continuous read in the background, assuming that an error will eventually trigger the need to recover synchronization. I can envision a case where the user embeds a sequence of several well-formed PDUs in such a payload to handle the case where resynchronization requires the detection of more than one good encapsulation. Charles > -----Original Message----- > From: Vi Chau [mailto:vchau@gadzoox.com] > Sent: Friday, March 09, 2001 3:21 PM > To: ips@ece.cmu.edu > Subject: RE: FCIP iFCP encapsulation proposal > > > Stephen, > > I am confused by your argument. The re-synchronization that we > proposed is between two FCIP devices/ports; why would one side want > to confuse the other side by sending patterns that cause false > re-synchronization? FCIP devices are defined to be the providers > of tunnels for FC data to pass through. So the users of the FCIP > devices are the FC end nodes that want to communicate with each > other so they do not have a tendency to intentionally send > confusing or "cooked" data patterns. > > Vi > > > -----Original Message----- > > From: Stephen Bailey [mailto:steph@cs.uchicago.edu] > > Sent: Friday, March 09, 2001 7:42 AM > > To: ips@ece.cmu.edu > > Subject: Re: FCIP iFCP encapsulation proposal > > > > > > > There is a remote possibility that a false compare could occur > > > from other data in the stream so it is necessary to continue to > > > check the "tentative" FCIP/iFCP payload and CRC also before > > > assuming a correct synchronization. If both CRC checks are good, > > > this certification should be at least as good as the data > > > integrity provided by the CRCs in a synchronized stream. > > > > Nonsense. > > > > The original FCIP proposal that the receiver scan for an EOF/HDR/SOF > > sandwich to recover synchronization has the same problem. This > > technique is fundamentally bogus. > > > > Any time you're scanning user-controllable data (as in from the > > completely uncontrolled, end application), you MUST assume absolute > > worst case (to fool the sync algorithm) data. In the worst > case, the > > probability of faking out your resynchronization heuristic is much > > (!!!) greater than the probability of an integrity CRC giving you a > > false positive acceptance check. > > > > In other words, the user can pack the data stream with `cooked' data > > patterns that are just waiting for a dropped frame and an attempt to > > resynchronize. If you assume that the shortest pattern required to > > fool the resynchronization algorithm is n words (no longer than the > > minimum well-formed PDU size), and TCP segmentation is effectively > > random (*), the chance of your resynch algorithm getting spoofed > > (resulting in the user getting some control of trusted > `machinery') is > > simply 1 in n. > > > > You can fix this problem by using a shared `secret' (from > the client) > > between the two endpoints. For example, if you put a 64-bit random > > number that both endpoints know in the end/begin sandwich, THEN you > > can talk about vanishingly small probabilities of the sandwich > > discovery going wrong. > > > > Personally, I think this running sync rediscovery is all makework > > because it's still a far distant second (or third, if you like > > periodic markers) to ensuring that each segment is sufficiently > > self-describing (AKA framing). A framing solves several other > > problems, such as the digest failure problem currently > being discussed > > by Julian & Somesh. > > > > Steph > > > > * It isn't but, AGAIN, you must make WORST CASE > assumptions, because, > > with properly periodic or aperiodic patterns, the user can actually > > encourage the segmentation algorithm to work in her favor. > > > From owner-ips@ece.cmu.edu Fri Mar 9 23:56:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA19126 for ; Fri, 9 Mar 2001 23:56:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2A3DJj26029 for ips-outgoing; Fri, 9 Mar 2001 22:13:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dns.LampreyNetworks.com (IDENT:root@mail.lampreynetworks.com [64.31.72.26]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2A23cr22275 for ; Fri, 9 Mar 2001 21:03:38 -0500 (EST) Received: from breinhold ([24.128.216.253]) by dns.LampreyNetworks.com (8.9.3/8.8.7) with SMTP id VAA18045; Fri, 9 Mar 2001 21:01:04 -0500 From: "Barry Reinhold - Lamprey POP3" To: "ISCSI" Cc: "Robert Russell" , "Ashish A. Palekar" , "Jon Sreekanth" , "James Smart" Subject: iSCSI: Towards a more effective PDU format Date: Fri, 9 Mar 2001 21:02:28 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit All, I know there are well defined feelings about how a PDU should be structured, and not everyone is going to be in the same camp on this issue. However, I am concerned that the current PDU format with the WN field has not really achieved the goals that I, for one, would like to see it achieve. So, let me list a few objectives for the PDU format and then suggest one possible solution. Objectives for the iSCSI PDU (that could be improved upon). 1. Simplicity - The PDU layout should be as straight forward as possible. Interoperability starts with simplicity. When the logical function is simple, documenting proper usage is a whole lot easier. When documentation is clear in a standard, the time to interoperability and market acceptance is reduced. 2. Efficient operation in the common execution path. I think "rough consensus" can probably be achieved in terms of what parts of an iSCSI header are going to be needed in the normal flow and what parts are exception cases. If we have consensus on this we should be able to agree on where to make things easy and where to add complexity. 3. Data integrity - If a header digest is going to be provided, the information it covers should be checked against the digest before being used. I think we can improve the current PDU format, in terms of these objectives, by doing the following: 1. Providing a fixed size header that contains the most commonly used fields. 2. Placing less commonly used fields in an optional portion of the header. 3. Providing a header digest over the fixed portion of the header and a separate digest for the optional portion of the header. Basic approach: 1. The fixed "BHS" portion of the header is in all iSCSI PDU headers. It is always the same size and is followed by a header digest that covers the "BHS", if header digests are being used. The fact that header digests are being used within a TCP byte stream is not encoded into the frame, but is obtained from state associated with the session. This allows the location of the header digest to be determined without using any information in the header. Only when the integrity of the header has been validated is the information in it used. 2. The header contains information on the size of the complete PDU, and a length field for any additional headers. The integrity of this information has been validated as it is part of the "BHS". The location of the additional headers, if they exist (in most cases they will not) is after the header digest. Thus additional headers can be located using validated information. 3. Each additional header is encoded using a standard TLV (type, length, value) format. In TLV format the fixed size type indicator tells how to interpret the data. The fixed size length field says how long the data is, and the variable sized value field contains the data. If an application doesn't understand the type, it can maintain sync. in the byte stream by skipping to the length field and then skipping the value. 4. The complete set of TLV values are covered by a distinct digest. This digest is considered a part of the header digest in terms of iSCSI option negotiation but is physically distinct. Although this requires another digest calculation its usage is expected to be fairly low and it is most likely calculated over a small number of bytes. 5. The data and data digest follow the TLV digest. In terms of layout, what you might have is the following: ================= Start of iSCSI PDU ============================= This portion of the PDU contains the information in the "BHS" plus: 1. The length field (4 bytes) 2. The length of the TLV information --------------- End of the fixed size header --------------------- --------------- Start of the header digest (optional) ------------ --------------- End of the header digest ------------------------- --------------- Start of TLV information (optional) -------------- (Additional header fields go here in a variable length space) --------------- End of TLV information --------------------------- ------ Start of "TLV header" digest (if header digest present)---- --------------- End of "TLV header" digest ----------------------- --------------- Start of data ------------------------------------ --------------- End of data -------------------------------------- --------------- Start of data digest (optional) ------------------ =============== End of data digest and PDU ======================= [David B: This looks better in a .pdf....are we allowed to use pdfs on this reflector?] I believe this to be a simple yet efficient structure for common usage. The optional headers require more involved processing, but allow incremental advancement of the TCP byte stream and avoids the loss of synchronization that might take place in other formats. The header can be verified, before being used. [Note 1: This PDU format does not attempt to address the problem of how to recover when there is a digest error. I believe this to be a distinct problem, though the PDU format could provide some additional level of robustness via redundancy of information at known locations.] [Note 2: Julian, If more detailed information on the PDU format is needed I would be glad to present it. However I think that most of the significant issues are addressed by this outline. The other items don't appear, to me, to be as significant.] Barry Reinhold Barry.Reinhold@trebia.com 603-659-0885 From owner-ips@ece.cmu.edu Sat Mar 10 00:57:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA20746 for ; Sat, 10 Mar 2001 00:57:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2A47Mp28863 for ips-outgoing; Fri, 9 Mar 2001 23:07:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2A46cr28838 for ; Fri, 9 Mar 2001 23:06:38 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel3.hp.com (Postfix) with ESMTP id 7E645955 for ; Fri, 9 Mar 2001 20:06:37 -0800 (PST) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id UAA05004 for ips@ece.cmu.edu; Fri, 9 Mar 2001 20:07:34 -0800 (PST) Message-Id: <200103100407.UAA05004@core.rose.hp.com> Subject: Re: iSCSI: unsolicited Vs immediate; restart delay To: ips@ece.cmu.edu Date: Fri, 09 Mar 2001 20:07:34 PST Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, >I would be glad to be corrected, if I am misinterpreting the usage of >UseR2T. >+++ the confusion stems from the role of UseR2T. UseR2T is all about >allowing an unsolicited first burst. After the first burst you need always >R2T. Thanks for the clarification. I was about to conclude this is the case earlier, but found the following in section 2.17, page67 - "In order to allow write operations without R2T, the initiator and target must have agreed to do so by both sending the UseR2T=no key- pair attribute to each other (either during Login or through the Text Command/Response mechanism)." And then the following in section 1.2.5, page 16 - "Targets operate in either solicited (R2T) data mode or unsolicited (non R2T) data mode." And then the following in Appendix D, page 109, item 19 (UseR2T key): "The UseR2T key is used to turn off the default use of R2T, thus allowing an initiator to send data to a target without the target having sent an R2T to the initiator." None of these give any indication that UseR2T only applies for the first burst! PLEASE rephrase all these appropriately to reflect the intent. >You are left now with only 2 parameters controlled by the two variables. >The only think still open is that for comletness it is probably advisable >to have for Immediate also a bit in the corresponding mode page bit+++ >+++++++++++++++++++++++++++++++++++++++ Immediate data is not the only thing that needs a bit. Even the unsolicited data needs one since we want to be able to control both independently. Also (as I have been harping on), SPC-2 defines a zero FirstBurstSize as "unlimited" than "not allowed". This needs to be redone to mean "not allowed". Should we request T10 for action on these (Rob Elliott helped the last time with SPC-2's definition of direction of FirstBurstSize)? Thanks. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Sat Mar 10 04:36:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA09042 for ; Sat, 10 Mar 2001 04:36:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2A7sPk10712 for ips-outgoing; Sat, 10 Mar 2001 02:54:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2A7rpr10691 for ; Sat, 10 Mar 2001 02:53:51 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id IAA65714; Sat, 10 Mar 2001 08:53:44 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA190930; Sat, 10 Mar 2001 08:52:41 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0B.002B3405 ; Sat, 10 Mar 2001 08:51:53 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: someshg@yahoo.com cc: ips@ece.cmu.edu Message-ID: Date: Fri, 9 Mar 2001 18:02:37 +0200 Subject: lengths and header digest error recovery Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, All protocols that have variable length elements and a check suffer from the same syndrome. You have to take some elements as good and try to recover. This holdsfor those protocols that have length as demarcation element and for those that have framing patterns for demarcation as framing patterns can also be bad. You recover at another frame boundary or another marker. I our case it might be possible to add to the level of confidence in the Next-Qualifier even within a bad block through parity bits or a specific pattern. This might enable implementations that don't use markers to reduce the number of times they have to resort to connection restart. But I am not sure that it is worth the effort. Regards, Julo From owner-ips@ece.cmu.edu Sat Mar 10 04:39:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA09117 for ; Sat, 10 Mar 2001 04:39:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2A7sRB10717 for ips-outgoing; Sat, 10 Mar 2001 02:54:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2A7rur10694 for ; Sat, 10 Mar 2001 02:53:56 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id IAA16748 for ; Sat, 10 Mar 2001 08:53:49 +0100 From: julian_satran@il.ibm.com Received: from d12mta05.de.ibm.com (d12mta06_cs0 [9.165.222.255]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA136896 for ; Sat, 10 Mar 2001 08:52:46 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0B.002B5D04 ; Sat, 10 Mar 2001 08:53:38 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: "KRUEGER,MARJORIE (HP-Roseville,ex1)" cc: ips@ece.cmu.edu Message-ID: Date: Fri, 9 Mar 2001 18:22:14 +0200 Subject: RE: description of recovery mechanisms Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Marjorie, Examples is one concrete item that I can understand is missing. And I will try to provide examples (or complete scenarios) in pseudocode in 06 Over specify - I think that is very bad. Recovery can't result in too many interoperability issues since the number of active elements we have for recovery is limited and the actions one has to follow when issuing or receiving those are simple. Julo "KRUEGER,MARJORIE (HP-Roseville,ex1)" on 09/03/2001 03:01:11 Please respond to "KRUEGER,MARJORIE (HP-Roseville,ex1)" To: ips@ece.cmu.edu cc: Subject: RE: description of recovery mechanisms More than just a description is necessary. I believe the proliferation of options regarding recovery will make for an interoperability nightmare similar to early iterations of Fibre Channel protocols. Too many options and too much ambiguity result in too many conflicting interpretations. We need to simplify, remove options, over specify with examples. Marjorie Krueger Networked Storage Architecture Networked Storage Solutions Org. Hewlett-Packard tel: +1 916 785 2656 fax: +1 916 785 0391 email: marjorie_krueger@hp.com > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Thursday, March 08, 2001 4:34 PM > To: someshg@yahoo.com; ips@ece.cmu.edu > Subject: RE: description of recovery mechanisms > > > What Somesh is asking for is necessary. We can > either put it in now, or discover it the hard > way later in interoperability testing with > considerably greater expenditure of time and > effort. > > --David > > > -----Original Message----- > > From: Somesh Gupta [SMTP:someshg@yahoo.com] > > Sent: Wednesday, March 07, 2001 10:24 PM > > To: ips@ece.cmu.edu > > Subject: description of recovery mechanisms > > > > I hope David and Julian will excuse me for using the following > > sentences from the Orlando minutes. > > > > ----------- start of quote ------------------------------- > > > > - There will be a significant connection recovery write-up, > > including details, procedures and examples added to the draft. > > > > ----------- end of quote --------------------------------- > > > > > > As an engineer, I believe that we do need detailed and thorough > > description of the usage of all the recovery tools in the > > protocol. This ensures > > > > 1. Determination that there are no holes. Presence of "holes" > > will lead to the mechanisms not being used (but implemented) > > > > or determine that there are no holes which will lead > > to testing nightmares. > > > > 2. Ensure interoperability among implementations > > > > I hope this does not sound like I am asking Julian to do > > my work for me. But it is better hashed out and debated > > in one place. > > > From owner-ips@ece.cmu.edu Sat Mar 10 04:39:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA09135 for ; Sat, 10 Mar 2001 04:39:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2A8CQw11672 for ips-outgoing; Sat, 10 Mar 2001 03:12:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2A8CBr11663 for ; Sat, 10 Mar 2001 03:12:11 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA24796 for ; Sat, 10 Mar 2001 09:12:04 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id JAA150936 for ; Sat, 10 Mar 2001 09:11:01 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0B.002CE48E ; Sat, 10 Mar 2001 09:10:20 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 10 Mar 2001 10:12:30 +0200 Subject: RE: Task Management Commands Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Robert, That was the easy way out (and I said so in the past too). Provided with enough access control you can limit the target reset to authorized initiators and then only for the group of LUs they are authorized to. Removing it completely removes the ability to do the operation atomically. Julo "Elliott, Robert" on 09/03/2001 08:15:12 Please respond to "Elliott, Robert" To: John Hufferd/San Jose/IBM@IBMUS, Julian Satran/Haifa/IBM@IBMIL cc: George Penokie , "'ENDL_TX@computer.org'" , Jim Hafner/Almaden/IBM@IBMUS Subject: RE: Task Management Commands Today, T10 approved 01-015r2. For devices compliant with SAM-2 revision 16, Logical Unit Reset is now mandatory and Target Reset is not. Protocols are allowed to mandate Target Reset, make it optional, or not support it at all. In response, the SCSI RDMA protocol working group (defining SCSI over InfiniBand) voted to remove Target Reset support from that protocol. See T10 document 01-108r0 for the (short) proposal. I recommend you do the same and remove both Target Warm Reset and Target Cold Reset task management functions from iSCSI. Other IP protocols don't offer similar resets - a web browser doesn't have a way to reset an http server, a NFS client cannot reset an NFS server, a telnet user cannot interfere with other telnet users. iSCSI shouldn't be any different. John's email mentioned 4 task management functions, which included Logical Unit Reset, and Clear Task Set. The Access Controls feature (by Jim Hafner; going into SPC-3 revision 1) can be used to prevent initiators from logging in and issuing Logical Unit Reset and Clear Task Set. It is not a cryptographically secure protocol, but will prevent accidents from software that doesn't know it is not supposed to access a logical unit. Many environments using iSCSI will want true authentication to be integrated with Access Controls. The general support for iSCSI authentication might cover this, or an Access Control-specific secure authentication might be needed. I think Access Controls was architected with that possibility in mind. --- Robert.Elliott@compaq.com Compaq Server Storage > -----Original Message----- > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > Sent: Tuesday, March 06, 2001 9:28 PM > To: John Hufferd > Cc: Elliott, Robert; George Penokie; Julian Satran > Subject: Re: Task Management Commands > > > John, Julian, > > Please review > > ftp://ftp.t10.org/t10/document.01/01-015r1.pdf > > If there are difficulties with the proposal, please contact > the author, > Rob Elliott whose e-mail address appears as a CC to this message. > > (Rob, thanks for taking on this task.) > > Best Wishes to All > > Ralph... > > John Hufferd wrote: > > > > > > > Ralph, George, > > The SCSI Task Management Commands, can impact other > initiators, and in some > > cases shuts down the other initiators, and the Target. I > think that, since > > this protocol can be issued from anyplace (now with iSCSI) > in the Internet, > > there is a Risk that some Users Systems might send a Target > Cold Reset, for > > example, either on purpose, or by accident. This seems to > be to be a very > > large hole. Basicly it is a SCSI issue, even if it is the > iSCSI protocol > > that brings the possibility of this to the "great > unwashed". When we > > build our Very Large Systems this seems to be a problem. > > > > Julian Satran, thought this was a T10 issue, and that you > folks could help > > push it through there. > > > > . > > . > > . > > John L. Hufferd > > Senior Technical Staff Member (STSM) > > IBM/SSG San Jose Ca > > (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > > Internet address: hufferd@us.ibm.com > > > > Julian Satran@IBMIL > > 03/02/2001 09:27 PM > > > > To: John Hufferd/San Jose/IBM@IBMUS@IBMDE > > cc: > > From: Julian Satran/Haifa/IBM@Haifa/IBM@IBMIL > > Subject: Re: Task Management Commands (Document link: > John Hufferd) > > > > As you might recall we discussed it several times and even > attempted to > > play with > > separate channels. Every solution we could think of > affected badly the > > good guys. > > I think that the best place to handle it is authorization > (T10). They > > have already limited the damage > > to resetting in fact only things that come from the > initiator that issued > > the reset. > > > > We have introduced the cold reset (drop all connections). We could > > specifically (or vendor specific) > > restrict that command to trusted entities with strict > authentication. > > > > I think that we could (should?) standardize an error code > that will tell > > the initiator > > "not authorized" but even this can be viewed as T10 turf. > > > > It would be a good idea to forward this exchange to George > Penokie and/or > > Ralph Weber. > > They have been helpful. > > > > Regards, > > Julo > > > > John Hufferd@IBMUS > > 02/03/2001 02:22 > > > > To: Julian Satran/Haifa/IBM@IBMIL@IBMDE > > cc: > > From: John Hufferd/San Jose/IBM@IBMUS > > Subject: Task Management Commands > > > > Julian, > > I got to thinking about the Task Management Commands, and > every thing > > dealing with functions 4-7, impacts other initiators, and > in some cases > > shuts down the other initiators, and the Target. I think > that, since this > > protocol can be issued from anyplace in the Internet, there > is a Risk that > > some Users Systems might send a Target Cold Reset, for > example, either on > > purpose, or by accident. This seems to be to be a very > large hole. I know > > it is a SCSI issue, but it is our protocol that brings the > possibility of > > this to the "great unwashed". When we build our Very > Large Systems this > > seems to be a problem. What do you think we should do > about this, both > > from the Standards side and from the Product side? > > > > . > > . > > . > > John L. Hufferd > > Senior Technical Staff Member (STSM) > > IBM/SSG San Jose Ca > > (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > > Internet address: hufferd@us.ibm.com > From owner-ips@ece.cmu.edu Sat Mar 10 10:52:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA16115 for ; Sat, 10 Mar 2001 10:52:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2ADhWa08245 for ips-outgoing; Sat, 10 Mar 2001 08:43:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from tiny-teddy.aarnet.edu.au (IDENT:root@tiny-teddy.aarnet.edu.au [203.21.37.30]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2A4aor00406 for ; Fri, 9 Mar 2001 23:36:50 -0500 (EST) Received: from aarnet.edu.au (sa085.dialup.csiro.au [144.110.4.85]) by tiny-teddy.aarnet.edu.au (8.9.3/8.9.3) with ESMTP id PAA26268; Sat, 10 Mar 2001 15:06:29 +1030 Message-ID: <3AA9AF65.8EB8CABC@aarnet.edu.au> Date: Sat, 10 Mar 2001 15:06:53 +1030 From: Glen Turner Organization: Australian Academic and Research Network X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.4.2-mpls0.800-gdt1 i686) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com CC: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: Re: description of recovery mechanisms References: <0F31E5C394DAD311B60C00E029101A0708015269@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian Satran wrote: > None of the major TCP/IP protocols have it and they have all added > details in a pragmatic fashion. Black David replied: > TCP error recovery is specified in excruciating detail down to > minor tweaks in the retransmit algorithms. That lack of detailed state diagrams for TCP notoriously lead to an undefined state that needed to be clarified in a later RFC. There was then a lot of work, recorded in the academic literature, to check that no other undefined states existed. And, as illustrated in earlier work in this list, we are still suffering in a lack of precision in the definition of TCP's Urgent field. Glen From owner-ips@ece.cmu.edu Sat Mar 10 14:26:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA20891 for ; Sat, 10 Mar 2001 14:26:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2AHge826763 for ips-outgoing; Sat, 10 Mar 2001 12:42:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2AHfrr26733 for ; Sat, 10 Mar 2001 12:41:54 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA44720 for ; Sat, 10 Mar 2001 18:41:46 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA35028 for ; Sat, 10 Mar 2001 18:40:42 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0B.00610C86 ; Sat, 10 Mar 2001 18:40:01 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 10 Mar 2001 18:58:14 +0200 Subject: Re: iSCSI : EndDataSN missing in READ Data PDU Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, Why would EndDataSN be needed? In separate statuses it is there to make sure that you have all all the data PDUs. In the "combined" you have the DataSN itself to tell you. A side remark - Over/Underrun are not always an error (causing separate status) - it depends on mode page settings? Regards, Julo Santosh Rao on 10/03/2001 02:44:52 Please respond to Santosh Rao To: ips@ece.cmu.edu (ips) cc: Subject: iSCSI : EndDataSN missing in READ Data PDU Julian & All, The EndDataSN is missing from the READ Data PDU. (Perhaps, this can replace the "Residual Count" field, since any READ that encountered an underrun would not result in Status being sent with the Data PDU.) Regards, Santosh -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Sat Mar 10 14:26:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA20904 for ; Sat, 10 Mar 2001 14:26:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2AHgc026759 for ips-outgoing; Sat, 10 Mar 2001 12:42:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2AHfpr26729 for ; Sat, 10 Mar 2001 12:41:51 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA147816 for ; Sat, 10 Mar 2001 18:41:44 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA35018 for ; Sat, 10 Mar 2001 18:40:39 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0B.00610990 ; Sat, 10 Mar 2001 18:39:54 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 10 Mar 2001 18:16:51 +0200 Subject: Re: iSCSI:draft05 SCSI Response Sense/Response Data Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Renato, The sense or response data (and text response) is part of the data segment. I changed some names in 06 to make this clear. Regards, Julo "Renato E. Maranon" on 09/03/2001 21:26:52 Please respond to "Renato E. Maranon" To: ips@ece.cmu.edu cc: Subject: iSCSI:draft05 SCSI Response Sense/Response Data I don't see a length field for the Sense Data or Response Data. Would Sense or Response Data be another segment (AHS)as part of the same PDU? If it is an AHS, would that be defined as a data segment (WN byte 0, Bit 7 = 0, WN byte 1-3 = length of sense/response data)? This question applies to Text Command and Text Response also. Thanks, Renato Maranon Maranti Networks, Inc 920 Hillview Court Milpitas, Ca 95035 Phone: 408-719-9600 x309 Fax: 408-719-9631 email: rmaranon@marantinetworks.com home: www.marantinetworks.com From owner-ips@ece.cmu.edu Sat Mar 10 14:26:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA20921 for ; Sat, 10 Mar 2001 14:26:36 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2AHgpq26784 for ips-outgoing; Sat, 10 Mar 2001 12:42:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2AHfbr26697 for ; Sat, 10 Mar 2001 12:41:37 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA04306 for ; Sat, 10 Mar 2001 18:41:33 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA35014 for ; Sat, 10 Mar 2001 18:40:29 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0B.00610673 ; Sat, 10 Mar 2001 18:39:46 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 10 Mar 2001 17:27:52 +0200 Subject: Re: iscsi: editorial comments to iSCSI-05 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Fixed - Thanks, Julo Matt Wakeley on 09/03/2001 12:50:34 Please respond to Matt Wakeley To: IPS Reflector cc: Subject: iscsi: editorial comments to iSCSI-05 editorial comments to iSCSI 05: Global: In 1.2 the iSCSI PDU is defined, yet the phrase "message" is used interchangeably with PDU. Change all occurences of "message" to the correct term "PDU", especially fix instances of blahblah message. Also, change "packets" to PDUs. Global: "MUST", "SHOULD", etc are sometimes all upper case, sometimes lowercase. Please make them consistent. Abstract, page 2; 1.2 page 10: change: "on top of TCP" to: "over TCP". 1.1, middle page 9: change: "SCSI is client-server..." to: "SCSI is a client-server..." 1.1, bottom page 9: change: "Command Descriptor Blocks (CDB) is ..." to: "The Command Descriptor Block (CDB) is ..." 1.2.2.1, last paragraph page 12: "iSCSI initiators and target ..." add "s" to target. 1.2.3, page 14, last paragraph: change: "Any message except login and text reaching a target on a TCP connection before the full feature phase MUST be ..." to: "... full feature phase is a protocol error (see 6.1) and MUST be..." 2.3.2, page 33: Include a reference to SAM-2. 2.3.6, page 34: Indicate that the "Extended CDB" what's next field must be used. Change "48-byte header" to "BHS". 2.4.5, page 37: Indicate that sense or response data is "data" and not part of the header. 2.7, page 43 and 44: Change "SCSI Data packet" to "iSCSI Data PDU" 2.8.3, page 48: move 'If the Text Response does not contain a key that was requested, the initiator must assume that the key was not understood by the target or, whenever appropriate, that the response was "none".' to after "Any other key not understood by the target may be ignored without affecting basic function." 2.10.1, page 51: change "command is performs" to "command performs" change "logout is not" to "logout was not" 2.10.6, page 52: change "This an" to "This is an". 2.11.3, page 54: The sentance "If the login phase involves two login responses then each of them will hold for the subsequent responses." does not make sense to me. 2.12.2, page 58: change "only if it issued" to "only if it is issued" 2.14.1, page 61: do you mean "close session" instead of "close connection"? 2.15, page 63: change "for the failed connection" to simply "for the connection". 2.16.4, page 65: change "additional missed" to "additional sequential missed". 2.17, page 66: "The last PDU should have the F bit set to 1". Shouldn't this be "must have the F bit set to 1"? 2.17, page 66: change: "The target may send several R2T PDUs and..." to to: The target may send several R2T PDUs (if negotiated) and..." 2.18, page 68: change: "the target specifies the status for..." to: "the target specifies the status and reason for..." 2.18.1, page 69: change: "the codes returned for..." to: "the codes sent for..." (and event is not "returned") 2.20, page 71: change "inexistent" to "non-existent" 6.1, page 80: change "format errors" to "protocol errors". 7.2, page 86: change "ACA helps preserving" to "ACA helps preserve" 8, page 87: The first paragraph says "iSCSI implementations MUST provide..." and the second says "should be provided". Make these consistent. From owner-ips@ece.cmu.edu Sat Mar 10 14:27:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA20942 for ; Sat, 10 Mar 2001 14:27:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2AHgsl26789 for ips-outgoing; Sat, 10 Mar 2001 12:42:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2AHfsr26734 for ; Sat, 10 Mar 2001 12:41:54 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA95030 for ; Sat, 10 Mar 2001 18:41:47 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA35032 for ; Sat, 10 Mar 2001 18:40:43 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0B.00610D08 ; Sat, 10 Mar 2001 18:40:03 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 10 Mar 2001 19:16:33 +0200 Subject: Re: iSCSI: Towards a more effective PDU format Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Barry, Thanks for outline. I will consider it as one of the variants I am going through to asses clarity. There are two things that are against it: 1- several digests 2- if I don't gather (how?) from BHS that TLVs are there then I have to do 2 reads for each additional segment - and besides this you are using a field before knowing it is correct. We could however rearrange in any fashion the BHS and NWs to make it look very close to what you propose and still have one read per segment and data used only after checking. Regards, Julo "Barry Reinhold - Lamprey POP3" on 10/03/2001 04:02:28 Please respond to "Barry Reinhold - Lamprey POP3" To: "ISCSI" cc: "Robert Russell" , "Ashish A. Palekar" , "Jon Sreekanth" , "James Smart" Subject: iSCSI: Towards a more effective PDU format All, I know there are well defined feelings about how a PDU should be structured, and not everyone is going to be in the same camp on this issue. However, I am concerned that the current PDU format with the WN field has not really achieved the goals that I, for one, would like to see it achieve. So, let me list a few objectives for the PDU format and then suggest one possible solution. Objectives for the iSCSI PDU (that could be improved upon). 1. Simplicity - The PDU layout should be as straight forward as possible. Interoperability starts with simplicity. When the logical function is simple, documenting proper usage is a whole lot easier. When documentation is clear in a standard, the time to interoperability and market acceptance is reduced. 2. Efficient operation in the common execution path. I think "rough consensus" can probably be achieved in terms of what parts of an iSCSI header are going to be needed in the normal flow and what parts are exception cases. If we have consensus on this we should be able to agree on where to make things easy and where to add complexity. 3. Data integrity - If a header digest is going to be provided, the information it covers should be checked against the digest before being used. I think we can improve the current PDU format, in terms of these objectives, by doing the following: 1. Providing a fixed size header that contains the most commonly used fields. 2. Placing less commonly used fields in an optional portion of the header. 3. Providing a header digest over the fixed portion of the header and a separate digest for the optional portion of the header. Basic approach: 1. The fixed "BHS" portion of the header is in all iSCSI PDU headers. It is always the same size and is followed by a header digest that covers the "BHS", if header digests are being used. The fact that header digests are being used within a TCP byte stream is not encoded into the frame, but is obtained from state associated with the session. This allows the location of the header digest to be determined without using any information in the header. Only when the integrity of the header has been validated is the information in it used. 2. The header contains information on the size of the complete PDU, and a length field for any additional headers. The integrity of this information has been validated as it is part of the "BHS". The location of the additional headers, if they exist (in most cases they will not) is after the header digest. Thus additional headers can be located using validated information. 3. Each additional header is encoded using a standard TLV (type, length, value) format. In TLV format the fixed size type indicator tells how to interpret the data. The fixed size length field says how long the data is, and the variable sized value field contains the data. If an application doesn't understand the type, it can maintain sync. in the byte stream by skipping to the length field and then skipping the value. 4. The complete set of TLV values are covered by a distinct digest. This digest is considered a part of the header digest in terms of iSCSI option negotiation but is physically distinct. Although this requires another digest calculation its usage is expected to be fairly low and it is most likely calculated over a small number of bytes. 5. The data and data digest follow the TLV digest. In terms of layout, what you might have is the following: ================= Start of iSCSI PDU ============================= This portion of the PDU contains the information in the "BHS" plus: 1. The length field (4 bytes) 2. The length of the TLV information --------------- End of the fixed size header --------------------- --------------- Start of the header digest (optional) ------------ --------------- End of the header digest ------------------------- --------------- Start of TLV information (optional) -------------- (Additional header fields go here in a variable length space) --------------- End of TLV information --------------------------- ------ Start of "TLV header" digest (if header digest present)---- --------------- End of "TLV header" digest ----------------------- --------------- Start of data ------------------------------------ --------------- End of data -------------------------------------- --------------- Start of data digest (optional) ------------------ =============== End of data digest and PDU ======================= [David B: This looks better in a .pdf....are we allowed to use pdfs on this reflector?] I believe this to be a simple yet efficient structure for common usage. The optional headers require more involved processing, but allow incremental advancement of the TCP byte stream and avoids the loss of synchronization that might take place in other formats. The header can be verified, before being used. [Note 1: This PDU format does not attempt to address the problem of how to recover when there is a digest error. I believe this to be a distinct problem, though the PDU format could provide some additional level of robustness via redundancy of information at known locations.] [Note 2: Julian, If more detailed information on the PDU format is needed I would be glad to present it. However I think that most of the significant issues are addressed by this outline. The other items don't appear, to me, to be as significant.] Barry Reinhold Barry.Reinhold@trebia.com 603-659-0885 From owner-ips@ece.cmu.edu Sat Mar 10 14:29:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA20980 for ; Sat, 10 Mar 2001 14:29:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2AHgg926768 for ips-outgoing; Sat, 10 Mar 2001 12:42:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2AHfnr26728 for ; Sat, 10 Mar 2001 12:41:50 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA95026; Sat, 10 Mar 2001 18:41:43 +0100 From: julian_satran@il.ibm.com Received: from d12mta05.de.ibm.com (d12mta06_cs0 [9.165.222.255]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA32728; Sat, 10 Mar 2001 18:39:54 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0B.00613241 ; Sat, 10 Mar 2001 18:41:38 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Black_David@emc.com cc: matt_wakeley@agilent.com, ips@ece.cmu.edu Message-ID: Date: Sat, 10 Mar 2001 18:19:43 +0200 Subject: RE: iscsi: editorial comments to iSCSI-05 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk correct - Julo Black_David@emc.com on 09/03/2001 22:34:19 Please respond to Black_David@emc.com To: matt_wakeley@agilent.com, ips@ece.cmu.edu cc: Subject: RE: iscsi: editorial comments to iSCSI-05 > Global: "MUST", "SHOULD", etc are sometimes all upper case, > sometimes lowercase. Please make them consistent. Actually, that's deliberate. Upper case versions of these and other terms express requirements, recommendations, and options for interoperability and the like and are defined in RFC 2119 (see statement on p.3 of the iSCSI draft). Lower case versions of these terms may be used, but do not have the RFC 2119 meanings and implications of the upper case terms. The decision to use upper case vs. lower case is a deliberate decision about whether this is an interoperability issue or one of similar import/impact - in particular RFC 2119 advises that these upper case terms "be used with care and sparingly". --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Sat Mar 10 14:38:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA21174 for ; Sat, 10 Mar 2001 14:38:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2AHgnU26781 for ips-outgoing; Sat, 10 Mar 2001 12:42:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2AHfar26696 for ; Sat, 10 Mar 2001 12:41:37 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA166462 for ; Sat, 10 Mar 2001 18:41:33 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA35012 for ; Sat, 10 Mar 2001 18:40:29 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0B.006106BD ; Sat, 10 Mar 2001 18:39:47 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 10 Mar 2001 17:52:48 +0200 Subject: Re: StatSn Questions Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun, Responses in text. Julo "Mallikarjun C." on 09/03/2001 21:02:50 Please respond to cbm@rose.hp.com To: ips@ece.cmu.edu cc: Subject: Re: StatSn Questions Matt and Julian, >Hi Kevin, > >The intent (I believe) is that all target-to-initiator messages will convey >the current StatSN. That way, if the "status" message that contained the >StatSN was discarded due to header digest error, the initiator would know >about it when the next PDU was received, and could then perform error recovery >(if possible). > >In the SCSI Task Response, Text Response and NOP-In PDUs,- StatSN has the same >meaning as SCSI Response (StatSN is incremented before PDU is sent). OK. >In Data >and R2T, it just transfers the previous value (except if Data is the last Data >and conveys status). StatSN in a read data PDU is *not* the previous value. Instead, it is a new (incremented) value and the field is valid only if the S-bit is set. +++ You are right - the reason I left StatSN out is that the Data PDU may originate close to the device where you don't know about StatRN +++ I hope Matt's interpretation of StatSN in R2T is correct - Julian, could you please add text explaining it? Also, please rename it as "Current StatSN"/"Running DataSN" (along those lines, since it is quoting a previous value) than a simple "StatSN" (which to me implies a unique StatSN assigned to this PDU). From the current names, I was inclined to think that the R2T PDU has two sequence numbers! +++ Matt is right - I've added a heading with the explanation on StatSN to R2T +++ -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com > >-Matt > > > >"LEMAY,KEVIN (A-Roseville,ex1)" wrote: >> >> The iSCSI spec implies that the StatSn handshake is supposed to be for >> command responses, but all target PDUs except Login response, logout >> response and reject contain StatSn. The asynchronous message has a comment >> that it is a acknowledgeable event. For what purpose is StatSn on the other >> PDUs and how is it used? >> >> Julian answered a question earlier about how to handle holes in the StatSn, >> but what about the case where the header is corrupted. It would be >> impossible to determine which command to retry. >> >> Additionally, any responses that may be discarded while we attempt to resync >> the iSCSI headers via markers would require to be retied also. Will the spec >> be changed to address the appropriate way to handle these cases? >> >> The cases shown in appendix B the command complete with the reception of the >> command response from the target. In reality, the command is not complete >> until the incremented ExpStatSn is received at the target. >> >> Kevin Lemay >> > > From owner-ips@ece.cmu.edu Sat Mar 10 14:39:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA21194 for ; Sat, 10 Mar 2001 14:39:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2AHgj326772 for ips-outgoing; Sat, 10 Mar 2001 12:42:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2AHfqr26731 for ; Sat, 10 Mar 2001 12:41:52 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA128438 for ; Sat, 10 Mar 2001 18:41:46 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA35026 for ; Sat, 10 Mar 2001 18:40:41 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0B.00610C32 ; Sat, 10 Mar 2001 18:40:01 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 10 Mar 2001 18:52:09 +0200 Subject: Re: iSCSI : SCSI Response PDU Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk responses in text: Thanks, Julo Santosh Rao on 10/03/2001 01:11:23 Please respond to Santosh Rao To: ips@ece.cmu.edu (ips) cc: santoshr@hpcuhe.cup.hp.com (Santosh Rao) Subject: iSCSI : SCSI Response PDU Julian/All, Some comments on recent changes to the SCSI Response PDU in Rev 04/05 : 1) Sense Length has been removed. This field is required and should be re-introduced. (Can anyone comment on the reasons behind its removal(?) ). +++ sense is in the data segment - data segment length is sense length +++ 2) There is a reference to Response Data [without a corresponding response length. Once again, response length is required, if a response data is used.]. Response Data is modelled on the FC response data and is useless in the current context. (The FC response data only contained a response code). Any target that wishes to send vendor unique information could do so through vendor unique sense data. All references to Response Data should be removed. +++ Responses are not sense +++ 3) A range of iSCSI Response codes should be assigned for vendor-unique responses. +++ Good Idea +++ 4) The spec should state explicitly whether implementors are to use the REJECT PDU or a Scsi Response PDU with iSCSI Response code to indicate failure to process a command due to invalid command pdu fields. If the SCSI Response PDU is to be used to convey this, an additional iSCSI response code must be added for "Invalid Command PDU". +++ section 6.1 covers this - Matt Wakeley proposed changing the name to protocol errors +++ 5) In section 2.5.1, pg 40, the spec states : "For all these functions, the SCSI Task Management Response MUST be returned using the Initiator Task Tag to identify the operation for which it is responding." The above MUST should be changed to SHOULD since a task mgmt command can be responded to with a REJECT response on a format error. +++ I will make " for all these functions, if executed ... +++ Comments ? Regards, Santosh Rao -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Sat Mar 10 14:42:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA21319 for ; Sat, 10 Mar 2001 14:42:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2AHgl126776 for ips-outgoing; Sat, 10 Mar 2001 12:42:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2AHg0r26739 for ; Sat, 10 Mar 2001 12:42:01 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA147826 for ; Sat, 10 Mar 2001 18:41:54 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA64438 for ; Sat, 10 Mar 2001 18:40:05 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0B.00610F7E ; Sat, 10 Mar 2001 18:40:09 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 10 Mar 2001 19:34:45 +0200 Subject: Re: iSCSI: unsolicited Vs immediate; restart delay Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun, I will make those statements clearer and perhaps change the name of the key from UseR2T to InitialR2T. Julo "Mallikarjun C." on 10/03/2001 06:07:34 Please respond to cbm@rose.hp.com To: ips@ece.cmu.edu cc: Subject: Re: iSCSI: unsolicited Vs immediate; restart delay Julian, >I would be glad to be corrected, if I am misinterpreting the usage of >UseR2T. >+++ the confusion stems from the role of UseR2T. UseR2T is all about >allowing an unsolicited first burst. After the first burst you need always >R2T. Thanks for the clarification. I was about to conclude this is the case earlier, but found the following in section 2.17, page67 - "In order to allow write operations without R2T, the initiator and target must have agreed to do so by both sending the UseR2T=no key- pair attribute to each other (either during Login or through the Text Command/Response mechanism)." And then the following in section 1.2.5, page 16 - "Targets operate in either solicited (R2T) data mode or unsolicited (non R2T) data mode." And then the following in Appendix D, page 109, item 19 (UseR2T key): "The UseR2T key is used to turn off the default use of R2T, thus allowing an initiator to send data to a target without the target having sent an R2T to the initiator." None of these give any indication that UseR2T only applies for the first burst! PLEASE rephrase all these appropriately to reflect the intent. >You are left now with only 2 parameters controlled by the two variables. >The only think still open is that for comletness it is probably advisable >to have for Immediate also a bit in the corresponding mode page bit+++ >+++++++++++++++++++++++++++++++++++++++ Immediate data is not the only thing that needs a bit. Even the unsolicited data needs one since we want to be able to control both independently. Also (as I have been harping on), SPC-2 defines a zero FirstBurstSize as "unlimited" than "not allowed". This needs to be redone to mean "not allowed". Should we request T10 for action on these (Rob Elliott helped the last time with SPC-2's definition of direction of FirstBurstSize)? Thanks. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Sat Mar 10 16:49:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA24085 for ; Sat, 10 Mar 2001 16:49:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2AJdeU03172 for ips-outgoing; Sat, 10 Mar 2001 14:39:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2AJcdr03136 for ; Sat, 10 Mar 2001 14:38:40 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Sat, 10 Mar 2001 14:38:37 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A070801526D@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: PDF usage Date: Sat, 10 Mar 2001 14:38:36 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Barry Reinhold asks: > [David B: This looks better in a .pdf....are we allowed to use pdfs on this > reflector?] Not exactly. PDFs, drafts, etc. (basically anything large) should *never* be sent to the reflector. Put them on a web site and send a URL to the reflector. It's ok to use PDFs for diagrams related to drafts or work underway in the WG, although any diagram that is to appear in a standards track RFC has to be reduced to ASCII graphics prior to Last Call on the draft that contains it. While it's ok to use PDFs for initial and interim versions of diagrams, I would strongly encourage ASCII for the sort of header layout and format issues that motivated Barry's original question. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Sat Mar 10 17:46:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA25182 for ; Sat, 10 Mar 2001 17:46:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2AKugB07366 for ips-outgoing; Sat, 10 Mar 2001 15:56:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from chmls05.mediaone.net (chmls05.mediaone.net [24.147.1.143]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2AKuJr07349 for ; Sat, 10 Mar 2001 15:56:19 -0500 (EST) Received: from breinhold ([24.128.216.253]) by chmls05.mediaone.net (8.11.1/8.11.1) with SMTP id f2AKuFS16067; Sat, 10 Mar 2001 15:56:15 -0500 (EST) From: "Barry Reinhold" To: , Subject: RE: iSCSI: Towards a more effective PDU format Date: Sat, 10 Mar 2001 15:55:06 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-reply-to: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, On the two drawbacks: 1. Agreed, there are two header digests and this isn't desirable. However, I suspect the TLVs will seldom be used so at least it is out of the fast path. [BTW - Can some digest expert suggest how bad this is vs. a single longer digest?] 2. I'm not sure I fully understand what you are asking under two, so I'm going to answer the question of "How would you read the TLV w/o using any unverified info." If this is an answer to the wrong question, my apologies. You'll have to ask again. The TLV length field is obtained from the BHS header as, or after, it has been verified. This TLV length field gives the size of the TLV data (surprise) that has to be buffered in reading the TCP stream to get to the TLV digest. The TLV digest is checked against the TLV data. The TLV's are then parsed. (Humm - I guess I have an assumption in here that TLV data is not *really* big, such that buffering the whole set of TLVs is a reasonable thing to do) I don't see where one ends up using unverified fields. I would expect to read a fixed N bytes from the TCP stream (through the header digest), verify the header, use the fixed offset of the "TLV length field" that is in the header, which has been verified, to find if I had TLV fields. If no, I'm on to the data. If yes I read the TCP stream with the number of bytes given by "TLV length field" + sizeof(TLV digest), verify the TLV fields, and then parse the TLVs as needed. A plus is that there is very little work to do for the fast path. Another plus is that the TLV mechanism allows vendor options, and/or future extensions with well understood backward compatibility behavior. A third plus is that there should be no special cases relative to how the PDU format has you read the TCP stream...read header, optionaly read TLV stuff, read data. This should be friendly to hardware accelerated stuff. A negative is that TLV parsing is sorta slow if you have complicated TLV structures such as in DOCSIS. Our TLVs are very simple....so far at least. >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >julian_satran@il.ibm.com >Sent: Saturday, March 10, 2001 12:17 PM >To: ips@ece.cmu.edu >Subject: Re: iSCSI: Towards a more effective PDU format > > > > >Barry, > >Thanks for outline. >I will consider it as one of the variants I am going through to asses >clarity. > >There are two things that are against it: > >1- several digests >2- if I don't gather (how?) from BHS that TLVs are there then I have to do >2 reads for each additional segment - and besides this you are using a >field before knowing it is correct. > >We could however rearrange in any fashion the BHS and NWs to make it look >very close to what you propose and still have one read per segment and data >used only after checking. > >Regards, >Julo > >"Barry Reinhold - Lamprey POP3" on 10/03/2001 >04:02:28 > >Please respond to "Barry Reinhold - Lamprey POP3" > >To: "ISCSI" >cc: "Robert Russell" , "Ashish A. Palekar" > , "Jon Sreekanth" , > "James Smart" >Subject: iSCSI: Towards a more effective PDU format > > > > >All, > >I know there are well defined feelings about how a PDU should be >structured, >and not everyone is going to be in the same camp on this issue. However, I >am concerned that the current PDU format with the WN field has not really >achieved the goals that I, for one, would like to see it achieve. So, let >me >list a few objectives for the PDU format and then suggest one possible >solution. > >Objectives for the iSCSI PDU (that could be improved upon). > >1. Simplicity - The PDU layout should be as straight forward as possible. >Interoperability starts with simplicity. When the logical function is >simple, documenting proper usage is a whole lot easier. When documentation >is clear in a standard, the time to interoperability and market acceptance >is reduced. > >2. Efficient operation in the common execution path. I think "rough >consensus" can probably be achieved in terms of what parts of an iSCSI >header are going to be needed in the normal flow and what parts are >exception cases. If we have consensus on this we should be able to agree on >where to make things easy and where to add complexity. > >3. Data integrity - If a header digest is going to be provided, the >information it covers should be checked against the digest before being >used. > >I think we can improve the current PDU format, in terms of these >objectives, >by doing the following: > >1. Providing a fixed size header that contains the most commonly used >fields. >2. Placing less commonly used fields in an optional portion of the header. >3. Providing a header digest over the fixed portion of the header and a >separate digest for the optional portion of the header. > >Basic approach: > >1. The fixed "BHS" portion of the header is in all iSCSI PDU headers. It is >always the same size and is followed by a header digest that covers the >"BHS", if header digests are being used. The fact that header digests are >being used within a TCP byte stream is not encoded into the frame, but is >obtained from state associated with the session. This allows the location >of >the header digest to be determined without using any information in the >header. Only when the integrity of the header has been validated is the >information in it used. > >2. The header contains information on the size of the complete PDU, and a >length field for any additional headers. The integrity of this information >has been validated as it is part of the "BHS". The location of the >additional headers, if they exist (in most cases they will not) is after >the >header digest. Thus additional headers can be located using validated >information. > >3. Each additional header is encoded using a standard TLV (type, length, >value) format. In TLV format the fixed size type indicator tells how to >interpret the data. The fixed size length field says how long the data is, >and the variable sized value field contains the data. If an application >doesn't understand the type, it can maintain sync. in the byte stream by >skipping to the length field and then skipping the value. > >4. The complete set of TLV values are covered by a distinct digest. This >digest is considered a part of the header digest in terms of iSCSI option >negotiation but is physically distinct. Although this requires another >digest calculation its usage is expected to be fairly low and it is most >likely calculated over a small number of bytes. > >5. The data and data digest follow the TLV digest. > >In terms of layout, what you might have is the following: > >================= Start of iSCSI PDU ============================= > > > > This portion of the PDU contains the information in > the "BHS" plus: > > 1. The length field (4 bytes) > 2. The length of the TLV information > >--------------- End of the fixed size header --------------------- >--------------- Start of the header digest (optional) ------------ > >--------------- End of the header digest ------------------------- >--------------- Start of TLV information (optional) -------------- > > (Additional header fields go here in a variable length space) > > >--------------- End of TLV information --------------------------- >------ Start of "TLV header" digest (if header digest present)---- > >--------------- End of "TLV header" digest ----------------------- >--------------- Start of data ------------------------------------ > > > > >--------------- End of data -------------------------------------- >--------------- Start of data digest (optional) ------------------ > >=============== End of data digest and PDU ======================= > >[David B: This looks better in a .pdf....are we allowed to use pdfs on this >reflector?] > >I believe this to be a simple yet efficient structure for common usage. >The optional headers require more involved processing, but allow >incremental >advancement of the TCP byte stream and avoids the loss of synchronization >that might take place in other formats. >The header can be verified, before being used. > >[Note 1: This PDU format does not attempt to address the problem of how to >recover when there is a digest error. I believe this to be a distinct >problem, though the PDU format could provide some additional level of >robustness via redundancy of information at known locations.] > >[Note 2: Julian, If more detailed information on the PDU format is needed I >would be glad to present it. However I think that most of the significant >issues are addressed by this outline. The other items don't appear, to me, >to be as significant.] > >Barry Reinhold >Barry.Reinhold@trebia.com >603-659-0885 > > > > From owner-ips@ece.cmu.edu Sat Mar 10 17:48:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA25220 for ; Sat, 10 Mar 2001 17:48:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2ALLgs08703 for ips-outgoing; Sat, 10 Mar 2001 16:21:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com (mxic1.isus.emc.com [168.159.211.82]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2ALLTr08692 for ; Sat, 10 Mar 2001 16:21:30 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Sat, 10 Mar 2001 16:22:45 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0708015271@corpmx9.isus.emc.com> From: Black_David@emc.com To: cmonia@NishanSystems.com, Black_David@emc.com Cc: ips@ece.cmu.edu Subject: RE: TCP/IP Framing Assists Date: Sat, 10 Mar 2001 16:21:21 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Copying the list on a response to a question of general interest ... Charles Monia writes: > At the last IPS WG, you said you felt there was a possibility that the IETF > community might be persuaded to accept some sort of framing assists for > TCP/IP (At the time, I characterized this as 'SCTP lite'). > > Has there been any progress on this front? This could impact the outcome of > the common encapsulation effort. It is still a definite possibility. There has been progress, but it's not exactly visible, yet. As indicated in the IPS Agenda for Minneapolis, work on this will be conducted through the tsvwg Working Group - I have not seen a Minneapolis Agenda for tsvwg, so I don't know if/when this issue will be taken up. I'll try to get an update (no discussion) on this topic included in the initial 10 minutes of Agenda Bashing and Administrivia. For the FCIP/iFCP common encapsulation, I would recommend separating framing from data representation, and adopting an approach along the lines of that in Section 1.2.8 of the iSCSI draft in which framing logically occurs between the block storage protocol and TCP. In particular, this means that the 0xfcfcfcfc reserved value/word-stuffing approach in Section 2 of the weber draft should be specified separately as a "shim" protocol at this level if it is to be carried forward - this is the right way to think about it in any case because any protocol headers at higher layers in the stack have to be word-stuffed if the reserved value sequence (4 occurrences of the reserved value in the weber draft) occurs there. The result could be three TCP framing approaches - SCTP-lite - Markers (iSCSI appendix) - Reserved value/word stuffing (weber draft) Each could be independently chosen/used by any of the IPS block storage protocols, provided that suitable negotiation mechanisms are specified to make sure that both sides of a connection understand exactly what is being used in which direction. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Sat Mar 10 19:24:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA26926 for ; Sat, 10 Mar 2001 19:23:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2AMOhh12082 for ips-outgoing; Sat, 10 Mar 2001 17:24:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dirty.research.bell-labs.com (dirty.research.bell-labs.com [204.178.16.6]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2AMO2r12059 for ; Sat, 10 Mar 2001 17:24:02 -0500 (EST) Received: from scummy.research.bell-labs.com ([135.104.2.10]) by dirty; Sat Mar 10 17:23:55 EST 2001 Received: from aura.research.bell-labs.com ([135.104.46.10]) by scummy; Sat Mar 10 17:23:54 EST 2001 Received: (from sandeepj@localhost) by aura.research.bell-labs.com (8.9.1/8.9.1) id RAA03513 for ips@ece.cmu.edu; Sat, 10 Mar 2001 17:23:54 -0500 (EST) Date: Sat, 10 Mar 2001 17:23:54 -0500 (EST) Message-Id: <200103102223.RAA03513@aura.research.bell-labs.com> From: sandeepj@research.bell-labs.com (Sandeep Joshi) To: ips@ece.cmu.edu Subject: comments on naming&discovery draft Sender: owner-ips@ece.cmu.edu Precedence: bulk 1) Section 4.2 last line before Section 4.2.1 "target MUST send any iSCSI-level async on this session, allowing the initiator to discover new targets.." The session mentioned here is a session to the canonical target. However, the iSCSI 05 draft does not mention any such condition in Sec 2.18 on Async Message. In there, a SCSI event (note: not iSCSI) is used to notify availability of new targets. 2) Appendix C, Section 5 "Stateful Inspection Firewall" It contains statements of the sort "I dont expect/think.." These statements could be rephrased to be impersonal. 3) Section 4.3 Middle of page 17 (After reference to RFC 2608) -> "A target can register either its canonical target, ..." Multiple references to term "target" is confusing. It can be changed to refer to the term "Network entity" introduced earlier in the document. -> "A network entity can register its canonical target,.." 4) Sec 4.2 SendTargets Command The status code 0x42 mentioned is now 0x02 in iSCSI document. A similar mismatch exists in some codes listed in B.4.5. It might help to just use error names and not numbers to avoid this problem. [readers can do error code "discovery"! ] -Sandeep From owner-ips@ece.cmu.edu Sat Mar 10 19:25:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA26925 for ; Sat, 10 Mar 2001 19:23:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2AMojU13422 for ips-outgoing; Sat, 10 Mar 2001 17:50:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dirty.research.bell-labs.com (dirty.research.bell-labs.com [204.178.16.6]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2AMo6r13383 for ; Sat, 10 Mar 2001 17:50:07 -0500 (EST) Received: from grubby.research.bell-labs.com ([135.104.2.9]) by dirty; Sat Mar 10 17:48:10 EST 2001 Received: from aura.research.bell-labs.com ([135.104.46.10]) by grubby; Sat Mar 10 17:48:09 EST 2001 Received: from research.bell-labs.com (IDENT:sandeepj@sandeepj-pcmh.research.bell-labs.com [135.104.47.90]) by aura.research.bell-labs.com (8.9.1/8.9.1) with ESMTP id RAA03869 for ; Sat, 10 Mar 2001 17:48:09 -0500 (EST) Message-ID: <3AAAAF29.8D91CF@research.bell-labs.com> Date: Sat, 10 Mar 2001 17:48:09 -0500 From: Sandeep Joshi X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.2.16-3 i686) X-Accept-Language: en MIME-Version: 1.0 To: "'ips@ece.cmu.edu'" Subject: Re: iSNS comments References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi Kevin, Thanks for the clarifications. Some more questions. What scenarios require changes in ESI IP address/port ? I would have assumed one wouldnt want to change these things but just keep them read-only. I can see DHCP being one reason for IP address changes, but in that case, question (2) below should result in something more akin to a new entity insert (NOT an update). The other reason could be a multi-NIC device where one interface goes down and one wants to switch the ESI/mgmt address to be the other interface. Anything else? Changing the ESI portnumber could be a headache if the iSNSdb is trashed, since one would then have to resort to port scans to find network entities. (or SLP/SNMP could help but then there is no way to "manually" find your way around). On a (port) related note, is SCN going to be a unicast service or can multicast(reliable/unreliable) also be defined for use. It might seem like all clients would be interested in generic events like "network changed" and such events might get generated more often than anticipated. One other comment. I noticed that iSNS will generate a new DDid for an iSNS client which does not supply one(Sec 6.6) This seems to conflict with soft-zoning since initiators can only find targets in the same DD. Am I interpreting this correctly? Perhaps the default DDid should be a wildcard of some sort (subnet or all within same domain) -Sandeep Kevin Gibbons wrote: > > Sandeep, > thanks for the iSNS feedback. Please see responses below. > Regards, Kevin > > -----Original Message----- > From: sandeepj@research.bell-labs.com > [mailto:sandeepj@research.bell-labs.com] > Sent: Thursday, March 08, 2001 7:35 AM > To: ips@ece.cmu.edu > Subject: Re: iSNS comments > > > some comments on the new iSNS draft > > > > 1) it lacks a Table of contents, unlike iSCSI. please add one. > > (given that, i am liable to have missed some section which > > might have answered the questions raised below!) > > kg> we can add a TOC to the next update of the document. > > > > > 2) What happens if an iSNS client tries to update its > > TCP/UDP port or IP address (for Portal/ESI/etc) ? > > > > So the entry exists, and now the client sends a RegDevAttr > > request with the update flag set for changing the above. > > > > If the port(s) is going to be well-known, the questions below > > may not arise. If not,... > > > > a) Will the old TCP connection be broken by the iSNS server ? > > kg> Who breaks the former connection first is implementation > dependent. However, the iSNS server initiates the TCP > connection to the new port for the next ESI message. The ip > address and port used for ESI messages does not have to be the > same as portal addresses that an entity has registered. > > > b) Would the RegDevResponse be sent to the old/new port ? > > kg> as described in section 7.2, the response will be sent > to the same IP Address and Port that the registration > message originated from. > > > c) Who initiates the new connection (client or server) ? > > kg> the iSNS server will initiate the connection for the next > ESI. There was concern during the design that the server, > rather then the client, be in control of the ESI period and > connection process to manage the ESI handling overhead. > > > d) How would the client know the request succeeded ? > > > i see what i missed here.. SLP, which is orthogonally managing > > the "control" plane. the questions above dont arise. > > kg> The client will know the request succeeded from the > RegDevResponse message status field, sent by the iSNS is > response to the request. > > > > > 3) Is there a requirement to provide (keyword=MUST) a secondary > > iSNS server ? If I am not mistaken, DNS does mandate a > > secondary server for every zone to avoid single-pt-of-failure. > > > > kg> There was concern that the design not mandate a requirement > for a secondary server for situations where High Availability > is not a concern. If desired, additional text can be added to > recommend secondary server in HA configurations. > > > > > -Sandeep From owner-ips@ece.cmu.edu Sat Mar 10 22:18:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA00648 for ; Sat, 10 Mar 2001 22:18:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2B1UmS21701 for ips-outgoing; Sat, 10 Mar 2001 20:30:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2B1Tvr21660 for ; Sat, 10 Mar 2001 20:29:57 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id UAA41892; Sat, 10 Mar 2001 20:29:59 -0500 Received: from f4n49e (d03nm065h.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id SAA295124; Sat, 10 Mar 2001 18:28:39 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: Group Test Period for iSCSI interoperability - call for interest To: "Barry Reinhold - Lamprey POP3" Cc: "ISCSI" X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Sat, 10 Mar 2001 17:28:22 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/10/2001 06:28:38 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Barry, This exact thing is will be setting up with the SNIA IP Storage Forum (iSCSI Subgroup) and the SNIA IP Storage Working Group. It will be using the SNIA interoperability Lab in Colo. Springs. It is not clear that we need still another one, and if you have not yet become a part of SNIA and this forum you might want to do that. SNIA can keep those types of things and issues off the IETF Reflectors, and keep us all from being abused by the Gods of IETF :-) . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com "Barry Reinhold - Lamprey POP3" @ece.cmu.edu on 03/08/2001 06:40:08 AM Sent by: owner-ips@ece.cmu.edu To: "ISCSI" cc: Subject: Group Test Period for iSCSI interoperability - call for interest To anyone who has an interest: A number of people have expressed some interest in getting together in a technical debugging environment to try and see what types of issues come up with early iSCSI prototypes. This is usually a great way to see how different people have read and understood a draft, and is also a big help in identifying interoperability issues before they get ingrained into implementatons. So, as a "toe in the water" step, I would like to propose a group test period, as outlined below, and get some feedback as to who would like to participate in such an event. Group Test Period Overview: A three day test period that would run Tuesday - Thursday. Monday and Friday would be open for people to make adhoc arrangements if they so desired. Very informal "pair up with someone else and try" process for the first two days followed by an attempt at putting together groups of devices on the third day. There would be no real expectation that anything has to "work right" and people could drop out, fix things, and rejoin on the first two days. This type of early testing is most effective when development engineers attend and bring the toys they need to fix bugs. The goal would be to work based on the current draft but implementations that were at previous draft levels could test against other implentations that were at that level. ISCSI would be the focus but if there are groups of people who want to get together and do other storage over IP protocols, and there is someone willing to coordinate part of the effort, it should be easy to accomodate. Time wise -- mid April (17-19)? This would be turned to accomodate the interested parties. A fee would be charged to cover whatever costs we incurred for food and facilities. That being said, if your organization has an interest, send me an email or post to the reflector. If there are six + implementations ready to try we'll give it a go as an "event". If there is a larger group (15 + players) the date may have to be pushed back due to organizational issues. The following information would be helpful: 1. Date desired 2. Protocol(s) you want to do 3. Roles your stuff can play (iSCSI initiator, target, bridge, gateway) 4. Draft level you want to test at 5. Contact email and phone. Barry Reinhold 603-868-5144 From owner-ips@ece.cmu.edu Sat Mar 10 22:18:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA00671 for ; Sat, 10 Mar 2001 22:18:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2B1Uqj21724 for ips-outgoing; Sat, 10 Mar 2001 20:30:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.bld.us.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2B1UPr21675 for ; Sat, 10 Mar 2001 20:30:25 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id TAA88614; Sat, 10 Mar 2001 19:24:32 -0600 Received: from f4n49e (d03nm065h.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id SAA201170; Sat, 10 Mar 2001 18:29:07 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: Group Test Period for iSCSI interoperability - call for interest To: "Barry Reinhold - Lamprey POP3" Cc: "ISCSI" X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Sat, 10 Mar 2001 17:28:46 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/10/2001 06:29:06 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Barry, This exact thing is will be setting up with the SNIA IP Storage Forum (iSCSI Subgroup) and the SNIA IP Storage Working Group. It will be using the SNIA interoperability Lab in Colo. Springs. It is not clear that we need still another one, and if you have not yet become a part of SNIA and this forum you might want to do that. SNIA can keep those types of things and issues off the IETF Reflectors, and keep us all from being abused by the Gods of IETF :-) . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com "Barry Reinhold - Lamprey POP3" @ece.cmu.edu on 03/08/2001 06:40:08 AM Sent by: owner-ips@ece.cmu.edu To: "ISCSI" cc: Subject: Group Test Period for iSCSI interoperability - call for interest To anyone who has an interest: A number of people have expressed some interest in getting together in a technical debugging environment to try and see what types of issues come up with early iSCSI prototypes. This is usually a great way to see how different people have read and understood a draft, and is also a big help in identifying interoperability issues before they get ingrained into implementatons. So, as a "toe in the water" step, I would like to propose a group test period, as outlined below, and get some feedback as to who would like to participate in such an event. Group Test Period Overview: A three day test period that would run Tuesday - Thursday. Monday and Friday would be open for people to make adhoc arrangements if they so desired. Very informal "pair up with someone else and try" process for the first two days followed by an attempt at putting together groups of devices on the third day. There would be no real expectation that anything has to "work right" and people could drop out, fix things, and rejoin on the first two days. This type of early testing is most effective when development engineers attend and bring the toys they need to fix bugs. The goal would be to work based on the current draft but implementations that were at previous draft levels could test against other implentations that were at that level. ISCSI would be the focus but if there are groups of people who want to get together and do other storage over IP protocols, and there is someone willing to coordinate part of the effort, it should be easy to accomodate. Time wise -- mid April (17-19)? This would be turned to accomodate the interested parties. A fee would be charged to cover whatever costs we incurred for food and facilities. That being said, if your organization has an interest, send me an email or post to the reflector. If there are six + implementations ready to try we'll give it a go as an "event". If there is a larger group (15 + players) the date may have to be pushed back due to organizational issues. The following information would be helpful: 1. Date desired 2. Protocol(s) you want to do 3. Roles your stuff can play (iSCSI initiator, target, bridge, gateway) 4. Draft level you want to test at 5. Contact email and phone. Barry Reinhold 603-868-5144 From owner-ips@ece.cmu.edu Sun Mar 11 00:53:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA04328 for ; Sun, 11 Mar 2001 00:52:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2B3aog28159 for ips-outgoing; Sat, 10 Mar 2001 22:36:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2B3Zor28083 for ; Sat, 10 Mar 2001 22:35:51 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Sat, 10 Mar 2001 19:35:43 -0800 Message-ID: From: Joshua Tseng To: Sandeep Joshi , "'ips@ece.cmu.edu'" Subject: RE: iSNS comments Date: Sat, 10 Mar 2001 19:35:37 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Sandeep, Thanks for your helpful comments. > > > Hi Kevin, > > Thanks for the clarifications. Some more questions. > > What scenarios require changes in ESI IP address/port ? > I would have assumed one wouldnt want to change these > things but just keep them read-only. The iSNS can send ESI messages to either the Portal IP address or the Management IP address. The ESI TCP/UDP Port attribute is merely there to allow the client to specify to the server that it wants to receive ESI messages on a different port, if that is what it wants. > > I can see DHCP being one reason for IP address changes, but > in that case, question (2) below should result in something > more akin to a new entity insert (NOT an update). The other > reason could be a multi-NIC device where one interface goes > down and one wants to switch the ESI/mgmt address to be the > other interface. Anything else? A new entity is akin to a new iSCSI instance or device being plugged into the network. An expiring DHCP lease resulting in a new IP address being assigned to a portal would be an example of an update or a change to an existing portal IP address. A new NIC being hot-plugged into an existing operational iSCSI device would be an example of a new portal added to an existing entity. > > Changing the ESI portnumber could be a headache if the iSNSdb > is trashed, since one would then have to resort to port scans > to find network entities. (or SLP/SNMP could help but then > there is no way to "manually" find your way around). If the iSNS database is trashed, then the ESI would be the least of your worries. You are right that SLP and SNMP would be alternative ways of finding your way around. > > On a (port) related note, is SCN going to be a unicast service > or can multicast(reliable/unreliable) also be defined for use. > It might seem like all clients would be interested in generic > events like "network changed" and such events might get generated > more often than anticipated. Currently, SCN is specified as a unicast service. However, I could imagine using vendor-defined attributes to store a multicast group for each Discovery Domain, and then multicasting SCN's. iSNS is structured to provide flexibility depending on the needs of the implementer. > > One other comment. I noticed that iSNS will generate a new DDid > for an iSNS client which does not supply one(Sec 6.6) This seems > to conflict with soft-zoning since initiators can only find > targets in the same DD. Am I interpreting this correctly? > Perhaps the default DDid should be a wildcard of some sort > (subnet or all within same domain) Section 6.6 is left over from a previous version of iSNS, and will be updated in the next revision--thanks. Section 3.2.2 describes how it is up to an implementer to decide whether a new device that hasn't been explicitly registered into a DD should be put into a "default DD", or remain without a DD. If a device is not a member of a DD, then it shall be inaccessible. It is up to the implementer to decide whether to put new devices with no DD into the "default DD". Regards, Josh > > -Sandeep > > Kevin Gibbons wrote: > > > > Sandeep, > > thanks for the iSNS feedback. Please see responses below. > > Regards, Kevin > > > > -----Original Message----- > > From: sandeepj@research.bell-labs.com > > [mailto:sandeepj@research.bell-labs.com] > > Sent: Thursday, March 08, 2001 7:35 AM > > To: ips@ece.cmu.edu > > Subject: Re: iSNS comments > > > > > some comments on the new iSNS draft > > > > > > 1) it lacks a Table of contents, unlike iSCSI. please add one. > > > (given that, i am liable to have missed some section which > > > might have answered the questions raised below!) > > > > kg> we can add a TOC to the next update of the document. > > > > > > > > 2) What happens if an iSNS client tries to update its > > > TCP/UDP port or IP address (for Portal/ESI/etc) ? > > > > > > So the entry exists, and now the client sends a RegDevAttr > > > request with the update flag set for changing the above. > > > > > > If the port(s) is going to be well-known, the questions below > > > may not arise. If not,... > > > > > > a) Will the old TCP connection be broken by the iSNS server ? > > > > kg> Who breaks the former connection first is implementation > > dependent. However, the iSNS server initiates the TCP > > connection to the new port for the next ESI message. The ip > > address and port used for ESI messages does not have to be the > > same as portal addresses that an entity has registered. > > > > > b) Would the RegDevResponse be sent to the old/new port ? > > > > kg> as described in section 7.2, the response will be sent > > to the same IP Address and Port that the registration > > message originated from. > > > > > c) Who initiates the new connection (client or server) ? > > > > kg> the iSNS server will initiate the connection for the next > > ESI. There was concern during the design that the server, > > rather then the client, be in control of the ESI period and > > connection process to manage the ESI handling overhead. > > > > > d) How would the client know the request succeeded ? > > > > > i see what i missed here.. SLP, which is orthogonally managing > > > the "control" plane. the questions above dont arise. > > > > kg> The client will know the request succeeded from the > > RegDevResponse message status field, sent by the iSNS is > > response to the request. > > > > > > > > 3) Is there a requirement to provide (keyword=MUST) a secondary > > > iSNS server ? If I am not mistaken, DNS does mandate a > > > secondary server for every zone to avoid single-pt-of-failure. > > > > > > > kg> There was concern that the design not mandate a requirement > > for a secondary server for situations where High Availability > > is not a concern. If desired, additional text can be added to > > recommend secondary server in HA configurations. > > > > > > > > -Sandeep > From owner-ips@ece.cmu.edu Sun Mar 11 04:11:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA07766 for ; Sun, 11 Mar 2001 04:11:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2B7uv211469 for ips-outgoing; Sun, 11 Mar 2001 02:56:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2B7uLr11418 for ; Sun, 11 Mar 2001 02:56:21 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 8E6C267C for ; Sun, 11 Mar 2001 00:56:20 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.cos.agilent.com (Postfix) with ESMTP id 1044C1C for ; Sun, 11 Mar 2001 00:56:20 -0700 (MST) Received: from agilent.com (cos1nai129020.cos.agilent.com [141.184.129.20]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id XAA24039 for ; Sat, 10 Mar 2001 23:55:04 -0800 (PST) Message-ID: <3AAB2F51.E7F4EE8F@agilent.com> Date: Sat, 10 Mar 2001 23:54:57 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iscsi: editorial comments to iSCSI-05 References: <0F31E5C394DAD311B60C00E029101A0708015268@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I understand this, but it's not consistent. For example, on page 15: "data and the status for a command must be sent over the same TCP connection that was used to deliver the SCSI command." and in the very same paragraph, "the target MUST return R2T, if any, and the status over the same TCP connection" These are requirements, but must is sometimes capitalized, sometimes not. And on page 56: "The target MUST not set the Status to 0x'0000'", the must is capitalized, but the not is not. -Matt Black_David@emc.com wrote: > > > Global: "MUST", "SHOULD", etc are sometimes all upper case, > > sometimes lowercase. Please make them consistent. > > Actually, that's deliberate. Upper case versions of these > and other terms express requirements, recommendations, and > options for interoperability and the like and are defined > in RFC 2119 (see statement on p.3 of the iSCSI draft). Lower > case versions of these terms may be used, but do not have > the RFC 2119 meanings and implications of the upper case terms. > The decision to use upper case vs. lower case is a deliberate > decision about whether this is an interoperability issue or > one of similar import/impact - in particular RFC 2119 advises > that these upper case terms "be used with care and sparingly". > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- From owner-ips@ece.cmu.edu Sun Mar 11 04:12:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA07801 for ; Sun, 11 Mar 2001 04:12:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2B7SuT10065 for ips-outgoing; Sun, 11 Mar 2001 02:28:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2B7Sbr10054 for ; Sun, 11 Mar 2001 02:28:37 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA94250 for ; Sun, 11 Mar 2001 08:28:30 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA75096 for ; Sun, 11 Mar 2001 08:27:25 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0C.0028DFB7 ; Sun, 11 Mar 2001 08:26:27 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 11 Mar 2001 09:28:35 +0200 Subject: RE: iSCSI: Towards a more effective PDU format Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Barry, If you have no idea what follows BHS the you have to start by reading the TLV first word or equivalent to. The rest folllows. What we did with the current scheme is that we cut a byte from the length (that was meant for data) and made out of it a descriptor of the first TLV and the length the length part of it. From here whatever follows BHS is known after BHS processing. In most case it is data - so that it is really simple - you read 48 bytes and you are done (your fast path is really that fast and simple). I am not sure that 48 bytes is that long and convoluted. The digest is where some trouble arises. If we use a single CRC/digest we have to keep reading while we don't know yet if the data is good or bad. The conventional TLV schemes have the TL at the beginnings of the block and you read them then the Value. Here again you use TL before knowing if it is good. All the speculation about good or bad is however practically solve in several attempts (you will get to a block with a good CRC (resynch). I suggested also adding some parity to the TL part to increase confidence in it. Regards, Julo Please respond to "Barry Reinhold" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI: Towards a more effective PDU format Julian, On the two drawbacks: 1. Agreed, there are two header digests and this isn't desirable. However, I suspect the TLVs will seldom be used so at least it is out of the fast path. [BTW - Can some digest expert suggest how bad this is vs. a single longer digest?] 2. I'm not sure I fully understand what you are asking under two, so I'm going to answer the question of "How would you read the TLV w/o using any unverified info." If this is an answer to the wrong question, my apologies. You'll have to ask again. The TLV length field is obtained from the BHS header as, or after, it has been verified. This TLV length field gives the size of the TLV data (surprise) that has to be buffered in reading the TCP stream to get to the TLV digest. The TLV digest is checked against the TLV data. The TLV's are then parsed. (Humm - I guess I have an assumption in here that TLV data is not *really* big, such that buffering the whole set of TLVs is a reasonable thing to do) I don't see where one ends up using unverified fields. I would expect to read a fixed N bytes from the TCP stream (through the header digest), verify the header, use the fixed offset of the "TLV length field" that is in the header, which has been verified, to find if I had TLV fields. If no, I'm on to the data. If yes I read the TCP stream with the number of bytes given by "TLV length field" + sizeof(TLV digest), verify the TLV fields, and then parse the TLVs as needed. A plus is that there is very little work to do for the fast path. Another plus is that the TLV mechanism allows vendor options, and/or future extensions with well understood backward compatibility behavior. A third plus is that there should be no special cases relative to how the PDU format has you read the TCP stream...read header, optionaly read TLV stuff, read data. This should be friendly to hardware accelerated stuff. A negative is that TLV parsing is sorta slow if you have complicated TLV structures such as in DOCSIS. Our TLVs are very simple....so far at least. >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >julian_satran@il.ibm.com >Sent: Saturday, March 10, 2001 12:17 PM >To: ips@ece.cmu.edu >Subject: Re: iSCSI: Towards a more effective PDU format > > > > >Barry, > >Thanks for outline. >I will consider it as one of the variants I am going through to asses >clarity. > >There are two things that are against it: > >1- several digests >2- if I don't gather (how?) from BHS that TLVs are there then I have to do >2 reads for each additional segment - and besides this you are using a >field before knowing it is correct. > >We could however rearrange in any fashion the BHS and NWs to make it look >very close to what you propose and still have one read per segment and data >used only after checking. > >Regards, >Julo > >"Barry Reinhold - Lamprey POP3" on 10/03/2001 >04:02:28 > >Please respond to "Barry Reinhold - Lamprey POP3" > >To: "ISCSI" >cc: "Robert Russell" , "Ashish A. Palekar" > , "Jon Sreekanth" , > "James Smart" >Subject: iSCSI: Towards a more effective PDU format > > > > >All, > >I know there are well defined feelings about how a PDU should be >structured, >and not everyone is going to be in the same camp on this issue. However, I >am concerned that the current PDU format with the WN field has not really >achieved the goals that I, for one, would like to see it achieve. So, let >me >list a few objectives for the PDU format and then suggest one possible >solution. > >Objectives for the iSCSI PDU (that could be improved upon). > >1. Simplicity - The PDU layout should be as straight forward as possible. >Interoperability starts with simplicity. When the logical function is >simple, documenting proper usage is a whole lot easier. When documentation >is clear in a standard, the time to interoperability and market acceptance >is reduced. > >2. Efficient operation in the common execution path. I think "rough >consensus" can probably be achieved in terms of what parts of an iSCSI >header are going to be needed in the normal flow and what parts are >exception cases. If we have consensus on this we should be able to agree on >where to make things easy and where to add complexity. > >3. Data integrity - If a header digest is going to be provided, the >information it covers should be checked against the digest before being >used. > >I think we can improve the current PDU format, in terms of these >objectives, >by doing the following: > >1. Providing a fixed size header that contains the most commonly used >fields. >2. Placing less commonly used fields in an optional portion of the header. >3. Providing a header digest over the fixed portion of the header and a >separate digest for the optional portion of the header. > >Basic approach: > >1. The fixed "BHS" portion of the header is in all iSCSI PDU headers. It is >always the same size and is followed by a header digest that covers the >"BHS", if header digests are being used. The fact that header digests are >being used within a TCP byte stream is not encoded into the frame, but is >obtained from state associated with the session. This allows the location >of >the header digest to be determined without using any information in the >header. Only when the integrity of the header has been validated is the >information in it used. > >2. The header contains information on the size of the complete PDU, and a >length field for any additional headers. The integrity of this information >has been validated as it is part of the "BHS". The location of the >additional headers, if they exist (in most cases they will not) is after >the >header digest. Thus additional headers can be located using validated >information. > >3. Each additional header is encoded using a standard TLV (type, length, >value) format. In TLV format the fixed size type indicator tells how to >interpret the data. The fixed size length field says how long the data is, >and the variable sized value field contains the data. If an application >doesn't understand the type, it can maintain sync. in the byte stream by >skipping to the length field and then skipping the value. > >4. The complete set of TLV values are covered by a distinct digest. This >digest is considered a part of the header digest in terms of iSCSI option >negotiation but is physically distinct. Although this requires another >digest calculation its usage is expected to be fairly low and it is most >likely calculated over a small number of bytes. > >5. The data and data digest follow the TLV digest. > >In terms of layout, what you might have is the following: > >================= Start of iSCSI PDU ============================= > > > > This portion of the PDU contains the information in > the "BHS" plus: > > 1. The length field (4 bytes) > 2. The length of the TLV information > >--------------- End of the fixed size header --------------------- >--------------- Start of the header digest (optional) ------------ > >--------------- End of the header digest ------------------------- >--------------- Start of TLV information (optional) -------------- > > (Additional header fields go here in a variable length space) > > >--------------- End of TLV information --------------------------- >------ Start of "TLV header" digest (if header digest present)---- > >--------------- End of "TLV header" digest ----------------------- >--------------- Start of data ------------------------------------ > > > > >--------------- End of data -------------------------------------- >--------------- Start of data digest (optional) ------------------ > >=============== End of data digest and PDU ======================= > >[David B: This looks better in a .pdf....are we allowed to use pdfs on this >reflector?] > >I believe this to be a simple yet efficient structure for common usage. >The optional headers require more involved processing, but allow >incremental >advancement of the TCP byte stream and avoids the loss of synchronization >that might take place in other formats. >The header can be verified, before being used. > >[Note 1: This PDU format does not attempt to address the problem of how to >recover when there is a digest error. I believe this to be a distinct >problem, though the PDU format could provide some additional level of >robustness via redundancy of information at known locations.] > >[Note 2: Julian, If more detailed information on the PDU format is needed I >would be glad to present it. However I think that most of the significant >issues are addressed by this outline. The other items don't appear, to me, >to be as significant.] > >Barry Reinhold >Barry.Reinhold@trebia.com >603-659-0885 > > > > From owner-ips@ece.cmu.edu Sun Mar 11 04:13:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA07972 for ; Sun, 11 Mar 2001 04:13:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2B8AvW12123 for ips-outgoing; Sun, 11 Mar 2001 03:10:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2B89vr12035 for ; Sun, 11 Mar 2001 03:09:57 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id CA273651; Sun, 11 Mar 2001 01:09:56 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.cos.agilent.com (Postfix) with ESMTP id 14C2041A; Sun, 11 Mar 2001 01:09:56 -0700 (MST) Received: from agilent.com (cos1nai129020.cos.agilent.com [141.184.129.20]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id AAA24098; Sun, 11 Mar 2001 00:08:41 -0800 (PST) Message-ID: <3AAB3283.28D6D4B4@agilent.com> Date: Sun, 11 Mar 2001 00:08:35 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Barry Reinhold - Lamprey POP3 , ISCSI Subject: Re: iSCSI: Towards a more effective PDU format References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Barry, The objective of the "fixed header" with all the "optional" fields in it was to improve the performance of software implementations of iSCSI. With the fixed header, software could to a single "read()" and get the whole header - everything (reads can incur context switch overhead into kernel space). Then it could to another "read()" to get the data (if any). I also don't like your suggestion of separate digests for optional headers. There is precedent in the IP header that options are covered under the same "check" as the rest of the header. As for the "whats next" headers, I understand that Julian was just trying to reduce the overhead of iSCSI on the link. I think a complex item is ok if it is clearly and simply documented (which it is currently not). -Matt Wakeley Agilent Technologies Barry Reinhold - Lamprey POP3 wrote: > > All, > > I know there are well defined feelings about how a PDU should be structured, > and not everyone is going to be in the same camp on this issue. However, I > am concerned that the current PDU format with the WN field has not really > achieved the goals that I, for one, would like to see it achieve. So, let me > list a few objectives for the PDU format and then suggest one possible > solution. > > Objectives for the iSCSI PDU (that could be improved upon). > > 1. Simplicity - The PDU layout should be as straight forward as possible. > Interoperability starts with simplicity. When the logical function is > simple, documenting proper usage is a whole lot easier. When documentation > is clear in a standard, the time to interoperability and market acceptance > is reduced. > > 2. Efficient operation in the common execution path. I think "rough > consensus" can probably be achieved in terms of what parts of an iSCSI > header are going to be needed in the normal flow and what parts are > exception cases. If we have consensus on this we should be able to agree on > where to make things easy and where to add complexity. > > 3. Data integrity - If a header digest is going to be provided, the > information it covers should be checked against the digest before being > used. > > I think we can improve the current PDU format, in terms of these objectives, > by doing the following: > > 1. Providing a fixed size header that contains the most commonly used > fields. > 2. Placing less commonly used fields in an optional portion of the header. > 3. Providing a header digest over the fixed portion of the header and a > separate digest for the optional portion of the header. > > Basic approach: > > 1. The fixed "BHS" portion of the header is in all iSCSI PDU headers. It is > always the same size and is followed by a header digest that covers the > "BHS", if header digests are being used. The fact that header digests are > being used within a TCP byte stream is not encoded into the frame, but is > obtained from state associated with the session. This allows the location of > the header digest to be determined without using any information in the > header. Only when the integrity of the header has been validated is the > information in it used. > > 2. The header contains information on the size of the complete PDU, and a > length field for any additional headers. The integrity of this information > has been validated as it is part of the "BHS". The location of the > additional headers, if they exist (in most cases they will not) is after the > header digest. Thus additional headers can be located using validated > information. > > 3. Each additional header is encoded using a standard TLV (type, length, > value) format. In TLV format the fixed size type indicator tells how to > interpret the data. The fixed size length field says how long the data is, > and the variable sized value field contains the data. If an application > doesn't understand the type, it can maintain sync. in the byte stream by > skipping to the length field and then skipping the value. > > 4. The complete set of TLV values are covered by a distinct digest. This > digest is considered a part of the header digest in terms of iSCSI option > negotiation but is physically distinct. Although this requires another > digest calculation its usage is expected to be fairly low and it is most > likely calculated over a small number of bytes. > > 5. The data and data digest follow the TLV digest. > > In terms of layout, what you might have is the following: > > ================= Start of iSCSI PDU ============================= > > This portion of the PDU contains the information in > the "BHS" plus: > > 1. The length field (4 bytes) > 2. The length of the TLV information > > --------------- End of the fixed size header --------------------- > --------------- Start of the header digest (optional) ------------ > > --------------- End of the header digest ------------------------- > --------------- Start of TLV information (optional) -------------- > > (Additional header fields go here in a variable length space) > > --------------- End of TLV information --------------------------- > ------ Start of "TLV header" digest (if header digest present)---- > > --------------- End of "TLV header" digest ----------------------- > --------------- Start of data ------------------------------------ > > --------------- End of data -------------------------------------- > --------------- Start of data digest (optional) ------------------ > > =============== End of data digest and PDU ======================= > > [David B: This looks better in a .pdf....are we allowed to use pdfs on this > reflector?] > > I believe this to be a simple yet efficient structure for common usage. > The optional headers require more involved processing, but allow incremental > advancement of the TCP byte stream and avoids the loss of synchronization > that might take place in other formats. > The header can be verified, before being used. > > [Note 1: This PDU format does not attempt to address the problem of how to > recover when there is a digest error. I believe this to be a distinct > problem, though the PDU format could provide some additional level of > robustness via redundancy of information at known locations.] > > [Note 2: Julian, If more detailed information on the PDU format is needed I > would be glad to present it. However I think that most of the significant > issues are addressed by this outline. The other items don't appear, to me, > to be as significant.] > > Barry Reinhold > Barry.Reinhold@trebia.com > 603-659-0885 From owner-ips@ece.cmu.edu Sun Mar 11 04:56:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA08880 for ; Sun, 11 Mar 2001 04:56:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2B8nwk13959 for ips-outgoing; Sun, 11 Mar 2001 03:49:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2B8nhr13949 for ; Sun, 11 Mar 2001 03:49:44 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA127722 for ; Sun, 11 Mar 2001 09:49:37 +0100 From: julian_satran@il.ibm.com Received: from d12mta05.de.ibm.com (d12mta06_cs0 [9.165.222.255]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id JAA118132 for ; Sun, 11 Mar 2001 09:47:46 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0C.00307AF8 ; Sun, 11 Mar 2001 09:49:32 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Matt Wakeley cc: ips@ece.cmu.edu Message-ID: Date: Sun, 11 Mar 2001 10:43:10 +0200 Subject: Re: iscsi: editorial comments to iSCSI-05 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, I fixed those that MUST and did not those that MUST NOT -:) Julo Matt Wakeley on 11/03/2001 09:54:57 Please respond to Matt Wakeley To: ips@ece.cmu.edu cc: Subject: Re: iscsi: editorial comments to iSCSI-05 I understand this, but it's not consistent. For example, on page 15: "data and the status for a command must be sent over the same TCP connection that was used to deliver the SCSI command." and in the very same paragraph, "the target MUST return R2T, if any, and the status over the same TCP connection" These are requirements, but must is sometimes capitalized, sometimes not. And on page 56: "The target MUST not set the Status to 0x'0000'", the must is capitalized, but the not is not. -Matt Black_David@emc.com wrote: > > > Global: "MUST", "SHOULD", etc are sometimes all upper case, > > sometimes lowercase. Please make them consistent. > > Actually, that's deliberate. Upper case versions of these > and other terms express requirements, recommendations, and > options for interoperability and the like and are defined > in RFC 2119 (see statement on p.3 of the iSCSI draft). Lower > case versions of these terms may be used, but do not have > the RFC 2119 meanings and implications of the upper case terms. > The decision to use upper case vs. lower case is a deliberate > decision about whether this is an interoperability issue or > one of similar import/impact - in particular RFC 2119 advises > that these upper case terms "be used with care and sparingly". > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- From owner-ips@ece.cmu.edu Sun Mar 11 04:56:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA08891 for ; Sun, 11 Mar 2001 04:56:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2B8Tvx12988 for ips-outgoing; Sun, 11 Mar 2001 03:29:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2B8Tnr12981 for ; Sun, 11 Mar 2001 03:29:49 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 07FBA862 for ; Sun, 11 Mar 2001 01:29:49 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.cos.agilent.com (Postfix) with ESMTP id 14D6523 for ; Sun, 11 Mar 2001 01:29:48 -0700 (MST) Received: from agilent.com (cos1nai129020.cos.agilent.com [141.184.129.20]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id AAA24137 for ; Sun, 11 Mar 2001 00:28:33 -0800 (PST) Message-ID: <3AAB372A.2FF9D3B6@agilent.com> Date: Sun, 11 Mar 2001 00:28:26 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI : SCSI Response PDU References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > > responses in text: > > Thanks, > Julo > > Santosh Rao on 10/03/2001 01:11:23 > > Please respond to Santosh Rao > > To: ips@ece.cmu.edu (ips) > cc: santoshr@hpcuhe.cup.hp.com (Santosh Rao) > Subject: iSCSI : SCSI Response PDU > > Julian/All, > > Some comments on recent changes to the SCSI Response PDU in Rev 04/05 : > > 1) Sense Length has been removed. This field is required and should be > re-introduced. (Can anyone comment on the reasons behind its removal(?) ). > > +++ sense is in the data segment - data segment length is sense length +++ I figured that out, but there is nothing in the document stating that. Since this is a public spec, it should spell everything out (otherwise you, as the author, will constantly get questions). > 2) There is a reference to Response Data [without a corresponding > response length. Once again, response length is required, if a > response data is used.]. Response Data is modelled on the FC > response data and is useless in the current context. > (The FC response data only contained a response code). > Any target that wishes to send vendor unique information > could do so through vendor unique sense data. > > All references to Response Data should be removed. > +++ Responses are not sense +++ > 3) A range of iSCSI Response codes should be assigned for vendor-unique > responses. > +++ Good Idea +++ Yeah, will contribute to interoperability... NOT! > 4) The spec should state explicitly whether implementors are to use the > REJECT PDU or a Scsi Response PDU with iSCSI Response code to indicate > failure to process a command due to invalid command pdu fields. If the > SCSI Response PDU is to be used to convey this, an additional iSCSI > response code must be added for "Invalid Command PDU". > +++ section 6.1 covers this - Matt Wakeley proposed changing the name to > protocol errors +++ > 5) In section 2.5.1, pg 40, the spec states : > "For all these functions, the SCSI Task Management Response MUST be > returned using the Initiator Task Tag to identify the operation for > which it is responding." > > The above MUST should be changed to SHOULD since a task mgmt command can be > responded to with a REJECT response on a format error. > +++ I will make " for all these functions, if executed ... +++ From owner-ips@ece.cmu.edu Sun Mar 11 04:57:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA08902 for ; Sun, 11 Mar 2001 04:57:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2B9Mxo24496 for ips-outgoing; Sun, 11 Mar 2001 04:22:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2B9MLr24477 for ; Sun, 11 Mar 2001 04:22:21 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 9A6C7644 for ; Sun, 11 Mar 2001 02:22:20 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.cos.agilent.com (Postfix) with ESMTP id 1CB269F for ; Sun, 11 Mar 2001 02:22:20 -0700 (MST) Received: from agilent.com (cos1nai129020.cos.agilent.com [141.184.129.20]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id BAA24291 for ; Sun, 11 Mar 2001 01:21:05 -0800 (PST) Message-ID: <3AAB437A.46DC020E@agilent.com> Date: Sun, 11 Mar 2001 01:20:58 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI 05.txt is out References: <3AAB4135.B38AFF41@agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt Wakeley wrote: > > Venkat Rangan wrote: > > > Page 12: > > Remove the sentence: > > "A large difference between StatSN and ExpStatSN may indicate a failed > > connection." > > > > This may have made sense when StatSN and ExpStatSN are session-wide. Now > > that it is per-connection, this may not be valid. As was pointed out in the > > "Holes in StatSN" thread, ExpStatSN may be "stuck" at a low value when a > > single PDU encountered a Digest Error, followed by several well-formed PDUs. > > So, if the connection is not frames (markers, etc), it must still be ^d > terminated in order to recover. And if it is framed, there will not be a > "stuck" low value... > > > Page 24 - Section 2.1 > > Add: The length of the PDU SHALL include any padding bytes added. > > Agreed. There is very little padding and how it is conveyed. ^ info > > > > > This raises a question as to whether it may be useful to include a two-bit > > "Fill Bytes" field in the header (BHS and AHS) which indicates the number of > > bytes that were added. Without this, it may be harder for the receiver to > > know how many bytes are part of the data. Fibre Channel has the Fill Byte > > specifier in its F_CTL part of the header. > > > > Page 25: > > This may be covered in another thread. > > > > What does the first Next-Qualifier (at byte 0) describe? Is it the BHS or > > the header following the BHS? If a PDU has a single BHS and a single AHS, is > > the sequence WN-BHS-AHS or is it WN-BHS-WN-AHS? > > (since Julian did not reply), the next qualifier always describes the header > after the following header (I know, it can be confusing - that's why all this > discussion and David Black's message on examples, more info was posted). > > > > > Page 25 and 26: > > The text next to the bit description and the headers in the text for > > sections 2.2.2.1 to 2.2.2.3 appear to be inconsistent. An example is > > "read-data transfer" vs. "Read Data" "Long Data" vs. "Long Data Transfer" > > > > Also, 2.2.2.3 appears to have text about Long Data Header that probably > > belongs in section 2.2.2.2 (or I am confused about how WN is supposed to > > work.) > > > > Page 43 > > Section 2.7 > > The PDU diagrams should not show "Digests if any..." etc. It should be > > covered by NQ and corresponding AHS. > > > > Page 44 > > Section 2.7.2 > > The requirement of having to specify a valid LUN as part of WRITE Data (and > > NOP-Out) may pose unnecessary overhead for Target processing. The fact that > > Targets are now REQUIRED to reject errant initiators who may place a LUN > > inconsistent with the one sent in the CommandPdu means that targets should > > save the LUN against each context entry for the task. This was discussed > > earlier, and I think we said that unless the folks who originally wanted it > > spoke up, we will remove it. Note that Fibre Channel does not have this > > feature, and its original purpose can be accomplished by using Target > > Transfer Tag. > > I completely agree. > > > If we still want it, targets should be given the option of negotiating this > > so that it can operate in a mode where a LUN specified as part of WRITE data > > will be ignored (as opposed to rejected). > > > > Section 2.17 > > The following text: > > "Within a connection, outstanding R2Ts MUST be fulfilled by the initiator in > > the order in which they were received." > > > > implies that R2Ts for the same WRITE command may be sent by the target on > > multiple connections? > > No. See 1.2.5: "the target MUST return R2T, if any, and the status over > the same TCP connection that was used to deliver the SCSI command." > > What this means is that if the initiator receives R2T for command 1, followed > by R2T for command 2, it must send the data for command 1 before command 2. I > don't know why this is a requirement, it seems like an unnecessary > restriction. > > > Since this is not possible, all R2Ts for a command are > > always on a single connection, so I am not sure how the above sentence > > should be interpreted. > > -Matt Wakeley > Agilent Technologies From owner-ips@ece.cmu.edu Sun Mar 11 04:57:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA08913 for ; Sun, 11 Mar 2001 04:57:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2B9D0517881 for ips-outgoing; Sun, 11 Mar 2001 04:13:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2B9Cfr16839 for ; Sun, 11 Mar 2001 04:12:41 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id B552A3F6 for ; Sun, 11 Mar 2001 02:12:40 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.cos.agilent.com (Postfix) with ESMTP id 2ABB022 for ; Sun, 11 Mar 2001 02:12:40 -0700 (MST) Received: from agilent.com (cos1nai129020.cos.agilent.com [141.184.129.20]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id BAA24254 for ; Sun, 11 Mar 2001 01:11:24 -0800 (PST) Message-ID: <3AAB4135.B38AFF41@agilent.com> Date: Sun, 11 Mar 2001 01:11:17 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI 05.txt is out References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Venkat Rangan wrote: > Page 12: > Remove the sentence: > "A large difference between StatSN and ExpStatSN may indicate a failed > connection." > > This may have made sense when StatSN and ExpStatSN are session-wide. Now > that it is per-connection, this may not be valid. As was pointed out in the > "Holes in StatSN" thread, ExpStatSN may be "stuck" at a low value when a > single PDU encountered a Digest Error, followed by several well-formed PDUs. So, if the connection is not frames (markers, etc), it must still be terminated in order to recover. And if it is framed, there will not be a "stuck" low value... > Page 24 - Section 2.1 > Add: The length of the PDU SHALL include any padding bytes added. Agreed. There is very little padding and how it is conveyed. > > This raises a question as to whether it may be useful to include a two-bit > "Fill Bytes" field in the header (BHS and AHS) which indicates the number of > bytes that were added. Without this, it may be harder for the receiver to > know how many bytes are part of the data. Fibre Channel has the Fill Byte > specifier in its F_CTL part of the header. > > Page 25: > This may be covered in another thread. > > What does the first Next-Qualifier (at byte 0) describe? Is it the BHS or > the header following the BHS? If a PDU has a single BHS and a single AHS, is > the sequence WN-BHS-AHS or is it WN-BHS-WN-AHS? (since Julian did not reply), the next qualifier always describes the header after the following header (I know, it can be confusing - that's why all this discussion and David Black's message on examples, more info was posted). > > Page 25 and 26: > The text next to the bit description and the headers in the text for > sections 2.2.2.1 to 2.2.2.3 appear to be inconsistent. An example is > "read-data transfer" vs. "Read Data" "Long Data" vs. "Long Data Transfer" > > Also, 2.2.2.3 appears to have text about Long Data Header that probably > belongs in section 2.2.2.2 (or I am confused about how WN is supposed to > work.) > > Page 43 > Section 2.7 > The PDU diagrams should not show "Digests if any..." etc. It should be > covered by NQ and corresponding AHS. > > Page 44 > Section 2.7.2 > The requirement of having to specify a valid LUN as part of WRITE Data (and > NOP-Out) may pose unnecessary overhead for Target processing. The fact that > Targets are now REQUIRED to reject errant initiators who may place a LUN > inconsistent with the one sent in the CommandPdu means that targets should > save the LUN against each context entry for the task. This was discussed > earlier, and I think we said that unless the folks who originally wanted it > spoke up, we will remove it. Note that Fibre Channel does not have this > feature, and its original purpose can be accomplished by using Target > Transfer Tag. I completely agree. > If we still want it, targets should be given the option of negotiating this > so that it can operate in a mode where a LUN specified as part of WRITE data > will be ignored (as opposed to rejected). > > Section 2.17 > The following text: > "Within a connection, outstanding R2Ts MUST be fulfilled by the initiator in > the order in which they were received." > > implies that R2Ts for the same WRITE command may be sent by the target on > multiple connections? No. See 1.2.5: "the target MUST return R2T, if any, and the status over the same TCP connection that was used to deliver the SCSI command." What this means is that if the initiator receives R2T for command 1, followed by R2T for command 2, it must send the data for command 1 before command 2. I don't know why this is a requirement, it seems like an unnecessary restriction. > Since this is not possible, all R2Ts for a command are > always on a single connection, so I am not sure how the above sentence > should be interpreted. -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Sun Mar 11 04:58:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA08929 for ; Sun, 11 Mar 2001 04:58:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2B8jw513798 for ips-outgoing; Sun, 11 Mar 2001 03:45:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2B8jjr13740 for ; Sun, 11 Mar 2001 03:45:45 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 9D95E882 for ; Sun, 11 Mar 2001 01:45:44 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.cos.agilent.com (Postfix) with ESMTP id 1FC9A22 for ; Sun, 11 Mar 2001 01:45:44 -0700 (MST) Received: from agilent.com (cos1nai129020.cos.agilent.com [141.184.129.20]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id AAA24178 for ; Sun, 11 Mar 2001 00:44:27 -0800 (PST) Message-ID: <3AAB3AE5.DFD79CF2@agilent.com> Date: Sun, 11 Mar 2001 00:44:21 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: StatSn Questions References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > >In Data > >and R2T, it just transfers the previous value (except if Data is the last > Data > >and conveys status). > > StatSN in a read data PDU is *not* the previous value. Instead, it is a > new (incremented) value and the field is valid only if the S-bit is set. > > +++ You are right - the reason I left StatSN out is that the Data PDU may > originate close to the device where you don't know about StatRN +++ I don't understand the above statement. The device is supposed to communicate through iSCSI, and iSCSI always knows what the StatSN is (not the device). Therefore, if the "S" bit is not set in a data PDU, then the StatSN conveys the current value. If "S" is set, then StatSN conveys the new value. -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Sun Mar 11 07:09:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA09327 for ; Sun, 11 Mar 2001 07:09:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2BAX0t27825 for ips-outgoing; Sun, 11 Mar 2001 05:33:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from anti_vir1.rad.co.il ([192.116.244.108]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2BAW0r27799 for ; Sun, 11 Mar 2001 05:32:01 -0500 (EST) Received: from anti_vir1.rad.co.il (localhost [127.0.0.1]) by anti_vir1.rad.co.il (8.9.3+Sun/8.9.3) with ESMTP id MAA23134 for ; Sun, 11 Mar 2001 12:30:44 +0200 (IST) Received: from radmail.rad.co.il (radmail [192.114.24.30]) by anti_vir1.rad.co.il (8.9.3+Sun/8.9.3) with ESMTP id MAA23127; Sun, 11 Mar 2001 12:30:43 +0200 (IST) Received: from yaron ([172.17.200.73]) by radmail.rad.co.il (Post.Office MTA v3.5.3 release 223 ID# 0-69802U1300L1200S0V35) with SMTP id il; Sun, 11 Mar 2001 11:34:47 +0200 Reply-To: From: "Yaron Klein" To: "'Tanjore K. Suresh'" Cc: Subject: RE: iSCSI: Naming and Discovery Draft... Date: Sun, 11 Mar 2001 10:32:09 -0000 Message-ID: <004701c0aa16$8956bf60$49c811ac@sanrad.co.il> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0) In-Reply-To: <200103081837.KAA24876@phys-ha3mpkb.Eng.Sun.COM> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Tanjore, In "Reject" I meant sending status of "Target Error", in no way it is related to the reject command (6f). Regards, Yaron Regards, -----Original Message----- From: Tanjore K. Suresh [mailto:Tanjore.Suresh@sun.com] Sent: Thursday, March 08, 2001 6:39 PM To: Tanjore.Suresh@sun.com; klein@sanrad.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI: Naming and Discovery Draft... Yaron, > Some more comments: > > The error statuses codes on Appendix B are not synchronized with the main > draft. We will fix it. > > The term "target conflict" was borrowed from HTTP. Mark clarified this > scenario well. I would like to add that this status enables better > resolution and knowledge to the target. That is, in those cases the target > can just not open the connection or just reject it like server error. I donot understand this part of your statement. " or just reject it like server error" I thought reject (0x6f) is used only when the target receives - a message with format error - a digest error. Am i missing something here or have i misunderstood this. > However, this will not give indication of the situation as described by > Mark. > I aggree with Yaron to give a better resolution error message depending on the scenerio. Today according to iSCSI revision 05 draft from Julian, it looks like two status codes are looking appropriate in the login response(0x43) 0x205 which means " Target Conflict" ==> this means target is in use by another Initiator and doesnot support Multiple initiators". 0x300 which means " Target Error" == > this means error occurred in iSCSI target( out of resources... etc). I think first one corresponds to the scenario " The Target is busy with another Initiator and cannot handle another one" The second one corresponds to " Case of simple device and target has reached the limit of its initiator capacity (out of resources)." IMO, May be splitting into two examples is better here. Thanks sureshtk > Regards, > > Yaron > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Mark > Bakke > Sent: Tuesday, March 06, 2001 6:51 PM > To: Tanjore K. Suresh > Cc: kaladhar@us.ibm.com; ips@ece.cmu.edu > Subject: Re: iSCSI: Naming and Discovery Draft... > > > Tanjore- > > Thanks for the feedback. I can comment on #3: > > "Tanjore K. Suresh" wrote: > > 3. Appendix B, B.4.5, > > Target Conflict 45 doesnot seem to be appropriate. > > > > I have not reviewed all the documents yet to give a > > recommendation and hence cannot give, but feel > > " Target Conflict" doesnot > > convey the meaning of the Scenario indicating > > case of " simple devices that can handle one device or > > the target had reached the limit of its Initiators' > capacity." > > Perhaps we chose the wrong term for this one. How about if call it > "Target Busy", and slightly re-word it? > > The target is busy with another initiator and cannot handle > another one. The initiator MAY try again later. This can be the case > for simple devices that can handle only one initiator at a time, or > for a target that has does not have the resources to support one more > initiator. In contrast to the previous examples, this rejection is > temporary. > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 > From owner-ips@ece.cmu.edu Sun Mar 11 07:09:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA09338 for ; Sun, 11 Mar 2001 07:09:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2BAP0N27391 for ips-outgoing; Sun, 11 Mar 2001 05:25:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2BAODr27370 for ; Sun, 11 Mar 2001 05:24:13 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id LAA55698 for ; Sun, 11 Mar 2001 11:24:06 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA101540 for ; Sun, 11 Mar 2001 11:23:01 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0C.0038F679 ; Sun, 11 Mar 2001 11:22:11 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 11 Mar 2001 12:24:26 +0200 Subject: RE: iSCSI 05.txt is out Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk After reading Matt's answer I realized that this probly did not get out to the list. So here it is again. Julo ---------------------- Forwarded by Julian Satran/Haifa/IBM on 11/03/2001 12:21 --------------------------- Julian Satran 06/03/2001 06:02 To: "Venkat Rangan" cc: From: Julian Satran/Haifa/IBM@IBMIL Subject: RE: iSCSI 05.txt is out (Document link: Julian Satran - Mail) Answers in text - +++ Thanks for the careful reading, Julo "Venkat Rangan" on 06/03/2001 00:40:18 Please respond to "Venkat Rangan" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI 05.txt is out Julian, Here are a few technical and editorial comments on the latest draft. Technical: Page 12: - CmdSN - the current Command Sequence Number advanced by 1 on each command shipped Add: skipping zero, which is reserved for immediate delivery. +++ will fix +++ Page 12: Remove the sentence: "A large difference between StatSN and ExpStatSN may indicate a failed connection." This may have made sense when StatSN and ExpStatSN are session-wide. Now that it is per-connection, this may not be valid. As was pointed out in the "Holes in StatSN" thread, ExpStatSN may be "stuck" at a low value when a single PDU encountered a Digest Error, followed by several well-formed PDUs. +++ if it cant't be recovered with a SACK it is stll a sign of error+++ Page 24 - Section 2.1 Add: The length of the PDU SHALL include any padding bytes added. +++ No the length does not include padding - padding is implicit +++ This raises a question as to whether it may be useful to include a two-bit "Fill Bytes" field in the header (BHS and AHS) which indicates the number of bytes that were added. Without this, it may be harder for the receiver to know how many bytes are part of the data. Fibre Channel has the Fill Byte specifier in its F_CTL part of the header. +++ No need - length does not include padding +++ Page 25: This may be covered in another thread. What does the first Next-Qualifier (at byte 0) describe? Is it the BHS or the header following the BHS? If a PDU has a single BHS and a single AHS, is the sequence WN-BHS-AHS or is it WN-BHS-WN-AHS? +++ in your example the second answer is correct and the second WN descibes the data payload +++ Page 25 and 26: The text next to the bit description and the headers in the text for sections 2.2.2.1 to 2.2.2.3 appear to be inconsistent. An example is "read-data transfer" vs. "Read Data" "Long Data" vs. "Long Data Transfer" Also, 2.2.2.3 appears to have text about Long Data Header that probably belongs in section 2.2.2.2 (or I am confused about how WN is supposed to work.) +++ will make consistent +++ Page 43 Section 2.7 The PDU diagrams should not show "Digests if any..." etc. It should be covered by NQ and corresponding AHS. +++ correct by tastes vary - and many people prefer a complete picture - I have a hard time now trying to find a way to reconcile all +++ Page 44 Section 2.7.2 The requirement of having to specify a valid LUN as part of WRITE Data (and NOP-Out) may pose unnecessary overhead for Target processing. The fact that Targets are now REQUIRED to reject errant initiators who may place a LUN inconsistent with the one sent in the CommandPdu means that targets should save the LUN against each context entry for the task. This was discussed earlier, and I think we said that unless the folks who originally wanted it spoke up, we will remove it. Note that Fibre Channel does not have this feature, and its original purpose can be accomplished by using Target Transfer Tag. If we still want it, targets should be given the option of negotiating this so that it can operate in a mode where a LUN specified as part of WRITE data will be ignored (as opposed to rejected). +++ Several people on this list reuested the freedom to build the tags at the device-server (part of the LU according to SAM) without regard to what other LUs the target has The simplest way to accomodate them is to have the initiator direct the dat through a LUN We can make checking that optional although I don't understand what is all the fuss about it+++ Section 2.17 The following text: "Within a connection, outstanding R2Ts MUST be fulfilled by the initiator in the order in which they were received." implies that R2Ts for the same WRITE command may be sent by the target on multiple connections? Since this is not possible, all R2Ts for a command are always on a single connection, so I am not sure how the above sentence should be interpreted. +++ even when received in order the initiator may decide to fulfill them out of order - this statement prevents it+++ Page 75: Add: "i.e., from Most Preferred to Least Preferred" to "reverse order of preference." +++ why? is there something ambiguous in the statement? ++ ------------ Editorial - typos and such 1. Page 11: Not covered are he means Change "he" to "the" +++ will fix ++ 2. Page 20: Change "later" to "latter" in "the later can be used in subsequent commands." +++ will fix ++ 3. Page 23: Change "isa" to "is a" +++ will fix ++ Page 42: Section 2.6.1 Change to: Initiator Task Tag of the task that was not found; used in conjuction with Response value 1. It MUST be set to zero in other cases. +++ will fix +++ Page 51: Change "CID does not change and this command is performs first" to "CID does not change and this command first performs" +++ will fix +++ Page 55: There is nothing that indicates the status codes in the table are sent as part of Status-Detail. +++ it says above the table +++ Page 66: Change "iSSCSI Data-out PDU" to "iSCSI Data (WRITE) PDU" The Data-out PDU is a new concept not introduced anywhere. +++ will fix +++ Page 75: The term "Security Context Complete" is referred to prior to its definition. Page 81: Change "later" to "latter" ++ will fix +++ ------------ Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com From owner-ips@ece.cmu.edu Sun Mar 11 08:15:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id IAA09862 for ; Sun, 11 Mar 2001 08:15:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2BBh3j01304 for ips-outgoing; Sun, 11 Mar 2001 06:43:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sphmraaa.compuserve.com (hs-img-rel-1.compuserve.com [149.174.177.156]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2BBgQr01283 for ; Sun, 11 Mar 2001 06:42:26 -0500 (EST) Received: (from mailgate@localhost) by sphmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id GAA04092 for ips@ece.cmu.edu; Sun, 11 Mar 2001 06:42:21 -0500 (EST) Received: from compuserve.com (dal-tgn-tkt-vty4.as.wcom.net [216.192.231.4]) by sphmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id GAA04034 for ; Sun, 11 Mar 2001 06:42:11 -0500 (EST) Message-ID: <3AAB656A.C1EBEFF8@compuserve.com> Date: Sun, 11 Mar 2001 05:45:46 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI: SCSI SAM-3 Model Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Last week I presented to the T10 SCSI working group the beginnings of an idea for a SCSI Architecture Model based on the I_T Nexus instead of on the SCSI Domain. The idea was well received and discussions are beginning for a SAM-3. In spite of the fact that it looks like the same protocol document (e.g., FCP-2) could fit under either the SAM-2 or the SAM-3 model, nobody thinks it would be fair to slip this kind of change in SAM-2. I've cleanup up bugs in the presentation and posted it as: ftp://ftp.t10.org/t10/document.01/01-093r1.pdf Those who think about the iSCSI model and how it fits with the SAM model may wish to examine the presentation and catch me at the Minneapolis meeting with questions. Thanks. Ralph... From owner-ips@ece.cmu.edu Sun Mar 11 08:16:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id IAA09887 for ; Sun, 11 Mar 2001 08:16:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2BBa2Z00968 for ips-outgoing; Sun, 11 Mar 2001 06:36:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2BBZAr00881 for ; Sun, 11 Mar 2001 06:35:10 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id MAA109884 for ; Sun, 11 Mar 2001 12:35:03 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id MAA109716 for ; Sun, 11 Mar 2001 12:33:58 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0C.003F7665 ; Sun, 11 Mar 2001 12:33:10 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 11 Mar 2001 13:35:24 +0200 Subject: Re: StatSn Questions Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, There are different ways to implement and use iSCSI. If the iSCSI target is only a "front" for a collection of boxes - it will be good served if the data traffic can go almost unchanged from the LU to the initiator (even True we loose a chance to update the StatSN. I am ambivalent also about R2T. Basically the same argument can be made about it and we should remove all the counters. Nevertheless let's hear what others have to say on this too. regards, Julo Matt Wakeley on 11/03/2001 10:44:21 Please respond to Matt Wakeley To: ips@ece.cmu.edu cc: Subject: Re: StatSn Questions julian_satran@il.ibm.com wrote: > >In Data > >and R2T, it just transfers the previous value (except if Data is the last > Data > >and conveys status). > > StatSN in a read data PDU is *not* the previous value. Instead, it is a > new (incremented) value and the field is valid only if the S-bit is set. > > +++ You are right - the reason I left StatSN out is that the Data PDU may > originate close to the device where you don't know about StatRN +++ I don't understand the above statement. The device is supposed to communicate through iSCSI, and iSCSI always knows what the StatSN is (not the device). Therefore, if the "S" bit is not set in a data PDU, then the StatSN conveys the current value. If "S" is set, then StatSN conveys the new value. -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Sun Mar 11 12:39:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA12448 for ; Sun, 11 Mar 2001 12:39:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2BFl8I13162 for ips-outgoing; Sun, 11 Mar 2001 10:47:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from anti_vir1.rad.co.il ([192.116.244.108]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2BFk5r13090 for ; Sun, 11 Mar 2001 10:46:06 -0500 (EST) Received: from anti_vir1.rad.co.il (localhost [127.0.0.1]) by anti_vir1.rad.co.il (8.9.3+Sun/8.9.3) with ESMTP id RAA08041 for ; Sun, 11 Mar 2001 17:44:46 +0200 (IST) Received: from radmail.rad.co.il (radmail [192.114.24.30]) by anti_vir1.rad.co.il (8.9.3+Sun/8.9.3) with ESMTP id RAA08034; Sun, 11 Mar 2001 17:44:46 +0200 (IST) Received: from yaron ([172.17.200.73]) by radmail.rad.co.il (Post.Office MTA v3.5.3 release 223 ID# 0-69802U1300L1200S0V35) with SMTP id il; Sun, 11 Mar 2001 16:48:50 +0200 Reply-To: From: "Yaron Klein" To: "'Mark Bakke'" , "'IPS'" Subject: RE: I-D ACTION:draft-bakke-iscsimib-02.txt Date: Sun, 11 Mar 2001 15:46:09 -0000 Message-ID: <000001c0aa42$65a99570$49c811ac@sanrad.co.il> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 In-Reply-To: <3AA791D6.416FFEBB@cisco.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mark, Great job with the MIB. Some questions and comments. 1. Date should be change to February 2001. 2. Can you take a student from UMN to fill the description fields of the parameters :-). 3. The StatusStatsEntry type parameters should be updated according to version 05 new status codes. 4. What does the iscsiSsnOtherHdrIntegrity means? The text format of the algorithm? 5. The iscsiSsnDataIntegrity table should include the kerberos options as well as the header digest (version 05). 6. Can the AccessList entry include IP address as well as WWUI for initial filtering? Regards, Yaron -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Mark Bakke Sent: Thursday, March 08, 2001 2:06 PM To: IPS Subject: Fwd: I-D ACTION:draft-bakke-iscsimib-02.txt FYI. For a pictorial view of this MIB, please see http://www.haifa.il.ibm.com/satran/ips/iscsi-mib-structure-02.pdf -- Mark > A New Internet-Draft is available from the on-line Internet-Drafts directories. > > > Title : Definitions of Managed Objects for SCSI over TCP > Author(s) : M. Bakke, J. Muchow, M. Krueger > Filename : draft-bakke-iscsimib-02.txt > Pages : 141 > Date : 06-Mar-01 > > This memo defines a portion of the Management Information Base (MIB) > for use with network management protocols in TCP/IP based internets. > In particular it defines objects for managing a client using the > iSCSI (SCSI over TCP) protocol. It is meant to match the latest > version of iSCSI defined in [ISCSI]. > > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-bakke-iscsimib-02.txt > > Internet-Drafts are also available by anonymous FTP. Login with the username > "anonymous" and a password of your e-mail address. After logging in, > type "cd internet-drafts" and then > "get draft-bakke-iscsimib-02.txt". > > A list of Internet-Drafts directories can be found in > http://www.ietf.org/shadow.html > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > > > Internet-Drafts can also be obtained by e-mail. > > Send a message to: > mailserv@ietf.org. > In the body type: > "FILE /internet-drafts/draft-bakke-iscsimib-02.txt". > > NOTE: The mail server at ietf.org can return the document in > MIME-encoded form by using the "mpack" utility. To use this > feature, insert the command "ENCODING mime" before the "FILE" > command. To decode the response(s), you will need "munpack" or > a MIME-compliant mail reader. Different MIME-compliant mail readers > exhibit different behavior, especially when dealing with > "multipart" MIME messages (i.e. documents which have been split > up into multiple messages), so check your local documentation on > how to manipulate these messages. > > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Sun Mar 11 19:31:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA15961 for ; Sun, 11 Mar 2001 19:31:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2BMMI104312 for ips-outgoing; Sun, 11 Mar 2001 17:22:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dirty.research.bell-labs.com (dirty.research.bell-labs.com [204.178.16.6]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2BMM2r04298 for ; Sun, 11 Mar 2001 17:22:02 -0500 (EST) Received: from scummy.research.bell-labs.com ([135.104.2.10]) by dirty; Sun Mar 11 17:20:05 EST 2001 Received: from aura.research.bell-labs.com ([135.104.46.10]) by scummy; Sun Mar 11 17:20:05 EST 2001 Received: (from sandeepj@localhost) by aura.research.bell-labs.com (8.9.1/8.9.1) id RAA21723 for ips@ece.cmu.edu; Sun, 11 Mar 2001 17:20:04 -0500 (EST) Date: Sun, 11 Mar 2001 17:20:04 -0500 (EST) Message-Id: <200103112220.RAA21723@aura.research.bell-labs.com> From: sandeepj@research.bell-labs.com (Sandeep Joshi) To: ips@ece.cmu.edu Subject: requirements draft (asym/sym reference) Sender: owner-ips@ece.cmu.edu Precedence: bulk The following para in the iSCSI requirements needs to be updated. Last paragraph of Section 2.4 beginning with -> "The question of asym/sym connections is yet to be resolved.." I was wondering if instead, it can be replaced with some sort of design rationale describing the issues discussed in each case (asym,sym) and what determined the final choice. This might be useful when considering a move to future TCP-friendly protocols (SCTP, TFRC, etc). It may also help people who missed the excitement :-) A design-space exploration document is hard to glean from the email archives. Thanks, -Sandeep From owner-ips@ece.cmu.edu Mon Mar 12 11:41:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA13666 for ; Mon, 12 Mar 2001 11:41:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2CE0kN02234 for ips-outgoing; Mon, 12 Mar 2001 09:00:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sandmail.sandburst.com (sandburst-gw.bstn-gw02.ma.us.intelilink.net [216.57.129.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2CE0Cr02195 for ; Mon, 12 Mar 2001 09:00:12 -0500 (EST) Received: from cs.uchicago.edu (dynamite-38.sandburst.com [172.16.5.38]) by sandmail.sandburst.com (Postfix) with ESMTP id F1CE094006 for ; Mon, 12 Mar 2001 08:59:52 -0500 (EST) To: ips@ece.cmu.edu Subject: Re: FCIP iFCP encapsulation proposal In-Reply-To: Message from Charles Monia of "Fri, 09 Mar 2001 17:57:04 PST." References: Date: Mon, 12 Mar 2001 08:58:53 -0500 From: Stephen Bailey Message-Id: <20010312135952.F1CE094006@sandmail.sandburst.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk > I can envision a case where the user embeds a sequence of several > well-formed PDUs in such a payload to handle the case where > resynchronization requires the detection of more than one good > encapsulation. Precisely. In fact, in fact, I just realized (once again), that I'm an idiot---the chance of user getting control with a repeating set of valid PDUs is actually MUCH higher than 1 in # of words in the PDU. If the TCP segment begins with the start of a PDU, the user will not get control. If the TCP segment begins at any other point, the scan will proceed (failing to resynch) until it reaches the beginning of one of the user's spoofing PDUs, at which point resynching will succeed (erroneously), and the user will have control, at least until the end of the current, actual PDU. In other words, the chances are remote that a stream of PDU patterns in the user data WON'T cause resynchronization to occur erroneously. As to why would this happen---to watch it burn. Why not? Script Kiddies abound. Personally, if I were a 12 year old, hacking on my school's shared 11/34 running RSTS/e, I'd blindly stumble around trying different SYS(xxx) calls too. Steph From owner-ips@ece.cmu.edu Mon Mar 12 11:50:26 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA13796 for ; Mon, 12 Mar 2001 11:50:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2CFCcH06942 for ips-outgoing; Mon, 12 Mar 2001 10:12:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2CFC8r06911 for ; Mon, 12 Mar 2001 10:12:08 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id KAA05156 for ; Mon, 12 Mar 2001 10:11:58 -0500 (EST) Message-ID: <3AACE746.583E0B1B@cisco.com> Date: Mon, 12 Mar 2001 09:12:06 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: IPS Subject: iSCSI MIB Object Model Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit iSCSI MIB enthusiasts- While building the iSCSI MIB table structure, we created a UML (more or less) model for iSCSI, from which the MIB was derived. Julian has placed a pdf of the model on his web site; it is available at: http://www.haifa.il.ibm.com/satran/ips/Visio-iscsi-uml-model-02.pdf The UML model does not show every table and attribute; it is meant as a higher-level abstraction, and should be helpful in reading the MIB, or in understanding how the iSCSI objects relate to one another. It has been updated to include a few fields that are missing in the current draft. Also available, for those who do not have the previous messages: iSCSI MIB table structure http://www.haifa.il.ibm.com/satran/ips/iscsi-mib-structure-02.pdf iSCSI MIB (zipped version) http://www.haifa.il.ibm.com/satran/ips/draft-bakke-iscsimib-02.zip iSCSI MIB (uncompressed) http://www.haifa.il.ibm.com/satran/ips/draft-bakke-iscsimib-02.txt Enjoy, -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Mon Mar 12 13:14:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA15892 for ; Mon, 12 Mar 2001 13:14:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2CFxiu10232 for ips-outgoing; Mon, 12 Mar 2001 10:59:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2CFwxr10185 for ; Mon, 12 Mar 2001 10:58:59 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id KAA17444; Mon, 12 Mar 2001 10:58:51 -0500 (EST) Message-ID: <3AACF243.9C30D52C@cisco.com> Date: Mon, 12 Mar 2001 09:58:59 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Sandeep Joshi CC: ips@ece.cmu.edu Subject: Re: comments on naming&discovery draft References: <200103102223.RAA03513@aura.research.bell-labs.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Sandeep- The problem you pointed out in item number 1 creates the need for an additional iSCSI-level event. Since the discovery of targets happens at the iSCSI level, rather than at the SCSI level, how about adding this to 2.18.1 (in iSCSI-05)? 4 Network entity indicates that a "target discovery" event has occurred. Upon receiving this message, the initiator should use SendTargets, or whatever other methods of discovery it is using, to find out what has changed. Usually, this would be due to adding a new target. We will fix items 2-4; thanks for pointing them out. Thanks, Mark Sandeep Joshi wrote: > > 1) Section 4.2 last line before Section 4.2.1 > "target MUST send any iSCSI-level async on this session, > allowing the initiator to discover new targets.." > > The session mentioned here is a session to the canonical target. > > However, the iSCSI 05 draft does not mention any such condition > in Sec 2.18 on Async Message. In there, a SCSI event (note: not > iSCSI) is used to notify availability of new targets. > > 2) Appendix C, Section 5 "Stateful Inspection Firewall" > It contains statements of the sort "I dont expect/think.." > These statements could be rephrased to be impersonal. I will fix these. > 3) Section 4.3 Middle of page 17 (After reference to RFC 2608) > -> "A target can register either its canonical target, ..." > > Multiple references to term "target" is confusing. It can be > changed to refer to the term "Network entity" introduced earlier > in the document. > -> "A network entity can register its canonical target,.." I will fix this as well. > 4) Sec 4.2 SendTargets Command > The status code 0x42 mentioned is now 0x02 in iSCSI document. > A similar mismatch exists in some codes listed in B.4.5. > > It might help to just use error names and not numbers to avoid > this problem. [readers can do error code "discovery"! ] Good idea; we will fix this, too. > -Sandeep -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Mon Mar 12 13:24:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA16184 for ; Mon, 12 Mar 2001 13:24:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2CGlfN14368 for ips-outgoing; Mon, 12 Mar 2001 11:47:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (mars.iol.unh.edu [132.177.121.222]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2CGZCr13366 for ; Mon, 12 Mar 2001 11:35:12 -0500 (EST) Received: from localhost (rdr@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id LAA27734; Mon, 12 Mar 2001 11:40:01 -0500 (EST) Date: Mon, 12 Mar 2001 11:40:01 -0500 From: "Robert D. Russell" To: ips@ece.cmu.edu cc: bbr@lampreyNetworks.com, ashishp@iol.unh.edu Subject: iSCSI: Towards a more effective PDU format Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk All, Some people are objecting to having 2 (there is no need for SEVERAL, just 2) header digests. But to have only 1 header digest for variable length headers leads to the following problem: If the header is variable length, how does the receiver find the header digest WITHOUT using the header data that is supposed to be protected by the digest (since that data might be unreliable)? The only answer I see is that the receiver MUST know where the digest is located in the PDU (WITHOUT looking at the header data first). And the only way that can be done is to put the digest at a FIXED offset from the start of the PDU. >From this it follows that the digest cannot be at the end of a variable-length header -- the receiver will NOT know where to find it! However, if the digest is NOT at the end of the header, then 2 possibilities arise: 1) If there is only one digest for ALL the header data, then some of that data must FOLLOW the digest. 2) If a digest must always follow the data it protects, then there has to be a second digest that follows and protects the data not covered by the first digest. Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 From owner-ips@ece.cmu.edu Mon Mar 12 13:26:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA16231 for ; Mon, 12 Mar 2001 13:26:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2CGVcB12967 for ips-outgoing; Mon, 12 Mar 2001 11:31:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2CGQmr12591 for ; Mon, 12 Mar 2001 11:26:48 -0500 (EST) Message-ID: <618471D08ABDD31188B6009027E5009352A37B@apollo.pirus.com> From: "Merhar, Milan" To: "'Vi Chau'" Cc: ips@ece.cmu.edu Subject: RE: FCIP iFCP encapsulation proposal Date: Mon, 12 Mar 2001 11:30:16 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Vi, I think the point that was being presented is that the proposed protocol is open to malicious attack by an end station that _would_ create such data patterns (e.g. - as part of an image written to a filesystem,) and then attempts to take down the FCIP link by repeatedly causing that pattern to traverse the link. The usual rule of thumb is to assume the worst-case situation - that a "unique" data pattern used for control is truly unique only if active measures are taken to prevent its occurance elsewhere. Examples include FC framing (using 10bit sequences that do not map to normal 8bit characters) and HDLC/SDLC (which actively bit-stuffs to prevent normal data from containing the special 6-ones-in-a-row pattern.) Neither of these approaches is probably viable here, as the overhead of bit-level manipulations and mappings without dedicated hardware would be onerous. Sonet introduces a bit-level scrambling onto the data stream, to prevent the introduction of a data pattern that might trigger the line fault detector. This doesn't so much eliminate the "magic sequence" of a long stream of zeroes, but instead obscures it, turning it into a long pseudo-random sequence. Even in the latter case, there have been concerns that some higher-level protocols, such as AAL5 over ATM over Sonet, allow too much flexibility in the payload data, potentially permitting someone to transmit the compliment of the bit-scrambling sequence, and thereby inducing a datalink fault! So, I guess the take-away from this conversation is that some people are very conservative (some would read that as "paranoid" ;-) but the measures they demand _will_ result in a more robust transport spec. And, if that's so, the real discussion IMHO is how to provide such robustness with reasonably low cost/complexity. Best regards, Milan > -----Original Message----- > From: Vi Chau [mailto:vchau@gadzoox.com] > Sent: Friday, March 09, 2001 6:21 PM > To: ips@ece.cmu.edu > Subject: RE: FCIP iFCP encapsulation proposal > > > Stephen, > > I am confused by your argument. The re-synchronization that we > proposed is between two FCIP devices/ports; why would one side want > to confuse the other side by sending patterns that cause false > re-synchronization? FCIP devices are defined to be the providers > of tunnels for FC data to pass through. So the users of the FCIP > devices are the FC end nodes that want to communicate with each > other so they do not have a tendency to intentionally send > confusing or "cooked" data patterns. > > Vi > > > -----Original Message----- > > From: Stephen Bailey [mailto:steph@cs.uchicago.edu] > > Sent: Friday, March 09, 2001 7:42 AM > > To: ips@ece.cmu.edu > > Subject: Re: FCIP iFCP encapsulation proposal > > > > > > > There is a remote possibility that a false compare could occur > > > from other data in the stream so it is necessary to continue to > > > check the "tentative" FCIP/iFCP payload and CRC also before > > > assuming a correct synchronization. If both CRC checks are good, > > > this certification should be at least as good as the data > > > integrity provided by the CRCs in a synchronized stream. > > > > Nonsense. > > > > The original FCIP proposal that the receiver scan for an EOF/HDR/SOF > > sandwich to recover synchronization has the same problem. This > > technique is fundamentally bogus. > > > > Any time you're scanning user-controllable data (as in from the > > completely uncontrolled, end application), you MUST assume absolute > > worst case (to fool the sync algorithm) data. In the worst > case, the > > probability of faking out your resynchronization heuristic is much > > (!!!) greater than the probability of an integrity CRC giving you a > > false positive acceptance check. > > > > In other words, the user can pack the data stream with `cooked' data > > patterns that are just waiting for a dropped frame and an attempt to > > resynchronize. If you assume that the shortest pattern required to > > fool the resynchronization algorithm is n words (no longer than the > > minimum well-formed PDU size), and TCP segmentation is effectively > > random (*), the chance of your resynch algorithm getting spoofed > > (resulting in the user getting some control of trusted > `machinery') is > > simply 1 in n. > > > > You can fix this problem by using a shared `secret' (from > the client) > > between the two endpoints. For example, if you put a 64-bit random > > number that both endpoints know in the end/begin sandwich, THEN you > > can talk about vanishingly small probabilities of the sandwich > > discovery going wrong. > > > > Personally, I think this running sync rediscovery is all makework > > because it's still a far distant second (or third, if you like > > periodic markers) to ensuring that each segment is sufficiently > > self-describing (AKA framing). A framing solves several other > > problems, such as the digest failure problem currently > being discussed > > by Julian & Somesh. > > > > Steph > > > > * It isn't but, AGAIN, you must make WORST CASE > assumptions, because, > > with properly periodic or aperiodic patterns, the user can actually > > encourage the segmentation algorithm to work in her favor. > > > From owner-ips@ece.cmu.edu Mon Mar 12 13:27:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA16251 for ; Mon, 12 Mar 2001 13:27:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2CGucU15053 for ips-outgoing; Mon, 12 Mar 2001 11:56:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2CGtpr14959 for ; Mon, 12 Mar 2001 11:55:51 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id LAA19423; Mon, 12 Mar 2001 11:55:37 -0500 (EST) Message-ID: <3AACFF91.58FA5F6C@cisco.com> Date: Mon, 12 Mar 2001 10:55:45 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: klein@sanrad.com CC: "'IPS'" Subject: Re: I-D ACTION:draft-bakke-iscsimib-02.txt References: <000001c0aa42$65a99570$49c811ac@sanrad.co.il> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Yaron- Thanks for looking through the draft. My comments are embedded. -- Mark Yaron Klein wrote: > > Mark, > > Great job with the MIB. Some questions and comments. > > 1. Date should be change to February 2001. Oops. I still have a Y2k bug in my chair-to-keyboard interface. I'll fix it. > 2. Can you take a student from UMN to fill the description fields of the > parameters :-). Actually, the team is working on descriptions for the fields. Doing these will take a while, and we did not have the time to get them done in time for the meeting. > 3. The StatusStatsEntry type parameters should be updated according to > version 05 new status codes. OK. > 4. What does the iscsiSsnOtherHdrIntegrity means? The text format of the > algorithm? This would be the text name of the algorithm used for header integrity, if there is no enumerated type for it. We plan to get rid of the enums in the next draft, and include only the UTF8 representations of header integrity, data integrity, initiator authentication, and target authentication. So it will look more like: iscsiSsnHeaderIntegrity UTF8String, iscsiSsnDataIntegrity UTF8String, iscsiSsnInitiatorAuth UTF8String, iscsiSsnTargetAuth UTF8String, This will make handling of non-standard schemes identical to that used by the standard schemes, and remove the need to keep updating enumerated types in the MIB as these evolve. > 5. The iscsiSsnDataIntegrity table should include the kerberos options as > well as the header digest (version 05). The new attributes from #4 will fix this. > 6. Can the AccessList entry include IP address as well as WWUI for initial > filtering? That's one of the open issues I will bring up during the MIB presentation next week. In general, there are a lot of different things that can be done with access lists; which of these will be universally-implemented to the extent that they should be included in the MIB, and which should be left to table augmentations in enterprise MIBs? > Regards, > > Yaron > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Mon Mar 12 14:57:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA18815 for ; Mon, 12 Mar 2001 14:57:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2CHrnl19296 for ips-outgoing; Mon, 12 Mar 2001 12:53:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2CHqdr19218 for ; Mon, 12 Mar 2001 12:52:39 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id D28D7C25 for ; Mon, 12 Mar 2001 09:52:20 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id JAA26497 for ; Mon, 12 Mar 2001 09:52:16 -0800 (PST) Message-ID: <3AAD0DC4.9E2ABEC5@cup.hp.com> Date: Mon, 12 Mar 2001 09:56:21 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI : SCSI Response PDU References: Content-Type: multipart/mixed; boundary="------------A039F071EB3A42DC8E75030F" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------A039F071EB3A42DC8E75030F Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > 2) There is a reference to Response Data [without a corresponding > response length. Once again, response length is required, if a > response data is used.]. Response Data is modelled on the FC > response data and is useless in the current context. > (The FC response data only contained a response code). > Any target that wishes to send vendor unique information > could do so through vendor unique sense data. > > All references to Response Data should be removed. > +++ Responses are not sense +++ Julian, What additional information does the response data convey which cannot be done through the response codes ? As stated below, any vendor unique response data can be communicated through the use of vendor unique response codes. (The response code itself constitutes response data, as is the case with FC.) Regards, Santosh > 3) A range of iSCSI Response codes should be assigned for vendor-unique > responses. > +++ Good Idea +++ --------------A039F071EB3A42DC8E75030F Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------A039F071EB3A42DC8E75030F-- From owner-ips@ece.cmu.edu Mon Mar 12 14:57:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA18847 for ; Mon, 12 Mar 2001 14:57:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2CI4d720130 for ips-outgoing; Mon, 12 Mar 2001 13:04:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gordan.pl.gadzoox.com (i248.gadzoox.com [216.52.31.248]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2CI3kr20077 for ; Mon, 12 Mar 2001 13:03:46 -0500 (EST) Received: by gordan.pl.gadzoox.com with Internet Mail Service (5.5.2653.19) id ; Mon, 12 Mar 2001 10:03:08 -0800 Message-ID: From: Vi Chau To: ips@ece.cmu.edu Subject: RE: FCIP iFCP encapsulation proposal Date: Mon, 12 Mar 2001 10:03:07 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk From reading the notes on this thread since Friday, it is clear to me that this is a security (authentication) problem and not a framing (re-synch) issue since without an authentication mechanism, no framing scheme is safe from spoofing or malicious actions. Speaking of which, there is this, from Julian's reply in the iSCSI "lengths and header digest error recovery" thread: > You have to take some elements as good and try to recover. This holdsfor > those protocols that have length as demarcation element and for those that > have framing patterns for demarcation as framing patterns can also be bad. > You recover at another frame boundary or another marker. Same issue as what we are talking about here. Using a special pattern for frame delineaton does not get around this problem as there is no guarantee that the same pattern does not appear in the payload (unlike the case of HDLC or Sonet which operate at a lower protocol layer); and a malicious user can create well-formed packets that carry the special framing pattern designed to fool the receiver. Perhaps security should be considered together with encapsulation if enough people think this is a serious concern. Vi > -----Original Message----- > From: Stephen Bailey [mailto:steph@cs.uchicago.edu] > Sent: Monday, March 12, 2001 5:59 AM > To: ips@ece.cmu.edu > Subject: Re: FCIP iFCP encapsulation proposal > > > > I can envision a case where the user embeds a sequence of several > > well-formed PDUs in such a payload to handle the case where > > resynchronization requires the detection of more than one good > > encapsulation. > > Precisely. > > In fact, in fact, I just realized (once again), that I'm an > idiot---the chance of user getting control with a repeating set of > valid PDUs is actually MUCH higher than 1 in # of words in the PDU. > > If the TCP segment begins with the start of a PDU, the user will not > get control. If the TCP segment begins at any other point, the scan > will proceed (failing to resynch) until it reaches the beginning of > one of the user's spoofing PDUs, at which point resynching will > succeed (erroneously), and the user will have control, at least until > the end of the current, actual PDU. > > In other words, the chances are remote that a stream of PDU patterns > in the user data WON'T cause resynchronization to occur erroneously. > > As to why would this happen---to watch it burn. Why not? Script > Kiddies abound. Personally, if I were a 12 year old, hacking on my > school's shared 11/34 running RSTS/e, I'd blindly stumble around > trying different SYS(xxx) calls too. > > Steph > From owner-ips@ece.cmu.edu Mon Mar 12 17:02:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA21864 for ; Mon, 12 Mar 2001 17:02:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2CJvhX28391 for ips-outgoing; Mon, 12 Mar 2001 14:57:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (mars.iol.unh.edu [132.177.121.222]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2CJusr28335 for ; Mon, 12 Mar 2001 14:56:55 -0500 (EST) Received: from localhost (rdr@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id PAA02856; Mon, 12 Mar 2001 15:01:39 -0500 (EST) Date: Mon, 12 Mar 2001 15:01:39 -0500 From: "Robert D. Russell" To: ips@ece.cmu.edu cc: bbrtrebia@mediaone.net, "Ashish P. Palekar" , Anshul Chadda , Narendran Ganapathy Subject: Re: iSCSI: Towards a more effective PDU format Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk All: Here is a proposal for an iSCSI PDU format without WN Next-Qualifiers: 1. Every PDU must start with a 48-byte Basic Header Segment (BHS). Assuming header digests are in use, this BHS is immediately followed by a header digest that covers it. 2. Every BHS contains 2 fields at fixed locations: a) AHS-length, containing the total length of all Additional Header Segments (AHS) that follow. This field is 0 if there are no AHSs. b) DATA-length, containing the total length of all Data that follows the AHSs. This field is 0 if there is no data. 3. If the AHS-length field in the BHS is non-zero, all the AHSs immediately follow the header digest of the BHS. An additional digest immediately follows that last AHS and covers all the AHSs. 4. If the DATA-length field in the BHS is non-zero, all the data immediately follows the AHS digest. A data digest immediately follows the data. 5. If space within the BHS is an issue, the AHS-length field could be restricted to a single byte, and could be in units of 4-byte words. This would allow up to 1020 bytes of AHSs to follow a BHS, and this seems to be more than enough for the uses foreseen (so far, only extended CDB and bidi-read info). 6. Each AHS should start with a word containing a TYPE field and a LENGTH field. The TYPE field should be enumerated rather than bit-field encoded, for easier decoding and future expansion. The LENGTH field is the number of bytes of additional information in this AHS that follows this word. Advantages of this proposal. 1. The receiver always knows where to find digests in a reliable manner, without having to use unreliable data to find them. 2. Because the BHS is fixed length, the receiver can reliably obtain the entire BHS and its header digest in a single "read" operation. 3. Because the receiver gets the AHS-length field from the BHS, it can reliably obtain the entire set of AHSs and the header digest covering them in a single "read" operation. 4. Because the receiver gets the DATA-length field from the BHS, it can reliably obtain all the data and the data digest covering it in a single "read" operation. 5. Since each AHS begins with a word containing the TYPE and LENGTH in fixed positions, an unknown TYPE (i.e., a type introduced in the future that is received by a legacy receiver) does NOT result in loss of synchronization during header processing -- the receiver knows the length of this AHS in any case, and can just skip over it. Disadvantages of this proposal. 1. A second header digest is required, but only if one or more AHSs are present. In practice this should be a rare event. Proposed iSCSI PDU Format +-------------+ | 48-byte BHS | +-------------+ | BHS digest | --+-------------+-- | AHS 1 |\ | - - - - - - | \ | AHS 2 | \ | - - - - - - | > total length in AHS_length field in BHS | . . . | / | - - - - - - | / | AHS n |/ +-------------+ | AHSs digest | --+-------------+-- | |\ | Data | > total length in DATA_length field in BHS | |/ +-------------+ | Data digest | +-------------+ Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 From owner-ips@ece.cmu.edu Mon Mar 12 17:03:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA21884 for ; Mon, 12 Mar 2001 17:03:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2CJdi227132 for ips-outgoing; Mon, 12 Mar 2001 14:39:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com (42s3082.isus.emc.com [168.159.211.82] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2CJdFr27101 for ; Mon, 12 Mar 2001 14:39:17 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 12 Mar 2001 14:40:30 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A070801527C@corpmx9.isus.emc.com> From: Black_David@emc.com To: vchau@gadzoox.com, ips@ece.cmu.edu Subject: RE: FCIP iFCP encapsulation proposal Date: Mon, 12 Mar 2001 14:38:58 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > From reading the notes on this thread since Friday, it is clear > to me that this is a security (authentication) problem and not a > framing (re-synch) issue since without an authentication mechanism, > no framing scheme is safe from spoofing or malicious actions. While the examples that people are using to critique the design involve malicious attacks, the resync mechanism must ensure that data is not mistaken for headers even in the absence of malicious attacks. Here's a perfectly reasonable scenario -- consider using a protocol tracer to dump every bit transmitted on an FCIP connection to a binary trace file for debugging. Now ftp that file to a server for further examination, and suppose there's another FCIP link between the server and its storage. There MUST NOT be any way for the far end of the FCIP link to get confused and start to treat the contents of the binary trace file that is being written to storage as FCIP frames, even though the contents look like FCIP frames (e.g., what if one of the frames in the trace file is a FORMAT command?). If a special data pattern is used to trip resync, the special pattern has to only occur at resync points. There are a number of worked examples of this, including PPP byte-stuffing, and the 8b/10b coding used for both Gigabit Ethernet and Fibre Channel. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Mar 12 17:54:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA23480 for ; Mon, 12 Mar 2001 17:54:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2CKse402517 for ips-outgoing; Mon, 12 Mar 2001 15:54:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2CKrmr02479 for ; Mon, 12 Mar 2001 15:53:48 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GA300962RD6PA@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Mon, 12 Mar 2001 12:53:33 -0800 (PST) Date: Mon, 12 Mar 2001 12:52:12 -0800 From: Douglas Otis Subject: RE: FCIP iFCP encapsulation proposal In-reply-to: <618471D08ABDD31188B6009027E5009352A37B@apollo.pirus.com> To: "Merhar, Milan" , "'Vi Chau'" Cc: ips@ece.cmu.edu Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Milan, With trusted routers and secure name servers, spoofing and not man in the middle is of concern. Variable length framing opens up many possibilities for spoofing, as spoofing could attempt a series of validations. A simple solution would be to use two levels of framing. One at a fixed interval and a sub-layer that allow FC frames to be fragmented to fit within these fixed interval frames or pseudo-frames. The sub-layer could be a vanilla version of SCTP padded out to fit if required as it also provides the means of acknowledging pseudo-frames and the extensibility desired in a general purpose framing algorithm. This does two things. It allows error checking to be done a the pseudo-frame level rather than on a PDU basis and also allows this well defined length pseudo-frame to be retried. The overhead for such a scheme would be low as it should be rare pseudo-frames would require padding. If done within TCP, simply substitute SCTP ports with a pseudo-frame length. This ensures error checking is limited to a reasonable length rather than large PDUs found with iSCSI as example, if done in a general purpose mode and ensures memory requirements are defined for the transport. Doug > Vi, > > I think the point that was being presented is that > the proposed protocol is open to malicious attack by an > end station that _would_ create such data patterns > (e.g. - as part of an image written to a filesystem,) > and then attempts to take down the FCIP link > by repeatedly causing that pattern to traverse the link. > > The usual rule of thumb is to assume the worst-case > situation - that a "unique" data pattern used for > control is truly unique only if active measures are > taken to prevent its occurance elsewhere. > Examples include FC framing (using 10bit sequences > that do not map to normal 8bit characters) and HDLC/SDLC > (which actively bit-stuffs to prevent normal data from > containing the special 6-ones-in-a-row pattern.) > Neither of these approaches is probably viable here, > as the overhead of bit-level manipulations and mappings > without dedicated hardware would be onerous. > > Sonet introduces a bit-level scrambling onto the data > stream, to prevent the introduction of a data pattern > that might trigger the line fault detector. This doesn't > so much eliminate the "magic sequence" of a long stream > of zeroes, but instead obscures it, turning it into a > long pseudo-random sequence. > > Even in the latter case, there have been concerns > that some higher-level protocols, such as AAL5 over > ATM over Sonet, allow too much flexibility in the > payload data, potentially permitting someone to transmit > the compliment of the bit-scrambling sequence, and thereby > inducing a datalink fault! > > So, I guess the take-away from this conversation is that > some people are very conservative (some would read that > as "paranoid" ;-) but the measures they demand _will_ > result in a more robust transport spec. > > And, if that's so, the real discussion IMHO is how to provide > such robustness with reasonably low cost/complexity. > > Best regards, > Milan > > > > > > -----Original Message----- > > From: Vi Chau [mailto:vchau@gadzoox.com] > > Sent: Friday, March 09, 2001 6:21 PM > > To: ips@ece.cmu.edu > > Subject: RE: FCIP iFCP encapsulation proposal > > > > > > Stephen, > > > > I am confused by your argument. The re-synchronization that we > > proposed is between two FCIP devices/ports; why would one side want > > to confuse the other side by sending patterns that cause false > > re-synchronization? FCIP devices are defined to be the providers > > of tunnels for FC data to pass through. So the users of the FCIP > > devices are the FC end nodes that want to communicate with each > > other so they do not have a tendency to intentionally send > > confusing or "cooked" data patterns. > > > > Vi > > > > > -----Original Message----- > > > From: Stephen Bailey [mailto:steph@cs.uchicago.edu] > > > Sent: Friday, March 09, 2001 7:42 AM > > > To: ips@ece.cmu.edu > > > Subject: Re: FCIP iFCP encapsulation proposal > > > > > > > > > > There is a remote possibility that a false compare could occur > > > > from other data in the stream so it is necessary to continue to > > > > check the "tentative" FCIP/iFCP payload and CRC also before > > > > assuming a correct synchronization. If both CRC checks are good, > > > > this certification should be at least as good as the data > > > > integrity provided by the CRCs in a synchronized stream. > > > > > > Nonsense. > > > > > > The original FCIP proposal that the receiver scan for an EOF/HDR/SOF > > > sandwich to recover synchronization has the same problem. This > > > technique is fundamentally bogus. > > > > > > Any time you're scanning user-controllable data (as in from the > > > completely uncontrolled, end application), you MUST assume absolute > > > worst case (to fool the sync algorithm) data. In the worst > > case, the > > > probability of faking out your resynchronization heuristic is much > > > (!!!) greater than the probability of an integrity CRC giving you a > > > false positive acceptance check. > > > > > > In other words, the user can pack the data stream with `cooked' data > > > patterns that are just waiting for a dropped frame and an attempt to > > > resynchronize. If you assume that the shortest pattern required to > > > fool the resynchronization algorithm is n words (no longer than the > > > minimum well-formed PDU size), and TCP segmentation is effectively > > > random (*), the chance of your resynch algorithm getting spoofed > > > (resulting in the user getting some control of trusted > > `machinery') is > > > simply 1 in n. > > > > > > You can fix this problem by using a shared `secret' (from > > the client) > > > between the two endpoints. For example, if you put a 64-bit random > > > number that both endpoints know in the end/begin sandwich, THEN you > > > can talk about vanishingly small probabilities of the sandwich > > > discovery going wrong. > > > > > > Personally, I think this running sync rediscovery is all makework > > > because it's still a far distant second (or third, if you like > > > periodic markers) to ensuring that each segment is sufficiently > > > self-describing (AKA framing). A framing solves several other > > > problems, such as the digest failure problem currently > > being discussed > > > by Julian & Somesh. > > > > > > Steph > > > > > > * It isn't but, AGAIN, you must make WORST CASE > > assumptions, because, > > > with properly periodic or aperiodic patterns, the user can actually > > > encourage the segmentation algorithm to work in her favor. > > > > > > From owner-ips@ece.cmu.edu Mon Mar 12 20:48:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA27095 for ; Mon, 12 Mar 2001 20:48:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2D0Aoj16327 for ips-outgoing; Mon, 12 Mar 2001 19:10:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2D09sr16273 for ; Mon, 12 Mar 2001 19:09:54 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Mon, 12 Mar 2001 16:09:47 -0800 Message-ID: From: Charles Monia To: "Ips (E-mail)" Cc: "David Black (E-mail)" Subject: RE: FCIP: iFCP/FCIP Framing assists (was TCP/IP Framing Assists) Date: Mon, 12 Mar 2001 16:09:47 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi David: > The result could be three TCP framing approaches > - SCTP-lite > - Markers (iSCSI appendix) > - Reserved value/word stuffing (weber draft) The above layered approaches assume the framing mechanism and data representations are totally othogonal. Using that criteria leads to a fourth negotiable alternative: ie. no TCP framing assist. In that case, assuming: a) The header and FC frame each have their own error digests and b) The header contains the information needed to extract the FC frame, the approach is to validate the header and, if error-free, forward the resulting de-encapsulated frame (with FC-generated CRC) to the FC fabric for processing. In this approach, of course, there's no fall-back in the event of a header error. So, in the case of iFCP, the response would be to terminate the session. Assuming the cross-section for such errors is small compared to that of the typical FC payload, a seat-of-the-pants guess is that such occurrences should be rare. Of course, FCIP would probably negotiate some other TCP framing alternative. Charles > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Saturday, March 10, 2001 1:21 PM > To: cmonia@nishansystems.com; Black_David@emc.com > Cc: ips@ece.cmu.edu > Subject: RE: TCP/IP Framing Assists > > > Copying the list on a response to a question of general interest ... > > Charles Monia writes: > > At the last IPS WG, you said you felt there was a > possibility that the > IETF > > community might be persuaded to accept some sort of framing > assists for > > TCP/IP (At the time, I characterized this as 'SCTP lite'). > > > > Has there been any progress on this front? This could > impact the outcome > of > > the common encapsulation effort. > > It is still a definite possibility. There has been progress, > but it's not > exactly > visible, yet. As indicated in the IPS Agenda for > Minneapolis, work on this > will be conducted through the tsvwg Working Group - I have not seen a > Minneapolis Agenda for tsvwg, so I don't know if/when this > issue will be > taken up. I'll try to get an update (no discussion) on this > topic included > in the initial 10 minutes of Agenda Bashing and Administrivia. > > For the FCIP/iFCP common encapsulation, I would recommend separating > framing from data representation, and adopting an approach > along the lines > of > that in Section 1.2.8 of the iSCSI draft in which framing > logically occurs > between the block storage protocol and TCP. In particular, this means > that the 0xfcfcfcfc reserved value/word-stuffing approach in > Section 2 of > the weber draft should be specified separately as a "shim" protocol > at this level if it is to be carried forward - this is the > right way to > think > about it in any case because any protocol headers at higher layers in > the stack have to be word-stuffed if the reserved value sequence (4 > occurrences of the reserved value in the weber draft) occurs there. > > The result could be three TCP framing approaches > - SCTP-lite > - Markers (iSCSI appendix) > - Reserved value/word stuffing (weber draft) > Each could be independently chosen/used by any of the IPS > block storage > protocols, provided that suitable negotiation mechanisms are > specified to > make sure that both sides of a connection understand exactly > what is being > used in which direction. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ece.cmu.edu Mon Mar 12 20:50:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA27110 for ; Mon, 12 Mar 2001 20:50:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2CNZhV14034 for ips-outgoing; Mon, 12 Mar 2001 18:35:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2CNYlr13993 for ; Mon, 12 Mar 2001 18:34:47 -0500 (EST) Received: from fedex.cisco.com (fedex.cisco.com [171.69.18.27]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id PAA19618 for ; Mon, 12 Mar 2001 15:35:01 -0800 (PST) Received: from CSAPUNTZW2K (ssh-sj1.cisco.com [171.68.225.134]) by fedex.cisco.com (Mirapoint) with SMTP id ABU06429; Mon, 12 Mar 2001 15:34:40 -0800 (PST) Message-ID: <005001c0ab4d$4945b330$0b00000a@cisco.com> From: "Constantine Sapuntzakis" To: Subject: Fw: TCP Framing Proposal Date: Mon, 12 Mar 2001 15:36:39 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit A draft on TCP ULP framing from the ad-hoc WARP RDMA group is now available. The latest version of the draft is at http://www.stanford.edu/~csapuntz/draft-williams-tcpulpframe-01.txt A slightly older version is available from the Internet Draft repository under the name draft-williams-tcpulpframe-00.txt The algorithm is the same as in the -01 draft but the explanatory text and analysis is poorer. The discussion of this draft will take place on the TSVWG reflector. To subscribe or browse the archives, go to http://www1.ietf.org/mailman/listinfo/tsvwg/ Thanks, -Costa ----- Original Message ----- From: "Constantine Sapuntzakis" To: Sent: Monday, March 12, 2001 3:32 PM Subject: TCP Framing Proposal > The authors of the TCP ULP framing draft kindly request your feedback on a > technique for > recovering upper-layer protocols headers in TCP segments, even when the > segments arrive > out-of-order. The upper-layer headers are used to place the data in memory; > however, the > upper-layer still does protocol processing (the control path) in-order. > > The most current version of the techniques is described in a draft available > at > > http://www.stanford.edu/~csapuntz/draft-williams-tcpulpframe-01.txt > > This draft has been submitted and should appear after the quiet period. > > A slightly older version is available from the Internet Draft repository > under the name > draft-williams-tcpulpframe-00.txt The algorithm is the same as in the -01 > draft but the explanatory > text and analysis is poorer. > > Thanks, > Costa > From owner-ips@ece.cmu.edu Mon Mar 12 22:39:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA29966 for ; Mon, 12 Mar 2001 22:39:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2D1kh622062 for ips-outgoing; Mon, 12 Mar 2001 20:46:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2D1jwr22032 for ; Mon, 12 Mar 2001 20:45:59 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GA400MMC4WDQQ@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Mon, 12 Mar 2001 17:45:49 -0800 (PST) Date: Mon, 12 Mar 2001 17:44:21 -0800 From: Douglas Otis Subject: RE: TCP Framing Proposal In-reply-to: <005001c0ab4d$4945b330$0b00000a@cisco.com> To: Constantine Sapuntzakis , ips@ece.cmu.edu Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Costa, Creating an alignment with the TCP segment and the PDU, even to the point of dictating how this PDU will be fragmented, does not seem practical in the event of network errors. In addition, if there is any extended error checking as there seems to be a consensus for including, this framing does not define a means of reissuing an invalidated pseudo-frame in a generic fashion. What is the difficulty of including SCTP as a layer within TCP where a fixed length pseudo-frame is negotiated as an extension to SCTP? The real frame length can be indicated by replacing the SCTP ports field with a length much as you have already done in your example. Using RAM to allow reconstruction of these pseudo-frames seems like a small price to pay and can be done without playing games with TCP. There would be little benefit from forcing an alignment between TCP and this pseudo-frame which is larger than the TCP frame in most cases. The amount of overhead consumed by this SCTP encapsulation scheme would be virtually identical to this proposal with the benefits of frame level acknowledgements, selective retries, multi-homing, session recovery, etc. With your proposed approach, the TCP stack must be altered to handle PDU alignments with virtually no benefit with respect to any error recovery. As you should expect fragmentation unless jumbo frames are used for sending FC frames, there will always be some reconstruction required. This does not change regardless of the frame alignment. SCTP isolates encapsulated PDUs or FC frames with STREAM and SEQUENCE. You have yet to make such a definition within your scheme except to expect such processing is always done sequentially but then again you have not consider the impact an error will cause. Doug > A draft on TCP ULP framing from the ad-hoc WARP RDMA group is now > available. > The latest > version of the draft is at > > http://www.stanford.edu/~csapuntz/draft-williams-tcpulpframe-01.txt > > A slightly older version is available from the Internet Draft repository > under the name > draft-williams-tcpulpframe-00.txt The algorithm is the same as > in the -01 > draft but the > explanatory text and analysis is poorer. > > The discussion of this draft will take place on the TSVWG reflector. To > subscribe > or browse the archives, go to > > http://www1.ietf.org/mailman/listinfo/tsvwg/ > > Thanks, > -Costa > > > ----- Original Message ----- > From: "Constantine Sapuntzakis" > To: > Sent: Monday, March 12, 2001 3:32 PM > Subject: TCP Framing Proposal > > > > The authors of the TCP ULP framing draft kindly request your > feedback on a > > technique for > > recovering upper-layer protocols headers in TCP segments, even when the > > segments arrive > > out-of-order. The upper-layer headers are used to place the data in > memory; > > however, the > > upper-layer still does protocol processing (the control path) in-order. > > > > The most current version of the techniques is described in a draft > available > > at > > > > http://www.stanford.edu/~csapuntz/draft-williams-tcpulpframe-01.txt > > > > This draft has been submitted and should appear after the quiet period. > > > > A slightly older version is available from the Internet Draft repository > > under the name > > draft-williams-tcpulpframe-00.txt The algorithm is the same as in > the -01 > > draft but the explanatory > > text and analysis is poorer. > > > > Thanks, > > Costa > > > > From owner-ips@ece.cmu.edu Mon Mar 12 22:41:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA00087 for ; Mon, 12 Mar 2001 22:41:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2D1tk522594 for ips-outgoing; Mon, 12 Mar 2001 20:55:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2D1ter22577 for ; Mon, 12 Mar 2001 20:55:41 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 1347DFF for ; Mon, 12 Mar 2001 17:48:56 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA09951 for ; Mon, 12 Mar 2001 17:48:51 -0800 (PST) Message-ID: <3AAD7D78.83C2DD5B@cup.hp.com> Date: Mon, 12 Mar 2001 17:52:57 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Section 4. Login Phase. Content-Type: multipart/mixed; boundary="------------52BA8BD241F0499A94891628" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------52BA8BD241F0499A94891628 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian & All, A couple of issues on Section 4 of the rev 05 iSCSI draft : 1) "The Login Final-Response can come only as a response to a Login command with the F bit set to 1 or a Text Command with the F bit set to 1." The above statement is not true and must be removed, since the login-final response is used by targets when rejecting login. As per the table in Section 2.11.4, the F bit is allowed to be set to 1 when the status class is non-zero. 2) "Operational parameters MAY be negotiated within or outside (after) the login phase." If text negotiation can be conducted outside of the login phase, what is the effect of changes in text keys on the existing login ? For ex, how does changes to keys like MaxConnections, UseR2T, BidiUseR2T, ImmediateData, DataPDULength, FirstBurstSize, ITagLength, etc affect their existing values negotiated during the current login ? The side-effects of such changes and the scope of their establishment must be documented. It may be simplest to prohibit such changes outside of the login phase, for the sake of simplicity. Regards, Santosh --------------52BA8BD241F0499A94891628 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------52BA8BD241F0499A94891628-- From owner-ips@ece.cmu.edu Mon Mar 12 22:42:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA00098 for ; Mon, 12 Mar 2001 22:42:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2D1d3521606 for ips-outgoing; Mon, 12 Mar 2001 20:39:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2D1bir21425 for ; Mon, 12 Mar 2001 20:37:44 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GA4006A94ID0B@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Mon, 12 Mar 2001 17:37:26 -0800 (PST) Date: Mon, 12 Mar 2001 17:35:58 -0800 From: Douglas Otis Subject: RE: FCIP: iFCP/FCIP Framing assists (was TCP/IP Framing Assists) In-reply-to: To: Charles Monia , "Ips (E-mail)" Cc: "David Black (E-mail)" Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Charles, Should you use SCTP within TCP by replacing the SCTP port with a pseudo-frame length, then conventions for SCTP allows for frame recovery independent of the encapsulated FC frame. SCTP even allows for setting retries for this frame as well as providing an authorities SACK independent of TCP. With this scheme, you could take advantage of many benefits offered by SCTP as well as eventually using SCTP directly. The pseudo-frame would then dictate memory overhead for frame reconstruction which will be required as FC and Ethernet often have incompatible lengths. The iFCP encapsulation would sit within an SCTP data chunk. Simply define a fixed length pseudo-frame as part of a modified SCTP negotiation. Doug > > Hi David: > > > The result could be three TCP framing approaches > > - SCTP-lite > > - Markers (iSCSI appendix) > > - Reserved value/word stuffing (weber draft) > > The above layered approaches assume the framing mechanism and data > representations are totally othogonal. Using that criteria leads > to a fourth > negotiable alternative: ie. no TCP framing assist. > > In that case, assuming: > > a) The header and FC frame each have their own error digests and > b) The header contains the information needed to extract the FC frame, > > the approach is to validate the header and, if error-free, forward the > resulting de-encapsulated frame (with FC-generated CRC) to the FC > fabric for > processing. In this approach, of course, there's no fall-back in > the event > of a header error. So, in the case of iFCP, the response would be to > terminate the session. > > Assuming the cross-section for such errors is small compared to > that of the > typical FC payload, a seat-of-the-pants guess is that such occurrences > should be rare. > > Of course, FCIP would probably negotiate some other TCP framing > alternative. > > Charles > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Saturday, March 10, 2001 1:21 PM > > To: cmonia@nishansystems.com; Black_David@emc.com > > Cc: ips@ece.cmu.edu > > Subject: RE: TCP/IP Framing Assists > > > > > > Copying the list on a response to a question of general interest ... > > > > Charles Monia writes: > > > At the last IPS WG, you said you felt there was a > > possibility that the > > IETF > > > community might be persuaded to accept some sort of framing > > assists for > > > TCP/IP (At the time, I characterized this as 'SCTP lite'). > > > > > > Has there been any progress on this front? This could > > impact the outcome > > of > > > the common encapsulation effort. > > > > It is still a definite possibility. There has been progress, > > but it's not > > exactly > > visible, yet. As indicated in the IPS Agenda for > > Minneapolis, work on this > > will be conducted through the tsvwg Working Group - I have not seen a > > Minneapolis Agenda for tsvwg, so I don't know if/when this > > issue will be > > taken up. I'll try to get an update (no discussion) on this > > topic included > > in the initial 10 minutes of Agenda Bashing and Administrivia. > > > > For the FCIP/iFCP common encapsulation, I would recommend separating > > framing from data representation, and adopting an approach > > along the lines > > of > > that in Section 1.2.8 of the iSCSI draft in which framing > > logically occurs > > between the block storage protocol and TCP. In particular, this means > > that the 0xfcfcfcfc reserved value/word-stuffing approach in > > Section 2 of > > the weber draft should be specified separately as a "shim" protocol > > at this level if it is to be carried forward - this is the > > right way to > > think > > about it in any case because any protocol headers at higher layers in > > the stack have to be word-stuffed if the reserved value sequence (4 > > occurrences of the reserved value in the weber draft) occurs there. > > > > The result could be three TCP framing approaches > > - SCTP-lite > > - Markers (iSCSI appendix) > > - Reserved value/word stuffing (weber draft) > > Each could be independently chosen/used by any of the IPS > > block storage > > protocols, provided that suitable negotiation mechanisms are > > specified to > > make sure that both sides of a connection understand exactly > > what is being > > used in which direction. > > > > --David > > > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > > > From owner-ips@ece.cmu.edu Tue Mar 13 00:06:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA00951 for ; Tue, 13 Mar 2001 00:06:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2D2skc26175 for ips-outgoing; Mon, 12 Mar 2001 21:54:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2D2rwr26149 for ; Mon, 12 Mar 2001 21:53:58 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 396C17B4; Mon, 12 Mar 2001 19:53:58 -0700 (MST) Received: from axcsbh2.cos.agilent.com (axcsbh2.cos.agilent.com [130.29.152.144]) by msgrel1.cos.agilent.com (Postfix) with SMTP id 0F23FA3; Mon, 12 Mar 2001 19:53:58 -0700 (MST) Received: from 130.29.152.144 by axcsbh2.cos.agilent.com (InterScan E-Mail VirusWall NT); Mon, 12 Mar 2001 19:53:58 -0700 (Mountain Standard Time) Received: by axcsbh2.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Mon, 12 Mar 2001 19:43:57 -0700 Message-ID: <1BEBA5E8600DD4119A50009027AF54A00530F2D3@axcs04.cos.agilent.com> From: pat_thaler@agilent.com To: julian_satran@il.ibm.com, ips@ece.cmu.edu Cc: pat_thaler@agilent.com, Dafna_Sheinwald@il.ibm.com Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Date: Mon, 12 Mar 2001 19:43:51 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian asked: "Do you have a formula to evaluate Pud for low noise channels with BER = e? Any formula to evaluate the conditional probability of an undetected error on bursts of 17, 18, 19 bits? Is there any reason to believe that we can approximate them to ones used for CRC?" Abstract: The following will show that the formula used in the case of CRC for Pud does not apply to checksums. Many data sets with small differences produce the same checksum while few such data sets produce the same CRC. Therefore, Pud for checksum is significantly worse than that for CRC. For independent bit errors, Pud is approximately 12,000 times worse for Fletcher32 than for CRC and 22,000 times worse for Adler32 than for CRC. Note: While Fletcher32 is better than Adler32 for the error case analyzed here, Fletcher32 has more susceptibility to two byte errors that Adler32 does not have. Overall, Fletcher32 is not necessarily a better choice. The grungy details: The formula for Pud for CRC in a low noise channel is: Pud = Pcrc * Cn,x * Pxerrors Where: Pcrc is the probability that the received CRC matches the errored message Cn,x is the number of combinations of 3 errors in an n bit message Pxerrors is the probability of getting a particular combination of 3 errors Pbe is the probability of a bit error. Inserting the values for 3 bit errors in a 1000 byte message where Pbe << 1: Pud = 2^-32 * C8000,3 * Pbe^3 = 2^-32 * 85e9 * Pbe^3 = 20 * Pbe^3 Note that this equation is assuming that the CRC in question does not detect all 3 bit errors. If the CRC detects all 3 bit errors as many do, the Pud would be 0. Also, actually equals Pxerrors = (Pbe ^ errored_bits) * ((1 - Pbe) ^ unerrored_bits, but the latter is very close to 1 for low noise channels. Approximately 20 positions of 3 bit errors cause undetectable errors in a 1000 byte message with the CRC. How many 3 bit errors cause undetectable errors for checksums? In the attached analysis of checksum Hamming distance and burst error detection, I showed that an undetectable error for both checksums is formed by sequence of three errored values with errors: x, -2x, and x. The same error pattern in three values with equal spacing between them will cause an undetectable error (see Undetectable 3 bit errors in non-consecutive values below). For Adler32: A value is one byte long. For x = 1, 2, 4, 8, 16, 32 or 64: The three errors can be caused by a 1 bit change each. There are 998 sets of 3 consecutive bytes in a 1000 byte frame. There are 996 sets of 3 bytes spaced with 1 bytes between them. There are 994 sets of 3 bytes spaced with 2 bytes between them. .... There are 2 sets of 3 bytes spaced with 498 bytes between them. Therefore, the undetectable errors analyzed below yield (998 + 2) * 49 / 2 sets of bytes that can have one of 7 sets of 3 bits change for an undetectable error in a 1000 byte frame. Of the 8 ways those three bits can be changed, 2 of them produce an undetectable error. If the pattern discussed here is the only one that produces 3 bit errors, then Pud for 3 bit errors in a 1000 byte message with Adler32 would be: Pud = Pcsum * Cerror * P3errors where Pcsum equals the probability that the 3 bit error results in a correct checksum which is 0.25 for the 3 bit errors. Cerror is the number of combinations of undetectable 3 bit errors. P3errors is the probability of getting a particular combination of 3 errors. Pud = 0.25 * (998 + 2) * 499/2 * 7 * Pbe^3 = 437 * 10^3 * Pbe^3 For Fletcher32: A value is two bytes long. For x = 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192 or 16384: The three errors can be caused by a 1 bit change each. In a 1000 byte message, there are 500 values. There are 498 sets of 3 consecutive values in a 1000 byte frame. There are 496 sets of 3 values spaced with 1 values between them. There are 494 sets of 3 values spaced with 2 values between them. .... There are 2 sets of 3 values spaced with 248 values between them. Therefore, the undetectable errors analyzed below yield (498 + 2) * 249/ 2 sets of bytes that can have one of 15 sets of 3 bits change for an undetectable error in a 1000 byte frame. Of the 8 ways those three bits can be changed, 2 of them produce an undetectable error. If the pattern discussed here is the only one that produces 3 bit errors, then Pud for 3 bit errors in a 1000 byte message with Fletcher32 would be: Pud = 0.25 (498 + 2) * 249/2 * 15 * Pbe^3 = 233 * 10^3 * Pbe^3 To recap: CRC Pud = 20 * Pbe^3 Adler32: Pud = 437 * 10^3 * Pbe^3 Fletcher32: Pud = 233 * 10^3 * Pbe^3 Regards, Pat Thaler ------- Analysis of checksum Hamming distance and burst error detection ----------- It is relatively easy to investigate the Hamming distance and burst detection of Adler32 and Fletcher32 using mathematics. In the argument below, I will use "value" to identify the quantities that get summed since Adler sums bytes and Fletcher32 sums 16-bits. Both Adler and Fletcher calculate for each value: S1 = (S1 + value) mod m S2 = (S2 + S1) mod m where m is 65521 for Adler and 65535 for Fletcher. Which means that for an n value string of values V1 to Vn: S1 = (V1 + V2 + ... + Vn (+ 1 for Adler)) mod m S2 = (n * V1 + (n-1) * V2 + ... + Vn (+ n for Adler)) mod m A two bit error can only change two values. For a two bit error to be undetectable the two changed values must produce no change to S1 and S2. Let x be the change to the first errored value y be the change to the second errored value k be the separation between the first and second errored values. Then for an undetectable error: delta S1 = (x + y) mod m = 0 delta S2 = ((k + 1) * x + y) mod m = 0 = (k * x + x + y) mod m =0 = (k * x) mod m = 0 because (x + y) mod m = 0 Therefore, for an undetectable 2 bit error, k * x must be a multiple of m. For Adler, m is prime and the maximum value of x is 255 so the only way to satisfy the condition is for k to equal 65521. Then x can be any value so it could be a one bit error such as changing the lsb of the first value from 0 to 1. y would then need to be the inverse changing the lsb of the second value from 1 to 0. For Fletcher, m is factorable to 3, 5, 17, 257. For x to provide one or more of those factors, it would have to have more than a 1 bit change and y would have to have the inverse of that change which would mean an error of at least 4 bits. The only way to get a two bit undetectable error is for k to equal 65535. So, Fletcher and Adler do have 2-bit undetectable errors if the messages are longer than the modulus. What about burst protection? For Fletcher, we know that a change of a value from all 0's to all 1's produces an undetectable error so we know that burst protection is less than 16. k = 1 for a burst spread across two consecutive values. Therefore, the only way for a burst spread across two consecutive values to produce an undetectable error is for x to equal 65535. A burst shorter than 16 can't do it. For Adler, corrupting two consecutive bytes can't produce an undetectable error because that would be the case above where k equals 1. There is no way for a burst across two consecutive values to be undetectable in Adler since x is less than the modulus. For a burst across three values: let x equal the error in the first value y equal the error in the second value z equal the error in the third value. For the burst to be undetectable Delta S1 = (x + y + z) mod m = 0 Delta S2 = (3x + 2y + z) mod m = 0 Since the maximum values of x, y, and z are 255 and m is more than 6 times that, the sums above will always be less than m and we can drop the mod m. Delta S2 = (3x + 2y + z) = 2x + y + (x + y + z) = 0 = 2x + y = 0 because x + y + z = 0 y = - 2x Delta S1 = x + y + z = 0 = x - 2x + z = 0 = -x + z z = x So an undetectable error is created when in three consecutive values the errors are x -2x x This could for instance be errors of 1, -2, and 1. Since the error to the first and third values are the same, the error must span at least 2 * the length of the value + 1. So for Adler, it must be a 17 bit burst at least. Also, note that this can be created by a 3 bit error and is equally applicable to Adler and Fletcher. Therefore, for messages shorter than the modulus + 1, the Hamming distance of Adler32 and Fletcher 32 is 3 and, for messages longer than that, the Hamming distance is 2. Regards, Pat Thaler ---------- Undetectable 3 bit errors in non-consecutive values --------------- For undetectable errors in 3 values spaced with k1 from the first error to second error and k2 values from the second error to the third: Delta S1 = (x + y + z) mod m = 0 Delta S2 = ((k1 + k2 + 1)x + (k2 + 1)y + z) mod m = 0 = ((k1 + k2)x + (k2)y +(x + y + z)) mod m = 0 = ((k1 + k2)x + (k2)y) mod m = 0 because (x + y + z) mod m = 0 y = - x(k1 + k2)/k2 z = -x - y = - x + x(k1 + k2)/k2 = x (-k2 + k1 + k2)/k2 = x(k2/k1) For k1 = k2, the result is the same as for 3 consecutive values. That is: y = -2x z = x Therefore, if x is 1, 2, ... 64 (for Adler32) or 1, 2, ... 16384 (for Fletcher32), then x, y, and z will be single bit errors. From owner-ips@ece.cmu.edu Tue Mar 13 00:48:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA01250 for ; Tue, 13 Mar 2001 00:48:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2D46n100358 for ips-outgoing; Mon, 12 Mar 2001 23:06:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraaa.compuserve.com (ds-img-rel-1.compuserve.com [149.174.206.140]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2D45kr00251 for ; Mon, 12 Mar 2001 23:05:46 -0500 (EST) Received: (from mailgate@localhost) by spdmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id XAA13399 for ips@ece.cmu.edu; Mon, 12 Mar 2001 23:05:41 -0500 (EST) Received: from compuserve.com (dal-tgn-tkt-vty48.as.wcom.net [216.192.231.48]) by spdmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id XAA13356 for ; Mon, 12 Mar 2001 23:05:33 -0500 (EST) Message-ID: <3AAD9D64.76CD9AB6@compuserve.com> Date: Mon, 12 Mar 2001 22:09:08 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: iscsi: rev 05 - section 2.5.1 is incomplete Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit The following statement is only half true: For the , if the SCSI control mode enables AE reporting, the target MUST send an Asynchronous Event to all other attached initiators to inform them that all pending tasks are cancelled. If the SCSI control mode [page] does not enable AE reporting, then the target MUST establish a unit attention condition for all other attached initiators. In fact, the target must establish a unit attention condition regardless of the contents of the SCSI control mode page. At this is specified in SAM-2. What the control mode pages does is tell the target how to report that unit attention condition; via a Check Condition on the next command received, or via an asynchronous event report. Someone might read the iSCSI version 05 text as releasing them from the SAM-2 requirement, which I doubt will be helpful to the SCSI community at large. The same problem exists in the following 2.5.1 text: For the and functions, the target cancels all pending operations and are both equivalent to the Target Reset as specified by SAM-2. Provided that SCSI control mode enables AE reporting, the target MUST send an Asynchronous Event to all attached initiators notifying them that the target is being reset. FYI Ralph... From owner-ips@ece.cmu.edu Tue Mar 13 00:49:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA01261 for ; Tue, 13 Mar 2001 00:49:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2D3smN29622 for ips-outgoing; Mon, 12 Mar 2001 22:54:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraaa.compuserve.com (ds-img-rel-1.compuserve.com [149.174.206.140]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2D3sQr29607 for ; Mon, 12 Mar 2001 22:54:26 -0500 (EST) Received: (from mailgate@localhost) by spdmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id WAA11773 for ips@ece.cmu.edu; Mon, 12 Mar 2001 22:54:21 -0500 (EST) Received: from compuserve.com (dal-tgn-tkt-vty48.as.wcom.net [216.192.231.48]) by spdmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id WAA11764 for ; Mon, 12 Mar 2001 22:54:17 -0500 (EST) Message-ID: <3AAD9AC0.80A94C4F@compuserve.com> Date: Mon, 12 Mar 2001 21:57:52 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: iscsi: rev 05 (2.18, Asynchronous Events) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit There's a typo in 2.18: Some Asynchronous Messages are strictly related to iSCSI while others are related to SCSI [SAM-2}. ... Probably should be "[SAM-2]". The following list is at best misguided and at worst totally misleading. 2.18.2 SCSI Event The following values are defined. (See [SAM2] for details): 1 An error condition was encountered after command completion. 2 A newly initialized device is available to this initiator. 3 Some other type of unit attention condition has occurred. 4 An asynchronous event has occurred. There are either hundreds of entries in this list (one for every possible unit attention condition in SCSI), or there are two: 1 A SCSI asynchronous event is NOT reported in the sense data, or 2 A SCSI asynchronous event IS reported in the sense data. Picking a few favorite conditions to report in this way suggests that they are the only ones that need to be reported this way, and if the Control mode page is set correctly that is an absolutely bogus assertion. Note also that what ever asynchronous condition exists is fully described by the sense data, leaving one to ask what a recipient is to do if the SCSI Event data fails to match the sense data. This needs to be fixed and I strongly recommend the binary-state solution. Thanks. Ralph... From owner-ips@ece.cmu.edu Tue Mar 13 00:51:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA01284 for ; Tue, 13 Mar 2001 00:51:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2D4DpE00764 for ips-outgoing; Mon, 12 Mar 2001 23:13:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraaa.compuserve.com (ds-img-rel-1.compuserve.com [149.174.206.140]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2D4DSr00749 for ; Mon, 12 Mar 2001 23:13:28 -0500 (EST) Received: (from mailgate@localhost) by spdmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id XAA14281 for ips@ece.cmu.edu; Mon, 12 Mar 2001 23:13:23 -0500 (EST) Received: from compuserve.com (dal-tgn-tkt-vty48.as.wcom.net [216.192.231.48]) by spdmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id XAA14274 for ; Mon, 12 Mar 2001 23:13:10 -0500 (EST) Message-ID: <3AAD9F2D.9F9B8673@compuserve.com> Date: Mon, 12 Mar 2001 22:16:45 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: iscsi: rev 05 2.5.1 requires ACA support Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit The following two 2.5.1 statements require the use of ACA and there is no way for an initiator that doesn't implement ACA to stop them. For the , ... The target MUST then enter the ACA state for any initiator for which it had pending tasks. In addition, for the the target enters the ACA state on all sessions and all LUs on which an AE was sent. Have you all signed the operating system vendors up for rewrites of their class drivers to support ACA? If not anticipate some serious deadlocks when iSCSI is first attempted with a non ACA class driver, since the driver will never clear these ACA conditions and no progress will be made after that. Alternatively, a way can be found to give the initiator control over the "MUST" statements. Thanks. Ralph... From owner-ips@ece.cmu.edu Tue Mar 13 03:36:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA17335 for ; Tue, 13 Mar 2001 03:36:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2D6bn608939 for ips-outgoing; Tue, 13 Mar 2001 01:37:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2D6b9r08909 for ; Tue, 13 Mar 2001 01:37:10 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA36630 for ; Tue, 13 Mar 2001 07:36:58 +0100 From: julian_satran@il.ibm.com Received: from d12mta05.de.ibm.com (d12mta06_cs0 [9.165.222.255]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id HAA178154 for ; Tue, 13 Mar 2001 07:35:49 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0E.002453D7 ; Tue, 13 Mar 2001 07:36:47 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: pat_thaler@agilent.com cc: ips@ece.cmu.edu, Dafna_Sheinwald@il.ibm.com Message-ID: Date: Tue, 13 Mar 2001 08:37:09 +0200 Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Pat, Thanks a lot for you patience and hard work. I assumed that was the case based on the experimental results on the effects of ATM splicing on CRC and checksums published in 1998 that show how data bias affects checksums. On another subject - can you comment on weighted checksums? Thanks, Julo pat_thaler@agilent.com on 13/03/2001 04:43:51 Please respond to pat_thaler@agilent.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: pat_thaler@agilent.com, Dafna_Sheinwald@il.ibm.com Subject: RE: I-D ACTION:draft-cavanna-iscsi-crc-vs-cksum-00.txt Julian asked: "Do you have a formula to evaluate Pud for low noise channels with BER = e? Any formula to evaluate the conditional probability of an undetected error on bursts of 17, 18, 19 bits? Is there any reason to believe that we can approximate them to ones used for CRC?" Abstract: The following will show that the formula used in the case of CRC for Pud does not apply to checksums. Many data sets with small differences produce the same checksum while few such data sets produce the same CRC. Therefore, Pud for checksum is significantly worse than that for CRC. For independent bit errors, Pud is approximately 12,000 times worse for Fletcher32 than for CRC and 22,000 times worse for Adler32 than for CRC. Note: While Fletcher32 is better than Adler32 for the error case analyzed here, Fletcher32 has more susceptibility to two byte errors that Adler32 does not have. Overall, Fletcher32 is not necessarily a better choice. The grungy details: The formula for Pud for CRC in a low noise channel is: Pud = Pcrc * Cn,x * Pxerrors Where: Pcrc is the probability that the received CRC matches the errored message Cn,x is the number of combinations of 3 errors in an n bit message Pxerrors is the probability of getting a particular combination of 3 errors Pbe is the probability of a bit error. Inserting the values for 3 bit errors in a 1000 byte message where Pbe << 1: Pud = 2^-32 * C8000,3 * Pbe^3 = 2^-32 * 85e9 * Pbe^3 = 20 * Pbe^3 Note that this equation is assuming that the CRC in question does not detect all 3 bit errors. If the CRC detects all 3 bit errors as many do, the Pud would be 0. Also, actually equals Pxerrors = (Pbe ^ errored_bits) * ((1 - Pbe) ^ unerrored_bits, but the latter is very close to 1 for low noise channels. Approximately 20 positions of 3 bit errors cause undetectable errors in a 1000 byte message with the CRC. How many 3 bit errors cause undetectable errors for checksums? In the attached analysis of checksum Hamming distance and burst error detection, I showed that an undetectable error for both checksums is formed by sequence of three errored values with errors: x, -2x, and x. The same error pattern in three values with equal spacing between them will cause an undetectable error (see Undetectable 3 bit errors in non-consecutive values below). For Adler32: A value is one byte long. For x = 1, 2, 4, 8, 16, 32 or 64: The three errors can be caused by a 1 bit change each. There are 998 sets of 3 consecutive bytes in a 1000 byte frame. There are 996 sets of 3 bytes spaced with 1 bytes between them. There are 994 sets of 3 bytes spaced with 2 bytes between them. .... There are 2 sets of 3 bytes spaced with 498 bytes between them. Therefore, the undetectable errors analyzed below yield (998 + 2) * 49 / 2 sets of bytes that can have one of 7 sets of 3 bits change for an undetectable error in a 1000 byte frame. Of the 8 ways those three bits can be changed, 2 of them produce an undetectable error. If the pattern discussed here is the only one that produces 3 bit errors, then Pud for 3 bit errors in a 1000 byte message with Adler32 would be: Pud = Pcsum * Cerror * P3errors where Pcsum equals the probability that the 3 bit error results in a correct checksum which is 0.25 for the 3 bit errors. Cerror is the number of combinations of undetectable 3 bit errors. P3errors is the probability of getting a particular combination of 3 errors. Pud = 0.25 * (998 + 2) * 499/2 * 7 * Pbe^3 = 437 * 10^3 * Pbe^3 For Fletcher32: A value is two bytes long. For x = 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192 or 16384: The three errors can be caused by a 1 bit change each. In a 1000 byte message, there are 500 values. There are 498 sets of 3 consecutive values in a 1000 byte frame. There are 496 sets of 3 values spaced with 1 values between them. There are 494 sets of 3 values spaced with 2 values between them. .... There are 2 sets of 3 values spaced with 248 values between them. Therefore, the undetectable errors analyzed below yield (498 + 2) * 249/ 2 sets of bytes that can have one of 15 sets of 3 bits change for an undetectable error in a 1000 byte frame. Of the 8 ways those three bits can be changed, 2 of them produce an undetectable error. If the pattern discussed here is the only one that produces 3 bit errors, then Pud for 3 bit errors in a 1000 byte message with Fletcher32 would be: Pud = 0.25 (498 + 2) * 249/2 * 15 * Pbe^3 = 233 * 10^3 * Pbe^3 To recap: CRC Pud = 20 * Pbe^3 Adler32: Pud = 437 * 10^3 * Pbe^3 Fletcher32: Pud = 233 * 10^3 * Pbe^3 Regards, Pat Thaler ------- Analysis of checksum Hamming distance and burst error detection ----------- It is relatively easy to investigate the Hamming distance and burst detection of Adler32 and Fletcher32 using mathematics. In the argument below, I will use "value" to identify the quantities that get summed since Adler sums bytes and Fletcher32 sums 16-bits. Both Adler and Fletcher calculate for each value: S1 = (S1 + value) mod m S2 = (S2 + S1) mod m where m is 65521 for Adler and 65535 for Fletcher. Which means that for an n value string of values V1 to Vn: S1 = (V1 + V2 + ... + Vn (+ 1 for Adler)) mod m S2 = (n * V1 + (n-1) * V2 + ... + Vn (+ n for Adler)) mod m A two bit error can only change two values. For a two bit error to be undetectable the two changed values must produce no change to S1 and S2. Let x be the change to the first errored value y be the change to the second errored value k be the separation between the first and second errored values. Then for an undetectable error: delta S1 = (x + y) mod m = 0 delta S2 = ((k + 1) * x + y) mod m = 0 = (k * x + x + y) mod m =0 = (k * x) mod m = 0 because (x + y) mod m = 0 Therefore, for an undetectable 2 bit error, k * x must be a multiple of m. For Adler, m is prime and the maximum value of x is 255 so the only way to satisfy the condition is for k to equal 65521. Then x can be any value so it could be a one bit error such as changing the lsb of the first value from 0 to 1. y would then need to be the inverse changing the lsb of the second value from 1 to 0. For Fletcher, m is factorable to 3, 5, 17, 257. For x to provide one or more of those factors, it would have to have more than a 1 bit change and y would have to have the inverse of that change which would mean an error of at least 4 bits. The only way to get a two bit undetectable error is for k to equal 65535. So, Fletcher and Adler do have 2-bit undetectable errors if the messages are longer than the modulus. What about burst protection? For Fletcher, we know that a change of a value from all 0's to all 1's produces an undetectable error so we know that burst protection is less than 16. k = 1 for a burst spread across two consecutive values. Therefore, the only way for a burst spread across two consecutive values to produce an undetectable error is for x to equal 65535. A burst shorter than 16 can't do it. For Adler, corrupting two consecutive bytes can't produce an undetectable error because that would be the case above where k equals 1. There is no way for a burst across two consecutive values to be undetectable in Adler since x is less than the modulus. For a burst across three values: let x equal the error in the first value y equal the error in the second value z equal the error in the third value. For the burst to be undetectable Delta S1 = (x + y + z) mod m = 0 Delta S2 = (3x + 2y + z) mod m = 0 Since the maximum values of x, y, and z are 255 and m is more than 6 times that, the sums above will always be less than m and we can drop the mod m. Delta S2 = (3x + 2y + z) = 2x + y + (x + y + z) = 0 = 2x + y = 0 because x + y + z = 0 y = - 2x Delta S1 = x + y + z = 0 = x - 2x + z = 0 = -x + z z = x So an undetectable error is created when in three consecutive values the errors are x -2x x This could for instance be errors of 1, -2, and 1. Since the error to the first and third values are the same, the error must span at least 2 * the length of the value + 1. So for Adler, it must be a 17 bit burst at least. Also, note that this can be created by a 3 bit error and is equally applicable to Adler and Fletcher. Therefore, for messages shorter than the modulus + 1, the Hamming distance of Adler32 and Fletcher 32 is 3 and, for messages longer than that, the Hamming distance is 2. Regards, Pat Thaler ---------- Undetectable 3 bit errors in non-consecutive values --------------- For undetectable errors in 3 values spaced with k1 from the first error to second error and k2 values from the second error to the third: Delta S1 = (x + y + z) mod m = 0 Delta S2 = ((k1 + k2 + 1)x + (k2 + 1)y + z) mod m = 0 = ((k1 + k2)x + (k2)y +(x + y + z)) mod m = 0 = ((k1 + k2)x + (k2)y) mod m = 0 because (x + y + z) mod m = 0 y = - x(k1 + k2)/k2 z = -x - y = - x + x(k1 + k2)/k2 = x (-k2 + k1 + k2)/k2 = x(k2/k1) For k1 = k2, the result is the same as for 3 consecutive values. That is: y = -2x z = x Therefore, if x is 1, 2, ... 64 (for Adler32) or 1, 2, ... 16384 (for Fletcher32), then x, y, and z will be single bit errors. From owner-ips@ece.cmu.edu Tue Mar 13 03:36:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA17346 for ; Tue, 13 Mar 2001 03:36:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2D6NnO08159 for ips-outgoing; Tue, 13 Mar 2001 01:23:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2D6Mvr08128 for ; Tue, 13 Mar 2001 01:22:57 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA179224 for ; Tue, 13 Mar 2001 07:22:49 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id HAA163006 for ; Tue, 13 Mar 2001 07:21:40 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0E.0022DF27 ; Tue, 13 Mar 2001 07:20:53 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 13 Mar 2001 08:23:06 +0200 Subject: Re: iSCSI: Towards a more effective PDU format Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, Interesting. This is close to one of the variants I had for IETF-50 (a clear header). The only adavantage it has is that you are able to read all the AHSs in one read. The (admitedly academic) disadvantage it has is that you are limited in the size of extensions and have some redundancy. The basic issue that was raised - and there is no simple way out - is that once you have lost a block (BHS in your suggested layout) you are out of synch. There is no simple way around it (at least not one that can be solved by changing layout) and an added digest (only the code or silicon to account for it) is IMHO not warranted by what you gain. If you are willing to spend silicon or code on this the making header failures less probable and recovering if you fail by dropping the connection is a better bet (using the redundancy for a coding gain). But this involves some complexity too. Julo "Robert D. Russell" on 12/03/2001 22:01:39 Please respond to "Robert D. Russell" To: ips@ece.cmu.edu cc: bbrtrebia@mediaone.net, "Ashish P. Palekar" , Anshul Chadda , Narendran Ganapathy Subject: Re: iSCSI: Towards a more effective PDU format All: Here is a proposal for an iSCSI PDU format without WN Next-Qualifiers: 1. Every PDU must start with a 48-byte Basic Header Segment (BHS). Assuming header digests are in use, this BHS is immediately followed by a header digest that covers it. 2. Every BHS contains 2 fields at fixed locations: a) AHS-length, containing the total length of all Additional Header Segments (AHS) that follow. This field is 0 if there are no AHSs. b) DATA-length, containing the total length of all Data that follows the AHSs. This field is 0 if there is no data. 3. If the AHS-length field in the BHS is non-zero, all the AHSs immediately follow the header digest of the BHS. An additional digest immediately follows that last AHS and covers all the AHSs. 4. If the DATA-length field in the BHS is non-zero, all the data immediately follows the AHS digest. A data digest immediately follows the data. 5. If space within the BHS is an issue, the AHS-length field could be restricted to a single byte, and could be in units of 4-byte words. This would allow up to 1020 bytes of AHSs to follow a BHS, and this seems to be more than enough for the uses foreseen (so far, only extended CDB and bidi-read info). 6. Each AHS should start with a word containing a TYPE field and a LENGTH field. The TYPE field should be enumerated rather than bit-field encoded, for easier decoding and future expansion. The LENGTH field is the number of bytes of additional information in this AHS that follows this word. Advantages of this proposal. 1. The receiver always knows where to find digests in a reliable manner, without having to use unreliable data to find them. 2. Because the BHS is fixed length, the receiver can reliably obtain the entire BHS and its header digest in a single "read" operation. 3. Because the receiver gets the AHS-length field from the BHS, it can reliably obtain the entire set of AHSs and the header digest covering them in a single "read" operation. 4. Because the receiver gets the DATA-length field from the BHS, it can reliably obtain all the data and the data digest covering it in a single "read" operation. 5. Since each AHS begins with a word containing the TYPE and LENGTH in fixed positions, an unknown TYPE (i.e., a type introduced in the future that is received by a legacy receiver) does NOT result in loss of synchronization during header processing -- the receiver knows the length of this AHS in any case, and can just skip over it. Disadvantages of this proposal. 1. A second header digest is required, but only if one or more AHSs are present. In practice this should be a rare event. Proposed iSCSI PDU Format +-------------+ | 48-byte BHS | +-------------+ | BHS digest | --+-------------+-- | AHS 1 |\ | - - - - - - | \ | AHS 2 | \ | - - - - - - | > total length in AHS_length field in BHS | . . . | / | - - - - - - | / | AHS n |/ +-------------+ | AHSs digest | --+-------------+-- | |\ | Data | > total length in DATA_length field in BHS | |/ +-------------+ | Data digest | +-------------+ Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 From owner-ips@ece.cmu.edu Tue Mar 13 03:43:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA17398 for ; Tue, 13 Mar 2001 03:43:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2D6usf09992 for ips-outgoing; Tue, 13 Mar 2001 01:56:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2D6uGr09961 for ; Tue, 13 Mar 2001 01:56:16 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA76334; Tue, 13 Mar 2001 07:56:06 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id HAA70292; Tue, 13 Mar 2001 07:54:09 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0E.0025EB55 ; Tue, 13 Mar 2001 07:54:10 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Sandeep Joshi cc: ips@ece.cmu.edu Message-ID: Date: Tue, 13 Mar 2001 08:56:25 +0200 Subject: Re: [Fwd: comments on naming&discovery draft] Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I am not sure. There are some SCSI items in it too (SCSI handles now the appearnce of new LUs). I will need a longer discussion with NDT to understand the semantics. Julo Sandeep Joshi on 12/03/2001 22:25:27 Please respond to Sandeep Joshi To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: [Fwd: comments on naming&discovery draft] Julian, in case you skip this one.. your response is required on point (1) for amending iSCSI draft. -sandeep Received: from scummy.research.bell-labs.com (guard.research.bell-labs.com [135.104.2.10]) by aura.research.bell-labs.com (8.9.1/8.9.1) with SMTP id KAA14926 for ; Mon, 12 Mar 2001 10:59:06 -0500 (EST) Received: from dusty.research.bell-labs.com ([135.104.2.7]) by scummy; Mon Mar 12 10:58:53 EST 2001 Received: from dogwood.cisco.com ([161.44.11.19]) by dusty; Mon Mar 12 10:58:52 EST 2001 Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id KAA17444; Mon, 12 Mar 2001 10:58:51 -0500 (EST) Sender: mbakke@cisco.com Message-ID: <3AACF243.9C30D52C@cisco.com> Date: Mon, 12 Mar 2001 09:58:59 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Sandeep Joshi CC: ips@ece.cmu.edu Subject: Re: comments on naming&discovery draft References: <200103102223.RAA03513@aura.research.bell-labs.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii Sandeep- The problem you pointed out in item number 1 creates the need for an additional iSCSI-level event. Since the discovery of targets happens at the iSCSI level, rather than at the SCSI level, how about adding this to 2.18.1 (in iSCSI-05)? 4 Network entity indicates that a "target discovery" event has occurred. Upon receiving this message, the initiator should use SendTargets, or whatever other methods of discovery it is using, to find out what has changed. Usually, this would be due to adding a new target. We will fix items 2-4; thanks for pointing them out. Thanks, Mark Sandeep Joshi wrote: > > 1) Section 4.2 last line before Section 4.2.1 > "target MUST send any iSCSI-level async on this session, > allowing the initiator to discover new targets.." > > The session mentioned here is a session to the canonical target. > > However, the iSCSI 05 draft does not mention any such condition > in Sec 2.18 on Async Message. In there, a SCSI event (note: not > iSCSI) is used to notify availability of new targets. > > 2) Appendix C, Section 5 "Stateful Inspection Firewall" > It contains statements of the sort "I dont expect/think.." > These statements could be rephrased to be impersonal. I will fix these. > 3) Section 4.3 Middle of page 17 (After reference to RFC 2608) > -> "A target can register either its canonical target, ..." > > Multiple references to term "target" is confusing. It can be > changed to refer to the term "Network entity" introduced earlier > in the document. > -> "A network entity can register its canonical target,.." I will fix this as well. > 4) Sec 4.2 SendTargets Command > The status code 0x42 mentioned is now 0x02 in iSCSI document. > A similar mismatch exists in some codes listed in B.4.5. > > It might help to just use error names and not numbers to avoid > this problem. [readers can do error code "discovery"! ] Good idea; we will fix this, too. > -Sandeep -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Tue Mar 13 04:23:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA17578 for ; Tue, 13 Mar 2001 04:23:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2D7VnZ11937 for ips-outgoing; Tue, 13 Mar 2001 02:31:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2D7Uwr11903 for ; Tue, 13 Mar 2001 02:30:58 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id IAA91712 for ; Tue, 13 Mar 2001 08:30:51 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA103884 for ; Tue, 13 Mar 2001 08:28:54 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0E.00291882 ; Tue, 13 Mar 2001 08:28:52 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 13 Mar 2001 09:31:08 +0200 Subject: Re: iscsi: rev 05 - section 2.5.1 is incomplete Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Will fix. It should read: For the , the target MUST enter a Unit Attention Condition for all other attached initiators to inform them that all pending tasks are cancelled. After reporting the Unit Attention Condition the target MUST enter the ACA state for any initiator for which it had pending tasks. For the and functions, the target cancels all pending operations and are both equivalent to the Target Reset as specified by SAM-2. The target MUST enter a Unit Attention Condition for all attached initiators notifying them that the target is being reset. In addition, for the , after reporting the Unit Attention Condition, the target enters the ACA state on all sessions and all LUs on which the condition was reported. In addition, for the the target then MUST terminate all of its TCP connections to all initiators (all sessions are terminated). However, if the target finds that it cannot send the required response or AEN, it MUST continue the reset operation and it SHOULD log the condition for later retrieval. The logging operation MUST be reported through the target MIB. Further actions on reset functions are specified in the relevant SCSI documents for the specific class of devices. Thanks, Julo Ralph Weber on 13/03/2001 06:09:08 Please respond to ENDL_TX@computer.org To: IPS Reflector cc: Subject: iscsi: rev 05 - section 2.5.1 is incomplete The following statement is only half true: For the , if the SCSI control mode enables AE reporting, the target MUST send an Asynchronous Event to all other attached initiators to inform them that all pending tasks are cancelled. If the SCSI control mode [page] does not enable AE reporting, then the target MUST establish a unit attention condition for all other attached initiators. In fact, the target must establish a unit attention condition regardless of the contents of the SCSI control mode page. At this is specified in SAM-2. What the control mode pages does is tell the target how to report that unit attention condition; via a Check Condition on the next command received, or via an asynchronous event report. Someone might read the iSCSI version 05 text as releasing them from the SAM-2 requirement, which I doubt will be helpful to the SCSI community at large. The same problem exists in the following 2.5.1 text: For the and functions, the target cancels all pending operations and are both equivalent to the Target Reset as specified by SAM-2. Provided that SCSI control mode enables AE reporting, the target MUST send an Asynchronous Event to all attached initiators notifying them that the target is being reset. FYI Ralph... From owner-ips@ece.cmu.edu Tue Mar 13 04:23:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA17589 for ; Tue, 13 Mar 2001 04:23:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2D7DnI10889 for ips-outgoing; Tue, 13 Mar 2001 02:13:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2D7Dbr10880 for ; Tue, 13 Mar 2001 02:13:38 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id IAA49592 for ; Tue, 13 Mar 2001 08:13:31 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA140948 for ; Tue, 13 Mar 2001 08:12:21 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0E.0027845C ; Tue, 13 Mar 2001 08:11:37 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 13 Mar 2001 09:13:54 +0200 Subject: Re: iscsi: rev 05 (2.18, Asynchronous Events) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk The new text reads: 1.1.1 iSCSI Event The codes used for iSCSI Asynchronous Messages (Events) are: 0 No iSCSI Event 1 Target is being reset. 2 Target requests Logout - the Parameter1 field indicates on what CID. 3 Target indicates it will/has dropped the connection - the Parameter1 field will indicate on what CID while the Parameter2 field indicates, in seconds, the minimum time to reconnect. 1.1.2 SCSI Event The following values are defined. (See [SAM2] for details): 0 No SCSI Asynchronous Event is reported. 1 A SCSI Asynchronous Event is reported in the sense data. Sense Data that accompanies the report identifies the condition. The Length parameter is set to the length of the Sense Data. Thanks, Julo Ralph Weber on 13/03/2001 05:57:52 Please respond to ENDL_TX@computer.org To: IPS Reflector cc: Subject: iscsi: rev 05 (2.18, Asynchronous Events) There's a typo in 2.18: Some Asynchronous Messages are strictly related to iSCSI while others are related to SCSI [SAM-2}. ... Probably should be "[SAM-2]". The following list is at best misguided and at worst totally misleading. 2.18.2 SCSI Event The following values are defined. (See [SAM2] for details): 1 An error condition was encountered after command completion. 2 A newly initialized device is available to this initiator. 3 Some other type of unit attention condition has occurred. 4 An asynchronous event has occurred. There are either hundreds of entries in this list (one for every possible unit attention condition in SCSI), or there are two: 1 A SCSI asynchronous event is NOT reported in the sense data, or 2 A SCSI asynchronous event IS reported in the sense data. Picking a few favorite conditions to report in this way suggests that they are the only ones that need to be reported this way, and if the Control mode page is set correctly that is an absolutely bogus assertion. Note also that what ever asynchronous condition exists is fully described by the sense data, leaving one to ask what a recipient is to do if the SCSI Event data fails to match the sense data. This needs to be fixed and I strongly recommend the binary-state solution. Thanks. Ralph... From owner-ips@ece.cmu.edu Tue Mar 13 05:38:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA17954 for ; Tue, 13 Mar 2001 05:38:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2D8FqA14313 for ips-outgoing; Tue, 13 Mar 2001 03:15:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2D8FMr14268 for ; Tue, 13 Mar 2001 03:15:22 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA162702 for ; Tue, 13 Mar 2001 09:15:15 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id JAA84478 for ; Tue, 13 Mar 2001 09:14:05 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0E.002D2A5F ; Tue, 13 Mar 2001 09:13:19 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 13 Mar 2001 10:15:35 +0200 Subject: Re: iSCSI : Section 4. Login Phase. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk answers in text - Julo Santosh Rao on 13/03/2001 03:52:57 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iSCSI : Section 4. Login Phase. Julian & All, A couple of issues on Section 4 of the rev 05 iSCSI draft : 1) "The Login Final-Response can come only as a response to a Login command with the F bit set to 1 or a Text Command with the F bit set to 1." The above statement is not true and must be removed, since the login-final response is used by targets when rejecting login. As per the table in Section 2.11.4, the F bit is allowed to be set to 1 when the status class is non-zero. +++ will fix wording +++ 2) "Operational parameters MAY be negotiated within or outside (after) the login phase." If text negotiation can be conducted outside of the login phase, what is the effect of changes in text keys on the existing login ? For ex, how does changes to keys like MaxConnections, UseR2T, BidiUseR2T, ImmediateData, DataPDULength, FirstBurstSize, ITagLength, etc affect their existing values negotiated during the current login ? The side-effects of such changes and the scope of their establishment must be documented. It may be simplest to prohibit such changes outside of the login phase, for the sake of simplicity. +++ most of the parameters you mentioned have the LO qualifier - i.e., they can't be changed after the Login that establishes the session. Some are not as I though that theuser may want them connection specific or even change them (e.g., R2T) +++ Regards, Santosh - santoshr.vcf From owner-ips@ece.cmu.edu Tue Mar 13 07:44:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA18926 for ; Tue, 13 Mar 2001 07:44:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2DAJqE29893 for ips-outgoing; Tue, 13 Mar 2001 05:19:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2DAJJr29875 for ; Tue, 13 Mar 2001 05:19:19 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id FAA236800 for ; Tue, 13 Mar 2001 05:12:05 -0500 Received: from f4n49e (d03nm065h.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id DAA122692 for ; Tue, 13 Mar 2001 03:19:16 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: iscsi: rev 05 - section 2.5.1 is incomplete To: "Julian Satran" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Tue, 13 Mar 2001 02:18:54 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/13/2001 03:19:16 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, based on the response that we got from Robert Elliott @ Compaq, we should probably at least change the draft to remove the Target Reset (Target Warm Reset and Target Cold Reset). This is good news, since these things would have been to much of a problem for iSCSI to permit. I do not really like the Logical Unit Resets either, but at least it should only effect the device you have a right to use, so the impact is a lot less of a problem there, and I think we can live with that. Rober said: Today, T10 approved 01-015r2. For devices compliant with SAM-2 revision 16, Logical Unit Reset is now mandatory and Target Reset is not. Protocols are allowed to mandate Target Reset, make it optional, or not support it at all. In response, the SCSI RDMA protocol working group (defining SCSI over InfiniBand) voted to remove Target Reset support from that protocol. See T10 document 01-108r0 for the (short) proposal. I recommend you do the same and remove both Target Warm Reset and Target Cold Reset task management functions from iSCSI. Other IP protocols don't offer similar resets - a web browser doesn't have a way to reset an http server, a NFS client cannot reset an NFS server, a telnet user cannot interfere with other telnet users. iSCSI shouldn't be any different. John's email mentioned 4 task management functions, which included Logical Unit Reset, and Clear Task Set. The Access Controls feature (by Jim Hafner; going into SPC-3 revision 1) can be used to prevent initiators from logging in and issuing Logical Unit Reset and Clear Task Set. .... --- Robert.Elliott@compaq.com Compaq Server Storage . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/12/2001 11:31:08 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iscsi: rev 05 - section 2.5.1 is incomplete Will fix. It should read: For the , the target MUST enter a Unit Attention Condition for all other attached initiators to inform them that all pending tasks are cancelled. After reporting the Unit Attention Condition the target MUST enter the ACA state for any initiator for which it had pending tasks. For the and functions, the target cancels all pending operations and are both equivalent to the Target Reset as specified by SAM-2. The target MUST enter a Unit Attention Condition for all attached initiators notifying them that the target is being reset. In addition, for the , after reporting the Unit Attention Condition, the target enters the ACA state on all sessions and all LUs on which the condition was reported. In addition, for the the target then MUST terminate all of its TCP connections to all initiators (all sessions are terminated). However, if the target finds that it cannot send the required response or AEN, it MUST continue the reset operation and it SHOULD log the condition for later retrieval. The logging operation MUST be reported through the target MIB. Further actions on reset functions are specified in the relevant SCSI documents for the specific class of devices. Thanks, Julo Ralph Weber on 13/03/2001 06:09:08 Please respond to ENDL_TX@computer.org To: IPS Reflector cc: Subject: iscsi: rev 05 - section 2.5.1 is incomplete The following statement is only half true: For the , if the SCSI control mode enables AE reporting, the target MUST send an Asynchronous Event to all other attached initiators to inform them that all pending tasks are cancelled. If the SCSI control mode [page] does not enable AE reporting, then the target MUST establish a unit attention condition for all other attached initiators. In fact, the target must establish a unit attention condition regardless of the contents of the SCSI control mode page. At this is specified in SAM-2. What the control mode pages does is tell the target how to report that unit attention condition; via a Check Condition on the next command received, or via an asynchronous event report. Someone might read the iSCSI version 05 text as releasing them from the SAM-2 requirement, which I doubt will be helpful to the SCSI community at large. The same problem exists in the following 2.5.1 text: For the and functions, the target cancels all pending operations and are both equivalent to the Target Reset as specified by SAM-2. Provided that SCSI control mode enables AE reporting, the target MUST send an Asynchronous Event to all attached initiators notifying them that the target is being reset. FYI Ralph... From owner-ips@ece.cmu.edu Tue Mar 13 09:50:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id JAA21160 for ; Tue, 13 Mar 2001 09:50:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2DCRsn06566 for ips-outgoing; Tue, 13 Mar 2001 07:27:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2DCR7r06539 for ; Tue, 13 Mar 2001 07:27:07 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id NAA172436 for ; Tue, 13 Mar 2001 13:27:00 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id NAA48988 for ; Tue, 13 Mar 2001 13:25:03 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0E.00443611 ; Tue, 13 Mar 2001 13:25:02 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 13 Mar 2001 14:27:19 +0200 Subject: Re: iscsi: rev 05 - section 2.5.1 is incomplete Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk John, I respectfully disagree (and I said so in the T10 meeting too). There are better ways to protect against Target Reset than to remove it from your command set. The Target Reset allows you to clean atomically all your LUs. If you are afraid about mistakes by unauthorized initiators we can leave it to the target to reject the command for unauthorized initiators. To obtain the same effect without Target Reset you will have to do a sequence of operations including a reservation for all the units and that might not work out. I bet that they will end-up having to put the same function back through some management interface (SNMP?) and then have to deal with the synchronization. I suggest we leave the commands in and invent a reject-response for initiators we don't want to allow them to do it based on the authentication and/or accessId. Regards, Julo John Hufferd@IBMUS 13/03/2001 12:18 To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: iscsi: rev 05 - section 2.5.1 is incomplete (Document link: Julian Satran - Mail) Julian, based on the response that we got from Robert Elliott @ Compaq, we should probably at least change the draft to remove the Target Reset (Target Warm Reset and Target Cold Reset). This is good news, since these things would have been to much of a problem for iSCSI to permit. I do not really like the Logical Unit Resets either, but at least it should only effect the device you have a right to use, so the impact is a lot less of a problem there, and I think we can live with that. Rober said: Today, T10 approved 01-015r2. For devices compliant with SAM-2 revision 16, Logical Unit Reset is now mandatory and Target Reset is not. Protocols are allowed to mandate Target Reset, make it optional, or not support it at all. In response, the SCSI RDMA protocol working group (defining SCSI over InfiniBand) voted to remove Target Reset support from that protocol. See T10 document 01-108r0 for the (short) proposal. I recommend you do the same and remove both Target Warm Reset and Target Cold Reset task management functions from iSCSI. Other IP protocols don't offer similar resets - a web browser doesn't have a way to reset an http server, a NFS client cannot reset an NFS server, a telnet user cannot interfere with other telnet users. iSCSI shouldn't be any different. John's email mentioned 4 task management functions, which included Logical Unit Reset, and Clear Task Set. The Access Controls feature (by Jim Hafner; going into SPC-3 revision 1) can be used to prevent initiators from logging in and issuing Logical Unit Reset and Clear Task Set. .... --- Robert.Elliott@compaq.com Compaq Server Storage . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/12/2001 11:31:08 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iscsi: rev 05 - section 2.5.1 is incomplete Will fix. It should read: For the , the target MUST enter a Unit Attention Condition for all other attached initiators to inform them that all pending tasks are cancelled. After reporting the Unit Attention Condition the target MUST enter the ACA state for any initiator for which it had pending tasks. For the and functions, the target cancels all pending operations and are both equivalent to the Target Reset as specified by SAM-2. The target MUST enter a Unit Attention Condition for all attached initiators notifying them that the target is being reset. In addition, for the , after reporting the Unit Attention Condition, the target enters the ACA state on all sessions and all LUs on which the condition was reported. In addition, for the the target then MUST terminate all of its TCP connections to all initiators (all sessions are terminated). However, if the target finds that it cannot send the required response or AEN, it MUST continue the reset operation and it SHOULD log the condition for later retrieval. The logging operation MUST be reported through the target MIB. Further actions on reset functions are specified in the relevant SCSI documents for the specific class of devices. Thanks, Julo Ralph Weber on 13/03/2001 06:09:08 Please respond to ENDL_TX@computer.org To: IPS Reflector cc: Subject: iscsi: rev 05 - section 2.5.1 is incomplete The following statement is only half true: For the , if the SCSI control mode enables AE reporting, the target MUST send an Asynchronous Event to all other attached initiators to inform them that all pending tasks are cancelled. If the SCSI control mode [page] does not enable AE reporting, then the target MUST establish a unit attention condition for all other attached initiators. In fact, the target must establish a unit attention condition regardless of the contents of the SCSI control mode page. At this is specified in SAM-2. What the control mode pages does is tell the target how to report that unit attention condition; via a Check Condition on the next command received, or via an asynchronous event report. Someone might read the iSCSI version 05 text as releasing them from the SAM-2 requirement, which I doubt will be helpful to the SCSI community at large. The same problem exists in the following 2.5.1 text: For the and functions, the target cancels all pending operations and are both equivalent to the Target Reset as specified by SAM-2. Provided that SCSI control mode enables AE reporting, the target MUST send an Asynchronous Event to all attached initiators notifying them that the target is being reset. FYI Ralph... From owner-ips@ece.cmu.edu Tue Mar 13 13:17:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA25212 for ; Tue, 13 Mar 2001 13:17:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2DFu2U19693 for ips-outgoing; Tue, 13 Mar 2001 10:56:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.bld.us.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2DFtbr19612 for ; Tue, 13 Mar 2001 10:55:37 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id JAA190550 for ; Tue, 13 Mar 2001 09:49:44 -0600 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id IAA206832 for ; Tue, 13 Mar 2001 08:55:24 -0700 Importance: Normal Subject: Re: iscsi: rev 05 - section 2.5.1 is incomplete To: "Julian Satran" Cc: ips@ece.cmu.edu From: "Jim Hafner" Date: Tue, 13 Mar 2001 07:55:22 -0800 Message-ID: X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/13/2001 07:55:24 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, You raise an interesting perspective on the desire to have an atomic target reset and to scope the use of that function to authorized initiators. Unfortunately, there isn't any way in either SAM or Access Controls to define the "initiators we don't want to allow...based on the authentication and/or accessId". Access controls grant access to a device and a subset of logical units. There is no "level of authority" that allows the target to distinguish a management entity from a non-management entity for any SAM task management function. SAM has no such notions. iSCSI could define such notions, but then why must the action be SAM's Target Reset (see "approaches" below). Also, I wasn't aware that a reservation was required to do a logical unit reset. Here's my perspective. 1) In the multi-initiator environments, the Target Reset type function (warm or cold) should be handled through management interfaces, not by any host with access to a part of the device. In fact, ANYTHING that impacts non-clustered/non-cooperating initiators in unsuspecting ways ought to have this property; this includes anything that smacks of management action. 2) There are many reasons for such a function (for example, to clear networking interfaces for the device, reboot, etc.), but it should NOT be just to clear task sets; it should be done with extreme care and by an entity that sees the "bigger picture". 3) The overhead of iterating a set of individual logical unit resets to clear task sets is minimal compared to the ill effects more global actions might have on other initiators. In short, I support disallowing Target Reset in iSCSI. If you want a "managed" form of this action, I see three approaches: a) define it as a iSCSI function (not a SAM function) and scope the rules on who/what/when/where/why. That's a big rat hole and is what you suggest will be required somewhere. b) define it as a SCSI Access Controls service action requiring the management key. c) leave it to the vendor-specific management interfaces. In all cases however, the function is NOT SAM's Target Reset Task Management. Jim Hafner Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03-13-2001 04:27:19 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iscsi: rev 05 - section 2.5.1 is incomplete John, I respectfully disagree (and I said so in the T10 meeting too). There are better ways to protect against Target Reset than to remove it from your command set. The Target Reset allows you to clean atomically all your LUs. If you are afraid about mistakes by unauthorized initiators we can leave it to the target to reject the command for unauthorized initiators. To obtain the same effect without Target Reset you will have to do a sequence of operations including a reservation for all the units and that might not work out. I bet that they will end-up having to put the same function back through some management interface (SNMP?) and then have to deal with the synchronization. I suggest we leave the commands in and invent a reject-response for initiators we don't want to allow them to do it based on the authentication and/or accessId. Regards, Julo John Hufferd@IBMUS 13/03/2001 12:18 To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: iscsi: rev 05 - section 2.5.1 is incomplete (Document link: Julian Satran - Mail) Julian, based on the response that we got from Robert Elliott @ Compaq, we should probably at least change the draft to remove the Target Reset (Target Warm Reset and Target Cold Reset). This is good news, since these things would have been to much of a problem for iSCSI to permit. I do not really like the Logical Unit Resets either, but at least it should only effect the device you have a right to use, so the impact is a lot less of a problem there, and I think we can live with that. Rober said: Today, T10 approved 01-015r2. For devices compliant with SAM-2 revision 16, Logical Unit Reset is now mandatory and Target Reset is not. Protocols are allowed to mandate Target Reset, make it optional, or not support it at all. In response, the SCSI RDMA protocol working group (defining SCSI over InfiniBand) voted to remove Target Reset support from that protocol. See T10 document 01-108r0 for the (short) proposal. I recommend you do the same and remove both Target Warm Reset and Target Cold Reset task management functions from iSCSI. Other IP protocols don't offer similar resets - a web browser doesn't have a way to reset an http server, a NFS client cannot reset an NFS server, a telnet user cannot interfere with other telnet users. iSCSI shouldn't be any different. John's email mentioned 4 task management functions, which included Logical Unit Reset, and Clear Task Set. The Access Controls feature (by Jim Hafner; going into SPC-3 revision 1) can be used to prevent initiators from logging in and issuing Logical Unit Reset and Clear Task Set. .... --- Robert.Elliott@compaq.com Compaq Server Storage . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/12/2001 11:31:08 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iscsi: rev 05 - section 2.5.1 is incomplete Will fix. It should read: For the , the target MUST enter a Unit Attention Condition for all other attached initiators to inform them that all pending tasks are cancelled. After reporting the Unit Attention Condition the target MUST enter the ACA state for any initiator for which it had pending tasks. For the and functions, the target cancels all pending operations and are both equivalent to the Target Reset as specified by SAM-2. The target MUST enter a Unit Attention Condition for all attached initiators notifying them that the target is being reset. In addition, for the , after reporting the Unit Attention Condition, the target enters the ACA state on all sessions and all LUs on which the condition was reported. In addition, for the the target then MUST terminate all of its TCP connections to all initiators (all sessions are terminated). However, if the target finds that it cannot send the required response or AEN, it MUST continue the reset operation and it SHOULD log the condition for later retrieval. The logging operation MUST be reported through the target MIB. Further actions on reset functions are specified in the relevant SCSI documents for the specific class of devices. Thanks, Julo Ralph Weber on 13/03/2001 06:09:08 Please respond to ENDL_TX@computer.org To: IPS Reflector cc: Subject: iscsi: rev 05 - section 2.5.1 is incomplete The following statement is only half true: For the , if the SCSI control mode enables AE reporting, the target MUST send an Asynchronous Event to all other attached initiators to inform them that all pending tasks are cancelled. If the SCSI control mode [page] does not enable AE reporting, then the target MUST establish a unit attention condition for all other attached initiators. In fact, the target must establish a unit attention condition regardless of the contents of the SCSI control mode page. At this is specified in SAM-2. What the control mode pages does is tell the target how to report that unit attention condition; via a Check Condition on the next command received, or via an asynchronous event report. Someone might read the iSCSI version 05 text as releasing them from the SAM-2 requirement, which I doubt will be helpful to the SCSI community at large. The same problem exists in the following 2.5.1 text: For the and functions, the target cancels all pending operations and are both equivalent to the Target Reset as specified by SAM-2. Provided that SCSI control mode enables AE reporting, the target MUST send an Asynchronous Event to all attached initiators notifying them that the target is being reset. FYI Ralph... From owner-ips@ece.cmu.edu Tue Mar 13 13:17:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA25223 for ; Tue, 13 Mar 2001 13:17:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2DGTJV22149 for ips-outgoing; Tue, 13 Mar 2001 11:29:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2DGSdr22103 for ; Tue, 13 Mar 2001 11:28:40 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id LAA214828 for ; Tue, 13 Mar 2001 11:21:24 -0500 Received: from f4n49e (d03nm065h.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id JAA35892 for ; Tue, 13 Mar 2001 09:28:34 -0700 X-Priority: 1 (High) Importance: Normal Subject: iSCSI: Target Resets are Management Functions To: "Julian Satran" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Tue, 13 Mar 2001 08:28:05 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/13/2001 09:28:34 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, Target Resets are management functions, that is where they belong, not as an iSCSI or SCSI action/command. It is not like this is going to be part of some automatic error recovery function. Target Resets need and deserve this function as part of administration management, it is up the vendors' products to perform, or not, this function, with information they get (or not) from an Admin interaction). I do not think we should support it in the normal iSCSI Protocols. And we should not have to specify the problem avoidance approaches that the implementer SHOULD/MUST take to support this function. You will probably say that it s an implementation decision as to how they avoid the problems, and that is what I am saying, and it belongs to the vendors' Admin function to implement or not. If it is a big enough problem for FC to take it out, with their limited network domain, we certainly should do that also. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/13/2001 04:27:19 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iscsi: rev 05 - section 2.5.1 is incomplete John, I respectfully disagree (and I said so in the T10 meeting too). There are better ways to protect against Target Reset than to remove it from your command set. The Target Reset allows you to clean atomically all your LUs. If you are afraid about mistakes by unauthorized initiators we can leave it to the target to reject the command for unauthorized initiators. To obtain the same effect without Target Reset you will have to do a sequence of operations including a reservation for all the units and that might not work out. I bet that they will end-up having to put the same function back through some management interface (SNMP?) and then have to deal with the synchronization. I suggest we leave the commands in and invent a reject-response for initiators we don't want to allow them to do it based on the authentication and/or accessId. Regards, Julo John Hufferd@IBMUS 13/03/2001 12:18 To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: iscsi: rev 05 - section 2.5.1 is incomplete (Document link: Julian Satran - Mail) Julian, based on the response that we got from Robert Elliott @ Compaq, we should probably at least change the draft to remove the Target Reset (Target Warm Reset and Target Cold Reset). This is good news, since these things would have been to much of a problem for iSCSI to permit. I do not really like the Logical Unit Resets either, but at least it should only effect the device you have a right to use, so the impact is a lot less of a problem there, and I think we can live with that. Rober said: Today, T10 approved 01-015r2. For devices compliant with SAM-2 revision 16, Logical Unit Reset is now mandatory and Target Reset is not. Protocols are allowed to mandate Target Reset, make it optional, or not support it at all. In response, the SCSI RDMA protocol working group (defining SCSI over InfiniBand) voted to remove Target Reset support from that protocol. See T10 document 01-108r0 for the (short) proposal. I recommend you do the same and remove both Target Warm Reset and Target Cold Reset task management functions from iSCSI. Other IP protocols don't offer similar resets - a web browser doesn't have a way to reset an http server, a NFS client cannot reset an NFS server, a telnet user cannot interfere with other telnet users. iSCSI shouldn't be any different. John's email mentioned 4 task management functions, which included Logical Unit Reset, and Clear Task Set. The Access Controls feature (by Jim Hafner; going into SPC-3 revision 1) can be used to prevent initiators from logging in and issuing Logical Unit Reset and Clear Task Set. .... --- Robert.Elliott@compaq.com Compaq Server Storage . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/12/2001 11:31:08 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iscsi: rev 05 - section 2.5.1 is incomplete Will fix. It should read: For the , the target MUST enter a Unit Attention Condition for all other attached initiators to inform them that all pending tasks are cancelled. After reporting the Unit Attention Condition the target MUST enter the ACA state for any initiator for which it had pending tasks. For the and functions, the target cancels all pending operations and are both equivalent to the Target Reset as specified by SAM-2. The target MUST enter a Unit Attention Condition for all attached initiators notifying them that the target is being reset. In addition, for the , after reporting the Unit Attention Condition, the target enters the ACA state on all sessions and all LUs on which the condition was reported. In addition, for the the target then MUST terminate all of its TCP connections to all initiators (all sessions are terminated). However, if the target finds that it cannot send the required response or AEN, it MUST continue the reset operation and it SHOULD log the condition for later retrieval. The logging operation MUST be reported through the target MIB. Further actions on reset functions are specified in the relevant SCSI documents for the specific class of devices. Thanks, Julo Ralph Weber on 13/03/2001 06:09:08 Please respond to ENDL_TX@computer.org To: IPS Reflector cc: Subject: iscsi: rev 05 - section 2.5.1 is incomplete The following statement is only half true: For the , if the SCSI control mode enables AE reporting, the target MUST send an Asynchronous Event to all other attached initiators to inform them that all pending tasks are cancelled. If the SCSI control mode [page] does not enable AE reporting, then the target MUST establish a unit attention condition for all other attached initiators. In fact, the target must establish a unit attention condition regardless of the contents of the SCSI control mode page. At this is specified in SAM-2. What the control mode pages does is tell the target how to report that unit attention condition; via a Check Condition on the next command received, or via an asynchronous event report. Someone might read the iSCSI version 05 text as releasing them from the SAM-2 requirement, which I doubt will be helpful to the SCSI community at large. The same problem exists in the following 2.5.1 text: For the and functions, the target cancels all pending operations and are both equivalent to the Target Reset as specified by SAM-2. Provided that SCSI control mode enables AE reporting, the target MUST send an Asynchronous Event to all attached initiators notifying them that the target is being reset. FYI Ralph... From owner-ips@ece.cmu.edu Tue Mar 13 14:31:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA26728 for ; Tue, 13 Mar 2001 14:31:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2DHQ0B26373 for ips-outgoing; Tue, 13 Mar 2001 12:26:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2DHPur26358 for ; Tue, 13 Mar 2001 12:25:57 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA40978 for ; Tue, 13 Mar 2001 18:25:47 +0100 From: julian_satran@il.ibm.com Received: from d12mta05.de.ibm.com (d12mta06_cs0 [9.165.222.255]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA113062 for ; Tue, 13 Mar 2001 18:24:37 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0E.005FBD54 ; Tue, 13 Mar 2001 18:25:43 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Chuck Micalizzi cc: ips@ece.cmu.edu Message-ID: Date: Tue, 13 Mar 2001 19:26:08 +0200 Subject: Re: inexistent LUN Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Chuk, You are right. I will remove this leftover. Julo Chuck Micalizzi on 13/03/2001 19:12:59 Please respond to Chuck Micalizzi To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: inexistent LUN Julian, Regarding version 5 of the iSCSI draft: On page 71, in Section 2.20 it states: It may happen that a target receives a message with a format error (e.g., inconsistent fields, reserved fields not 0, inexistent LUN etc.) or a digest error (e.g., invalid payload or header)... I'm concerned about the iSCSI layer having the responsibility of validating the LUN. In my opinion this is the responsibility of the SCSI layer. The target iSCSI layer should ignore the LUN field and let the SCSI layer validate the LUN. In SCSI, a command directed to a non-existent or missing LUN is not always an error. The SCSI Command Inquiry and Request Sense won't return "check condition" status if addressed to a non-existent LUN. chuck micalizzi From owner-ips@ece.cmu.edu Tue Mar 13 14:33:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA26868 for ; Tue, 13 Mar 2001 14:33:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2DHF1f25461 for ips-outgoing; Tue, 13 Mar 2001 12:15:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ntmail.qlc.com (216-231-2-8.qlc.com [216.231.2.8]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2DHEHr25423 for ; Tue, 13 Mar 2001 12:14:17 -0500 (EST) Received: by ntmail.qlc.com with Internet Mail Service (5.5.2653.19) id ; Tue, 13 Mar 2001 09:13:00 -0800 Message-ID: From: Chuck Micalizzi To: "'julian_satran@il.ibm.com'" Cc: ips@ece.cmu.edu Subject: inexistent LUN Date: Tue, 13 Mar 2001 09:12:59 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, Regarding version 5 of the iSCSI draft: On page 71, in Section 2.20 it states: It may happen that a target receives a message with a format error (e.g., inconsistent fields, reserved fields not 0, inexistent LUN etc.) or a digest error (e.g., invalid payload or header)... I'm concerned about the iSCSI layer having the responsibility of validating the LUN. In my opinion this is the responsibility of the SCSI layer. The target iSCSI layer should ignore the LUN field and let the SCSI layer validate the LUN. In SCSI, a command directed to a non-existent or missing LUN is not always an error. The SCSI Command Inquiry and Request Sense won't return "check condition" status if addressed to a non-existent LUN. chuck micalizzi From owner-ips@ece.cmu.edu Tue Mar 13 14:35:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA26913 for ; Tue, 13 Mar 2001 14:35:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2DHW3e26785 for ips-outgoing; Tue, 13 Mar 2001 12:32:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (mars.iol.unh.edu [132.177.121.222]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2DHVqr26744 for ; Tue, 13 Mar 2001 12:31:52 -0500 (EST) Received: from localhost (rdr@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id MAA28435; Tue, 13 Mar 2001 12:36:31 -0500 (EST) Date: Tue, 13 Mar 2001 12:36:31 -0500 From: "Robert D. Russell" To: julian_satran@il.ibm.com cc: ips@ece.cmu.edu Subject: Re: iSCSI: Towards a more effective PDU format In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo: Of course you are correct that once the BHS is lost, you are out of synch and have to drop the connection. However, there is more to this issue, and that is how quickly you can detect the failure and start recovery. In both my proposal and the current WN Next-Qualifier scheme, a PDU that is shortened due to loss of data can lead to a situation where the receiver blocks waiting for the rest of the PDU to arrive before it can be processed. The only way to detect this would seem to be to put timers on every read, so the read does not block indefinitely. When the timer goes off, the connection has to be dropped. However, in the WN Next-Qualifier scheme, the receiver can also block in situations where data is simply corrupted but not dropped, which is probably a more common situation. This will NOT happen in my proposal. For example, suppose there is a PDU consisting of a header with some of the extensions (extended CDB and/or bidi-read), but no data. Further, suppose the length field in one of the WN items gets corrupted, so that its value becomes longer than it should be and points off the end of the header and all its extensions. The receiver cannot know that this length has been corrupted, since it has not yet gotten to the data digest, and will therefore block on a read waiting for the rest of that WN item to arrive, but it never will. This cannot happen in my proposal, because the total length of the AHSs part of the PDU is given as the AHS_length field in the fixed-length BHS, and if that length field is corrupted then the data digest for the BHS will immediately detect that. It is true that the response will be to drop the connection, but it happens right away, without blocking on a read until a timeout occurs. Similarly, if the AHS_length field in the BHS is ok, but the length field in one of the AHS items itself is corrupted, that will also be known immediately in my proposal, because the data digest at the end of the AHSs part of the PDU is checked BEFORE using any of the length fields in the AHS items themselves. This points out the weakness of the WN Next-Qualifier scheme -- that the receiver has to use unreliable data to decode the header EVEN IF A DIGEST IS USED. This NEVER happens in my proposal. Thanks for your consideration, Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 On Tue, 13 Mar 2001 julian_satran@il.ibm.com wrote: > > > Bob, > > Interesting. This is close to one of the variants I had for IETF-50 (a > clear header). > The only adavantage it has is that you are able to read all the AHSs in one > read. > The (admitedly academic) disadvantage it has is that you are limited in the > size of extensions and have some redundancy. > The basic issue that was raised - and there is no simple way out - is that > once you have lost a block (BHS in your suggested layout) you are out of > synch. There is no simple way around it (at least not one that can be > solved by changing layout) and an added digest (only the code or silicon to > account for it) is IMHO not warranted by what you gain. If you are willing > to spend silicon or code on this the making header failures less probable > and recovering if you fail by dropping the connection is a better bet > (using the redundancy for a coding gain). But this involves some > complexity too. > > Julo > > "Robert D. Russell" on 12/03/2001 22:01:39 > > Please respond to "Robert D. Russell" > > To: ips@ece.cmu.edu > cc: bbrtrebia@mediaone.net, "Ashish P. Palekar" > , Anshul Chadda , > Narendran Ganapathy > Subject: Re: iSCSI: Towards a more effective PDU format > > > > > All: > > Here is a proposal for an iSCSI PDU format without WN Next-Qualifiers: > > 1. Every PDU must start with a 48-byte Basic Header Segment (BHS). > Assuming header digests are in use, this BHS is immediately > followed by a header digest that covers it. > > 2. Every BHS contains 2 fields at fixed locations: > a) AHS-length, containing the total length of all Additional Header > Segments (AHS) that follow. This field is 0 if there are no > AHSs. > b) DATA-length, containing the total length of all Data that follows > the AHSs. This field is 0 if there is no data. > > 3. If the AHS-length field in the BHS is non-zero, all the AHSs > immediately follow the header digest of the BHS. An additional > digest immediately follows that last AHS and covers all the AHSs. > > 4. If the DATA-length field in the BHS is non-zero, all the data > immediately follows the AHS digest. A data digest immediately > follows the data. > > 5. If space within the BHS is an issue, the AHS-length field could > be restricted to a single byte, and could be in units of 4-byte > words. This would allow up to 1020 bytes of AHSs to follow a > BHS, and this seems to be more than enough for the uses foreseen > (so far, only extended CDB and bidi-read info). > > 6. Each AHS should start with a word containing a TYPE field and a > LENGTH field. The TYPE field should be enumerated rather than > bit-field encoded, for easier decoding and future expansion. > The LENGTH field is the number of bytes of additional information > in this AHS that follows this word. > > > Advantages of this proposal. > > 1. The receiver always knows where to find digests in a reliable manner, > without having to use unreliable data to find them. > > 2. Because the BHS is fixed length, the receiver can reliably obtain the > entire BHS and its header digest in a single "read" operation. > > 3. Because the receiver gets the AHS-length field from the BHS, it can > reliably obtain the entire set of AHSs and the header digest covering > them in a single "read" operation. > > 4. Because the receiver gets the DATA-length field from the BHS, it can > reliably obtain all the data and the data digest covering it > in a single "read" operation. > > 5. Since each AHS begins with a word containing the TYPE and LENGTH in > fixed positions, an unknown TYPE (i.e., a type introduced in the > future that is received by a legacy receiver) does NOT result in > loss of synchronization during header processing -- the receiver > knows the length of this AHS in any case, and can just skip over it. > > > Disadvantages of this proposal. > > 1. A second header digest is required, but only if one or more AHSs are > present. In practice this should be a rare event. > > > Proposed iSCSI PDU Format > > > +-------------+ > | 48-byte BHS | > +-------------+ > | BHS digest | > --+-------------+-- > | AHS 1 |\ > | - - - - - - | \ > | AHS 2 | \ > | - - - - - - | > total length in AHS_length field in BHS > | . . . | / > | - - - - - - | / > | AHS n |/ > +-------------+ > | AHSs digest | > --+-------------+-- > | |\ > | Data | > total length in DATA_length field in BHS > | |/ > +-------------+ > | Data digest | > +-------------+ > > > Bob Russell > InterOperability Lab > University of New Hampshire > rdr@iol.unh.edu > 603-862-3774 > > > > > From owner-ips@ece.cmu.edu Tue Mar 13 19:33:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA00938 for ; Tue, 13 Mar 2001 19:33:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2DMC6X17369 for ips-outgoing; Tue, 13 Mar 2001 17:12:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from med.corp.rhapsodynetworks.com (64-160-62-201.rhapsodynetworks.com [64.160.62.201] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2DMBnr17323 for ; Tue, 13 Mar 2001 17:11:49 -0500 (EST) Received: by med.corp.rhapsodynetworks.com with Internet Mail Service (5.5.2653.19) id ; Tue, 13 Mar 2001 14:11:39 -0800 Message-ID: <15851BD69CFCD41186B100B0D0AABE650C1AD6@med.corp.rhapsodynetworks.com> From: Venkat Rangan To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: iSCSI: CRC-64 a "MUST"? Date: Tue, 13 Mar 2001 14:11:38 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk All, I know this was discussed at length, but I tried locating an email stating a consensus position that supports the following paragraph on page 93. "The following table lists cyclic integrity checksums that can be negotiated for the digests and MUST be implemented by every iSCSI initiator and target. Note that these digest options have only error detection significance." And CRC-64 is listed in the table as a "MUST" for implementation, with a TBD on the polynomial. Is CRC-64 required if an iSCSI session never negotiates a PDU size larger than 8K? Why is this requirement being imposed when it is known to be not very useful at shorter lengths? The draft-cavanna-iscsi-crc-vs-cksum-01.txt states that "CRC-64 is overkill" in Section 5. Can we therefore not be required to implement CRC-64? Also, even if a particular instance implements it, and always negotiates it down to CRC-32Q (or its equivalent), what is the purpose of carrying the implementation burden of a never-used option? Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com From owner-ips@ece.cmu.edu Tue Mar 13 21:41:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA02815 for ; Tue, 13 Mar 2001 21:41:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2E0Q5525690 for ips-outgoing; Tue, 13 Mar 2001 19:26:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.brocade.com (asbestos.brocade.com [63.121.140.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2E0J5r25274 for ; Tue, 13 Mar 2001 19:19:05 -0500 (EST) Received: from thor.brocade.com (thor [192.168.126.45]) by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id QAA24262 for ; Tue, 13 Mar 2001 16:19:03 -0800 (PST) Received: by thor.brocade.com with Internet Mail Service (5.5.2653.19) id ; Tue, 13 Mar 2001 16:19:03 -0800 Message-ID: From: Robert Snively To: ips@ece.cmu.edu Subject: RE: FCIP iFCP encapsulation proposal Date: Tue, 13 Mar 2001 16:19:01 -0800 X-Mailer: Internet Mail Service (5.5.2653.19) Sender: owner-ips@ece.cmu.edu Precedence: bulk Sorry, I think we have a severe case of red herring here. Any number of unlikely scenarios has been hypothesized, none of which (assuming there is any security in creating a TCP/IP connection at all) will cause any useful or dangerous behavior in either a SCSI target or a SCSI initiator. As an example, the information transmitted by FTP is first processed into FTP messages, which are then inserted on IP transfer units, which are then placed into Ethernet or other transport frames. Those frames have addressing which routes them to a destination. The IP layer unpackages the IP units, which are in turn unpackaged into FTP units for treatment by an FTP protocol only. Even if you delivered an FTP string transferring a bit-level trace to an FCIP device (not normally possible, because there would have had to be a TCP/IP connection formed between the FTP emitter and the FCIP receiver) the FTP encapsulation of the data could never be interpreted as legal SCSI commands by a SCSI device. The FTP string would not invoke the necessary valid responses for the statefully responding SCSI device. Bob From owner-ips@ece.cmu.edu Tue Mar 13 23:19:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA04387 for ; Tue, 13 Mar 2001 23:18:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2E2Y7S03421 for ips-outgoing; Tue, 13 Mar 2001 21:34:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2E2Y3r03414 for ; Tue, 13 Mar 2001 21:34:03 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GA60039H1NFSM@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Tue, 13 Mar 2001 18:32:06 -0800 (PST) Date: Tue, 13 Mar 2001 18:29:07 -0800 From: Douglas Otis Subject: RE: FCIP iFCP encapsulation proposal In-reply-to: To: Robert Snively , ips@ece.cmu.edu Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Bob, Such data could be stored in a binary file and this image could be relayed within the encapsulation as components of this file image. Doug > Sorry, I think we have a severe case of red herring here. Any number of > unlikely scenarios has been hypothesized, none of which (assuming > there is any security in creating a TCP/IP connection at all) will > cause any useful or dangerous behavior in either a SCSI target or > a SCSI initiator. > > As an example, the information transmitted by FTP is first processed into > FTP messages, which are then inserted on IP transfer units, which are > then placed into Ethernet or other transport frames. Those frames > have addressing which routes them to a destination. The IP layer > unpackages the IP units, which are in turn unpackaged into FTP units > for treatment by an FTP protocol only. > > Even if you delivered an FTP string transferring a bit-level trace > to an FCIP device (not normally > possible, because there would have had to be a TCP/IP connection formed > between the FTP emitter and the FCIP receiver) the FTP encapsulation of > the data could never be interpreted as legal SCSI commands by > a SCSI device. The FTP string would not invoke the necessary > valid responses for the statefully responding SCSI device. > > Bob > > > From owner-ips@ece.cmu.edu Tue Mar 13 23:20:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA04402 for ; Tue, 13 Mar 2001 23:20:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2E2oCd04303 for ips-outgoing; Tue, 13 Mar 2001 21:50:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2E2nQr04272 for ; Tue, 13 Mar 2001 21:49:27 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GA600CE02HPEY@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Tue, 13 Mar 2001 18:49:02 -0800 (PST) Date: Tue, 13 Mar 2001 18:47:15 -0800 From: Douglas Otis Subject: RE: TCP Framing Proposal In-reply-to: <015c01c0abf8$fc683220$0b00000a@cisco.com> To: Constantine Sapuntzakis Cc: Ips Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Costa, I understand a decision to drop a connection should there be a problem but then you can not assure a sequence within any activity unless there is an additional transport layer above TCP. SCTP could provide this additional layer to assist in maintaining sequences and without re-establishing a connection and all that it entails. Dropping is not required as SCTP would sit within a pseudo-frame. Normally SCTP sits within an Ethernet frame, and from this, it naturally establishes alignment. With TCP that is not practical or possible depending on the number of standards you wish to bend. | Ethernet Frame | | | | | | | Pseudo-Frame | Pseudo-Frame | Pseudo-Frame | |xxxxxxxxxxx00000|xxxxxxxxxxxxxxxxxx0|xxxxxxxxxxxxxxxx| [PDU| PDU |-----[ PDU | PDU [ | PDU [... [ = SCTP header 0 = Padding x = payload | = boundary The Ethernet frame represents the physical frame and SCTP sits within a fixed interval pseudo-frame which includes a length component (SCTP port field replaced with length). Within the SCTP Data Chunk sits the PDU which can be fragmented as are the Pseudo-Frames. Should there be an error, the Pseudo-Frame becomes a defined unit for providing both acknowledgement and retry as their fixed interval allows resynchronization in the event of an error. By placing SCTP as the layer above TCP, you enjoy features offered by SCTP such as multi-homing nor are you required to drop a connection should TCP acknowledge bad data. TCP will feed a process that merges the TCP payloads and parses SCTP at these merged payloads at regular intervals. SCTP parser then builds the PDU components limited to intervals of 4GB. The transport layers never need to deal with units larger than the Pseudo-Frame. You have defined a low level template devoid of any details of this additional transport layer. SCTP seems as good as any for this purpose. Of course, there would be a desire to use SCTP directly. Doug From owner-ips@ece.cmu.edu Tue Mar 13 23:21:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA04424 for ; Tue, 13 Mar 2001 23:21:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2E1v6o01179 for ips-outgoing; Tue, 13 Mar 2001 20:57:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2E1urr01146 for ; Tue, 13 Mar 2001 20:56:53 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Tue, 13 Mar 2001 17:56:46 -0800 Message-ID: From: Joshua Tseng To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: iSCSI: Logout needs ISID Date: Tue, 13 Mar 2001 17:56:45 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, Section 2.14 states that the logout command may be sent on a second connection to clean up the a single connection iSCSI session. If the reason code is 0 (close the session), then ISID is needed to identify the iSCSI session to close. Josh From owner-ips@ece.cmu.edu Wed Mar 14 03:03:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA20714 for ; Wed, 14 Mar 2001 03:03:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2E5nBp14800 for ips-outgoing; Wed, 14 Mar 2001 00:49:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2E5mKr14768 for ; Wed, 14 Mar 2001 00:48:20 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id GAA93152 for ; Wed, 14 Mar 2001 06:48:13 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id GAA153510 for ; Wed, 14 Mar 2001 06:47:02 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0F.001FB20C ; Wed, 14 Mar 2001 06:46:11 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 14 Mar 2001 07:48:30 +0200 Subject: Re: iSCSI: CRC-64 a "MUST"? Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Venkat, We will discuss this at Minneapolis. The draft you are quoting is very weak on numbers. We don't have any good candidate for a CRC-64 but we have a (new) CRC-32 that can work up to blocks of 2^31-1 bits with the same protection level we had with IEEE-802 or with CRC32Q up to 64kbits. Than could satisfy the most exacting needs for a while and we can put to rest the CRC-64 line for a while. Julo Venkat Rangan on 14/03/2001 00:11:38 Please respond to Venkat Rangan To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: iSCSI: CRC-64 a "MUST"? All, I know this was discussed at length, but I tried locating an email stating a consensus position that supports the following paragraph on page 93. "The following table lists cyclic integrity checksums that can be negotiated for the digests and MUST be implemented by every iSCSI initiator and target. Note that these digest options have only error detection significance." And CRC-64 is listed in the table as a "MUST" for implementation, with a TBD on the polynomial. Is CRC-64 required if an iSCSI session never negotiates a PDU size larger than 8K? Why is this requirement being imposed when it is known to be not very useful at shorter lengths? The draft-cavanna-iscsi-crc-vs-cksum-01.txt states that "CRC-64 is overkill" in Section 5. Can we therefore not be required to implement CRC-64? Also, even if a particular instance implements it, and always negotiates it down to CRC-32Q (or its equivalent), what is the purpose of carrying the implementation burden of a never-used option? Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com From owner-ips@ece.cmu.edu Wed Mar 14 03:11:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA20751 for ; Wed, 14 Mar 2001 03:11:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2E5wAM15340 for ips-outgoing; Wed, 14 Mar 2001 00:58:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2E5vPr15285 for ; Wed, 14 Mar 2001 00:57:25 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id GAA74162 for ; Wed, 14 Mar 2001 06:57:14 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id GAA189516 for ; Wed, 14 Mar 2001 06:56:03 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0F.00208564 ; Wed, 14 Mar 2001 06:55:13 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 14 Mar 2001 07:57:29 +0200 Subject: Re: iSCSI: Logout needs ISID Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Josh, There is something I am missing. The Logout can be issued only after Login (authentication and the rest). The session is then implied. Regards, Julo Joshua Tseng on 14/03/2001 03:56:45 Please respond to Joshua Tseng To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: iSCSI: Logout needs ISID Julian, Section 2.14 states that the logout command may be sent on a second connection to clean up the a single connection iSCSI session. If the reason code is 0 (close the session), then ISID is needed to identify the iSCSI session to close. Josh From owner-ips@ece.cmu.edu Wed Mar 14 11:45:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA27621 for ; Wed, 14 Mar 2001 11:45:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2EFAQZ25560 for ips-outgoing; Wed, 14 Mar 2001 10:10:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2EFA1r25521 for ; Wed, 14 Mar 2001 10:10:01 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Wed, 14 Mar 2001 07:09:55 -0800 Message-ID: From: Joshua Tseng To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI: Logout needs ISID Date: Wed, 14 Mar 2001 07:09:48 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, In 2.14, you state that a logout command can be sent on a different connection to destroy a single- connection iSCSI session. If you have multiple sessions ongoing, and the logout command is sent on a different connection than the one used to support the session that is to be killed, then you will need ISID to distinguish the session that you want killed. Regards, Josh > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Tuesday, March 13, 2001 9:57 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Logout needs ISID > > > > > Josh, > > There is something I am missing. The Logout can be issued > only after Login > (authentication and the rest). > The session is then implied. > > Regards, > Julo > > Joshua Tseng on 14/03/2001 03:56:45 > > Please respond to Joshua Tseng > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: iSCSI: Logout needs ISID > > > > > Julian, > > Section 2.14 states that the logout command may > be sent on a second connection to clean up the > a single connection iSCSI session. If the reason > code is 0 (close the session), then ISID is needed > to identify the iSCSI session to close. > > Josh > > > From owner-ips@ece.cmu.edu Wed Mar 14 11:46:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA27636 for ; Wed, 14 Mar 2001 11:45:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2EEEJ621948 for ips-outgoing; Wed, 14 Mar 2001 09:14:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zoetermeer.tripace.com (mail.tripace.com [212.136.147.1]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2EDb6r19714 for ; Wed, 14 Mar 2001 08:37:06 -0500 (EST) Received: by ZOETERMEER with Internet Mail Service (5.5.2650.21) id ; Wed, 14 Mar 2001 14:45:01 +0100 Message-ID: <8C59010722BBD31194640050DA6EC69711F27A@ZOETERMEER> From: "Sanjeev Bhagat (TRIPACE/Zoetermeer)" To: "'julian_satran@il.ibm.com'" Cc: ips@ece.cmu.edu Subject: FW: unsolicited data sequence Date: Wed, 14 Mar 2001 14:44:58 +0100 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C0AC8C.F75FEB12" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C0AC8C.F75FEB12 Content-Type: text/plain; charset="iso-8859-1" Hello julian, I have tried to post this question 2-3 times since yesterday but it does not show up in the mailing list? Any reason why? -----Original Message----- From: Sanjeev Bhagat (TRIPACE/Zoetermeer) Sent: Wednesday, March 14, 2001 6:32 AM To: 'IPS Reflector' Subject: unsolicited data sequence Hello, I have a question. In case the initiator and target negotiate the login parameter for the data transfer with UseR2T = No, what would be the sequence of data transfer in case of Write command wher the data transfer length is greater the immediate data +firstburst size? Can the taget keep makingextra iSCSI data PDUs and send it to target or it must wait for the acknowledgement for the firstburst size data. Sanjeev ------_=_NextPart_001_01C0AC8C.F75FEB12 Content-Type: text/html; charset="iso-8859-1"
Hello julian,
 
I have tried to post this question 2-3 times since yesterday but it does not show up in the mailing list? Any reason why?
-----Original Message-----
From: Sanjeev Bhagat (TRIPACE/Zoetermeer)
Sent: Wednesday, March 14, 2001 6:32 AM
To: 'IPS Reflector'
Subject: unsolicited data sequence

Hello,

I have a question.
 
 

In case the initiator and target negotiate the login parameter for the
data transfer with UseR2T = No, what would be the sequence of data
transfer in case of Write command wher the data transfer length is
greater the immediate data +firstburst size? Can the taget keep
makingextra iSCSI data PDUs and send it to target or it must wait for
the acknowledgement for the firstburst size data.

Sanjeev
------_=_NextPart_001_01C0AC8C.F75FEB12-- From owner-ips@ece.cmu.edu Wed Mar 14 12:33:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA29131 for ; Wed, 14 Mar 2001 12:33:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2EFS1w26890 for ips-outgoing; Wed, 14 Mar 2001 10:28:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2EFQPr26755 for ; Wed, 14 Mar 2001 10:26:28 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA89138 for ; Wed, 14 Mar 2001 16:26:10 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA31688 for ; Wed, 14 Mar 2001 16:24:59 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0F.00549C94 ; Wed, 14 Mar 2001 16:24:11 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 14 Mar 2001 17:26:31 +0200 Subject: RE: iSCSI: Logout needs ISID Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Josh, No command, including logout, can be sent without a login. The complete scenario is: -open a new connection -login in the same session as the old connection (this has the ISID & TSID) -logout the old connection -recover commands You are not supposed to be able to kill a session from outside (at least not an iSCSI defined mode). You will need management support for that (SNMP?) Regards, Julo Joshua Tseng on 14/03/2001 17:09:48 Please respond to Joshua Tseng To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI: Logout needs ISID Julian, In 2.14, you state that a logout command can be sent on a different connection to destroy a single- connection iSCSI session. If you have multiple sessions ongoing, and the logout command is sent on a different connection than the one used to support the session that is to be killed, then you will need ISID to distinguish the session that you want killed. Regards, Josh > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Tuesday, March 13, 2001 9:57 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Logout needs ISID > > > > > Josh, > > There is something I am missing. The Logout can be issued > only after Login > (authentication and the rest). > The session is then implied. > > Regards, > Julo > > Joshua Tseng on 14/03/2001 03:56:45 > > Please respond to Joshua Tseng > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: iSCSI: Logout needs ISID > > > > > Julian, > > Section 2.14 states that the logout command may > be sent on a second connection to clean up the > a single connection iSCSI session. If the reason > code is 0 (close the session), then ISID is needed > to identify the iSCSI session to close. > > Josh > > > From owner-ips@ece.cmu.edu Wed Mar 14 13:53:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA01212 for ; Wed, 14 Mar 2001 13:53:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2EFdaP27710 for ips-outgoing; Wed, 14 Mar 2001 10:39:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from opus.ece.cmu.edu (root@OPUS.ECE.CMU.EDU [128.2.134.91]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2EFcQr27629 for ; Wed, 14 Mar 2001 10:38:26 -0500 (EST) Received: from opus.ece.cmu.edu (bassoon@localhost [127.0.0.1]) by opus.ece.cmu.edu (8.9.3/8.9.3/SuSE Linux 8.9.3-0.1) with ESMTP id KAA02398 for ; Wed, 14 Mar 2001 10:38:25 -0500 Message-Id: <200103141538.KAA02398@opus.ece.cmu.edu> To: ips@ece.cmu.edu Subject: unsolicited data sequence Date: Wed, 14 Mar 2001 10:38:25 -0500 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk From: "Sanjeev Bhagat (TRIPACE/Zoetermeer)" To: "'IPS Reflector'" Subject: unsolicited data sequence Date: Wed, 14 Mar 2001 15:26:03 +0100 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Hello, Well i tried to send this question quiet a few times, but it never reflected in mailing list, probably because my mail was going in HTML format. so now i am trying to send in plain text mode. Please excuse me if the question shows up too many times in mailing list. I am trying to write a module to send a write command to the target in UseR2T= No mode so my question is as follows In case the initiator and target negotiate the login parameter for the data transfer with UseR2T = No, what would be the sequence of data transfer in case of Write command wher the data transfer length is greater the immediate data +firstburst size? Can the taget keep making extra iSCSI data PDUs and send it to target or it must wait for the acknowledgement for the firstburst size data. Sincerely, Sanjeev Bhagat Tripace Europe Tel # +31 (79) 361 2444 Fax# +31 (79) 361 2152 Please, do visit our web site for more info: SCSI Chip Set(s), SCSI Host Adapters, SCSI Software Drivers to various Operating Systems SCSI, USB 2.0 & ATAPI/ATA Interconnectivity Chip Solutions C R E A T O R S O F D A T A M O V I N G W O R L D S This e-mail is confidential and may contain legally privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender above mentioned immediately. ------- End of Forwarded Message From owner-ips@ece.cmu.edu Wed Mar 14 13:53:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA01223 for ; Wed, 14 Mar 2001 13:53:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2EGVRZ01855 for ips-outgoing; Wed, 14 Mar 2001 11:31:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2EGV5r01820 for ; Wed, 14 Mar 2001 11:31:05 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id RAA138508 for ; Wed, 14 Mar 2001 17:30:57 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA62128 for ; Wed, 14 Mar 2001 17:29:45 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A0F.005A8ADB ; Wed, 14 Mar 2001 17:28:57 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 14 Mar 2001 18:31:19 +0200 Subject: Re: iSCSI: Logout needs ISID Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mark, If you have only one connection you are supposed to use a Login with the restart bit one - and achieve a similar effect as a Login/Logout - i.e. keep the session alive. Even this might be a problem for a target that is so poor it has only one socket (even SNMP won't work there). For this case the only way out is to drop the connection and hope the target will hear you (it probably won't -:)). The comment is the draft is a memento for implementers to keep looking for new connections always (even for a one connection target) but probably it does not come through clear enough. Regards, Julo Mark Bakke on 14/03/2001 18:17:14 Please respond to Mark Bakke To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI: Logout needs ISID Julian- A target that does not support multiple connections per session will return "Initiator SID Error" when the login attempt is made. In this case, there is no way to log in, so there's no way to log out the old connection. The initiator would be stuck waiting until the target's side of the connection times out and goes away, which could be a very long time. There is an open question within the MIB team about whether anyone needs to be able to kill connections or sessions from SNMP; however, I don't think that anyone will want to use SNMP as part of error recovery. -- Mark julian_satran@il.ibm.com wrote: > > Josh, > > No command, including logout, can be sent without a login. > The complete scenario is: > > -open a new connection > -login in the same session as the old connection (this has the ISID & TSID) > -logout the old connection > -recover commands > > You are not supposed to be able to kill a session from outside (at least > not an iSCSI defined mode). > You will need management support for that (SNMP?) > > Regards, > Julo > > Joshua Tseng on 14/03/2001 17:09:48 > > Please respond to Joshua Tseng > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: iSCSI: Logout needs ISID > > Julian, > > In 2.14, you state that a logout command can be > sent on a different connection to destroy a single- > connection iSCSI session. If you have multiple > sessions ongoing, and the logout command is sent > on a different connection than the one used > to support the session that is to be killed, then > you will need ISID to distinguish the session that > you want killed. > > Regards, > Josh > > > -----Original Message----- > > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > > Sent: Tuesday, March 13, 2001 9:57 PM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: Logout needs ISID > > > > > > > > > > Josh, > > > > There is something I am missing. The Logout can be issued > > only after Login > > (authentication and the rest). > > The session is then implied. > > > > Regards, > > Julo > > > > Joshua Tseng on 14/03/2001 03:56:45 > > > > Please respond to Joshua Tseng > > > > To: Julian Satran/Haifa/IBM@IBMIL > > cc: ips@ece.cmu.edu > > Subject: iSCSI: Logout needs ISID > > > > > > > > > > Julian, > > > > Section 2.14 states that the logout command may > > be sent on a second connection to clean up the > > a single connection iSCSI session. If the reason > > code is 0 (close the session), then ISID is needed > > to identify the iSCSI session to close. > > > > Josh > > > > > > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Wed Mar 14 13:54:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA01260 for ; Wed, 14 Mar 2001 13:54:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2EGIUx00830 for ips-outgoing; Wed, 14 Mar 2001 11:18:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2EGHSr00731 for ; Wed, 14 Mar 2001 11:17:28 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id LAA11082; Wed, 14 Mar 2001 11:17:11 -0500 (EST) Message-ID: <3AAF998A.21B75C5A@cisco.com> Date: Wed, 14 Mar 2001 10:17:14 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: julian_satran@il.ibm.com CC: ips@ece.cmu.edu Subject: Re: iSCSI: Logout needs ISID References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian- A target that does not support multiple connections per session will return "Initiator SID Error" when the login attempt is made. In this case, there is no way to log in, so there's no way to log out the old connection. The initiator would be stuck waiting until the target's side of the connection times out and goes away, which could be a very long time. There is an open question within the MIB team about whether anyone needs to be able to kill connections or sessions from SNMP; however, I don't think that anyone will want to use SNMP as part of error recovery. -- Mark julian_satran@il.ibm.com wrote: > > Josh, > > No command, including logout, can be sent without a login. > The complete scenario is: > > -open a new connection > -login in the same session as the old connection (this has the ISID & TSID) > -logout the old connection > -recover commands > > You are not supposed to be able to kill a session from outside (at least > not an iSCSI defined mode). > You will need management support for that (SNMP?) > > Regards, > Julo > > Joshua Tseng on 14/03/2001 17:09:48 > > Please respond to Joshua Tseng > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: iSCSI: Logout needs ISID > > Julian, > > In 2.14, you state that a logout command can be > sent on a different connection to destroy a single- > connection iSCSI session. If you have multiple > sessions ongoing, and the logout command is sent > on a different connection than the one used > to support the session that is to be killed, then > you will need ISID to distinguish the session that > you want killed. > > Regards, > Josh > > > -----Original Message----- > > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > > Sent: Tuesday, March 13, 2001 9:57 PM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: Logout needs ISID > > > > > > > > > > Josh, > > > > There is something I am missing. The Logout can be issued > > only after Login > > (authentication and the rest). > > The session is then implied. > > > > Regards, > > Julo > > > > Joshua Tseng on 14/03/2001 03:56:45 > > > > Please respond to Joshua Tseng > > > > To: Julian Satran/Haifa/IBM@IBMIL > > cc: ips@ece.cmu.edu > > Subject: iSCSI: Logout needs ISID > > > > > > > > > > Julian, > > > > Section 2.14 states that the logout command may > > be sent on a second connection to clean up the > > a single connection iSCSI session. If the reason > > code is 0 (close the session), then ISID is needed > > to identify the iSCSI session to close. > > > > Josh > > > > > > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Wed Mar 14 13:54:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA01261 for ; Wed, 14 Mar 2001 13:54:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2EGIQV00824 for ips-outgoing; Wed, 14 Mar 2001 11:18:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com (42s3082.isus.emc.com [168.159.211.82] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2EGHJr00723 for ; Wed, 14 Mar 2001 11:17:19 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Wed, 14 Mar 2001 11:18:40 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0708015299@corpmx9.isus.emc.com> From: Black_David@emc.com To: rsnively@Brocade.COM, ips@ece.cmu.edu Subject: RE: FCIP iFCP encapsulation proposal Date: Wed, 14 Mar 2001 11:17:15 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, > Sorry, I think we have a severe case of red herring here. Any number of > unlikely scenarios has been hypothesized, none of which (assuming > there is any security in creating a TCP/IP connection at all) will > cause any useful or dangerous behavior in either a SCSI target or > a SCSI initiator. > > As an example, the information transmitted by FTP is first processed into > FTP messages, The ftp analogy is not indicative of what's going on here because ftp does not have a resync mechanism - in other words, ftp never scans the inbound byte stream from TCP looking for some data pattern that indicates the start of an ftp message. Some of the "framing" proposals for iSCSI, iFCP, and FCIP propose to do this or something close to it. The most severe case is one in which the framing data pattern is used to identify a message that will be processed by SCSI or FC logic in an upper layer. iSCSI needs to do this after a CRC failure if the CRC failure requires discarding the length information in a header and the connection is not closed. In this case, occurrence of the framing pattern in data could cause a PDU to be extracted from the data and processed - this would be very bad. If framing is used only for data steering (i.e., organization of buffer memory for data that has been received, but not yet processed), the situation is better, but it's necessary for logic somewhere to know that the memory organization done by the framing may not always be correct, and be prepared to copy the data to get it correctly organized (e.g., 4k data block on a 4k boundary) if necessary. Going back to Vi Chau's encapsulation proposal at the start of this thread, there's a separate header CRC. If the header CRC check fails, the frame length info in that header cannot be used. If the TCP connection is not closed at that point, the arriving data has to be scanned looking for the next header. Vi wrote: > In the event of loss of synchronization, a state machine which normally > checks the FCIP/iFCP header CRC can be continuously enabled on the > data stream which should provide for a "good CRC" compare when an > FCIP/iFCP header arrives. This is subject to exactly the "frame in data" problem caused by storing a trace file - the state machine will find what looks like a good header in what's actually data, process it (and perhaps several, if a number of short trace frames are included in the actual data frame), only realizing its mistake at some later point. This MUST NOT be allowed to happen. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Wed Mar 14 15:21:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA03436 for ; Wed, 14 Mar 2001 15:21:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2EIwUC13146 for ips-outgoing; Wed, 14 Mar 2001 13:58:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from med.corp.rhapsodynetworks.com (64-160-62-201.rhapsodynetworks.com [64.160.62.201] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2EIvtr13082 for ; Wed, 14 Mar 2001 13:57:56 -0500 (EST) Received: by med.corp.rhapsodynetworks.com with Internet Mail Service (5.5.2653.19) id ; Wed, 14 Mar 2001 10:57:23 -0800 Message-ID: <15851BD69CFCD41186B100B0D0AABE650C1AE0@med.corp.rhapsodynetworks.com> From: Venkat Rangan To: "'Mark Bakke'" , IPS Subject: RE: iSCSI MIB Object Model Date: Wed, 14 Mar 2001 10:57:22 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Mark, The UML is very nice, and the new object model, covering both Initiator and Target instances is also nice. Overall, we do have some concern over the need for maintaining per-LU and per-LUN statistics. In some iSCSI implementations, the iSCSI entity is merely an intermediate layer, with no knowledge of LUs and LUNs. Placing this in the GeneralInfo category and making it mandatory would add undue burden on these implementations. In Section 5.9: "The iscsiLUTable contains a list of iSCSI Logical Units that are accessible via the iSCSI target." Would this list be the same as what the "Report LUNs" command would give? In Section 5.7: "The iscsiLunTable contains an entry for each known LUN that is being accessed using this session. Note that is may not contain all of the LUNs that could be accessed; the only ones available to iSCSI are the ones that have been addressed specifically by iSCSI commands. Therefore, LUNs that have been discovered via mechanisms such as the SCSI "report LUNs" will not appear in this table until they are specifically addressed by the initiator. How long do these entries live in the table? If iSCSI sessions logout and close, do these entries continue to remain? Thanks, Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com -----Original Message----- From: Mark Bakke [mailto:mbakke@cisco.com] Sent: Monday, March 12, 2001 7:12 AM To: IPS Subject: iSCSI MIB Object Model iSCSI MIB enthusiasts- While building the iSCSI MIB table structure, we created a UML (more or less) model for iSCSI, from which the MIB was derived. Julian has placed a pdf of the model on his web site; it is available at: http://www.haifa.il.ibm.com/satran/ips/Visio-iscsi-uml-model-02.pdf The UML model does not show every table and attribute; it is meant as a higher-level abstraction, and should be helpful in reading the MIB, or in understanding how the iSCSI objects relate to one another. It has been updated to include a few fields that are missing in the current draft. Also available, for those who do not have the previous messages: iSCSI MIB table structure http://www.haifa.il.ibm.com/satran/ips/iscsi-mib-structure-02.pdf iSCSI MIB (zipped version) http://www.haifa.il.ibm.com/satran/ips/draft-bakke-iscsimib-02.zip iSCSI MIB (uncompressed) http://www.haifa.il.ibm.com/satran/ips/draft-bakke-iscsimib-02.txt Enjoy, -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Wed Mar 14 15:21:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA03447 for ; Wed, 14 Mar 2001 15:21:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2EIVQH11173 for ips-outgoing; Wed, 14 Mar 2001 13:31:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.bld.us.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2EIVKr11159 for ; Wed, 14 Mar 2001 13:31:20 -0500 (EST) Received: from f2n22e. (westrelay01.boulder.ibm.com [9.99.140.22]) by e33.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id MAA12902 for ; Wed, 14 Mar 2001 12:25:21 -0600 Received: from f4n49e (d03nm065h.boulder.ibm.com [9.99.140.49]) by f2n22e. (8.8.8m3/NCO v4.95) with ESMTP id LAA180758 for ; Wed, 14 Mar 2001 11:31:12 -0700 X-Priority: 1 (High) Importance: Normal Subject: iSCSI: Need to Kill Session from Surviving Node To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Wed, 14 Mar 2001 10:30:50 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/14/2001 11:31:12 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk There is a related problem that I think we need to address, and that is -- the case where the Storage Controller is an Active-Active Fail-Over unit with very little state shared between the Storage Controller Nodes. (I think a number of you folks have told me that you are planning such a design.) This would mean that a specific initiator OS (WWUI) would have separate sessions between the Nodes. At fail-over, the Take-Over Node will want to cause the Initiator, to Stop messing around with the Failing node ASAP, and start using the existing session, or start a new session with the Take_Over Node. The Initiator, will normally just time-out the TCP/IP connection, and then turn the recovery over to the SCSI Layer. Then, when SCSI retries, iSCSI will either use the Session to the Surviving Node or Create a New Session to that Surviving Node. The problem is, that this will take a longer time then a system might want. So the question: is there a way for a surviving Target Node to tell the Initiator to "Kill" the Session to the Failing Target Node? Comments? . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/14/2001 08:31:19 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iSCSI: Logout needs ISID Mark, If you have only one connection you are supposed to use a Login with the restart bit one - and achieve a similar effect as a Login/Logout - i.e. keep the session alive. Even this might be a problem for a target that is so poor it has only one socket (even SNMP won't work there). For this case the only way out is to drop the connection and hope the target will hear you (it probably won't -:)). The comment is the draft is a memento for implementers to keep looking for new connections always (even for a one connection target) but probably it does not come through clear enough. Regards, Julo Mark Bakke on 14/03/2001 18:17:14 Please respond to Mark Bakke To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI: Logout needs ISID Julian- A target that does not support multiple connections per session will return "Initiator SID Error" when the login attempt is made. In this case, there is no way to log in, so there's no way to log out the old connection. The initiator would be stuck waiting until the target's side of the connection times out and goes away, which could be a very long time. There is an open question within the MIB team about whether anyone needs to be able to kill connections or sessions from SNMP; however, I don't think that anyone will want to use SNMP as part of error recovery. -- Mark julian_satran@il.ibm.com wrote: > > Josh, > > No command, including logout, can be sent without a login. > The complete scenario is: > > -open a new connection > -login in the same session as the old connection (this has the ISID & TSID) > -logout the old connection > -recover commands > > You are not supposed to be able to kill a session from outside (at least > not an iSCSI defined mode). > You will need management support for that (SNMP?) > > Regards, > Julo > > Joshua Tseng on 14/03/2001 17:09:48 > > Please respond to Joshua Tseng > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: iSCSI: Logout needs ISID > > Julian, > > In 2.14, you state that a logout command can be > sent on a different connection to destroy a single- > connection iSCSI session. If you have multiple > sessions ongoing, and the logout command is sent > on a different connection than the one used > to support the session that is to be killed, then > you will need ISID to distinguish the session that > you want killed. > > Regards, > Josh > > > -----Original Message----- > > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > > Sent: Tuesday, March 13, 2001 9:57 PM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: Logout needs ISID > > > > > > > > > > Josh, > > > > There is something I am missing. The Logout can be issued > > only after Login > > (authentication and the rest). > > The session is then implied. > > > > Regards, > > Julo > > > > Joshua Tseng on 14/03/2001 03:56:45 > > > > Please respond to Joshua Tseng > > > > To: Julian Satran/Haifa/IBM@IBMIL > > cc: ips@ece.cmu.edu > > Subject: iSCSI: Logout needs ISID > > > > > > > > > > Julian, > > > > Section 2.14 states that the logout command may > > be sent on a second connection to clean up the > > a single connection iSCSI session. If the reason > > code is 0 (close the session), then ISID is needed > > to identify the iSCSI session to close. > > > > Josh > > > > > > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Wed Mar 14 15:25:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA03546 for ; Wed, 14 Mar 2001 15:25:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2EI1Og08716 for ips-outgoing; Wed, 14 Mar 2001 13:01:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.brocade.com (asbestos.brocade.com [63.121.140.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2EHlSr07612 for ; Wed, 14 Mar 2001 12:47:28 -0500 (EST) Received: from thor.brocade.com (thor [192.168.126.45]) by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id JAA02179; Wed, 14 Mar 2001 09:47:14 -0800 (PST) Received: by thor.brocade.com with Internet Mail Service (5.5.2653.19) id ; Wed, 14 Mar 2001 09:47:14 -0800 Message-ID: From: Robert Snively To: "'Black_David@emc.com'" , Robert Snively , ips@ece.cmu.edu Subject: RE: FCIP iFCP encapsulation proposal Date: Wed, 14 Mar 2001 09:47:13 -0800 X-Mailer: Internet Mail Service (5.5.2653.19) Sender: owner-ips@ece.cmu.edu Precedence: bulk Dave, The byte stream that is processed by the iSCSI or FCIP layer is not a byte stream that can be generated by a hardware spoofer. It is the byte stream that is delivered by a TCP connection, riding on PDU's containing IP headers, riding on Ethernet frames carrying their own identification information. The spoofer must successfully generate an output that will meet the Ethernet format requirements for delivery, including device addressing; will then meet the IP requirements for delivery, including frame identification, IP addressing, fragmentation and reassembly, and CRC; will then meet the TCP/IP requirements for delivery, including proper TCP header formatting, proper checksumming, proper identification of the order and location of data, and the definition of a properly defined TCP connection, which can in turn only be created by a secure action which cannot be replicated; will then meet the requirements of an FC frame, including sequence id and frame count, exchange identification by both the source and sink, address verification by the sink, consistency with login state, and 32-bit CRC; and will finally meet the logical and state dependent requirements of the upper layer protocol, including logical unit addressing, command set requirements, consistency with read and write operations, and status presentation; all without being noticed. It will be very hard for you to convince me that this can all be done by dropping Ethernet frames with a bunch of replicated FCIP frame headers into a fabric. What you have to do is realistically and undetectably meld the TCP data stream from the legitimate connection and the TCP data stream from the false connection in a manner seamless enough so that the data will get delivered. Then, after delivering the data, you have to reconstitute a legal SCSI, VI, or other Fibre Channel protocol in a manner that will not cause it to be rejected or timed out by the higher level protocol. This by the way is one of the advantages of having long connection periods. There are very infrequent opportunities to formulate a potentially legitimate TCP/IP connection capable of carrying falsified data. There is of course the unfortunate side effect that, once you have succeeded in creating such a connection, you can obtain all the rights associated with such a connection, in which case no framing behavior could protect you anyway. In effect, the problem is not a "frame in data" problem. It is a "frame in successfully delivered TCP/IP data across a legitimate TCP/IP connection consistent with the fully defined states of the Fibre Channel FC-2 layer and consistent with the states, requirements, requests, and acknowledgements of the upper layer protocol" problem, which is darned unlikely even if you are trying maliciously to do it. Bob > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Wednesday, March 14, 2001 8:17 AM > To: rsnively@brocade.com; ips@ece.cmu.edu > Subject: RE: FCIP iFCP encapsulation proposal > > > Bob, > > > Sorry, I think we have a severe case of red herring here. > Any number of > > unlikely scenarios has been hypothesized, none of which (assuming > > there is any security in creating a TCP/IP connection at all) will > > cause any useful or dangerous behavior in either a SCSI target or > > a SCSI initiator. > > > > As an example, the information transmitted by FTP is first > processed into > > FTP messages, > > The ftp analogy is not indicative of what's going on here > because ftp does > not > have a resync mechanism - in other words, ftp never scans > the inbound byte > stream from TCP looking for some data pattern that indicates > the start of > an ftp message. Some of the "framing" proposals for iSCSI, iFCP, and > FCIP propose to do this or something close to it. > > The most severe case is one in which the framing data > pattern is used to > identify > a message that will be processed by SCSI or FC logic in an > upper layer. > iSCSI > needs to do this after a CRC failure if the CRC failure > requires discarding > the > length information in a header and the connection is not > closed. In this > case, > occurrence of the framing pattern in data could cause a PDU > to be extracted > from the data and processed - this would be very bad. > > If framing is used only for data steering (i.e., > organization of buffer > memory for > data that has been received, but not yet processed), the situation is > better, but > it's necessary for logic somewhere to know that the memory > organization > done by the framing may not always be correct, and be > prepared to copy the > data to get it correctly organized (e.g., 4k data block on a > 4k boundary) if > necessary. > > Going back to Vi Chau's encapsulation proposal at the start > of this thread, > there's a separate header CRC. If the header CRC check > fails, the frame > length info in that header cannot be used. If the TCP > connection is not > closed at that point, the arriving data has to be scanned > looking for the > next header. Vi wrote: > > > In the event of loss of synchronization, a state machine > which normally > > checks the FCIP/iFCP header CRC can be continuously enabled on the > > data stream which should provide for a "good CRC" compare when an > > FCIP/iFCP header arrives. > > This is subject to exactly the "frame in data" problem > caused by storing > a trace file - the state machine will find what looks like a > good header in > what's actually data, process it (and perhaps several, if a > number of short > trace frames are included in the actual data frame), only > realizing its > mistake > at some later point. This MUST NOT be allowed to happen. > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Wed Mar 14 17:35:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA05779 for ; Wed, 14 Mar 2001 17:35:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2EKGWW19033 for ips-outgoing; Wed, 14 Mar 2001 15:16:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2EKFur18996 for ; Wed, 14 Mar 2001 15:15:56 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel3.hp.com (Postfix) with ESMTP id 08406F30 for ; Wed, 14 Mar 2001 12:14:48 -0800 (PST) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id MAA15518 for ips@ece.cmu.edu; Wed, 14 Mar 2001 12:15:46 -0800 (PST) Message-Id: <200103142015.MAA15518@core.rose.hp.com> Subject: Re: iSCSI: Need to Kill Session from Surviving Node To: ips@ece.cmu.edu Date: Wed, 14 Mar 2001 12:15:45 PST In-Reply-To: ; from "John Hufferd" at Mar 14, 101 11:42 am Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk >There is a related problem that I think we need to address, and that is -- >the case where the Storage Controller is an Active-Active Fail-Over unit >with very little state shared between the Storage Controller Nodes. (I >think a number of you folks have told me that you are planning such a >design.) This would mean that a specific initiator OS (WWUI) would have >separate sessions between the Nodes. At fail-over, the Take-Over Node will >want to cause the Initiator, to Stop messing around with the Failing node >ASAP, and start using the existing session, or start a new session with the >Take_Over Node. The Initiator, will normally just time-out the TCP/IP >connection, and then turn the recovery over to the SCSI Layer. Then, when >SCSI retries, iSCSI will either use the Session to the Surviving Node or >Create a New Session to that Surviving Node. The problem is, that this >will take a longer time then a system might want. > >So the question: is there a way for a surviving Target Node to tell the >Initiator to "Kill" the Session to the Failing Target Node? > >Comments? Assuming that the failing Node is still able to execute commands, a cold target reset from the Initiator to the failing node should "kill" all the sessions to it. The answer to two hosts configured in a failover configuration sharing a single Storage Controller Node is also the same. If it is active-active configuration, it appears to me that there's a need to somehow prompt the Initiator to doing the reset on a third party (Async PDU?). This is a reason why we'd still need target reset in iSCSI. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com > >. >. >. >John L. Hufferd >Senior Technical Staff Member (STSM) >IBM/SSG San Jose Ca >(408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 >Internet address: hufferd@us.ibm.com > > >Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/14/2001 08:31:19 AM > >Sent by: owner-ips@ece.cmu.edu > > >To: ips@ece.cmu.edu >cc: >Subject: Re: iSCSI: Logout needs ISID > > > > > >Mark, > >If you have only one connection you are supposed to use a Login with the >restart bit one - and achieve a similar effect as a Login/Logout - i.e. >keep the session alive. Even this might be a problem for a target that is >so poor it has only one socket (even SNMP won't work there). For this case >the only way out is to drop the connection and hope the target will hear >you (it probably won't -:)). The comment is the draft is a memento for >implementers to keep looking for new connections always (even for a one >connection target) but probably it does not come through clear enough. > >Regards, >Julo > >Mark Bakke on 14/03/2001 18:17:14 > >Please respond to Mark Bakke > >To: Julian Satran/Haifa/IBM@IBMIL >cc: ips@ece.cmu.edu >Subject: Re: iSCSI: Logout needs ISID > > > > >Julian- > >A target that does not support multiple connections per session >will return "Initiator SID Error" when the login attempt is made. >In this case, there is no way to log in, so there's no way to >log out the old connection. The initiator would be stuck waiting >until the target's side of the connection times out and goes >away, which could be a very long time. > >There is an open question within the MIB team about whether anyone >needs to be able to kill connections or sessions from SNMP; however, >I don't think that anyone will want to use SNMP as part of error >recovery. > >-- >Mark > >julian_satran@il.ibm.com wrote: >> >> Josh, >> >> No command, including logout, can be sent without a login. >> The complete scenario is: >> >> -open a new connection >> -login in the same session as the old connection (this has the ISID & >TSID) >> -logout the old connection >> -recover commands >> >> You are not supposed to be able to kill a session from outside (at least >> not an iSCSI defined mode). >> You will need management support for that (SNMP?) >> >> Regards, >> Julo >> >> Joshua Tseng on 14/03/2001 17:09:48 >> >> Please respond to Joshua Tseng >> >> To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu >> cc: >> Subject: RE: iSCSI: Logout needs ISID >> >> Julian, >> >> In 2.14, you state that a logout command can be >> sent on a different connection to destroy a single- >> connection iSCSI session. If you have multiple >> sessions ongoing, and the logout command is sent >> on a different connection than the one used >> to support the session that is to be killed, then >> you will need ISID to distinguish the session that >> you want killed. >> >> Regards, >> Josh >> >> > -----Original Message----- >> > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] >> > Sent: Tuesday, March 13, 2001 9:57 PM >> > To: ips@ece.cmu.edu >> > Subject: Re: iSCSI: Logout needs ISID >> > >> > >> > >> > >> > Josh, >> > >> > There is something I am missing. The Logout can be issued >> > only after Login >> > (authentication and the rest). >> > The session is then implied. >> > >> > Regards, >> > Julo >> > >> > Joshua Tseng on 14/03/2001 03:56:45 >> > >> > Please respond to Joshua Tseng >> > >> > To: Julian Satran/Haifa/IBM@IBMIL >> > cc: ips@ece.cmu.edu >> > Subject: iSCSI: Logout needs ISID >> > >> > >> > >> > >> > Julian, >> > >> > Section 2.14 states that the logout command may >> > be sent on a second connection to clean up the >> > a single connection iSCSI session. If the reason >> > code is 0 (close the session), then ISID is needed >> > to identify the iSCSI session to close. >> > >> > Josh >> > >> > >> > > >-- >Mark A. Bakke >Cisco Systems >mbakke@cisco.com >763.398.1054 > > > > > > > From owner-ips@ece.cmu.edu Wed Mar 14 17:35:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA05791 for ; Wed, 14 Mar 2001 17:35:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2EKdhU20722 for ips-outgoing; Wed, 14 Mar 2001 15:39:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.brocade.com (asbestos.brocade.com [63.121.140.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2EHlSr07612 for ; Wed, 14 Mar 2001 12:47:28 -0500 (EST) Received: from thor.brocade.com (thor [192.168.126.45]) by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id JAA02179; Wed, 14 Mar 2001 09:47:14 -0800 (PST) Received: by thor.brocade.com with Internet Mail Service (5.5.2653.19) id ; Wed, 14 Mar 2001 09:47:14 -0800 Message-ID: From: Robert Snively To: "'Black_David@emc.com'" , Robert Snively , ips@ece.cmu.edu Subject: RE: FCIP iFCP encapsulation proposal Date: Wed, 14 Mar 2001 09:47:13 -0800 X-Mailer: Internet Mail Service (5.5.2653.19) Sender: owner-ips@ece.cmu.edu Precedence: bulk Dave, The byte stream that is processed by the iSCSI or FCIP layer is not a byte stream that can be generated by a hardware spoofer. It is the byte stream that is delivered by a TCP connection, riding on PDU's containing IP headers, riding on Ethernet frames carrying their own identification information. The spoofer must successfully generate an output that will meet the Ethernet format requirements for delivery, including device addressing; will then meet the IP requirements for delivery, including frame identification, IP addressing, fragmentation and reassembly, and CRC; will then meet the TCP/IP requirements for delivery, including proper TCP header formatting, proper checksumming, proper identification of the order and location of data, and the definition of a properly defined TCP connection, which can in turn only be created by a secure action which cannot be replicated; will then meet the requirements of an FC frame, including sequence id and frame count, exchange identification by both the source and sink, address verification by the sink, consistency with login state, and 32-bit CRC; and will finally meet the logical and state dependent requirements of the upper layer protocol, including logical unit addressing, command set requirements, consistency with read and write operations, and status presentation; all without being noticed. It will be very hard for you to convince me that this can all be done by dropping Ethernet frames with a bunch of replicated FCIP frame headers into a fabric. What you have to do is realistically and undetectably meld the TCP data stream from the legitimate connection and the TCP data stream from the false connection in a manner seamless enough so that the data will get delivered. Then, after delivering the data, you have to reconstitute a legal SCSI, VI, or other Fibre Channel protocol in a manner that will not cause it to be rejected or timed out by the higher level protocol. This by the way is one of the advantages of having long connection periods. There are very infrequent opportunities to formulate a potentially legitimate TCP/IP connection capable of carrying falsified data. There is of course the unfortunate side effect that, once you have succeeded in creating such a connection, you can obtain all the rights associated with such a connection, in which case no framing behavior could protect you anyway. In effect, the problem is not a "frame in data" problem. It is a "frame in successfully delivered TCP/IP data across a legitimate TCP/IP connection consistent with the fully defined states of the Fibre Channel FC-2 layer and consistent with the states, requirements, requests, and acknowledgements of the upper layer protocol" problem, which is darned unlikely even if you are trying maliciously to do it. Bob > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Wednesday, March 14, 2001 8:17 AM > To: rsnively@brocade.com; ips@ece.cmu.edu > Subject: RE: FCIP iFCP encapsulation proposal > > > Bob, > > > Sorry, I think we have a severe case of red herring here. > Any number of > > unlikely scenarios has been hypothesized, none of which (assuming > > there is any security in creating a TCP/IP connection at all) will > > cause any useful or dangerous behavior in either a SCSI target or > > a SCSI initiator. > > > > As an example, the information transmitted by FTP is first > processed into > > FTP messages, > > The ftp analogy is not indicative of what's going on here > because ftp does > not > have a resync mechanism - in other words, ftp never scans > the inbound byte > stream from TCP looking for some data pattern that indicates > the start of > an ftp message. Some of the "framing" proposals for iSCSI, iFCP, and > FCIP propose to do this or something close to it. > > The most severe case is one in which the framing data > pattern is used to > identify > a message that will be processed by SCSI or FC logic in an > upper layer. > iSCSI > needs to do this after a CRC failure if the CRC failure > requires discarding > the > length information in a header and the connection is not > closed. In this > case, > occurrence of the framing pattern in data could cause a PDU > to be extracted > from the data and processed - this would be very bad. > > If framing is used only for data steering (i.e., > organization of buffer > memory for > data that has been received, but not yet processed), the situation is > better, but > it's necessary for logic somewhere to know that the memory > organization > done by the framing may not always be correct, and be > prepared to copy the > data to get it correctly organized (e.g., 4k data block on a > 4k boundary) if > necessary. > > Going back to Vi Chau's encapsulation proposal at the start > of this thread, > there's a separate header CRC. If the header CRC check > fails, the frame > length info in that header cannot be used. If the TCP > connection is not > closed at that point, the arriving data has to be scanned > looking for the > next header. Vi wrote: > > > In the event of loss of synchronization, a state machine > which normally > > checks the FCIP/iFCP header CRC can be continuously enabled on the > > data stream which should provide for a "good CRC" compare when an > > FCIP/iFCP header arrives. > > This is subject to exactly the "frame in data" problem > caused by storing > a trace file - the state machine will find what looks like a > good header in > what's actually data, process it (and perhaps several, if a > number of short > trace frames are included in the actual data frame), only > realizing its > mistake > at some later point. This MUST NOT be allowed to happen. > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Wed Mar 14 17:36:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA05802 for ; Wed, 14 Mar 2001 17:36:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2EKnQp21454 for ips-outgoing; Wed, 14 Mar 2001 15:49:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2EKmpr21417 for ; Wed, 14 Mar 2001 15:48:51 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id PAA05395; Wed, 14 Mar 2001 15:48:44 -0500 (EST) Message-ID: <3AAFD92F.B799BA8@cisco.com> Date: Wed, 14 Mar 2001 14:48:47 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Venkat Rangan CC: IPS , jmuchow@cisco.com Subject: Re: iSCSI MIB Object Model References: <15851BD69CFCD41186B100B0D0AABE650C1AE0@med.corp.rhapsodynetworks.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Venkat- You have captured many of our open issues in this message. Please see my comments below. Thanks, Mark Venkat Rangan wrote: > > Mark, > > The UML is very nice, and the new object model, covering both > Initiator and Target instances is also nice. > > Overall, we do have some concern over the need for maintaining per-LU and > per-LUN statistics. In some iSCSI implementations, the iSCSI entity is > merely an intermediate layer, with no knowledge of LUs and LUNs. Placing > this in the GeneralInfo category and making it mandatory would add undue > burden on these implementations. One of our open questions is "how much is too much?". We added the LU and LUN statistics for a few reasons: 1. The iSCSI layer does see the LUN; it is included in the SCSI Command message. 2. For implementations that include the actual target and LUs, such as storage controllers, there is no other way to get information on LUNs through SNMP, since there is no SCSI MIB. 3. We felt that in this case customers might require LU- or LUN-level accounting. However, keeping statistics at this level does create some work; perhaps there is a way to make it optional. > In Section 5.9: > "The iscsiLUTable contains a list of iSCSI Logical Units that are > accessible via the iSCSI target." > > Would this list be the same as what the "Report LUNs" command would give? It depends. A LUN is just the address of a logical unit, and in the general case, is only valid between a particular initiator and a particular target. If a second initiator issues a "Report LUNs" to the same target, it may get back a different list if the target is performing some sort of LUN mapping. Some targets will show the same set of LUNs regardless of the initiator; some won't. There is no reliable way given the current MIB to find out what "Report LUNs" would return; this is probably the domain of a SCSI MIB. > In Section 5.7: > "The iscsiLunTable contains an entry for each known LUN that is being > accessed using this session. Note that is may not contain all of the > LUNs that could be accessed; the only ones available to iSCSI are the > ones that have been addressed specifically by iSCSI commands. > Therefore, LUNs that have been discovered via mechanisms such as the > SCSI "report LUNs" will not appear in this table until they are > specifically addressed by the initiator. > > How long do these entries live in the table? If iSCSI sessions logout and > close, do these entries continue to remain? The current intent is to have entries live only as long as the sessions; these would be removed when the session is removed. This is the simplest solution and would put the smallest burden on the implementation. However, we still have an open "requirements" question on whether anyone wants these to hang around longer. > Thanks, > > Venkat Rangan > Rhapsody Networks Inc. > http://www.rhapsodynetworks.com > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Wed Mar 14 18:24:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA06383 for ; Wed, 14 Mar 2001 18:24:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2ELJTe23594 for ips-outgoing; Wed, 14 Mar 2001 16:19:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2ELIcr23520 for ; Wed, 14 Mar 2001 16:18:42 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GA700CL7HRAQO@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Wed, 14 Mar 2001 13:16:23 -0800 (PST) Date: Wed, 14 Mar 2001 13:15:00 -0800 From: Douglas Otis Subject: RE: FCIP iFCP encapsulation proposal In-reply-to: To: Robert Snively , Black_David@emc.com, ips@ece.cmu.edu Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Bob, With out discussing spoofing where attackers successfully guess TCP sequences (made too easy in some cases), a binary image is stored and then legitimately sent as a payload, with the example being binary content of a debug analyzer. In this case, headers contained within the payload could be seen as valid. The valid header within the payload may fool a process that attempts to recover header synchronization following a dropped packet. This header may carry the same information in current use and be acted upon or send the connection into error oblivion. It would appear to represent a weakness that can be exploited. Dropped packets happen. Doug > Dave, > > The byte stream that is processed by the iSCSI or FCIP layer is > not a byte stream that can be generated by a hardware spoofer. > It is the byte stream that is delivered by a TCP connection, riding > on PDU's containing IP headers, riding on Ethernet frames carrying their > own identification information. The spoofer must successfully generate > an output that will meet the Ethernet format requirements for delivery, > including device addressing; will then meet the IP requirements > for delivery, > including frame identification, IP addressing, fragmentation and > reassembly, and CRC; will then meet the TCP/IP requirements for delivery, > including proper TCP header formatting, proper checksumming, proper > identification of the order and location of data, and the definition of > a properly defined TCP connection, which can in turn only be > created by a secure action which cannot be replicated; will then meet the > requirements of an FC frame, including sequence id and frame > count, exchange > identification by both the source and sink, address verification by > the sink, consistency with login state, and 32-bit CRC; and will finally > meet the logical and state dependent requirements of the upper layer > protocol, including logical unit addressing, command set requirements, > consistency with read and write operations, and status presentation; > all without being noticed. > > It will be very hard for you to convince me that this can all be done > by dropping Ethernet frames with a bunch of replicated FCIP frame headers > into a fabric. What you have to do is realistically and undetectably > meld the TCP data stream from the legitimate connection and the TCP > data stream from the false connection in a manner seamless enough so > that the data will get delivered. Then, after delivering the data, you > have to reconstitute a legal SCSI, VI, or other Fibre Channel protocol > in a manner that will not cause it to be rejected or timed out by the > higher level protocol. This by the way is one of the advantages of having > long connection periods. There are very infrequent opportunities to > formulate a potentially legitimate TCP/IP connection capable of carrying > falsified data. There is of course the unfortunate side effect that, > once you have succeeded in creating such a connection, you can obtain > all the rights associated with such a connection, in which case no > framing behavior could protect you anyway. > > In effect, the problem is not a "frame in data" problem. It is a > "frame in successfully delivered TCP/IP data across a legitimate > TCP/IP connection consistent with the fully defined states of > the Fibre Channel FC-2 layer and consistent with the states, > requirements, requests, and acknowledgements of the upper layer protocol" > problem, which is darned unlikely even if you are trying maliciously > to do it. > > Bob > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Wednesday, March 14, 2001 8:17 AM > > To: rsnively@brocade.com; ips@ece.cmu.edu > > Subject: RE: FCIP iFCP encapsulation proposal > > > > > > Bob, > > > > > Sorry, I think we have a severe case of red herring here. > > Any number of > > > unlikely scenarios has been hypothesized, none of which (assuming > > > there is any security in creating a TCP/IP connection at all) will > > > cause any useful or dangerous behavior in either a SCSI target or > > > a SCSI initiator. > > > > > > As an example, the information transmitted by FTP is first > > processed into > > > FTP messages, > > > > The ftp analogy is not indicative of what's going on here > > because ftp does > > not > > have a resync mechanism - in other words, ftp never scans > > the inbound byte > > stream from TCP looking for some data pattern that indicates > > the start of > > an ftp message. Some of the "framing" proposals for iSCSI, iFCP, and > > FCIP propose to do this or something close to it. > > > > The most severe case is one in which the framing data > > pattern is used to > > identify > > a message that will be processed by SCSI or FC logic in an > > upper layer. > > iSCSI > > needs to do this after a CRC failure if the CRC failure > > requires discarding > > the > > length information in a header and the connection is not > > closed. In this > > case, > > occurrence of the framing pattern in data could cause a PDU > > to be extracted > > from the data and processed - this would be very bad. > > > > If framing is used only for data steering (i.e., > > organization of buffer > > memory for > > data that has been received, but not yet processed), the situation is > > better, but > > it's necessary for logic somewhere to know that the memory > > organization > > done by the framing may not always be correct, and be > > prepared to copy the > > data to get it correctly organized (e.g., 4k data block on a > > 4k boundary) if > > necessary. > > > > Going back to Vi Chau's encapsulation proposal at the start > > of this thread, > > there's a separate header CRC. If the header CRC check > > fails, the frame > > length info in that header cannot be used. If the TCP > > connection is not > > closed at that point, the arriving data has to be scanned > > looking for the > > next header. Vi wrote: > > > > > In the event of loss of synchronization, a state machine > > which normally > > > checks the FCIP/iFCP header CRC can be continuously enabled on the > > > data stream which should provide for a "good CRC" compare when an > > > FCIP/iFCP header arrives. > > > > This is subject to exactly the "frame in data" problem > > caused by storing > > a trace file - the state machine will find what looks like a > > good header in > > what's actually data, process it (and perhaps several, if a > > number of short > > trace frames are included in the actual data frame), only > > realizing its > > mistake > > at some later point. This MUST NOT be allowed to happen. > > > > --David > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > > > > > From owner-ips@ece.cmu.edu Wed Mar 14 22:09:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA09318 for ; Wed, 14 Mar 2001 22:09:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2F1AXU09306 for ips-outgoing; Wed, 14 Mar 2001 20:10:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from crufty.research.bell-labs.com (crufty.research.bell-labs.com [204.178.16.49]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2F19wr09278 for ; Wed, 14 Mar 2001 20:09:58 -0500 (EST) Received: from grubby.research.bell-labs.com ([135.104.2.9]) by crufty; Wed Mar 14 20:07:28 EST 2001 Received: from aura.research.bell-labs.com ([135.104.46.10]) by grubby; Wed Mar 14 20:09:21 EST 2001 Received: from research.bell-labs.com (IDENT:sandeepj@sandeepj-pcmh.research.bell-labs.com [135.104.47.90]) by aura.research.bell-labs.com (8.9.1/8.9.1) with ESMTP id UAA15838 for ; Wed, 14 Mar 2001 20:09:21 -0500 (EST) Message-ID: <3AB01641.D6FEF78F@research.bell-labs.com> Date: Wed, 14 Mar 2001 20:09:21 -0500 From: Sandeep Joshi X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.2.16-3 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI: connection recovery Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit some connection recovery issues and some misc stuff: (1) SACK & expStatSN -------------------- Assume a SACK has been sent for say (type=statSN,begin=5,len=5) The target keeps sending more response PDUs (statSN=11,12,13..) before it receives the SACK. The initiator has held expStatSN back at expStatSN=5 until it gets the missing statSN PDUs. There is no pre-negotiated limit allowed between expStatSN and statSN. See note at bottom of Sec 1.2.2.2. "...a large difference at target indicates a failed connection". The lack of expStatSN increments will make the target think the connection is bad and can cause termination. This kind of defeats the purpose of connection recovery since the SACK was meant to prevent connection failure in the first place. So perhaps the statement above should say the target should continue with a large statSN difference when SACKs are outstanding ..? (2) Login-phase & connection recovery ------------------------------------- There should be a statement indicating that only connections in full-feature phase can be recovered. It seems pointless to hold target resources for recovering a connection on which no SCSI (i.e.only iSCSI) activity occurred. (e.g. LoginCmd/Response was completed for the connection and TextCmd, etc negotiation was in progress when the connection failed) (3) Ping PDUs & tasks --------------------- The NOP-Out & Nop-IN PDU have taskTag identifiers. This means the target & initiator have to hold resources for these ping commands. During connection recovery, some outdated Nop-In PDU will be resent just to clear out some outdated Nop-OUT task. And vice-versa. The purpose of NOPs is to test the connection or peer. If you get a ping, you would send an immediate response to avoid breaking the connection. If you get a "pong" (even for pings sent long ago), you *instantly* know the connection is fine. Unless I am missing some other purpose, the association of a ping with an iSCSI task seems superfluous and could be removed. (4) maxCmdSN limit ------------------- Looking at Sec 7.1, the case where independent network adapters are handling individiaul connections for a session at the target. I dont see how the maxCmdSN value sent in response PDUs can be incremented by independent network adapters, WITHOUT state sharing, to accurately reflect the global&local resource capacity. Who decides the next good value for maxCmdSN, and if it's ok across all boards ? This maxCmdSN value should really be replaced with something local and per-connection. The initiator can then decide the appropriate routing of commands. Regards, -Sandeep From owner-ips@ece.cmu.edu Wed Mar 14 22:11:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA09338 for ; Wed, 14 Mar 2001 22:11:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2F1TXX10501 for ips-outgoing; Wed, 14 Mar 2001 20:29:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2F1TRr10495 for ; Wed, 14 Mar 2001 20:29:27 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 935121143 for ; Wed, 14 Mar 2001 17:29:26 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA16989 for ; Wed, 14 Mar 2001 17:29:22 -0800 (PST) Message-ID: <3AB01BED.71384E55@cup.hp.com> Date: Wed, 14 Mar 2001 17:33:33 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Section 1.2.3. PDUs received prior to login. Content-Type: multipart/mixed; boundary="------------47F67A20BC87F9D693CDA209" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------47F67A20BC87F9D693CDA209 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit All, Rev 05 now advocates a reject policy as compared to the previous discard policy for all PDUs received by a target prior to login. Can anybody comment on why this change was made ? It seems like the policy to discard any PDUs received prior to login is safer, simpler and preferred compared to the current policy of responding with a Reject. Regards, Santosh --------------47F67A20BC87F9D693CDA209 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------47F67A20BC87F9D693CDA209-- From owner-ips@ece.cmu.edu Wed Mar 14 22:17:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA09376 for ; Wed, 14 Mar 2001 22:17:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2F0vY608506 for ips-outgoing; Wed, 14 Mar 2001 19:57:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp012.mail.yahoo.com (smtp012.mail.yahoo.com [216.136.173.32]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2F0ukr08431 for ; Wed, 14 Mar 2001 19:56:46 -0500 (EST) Received: from unknown (HELO somesh) (206.112.122.117) by smtp.mail.vip.sc5.yahoo.com with SMTP; 15 Mar 2001 00:56:42 -0000 X-Apparently-From: Reply-To: From: "Somesh Gupta" To: "Mark Bakke" , "Venkat Rangan" Cc: "IPS" , Subject: RE: iSCSI MIB Object Model Date: Wed, 14 Mar 2001 16:52:07 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <3AAFD92F.B799BA8@cisco.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mark, > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Mark Bakke > Sent: Wednesday, March 14, 2001 12:49 PM > To: Venkat Rangan > Cc: IPS; jmuchow@cisco.com > Subject: Re: iSCSI MIB Object Model > > > Venkat- > > You have captured many of our open issues in this > message. Please see my comments below. > > Thanks, > > Mark > > Venkat Rangan wrote: > > > > Mark, > > > > The UML is very nice, and the new object model, covering both > > Initiator and Target instances is also nice. > > > > Overall, we do have some concern over the need for maintaining > per-LU and > > per-LUN statistics. In some iSCSI implementations, the iSCSI entity is > > merely an intermediate layer, with no knowledge of LUs and LUNs. Placing > > this in the GeneralInfo category and making it mandatory would add undue > > burden on these implementations. > > One of our open questions is "how much is too much?". We added the LU > and LUN statistics for a few reasons: > > 1. The iSCSI layer does see the LUN; it is included in the SCSI Command > message. > > 2. For implementations that include the actual target and LUs, such > as storage controllers, there is no other way to get information > on LUNs through SNMP, since there is no SCSI MIB. > > 3. We felt that in this case customers might require LU- or LUN-level > accounting. I just did a rough (very rough) calculation for the amount of data required per lu and per lun - It is of the order of 256 bytes per lu and 512 bytes (actually more like 350) per lun. The number of Luns/lus can be in the 10s of thousands. Even though the iSCSI PDU header carries a LUN number, this value is really transparent to the iSCSI layer. The iSCSI layer should not even know how many LUNs there are in the system. The size of the data set, and the cost of managing it are fairly significant. The ability of systems to dynamically add & delete LUNs/LUs should not be tied to the ability of the iSCSI layer to track the stats. I think reasons 1 & 2 are not very good reasons. An analogy would be asking TCP to perpetually keep statistics on every 4-tuple that was ever used to create a connection. I would recommend focusing on the error cases only for one - and perhaps have a table per error type tracking the last "n" errors. What does it matter if an R2T was received for a LUN or not? It just becomes a bunch of probably unneeded information to weed through to get to the real info. > > However, keeping statistics at this level does create some work; perhaps > there is a way to make it optional. > > > In Section 5.9: > > "The iscsiLUTable contains a list of iSCSI Logical Units that are > > accessible via the iSCSI target." > > > > Would this list be the same as what the "Report LUNs" command > would give? > > It depends. A LUN is just the address of a logical unit, and in the > general case, is only valid between a particular initiator and a > particular > target. If a second initiator issues a "Report LUNs" to the same target, > it may get back a different list if the target is performing some sort of > LUN mapping. Some targets will show the same set of LUNs > regardless of the > initiator; some won't. There is no reliable way given the current MIB to > find out what "Report LUNs" would return; this is probably the domain of > a SCSI MIB. > > > In Section 5.7: > > "The iscsiLunTable contains an entry for each known LUN that is being > > accessed using this session. Note that is may not contain all of the > > LUNs that could be accessed; the only ones available to iSCSI are the > > ones that have been addressed specifically by iSCSI commands. > > Therefore, LUNs that have been discovered via mechanisms such as the > > SCSI "report LUNs" will not appear in this table until they are > > specifically addressed by the initiator. > > > > How long do these entries live in the table? If iSCSI sessions > logout and > > close, do these entries continue to remain? > > The current intent is to have entries live only as long as the sessions; > these would be removed when the session is removed. This is the simplest > solution and would put the smallest burden on the implementation. > > However, we still have an open "requirements" question on whether anyone > wants these to hang around longer. > > > Thanks, > > > > Venkat Rangan > > Rhapsody Networks Inc. > > http://www.rhapsodynetworks.com > > > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 Somesh _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Wed Mar 14 23:02:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA11930 for ; Wed, 14 Mar 2001 23:02:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2F1jcS11485 for ips-outgoing; Wed, 14 Mar 2001 20:45:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp015.mail.yahoo.com (smtp015.mail.yahoo.com [216.136.173.59]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2F1jRr11477 for ; Wed, 14 Mar 2001 20:45:27 -0500 (EST) Received: from unknown (HELO somesh) (206.112.122.117) by smtp.mail.vip.sc5.yahoo.com with SMTP; 15 Mar 2001 01:45:25 -0000 X-Apparently-From: Reply-To: From: "Somesh Gupta" To: , Subject: RE: iSCSI: Need to Kill Session from Surviving Node Date: Wed, 14 Mar 2001 17:40:49 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <200103142015.MAA15518@core.rose.hp.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Do you need both a "warm-reset" and a "cold-reset"? The "warm-reset" seems to be an over-optimization. Somesh > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Mallikarjun C. > Sent: Wednesday, March 14, 2001 12:16 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Need to Kill Session from Surviving Node > > > >There is a related problem that I think we need to address, and > that is -- > >the case where the Storage Controller is an Active-Active Fail-Over unit > >with very little state shared between the Storage Controller Nodes. (I > >think a number of you folks have told me that you are planning such a > >design.) This would mean that a specific initiator OS (WWUI) would have > >separate sessions between the Nodes. At fail-over, the > Take-Over Node will > >want to cause the Initiator, to Stop messing around with the Failing node > >ASAP, and start using the existing session, or start a new > session with the > >Take_Over Node. The Initiator, will normally just time-out the TCP/IP > >connection, and then turn the recovery over to the SCSI Layer. Then, when > >SCSI retries, iSCSI will either use the Session to the Surviving Node or > >Create a New Session to that Surviving Node. The problem is, that this > >will take a longer time then a system might want. > > > >So the question: is there a way for a surviving Target Node to tell the > >Initiator to "Kill" the Session to the Failing Target Node? > > > >Comments? > > Assuming that the failing Node is still able to execute commands, > a cold target reset from the Initiator to the failing node should "kill" > all the sessions to it. > > The answer to two hosts configured in a failover configuration sharing > a single Storage Controller Node is also the same. > > If it is active-active configuration, it appears to me that there's a > need to somehow prompt the Initiator to doing the reset on a third party > (Async PDU?). > > This is a reason why we'd still need target reset in iSCSI. > -- > Mallikarjun > > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > > > >. > >. > >. > >John L. Hufferd > >Senior Technical Staff Member (STSM) > >IBM/SSG San Jose Ca > >(408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > >Internet address: hufferd@us.ibm.com > > > > > >Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/14/2001 08:31:19 AM > > > >Sent by: owner-ips@ece.cmu.edu > > > > > >To: ips@ece.cmu.edu > >cc: > >Subject: Re: iSCSI: Logout needs ISID > > > > > > > > > > > >Mark, > > > >If you have only one connection you are supposed to use a Login with the > >restart bit one - and achieve a similar effect as a Login/Logout - i.e. > >keep the session alive. Even this might be a problem for a > target that is > >so poor it has only one socket (even SNMP won't work there). > For this case > >the only way out is to drop the connection and hope the target will hear > >you (it probably won't -:)). The comment is the draft is a memento for > >implementers to keep looking for new connections always (even for a one > >connection target) but probably it does not come through clear enough. > > > >Regards, > >Julo > > > >Mark Bakke on 14/03/2001 18:17:14 > > > >Please respond to Mark Bakke > > > >To: Julian Satran/Haifa/IBM@IBMIL > >cc: ips@ece.cmu.edu > >Subject: Re: iSCSI: Logout needs ISID > > > > > > > > > >Julian- > > > >A target that does not support multiple connections per session > >will return "Initiator SID Error" when the login attempt is made. > >In this case, there is no way to log in, so there's no way to > >log out the old connection. The initiator would be stuck waiting > >until the target's side of the connection times out and goes > >away, which could be a very long time. > > > >There is an open question within the MIB team about whether anyone > >needs to be able to kill connections or sessions from SNMP; however, > >I don't think that anyone will want to use SNMP as part of error > >recovery. > > > >-- > >Mark > > > >julian_satran@il.ibm.com wrote: > >> > >> Josh, > >> > >> No command, including logout, can be sent without a login. > >> The complete scenario is: > >> > >> -open a new connection > >> -login in the same session as the old connection (this has the ISID & > >TSID) > >> -logout the old connection > >> -recover commands > >> > >> You are not supposed to be able to kill a session from outside > (at least > >> not an iSCSI defined mode). > >> You will need management support for that (SNMP?) > >> > >> Regards, > >> Julo > >> > >> Joshua Tseng on 14/03/2001 17:09:48 > >> > >> Please respond to Joshua Tseng > >> > >> To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > >> cc: > >> Subject: RE: iSCSI: Logout needs ISID > >> > >> Julian, > >> > >> In 2.14, you state that a logout command can be > >> sent on a different connection to destroy a single- > >> connection iSCSI session. If you have multiple > >> sessions ongoing, and the logout command is sent > >> on a different connection than the one used > >> to support the session that is to be killed, then > >> you will need ISID to distinguish the session that > >> you want killed. > >> > >> Regards, > >> Josh > >> > >> > -----Original Message----- > >> > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > >> > Sent: Tuesday, March 13, 2001 9:57 PM > >> > To: ips@ece.cmu.edu > >> > Subject: Re: iSCSI: Logout needs ISID > >> > > >> > > >> > > >> > > >> > Josh, > >> > > >> > There is something I am missing. The Logout can be issued > >> > only after Login > >> > (authentication and the rest). > >> > The session is then implied. > >> > > >> > Regards, > >> > Julo > >> > > >> > Joshua Tseng on 14/03/2001 03:56:45 > >> > > >> > Please respond to Joshua Tseng > >> > > >> > To: Julian Satran/Haifa/IBM@IBMIL > >> > cc: ips@ece.cmu.edu > >> > Subject: iSCSI: Logout needs ISID > >> > > >> > > >> > > >> > > >> > Julian, > >> > > >> > Section 2.14 states that the logout command may > >> > be sent on a second connection to clean up the > >> > a single connection iSCSI session. If the reason > >> > code is 0 (close the session), then ISID is needed > >> > to identify the iSCSI session to close. > >> > > >> > Josh > >> > > >> > > >> > > > > >-- > >Mark A. Bakke > >Cisco Systems > >mbakke@cisco.com > >763.398.1054 > > > > > > > > > > > > > > > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Wed Mar 14 23:02:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA11941 for ; Wed, 14 Mar 2001 23:02:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2F2aYM14520 for ips-outgoing; Wed, 14 Mar 2001 21:36:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2F2Rsr14007 for ; Wed, 14 Mar 2001 21:27:54 -0500 (EST) Received: from ypportable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id GQ9BKBJA; Wed, 14 Mar 2001 18:27:52 -0800 From: "Y P Cheng" To: Subject: RE: FCIP iFCP encapsulation proposal Date: Wed, 14 Mar 2001 18:23:19 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > Bob, > With out discussing spoofing where attackers successfully guess TCP > sequences (made too easy in some cases), a binary image is stored and then > legitimately sent as a payload, with the example being binary content of a > debug analyzer. In this case, headers contained within the > payload could be seen as valid. The valid header within the payload > may fool a process that attempts to recover header synchronization > following a dropped packet. This header may carry the same information > in current use and be acted upon or send the connection into error oblivion. > It would appear to represent a weakness that can be exploited. > Dropped packets happen. > > Doug Like Bob said, this is a lot of red herring. First, I don't agree with scanning the payload to resync. There are better ways. But, if scanning is done, the probability of sending the TCP connection into error oblivision by mistaking the payload for a valid "frame" is almost non-existence. There is a lot information inside a FC frame that must be checked. Let me just name a few, SOF, S_ID, D_ID, OX_ID, RX_ID, Seq_ID, Sequence Count, R_CTL, F_CTL, CS_CTL, DF_CTL, Data Offset, and EOF etc. OK, so you got the trace data that seem to have all the valid stuff. It is NOT just these fields looking like a valid frame; they MUST match the saved information inside a valid Exchange Control Block (ECB) refer to by the OX_ID and RX_ID. Without matching the saved information in an ECB, the frame is always discarded. Y.P. Cheng, ConnectCom Solutions. From owner-ips@ece.cmu.edu Wed Mar 14 23:10:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA11984 for ; Wed, 14 Mar 2001 23:10:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2F2MdM13726 for ips-outgoing; Wed, 14 Mar 2001 21:22:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2F2Lrr13663 for ; Wed, 14 Mar 2001 21:21:53 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id BB0FAB7A2 for ; Wed, 14 Mar 2001 17:37:48 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA17461 for ; Wed, 14 Mar 2001 17:37:43 -0800 (PST) Message-ID: <3AB01DE1.2DD3C455@cup.hp.com> Date: Wed, 14 Mar 2001 17:41:53 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Section 2.4.2. Error Communication Latencies. Content-Type: multipart/mixed; boundary="------------20F1DB16A9F0831F3832B198" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------20F1DB16A9F0831F3832B198 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit All, The rev 05 draft mandates targets to wait until all the Data PDUs of an exchange have been received, prior to communicating any error with the I/O that may have occurred on an earlier I/O. Why is this needed ? This only adds additional latency to error communication from target to initiator and can cause large delays in communicating an error with long lasting I/Os like tape. The ability to communicate an error on its occurrence is something other SCSI transports provide and it requires the target to be capable of safeguarding against the arrival of stale frames of a PDU from an old exchange, something which should be possible and required in any case. Regards, Santosh Section 2.4.2, pg 36 --------------- "If a SCSI device error is detected while data from the initiator is still expected (the command PDU did not contain all the data and the target has not received a Data PDU with the final bit Set) the target MUST wait until it receives the a Data PDU with the F bit set before sending the Response PDU." --------------20F1DB16A9F0831F3832B198 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------20F1DB16A9F0831F3832B198-- From owner-ips@ece.cmu.edu Thu Mar 15 00:54:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA12934 for ; Thu, 15 Mar 2001 00:54:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2F4Vbj21648 for ips-outgoing; Wed, 14 Mar 2001 23:31:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp016.mail.yahoo.com (smtp016.mail.yahoo.com [216.136.174.113]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2F4Udr21561 for ; Wed, 14 Mar 2001 23:30:40 -0500 (EST) Received: from unknown (HELO somesh) (206.112.122.117) by smtp.mail.vip.sc5.yahoo.com with SMTP; 15 Mar 2001 04:30:35 -0000 X-Apparently-From: Reply-To: From: "Somesh Gupta" To: , Cc: Subject: RE: iSCSI: Need to Kill Session from Surviving Node Date: Wed, 14 Mar 2001 20:25:58 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <200103150331.TAA02310@core.rose.hp.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mallikarjun, I am not disagreeing with the desire to do a graceful termination. However, when recovery states at the TCP level, and at the iSCSI level get intertwined with this, it is probably better to focus on a clean (possibly graceful) termination and a clean bringup. Somesh > -----Original Message----- > From: cbm@core.rose.hp.com [mailto:cbm@core.rose.hp.com]On Behalf Of > Mallikarjun C. > Sent: Wednesday, March 14, 2001 7:31 PM > To: someshg@yahoo.com > Cc: ips@ece.cmu.edu > Subject: RE: iSCSI: Need to Kill Session from Surviving Node > > > Somesh, > > Please refer to the following URL (and emails around that) for > a long discussion on this topic. Both flavors of resets have value, > and I agree with Julian that iSCSI should support both. > > http://ips.pdl.cs.cmu.edu/mail/msg00740.html > > -- > Mallikarjun > > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > > >Do you need both a "warm-reset" and a "cold-reset"? > >The "warm-reset" seems to be an over-optimization. > > > >Somesh > > > >> -----Original Message----- > >> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > >> Mallikarjun C. > >> Sent: Wednesday, March 14, 2001 12:16 PM > >> To: ips@ece.cmu.edu > >> Subject: Re: iSCSI: Need to Kill Session from Surviving Node > >> > >> > >> >There is a related problem that I think we need to address, and > >> that is -- > >> >the case where the Storage Controller is an Active-Active > Fail-Over unit > >> >with very little state shared between the Storage Controller > Nodes. (I > >> >think a number of you folks have told me that you are planning such a > >> >design.) This would mean that a specific initiator OS (WWUI) > would have > >> >separate sessions between the Nodes. At fail-over, the > >> Take-Over Node will > >> >want to cause the Initiator, to Stop messing around with the > Failing node > >> >ASAP, and start using the existing session, or start a new > >> session with the > >> >Take_Over Node. The Initiator, will normally just time-out the TCP/IP > >> >connection, and then turn the recovery over to the SCSI > Layer. Then, when > >> >SCSI retries, iSCSI will either use the Session to the > Surviving Node or > >> >Create a New Session to that Surviving Node. The problem is, > that this > >> >will take a longer time then a system might want. > >> > > >> >So the question: is there a way for a surviving Target Node > to tell the > >> >Initiator to "Kill" the Session to the Failing Target Node? > >> > > >> >Comments? > >> > >> Assuming that the failing Node is still able to execute commands, > >> a cold target reset from the Initiator to the failing node > should "kill" > >> all the sessions to it. > >> > >> The answer to two hosts configured in a failover configuration sharing > >> a single Storage Controller Node is also the same. > >> > >> If it is active-active configuration, it appears to me that there's a > >> need to somehow prompt the Initiator to doing the reset on a > third party > >> (Async PDU?). > >> > >> This is a reason why we'd still need target reset in iSCSI. > >> -- > >> Mallikarjun > >> > >> > >> Mallikarjun Chadalapaka > >> Networked Storage Architecture > >> Network Storage Solutions Organization > >> MS 5668 Hewlett-Packard, Roseville. > >> cbm@rose.hp.com > >> > >> > > >> >. > >> >. > >> >. > >> >John L. Hufferd > >> >Senior Technical Staff Member (STSM) > >> >IBM/SSG San Jose Ca > >> >(408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > >> >Internet address: hufferd@us.ibm.com > >> > > >> > > >> >Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/14/2001 08:31:19 AM > >> > > >> >Sent by: owner-ips@ece.cmu.edu > >> > > >> > > >> >To: ips@ece.cmu.edu > >> >cc: > >> >Subject: Re: iSCSI: Logout needs ISID > >> > > >> > > >> > > >> > > >> > > >> >Mark, > >> > > >> >If you have only one connection you are supposed to use a > Login with the > >> >restart bit one - and achieve a similar effect as a > Login/Logout - i.e. > >> >keep the session alive. Even this might be a problem for a > >> target that is > >> >so poor it has only one socket (even SNMP won't work there). > >> For this case > >> >the only way out is to drop the connection and hope the > target will hear > >> >you (it probably won't -:)). The comment is the draft is a memento for > >> >implementers to keep looking for new connections always (even > for a one > >> >connection target) but probably it does not come through clear enough. > >> > > >> >Regards, > >> >Julo > >> > > >> >Mark Bakke on 14/03/2001 18:17:14 > >> > > >> >Please respond to Mark Bakke > >> > > >> >To: Julian Satran/Haifa/IBM@IBMIL > >> >cc: ips@ece.cmu.edu > >> >Subject: Re: iSCSI: Logout needs ISID > >> > > >> > > >> > > >> > > >> >Julian- > >> > > >> >A target that does not support multiple connections per session > >> >will return "Initiator SID Error" when the login attempt is made. > >> >In this case, there is no way to log in, so there's no way to > >> >log out the old connection. The initiator would be stuck waiting > >> >until the target's side of the connection times out and goes > >> >away, which could be a very long time. > >> > > >> >There is an open question within the MIB team about whether anyone > >> >needs to be able to kill connections or sessions from SNMP; however, > >> >I don't think that anyone will want to use SNMP as part of error > >> >recovery. > >> > > >> >-- > >> >Mark > >> > > >> >julian_satran@il.ibm.com wrote: > >> >> > >> >> Josh, > >> >> > >> >> No command, including logout, can be sent without a login. > >> >> The complete scenario is: > >> >> > >> >> -open a new connection > >> >> -login in the same session as the old connection (this has > the ISID & > >> >TSID) > >> >> -logout the old connection > >> >> -recover commands > >> >> > >> >> You are not supposed to be able to kill a session from outside > >> (at least > >> >> not an iSCSI defined mode). > >> >> You will need management support for that (SNMP?) > >> >> > >> >> Regards, > >> >> Julo > >> >> > >> >> Joshua Tseng on 14/03/2001 17:09:48 > >> >> > >> >> Please respond to Joshua Tseng > >> >> > >> >> To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > >> >> cc: > >> >> Subject: RE: iSCSI: Logout needs ISID > >> >> > >> >> Julian, > >> >> > >> >> In 2.14, you state that a logout command can be > >> >> sent on a different connection to destroy a single- > >> >> connection iSCSI session. If you have multiple > >> >> sessions ongoing, and the logout command is sent > >> >> on a different connection than the one used > >> >> to support the session that is to be killed, then > >> >> you will need ISID to distinguish the session that > >> >> you want killed. > >> >> > >> >> Regards, > >> >> Josh > >> >> > >> >> > -----Original Message----- > >> >> > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > >> >> > Sent: Tuesday, March 13, 2001 9:57 PM > >> >> > To: ips@ece.cmu.edu > >> >> > Subject: Re: iSCSI: Logout needs ISID > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > Josh, > >> >> > > >> >> > There is something I am missing. The Logout can be issued > >> >> > only after Login > >> >> > (authentication and the rest). > >> >> > The session is then implied. > >> >> > > >> >> > Regards, > >> >> > Julo > >> >> > > >> >> > Joshua Tseng on 14/03/2001 03:56:45 > >> >> > > >> >> > Please respond to Joshua Tseng > >> >> > > >> >> > To: Julian Satran/Haifa/IBM@IBMIL > >> >> > cc: ips@ece.cmu.edu > >> >> > Subject: iSCSI: Logout needs ISID > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > Julian, > >> >> > > >> >> > Section 2.14 states that the logout command may > >> >> > be sent on a second connection to clean up the > >> >> > a single connection iSCSI session. If the reason > >> >> > code is 0 (close the session), then ISID is needed > >> >> > to identify the iSCSI session to close. > >> >> > > >> >> > Josh > >> >> > > >> >> > > >> >> > > >> > > >> >-- > >> >Mark A. Bakke > >> >Cisco Systems > >> >mbakke@cisco.com > >> >763.398.1054 > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > > > > >_________________________________________________________ > >Do You Yahoo!? > >Get your free @yahoo.com address at http://mail.yahoo.com > > > > > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Thu Mar 15 00:55:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA12955 for ; Thu, 15 Mar 2001 00:55:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2F3V2717842 for ips-outgoing; Wed, 14 Mar 2001 22:31:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2F3UXr17760 for ; Wed, 14 Mar 2001 22:30:33 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel2.hp.com (Postfix) with ESMTP id 7478C1403; Wed, 14 Mar 2001 22:30:32 -0500 (EST) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id TAA02310; Wed, 14 Mar 2001 19:31:30 -0800 (PST) Message-Id: <200103150331.TAA02310@core.rose.hp.com> Subject: RE: iSCSI: Need to Kill Session from Surviving Node To: someshg@yahoo.com () Date: Wed, 14 Mar 2001 19:31:29 PST Cc: ips@ece.cmu.edu In-Reply-To: ; from "Somesh Gupta" at Mar 14, 101 7:00 pm Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, Please refer to the following URL (and emails around that) for a long discussion on this topic. Both flavors of resets have value, and I agree with Julian that iSCSI should support both. http://ips.pdl.cs.cmu.edu/mail/msg00740.html -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com >Do you need both a "warm-reset" and a "cold-reset"? >The "warm-reset" seems to be an over-optimization. > >Somesh > >> -----Original Message----- >> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >> Mallikarjun C. >> Sent: Wednesday, March 14, 2001 12:16 PM >> To: ips@ece.cmu.edu >> Subject: Re: iSCSI: Need to Kill Session from Surviving Node >> >> >> >There is a related problem that I think we need to address, and >> that is -- >> >the case where the Storage Controller is an Active-Active Fail-Over unit >> >with very little state shared between the Storage Controller Nodes. (I >> >think a number of you folks have told me that you are planning such a >> >design.) This would mean that a specific initiator OS (WWUI) would have >> >separate sessions between the Nodes. At fail-over, the >> Take-Over Node will >> >want to cause the Initiator, to Stop messing around with the Failing node >> >ASAP, and start using the existing session, or start a new >> session with the >> >Take_Over Node. The Initiator, will normally just time-out the TCP/IP >> >connection, and then turn the recovery over to the SCSI Layer. Then, when >> >SCSI retries, iSCSI will either use the Session to the Surviving Node or >> >Create a New Session to that Surviving Node. The problem is, that this >> >will take a longer time then a system might want. >> > >> >So the question: is there a way for a surviving Target Node to tell the >> >Initiator to "Kill" the Session to the Failing Target Node? >> > >> >Comments? >> >> Assuming that the failing Node is still able to execute commands, >> a cold target reset from the Initiator to the failing node should "kill" >> all the sessions to it. >> >> The answer to two hosts configured in a failover configuration sharing >> a single Storage Controller Node is also the same. >> >> If it is active-active configuration, it appears to me that there's a >> need to somehow prompt the Initiator to doing the reset on a third party >> (Async PDU?). >> >> This is a reason why we'd still need target reset in iSCSI. >> -- >> Mallikarjun >> >> >> Mallikarjun Chadalapaka >> Networked Storage Architecture >> Network Storage Solutions Organization >> MS 5668 Hewlett-Packard, Roseville. >> cbm@rose.hp.com >> >> > >> >. >> >. >> >. >> >John L. Hufferd >> >Senior Technical Staff Member (STSM) >> >IBM/SSG San Jose Ca >> >(408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 >> >Internet address: hufferd@us.ibm.com >> > >> > >> >Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/14/2001 08:31:19 AM >> > >> >Sent by: owner-ips@ece.cmu.edu >> > >> > >> >To: ips@ece.cmu.edu >> >cc: >> >Subject: Re: iSCSI: Logout needs ISID >> > >> > >> > >> > >> > >> >Mark, >> > >> >If you have only one connection you are supposed to use a Login with the >> >restart bit one - and achieve a similar effect as a Login/Logout - i.e. >> >keep the session alive. Even this might be a problem for a >> target that is >> >so poor it has only one socket (even SNMP won't work there). >> For this case >> >the only way out is to drop the connection and hope the target will hear >> >you (it probably won't -:)). The comment is the draft is a memento for >> >implementers to keep looking for new connections always (even for a one >> >connection target) but probably it does not come through clear enough. >> > >> >Regards, >> >Julo >> > >> >Mark Bakke on 14/03/2001 18:17:14 >> > >> >Please respond to Mark Bakke >> > >> >To: Julian Satran/Haifa/IBM@IBMIL >> >cc: ips@ece.cmu.edu >> >Subject: Re: iSCSI: Logout needs ISID >> > >> > >> > >> > >> >Julian- >> > >> >A target that does not support multiple connections per session >> >will return "Initiator SID Error" when the login attempt is made. >> >In this case, there is no way to log in, so there's no way to >> >log out the old connection. The initiator would be stuck waiting >> >until the target's side of the connection times out and goes >> >away, which could be a very long time. >> > >> >There is an open question within the MIB team about whether anyone >> >needs to be able to kill connections or sessions from SNMP; however, >> >I don't think that anyone will want to use SNMP as part of error >> >recovery. >> > >> >-- >> >Mark >> > >> >julian_satran@il.ibm.com wrote: >> >> >> >> Josh, >> >> >> >> No command, including logout, can be sent without a login. >> >> The complete scenario is: >> >> >> >> -open a new connection >> >> -login in the same session as the old connection (this has the ISID & >> >TSID) >> >> -logout the old connection >> >> -recover commands >> >> >> >> You are not supposed to be able to kill a session from outside >> (at least >> >> not an iSCSI defined mode). >> >> You will need management support for that (SNMP?) >> >> >> >> Regards, >> >> Julo >> >> >> >> Joshua Tseng on 14/03/2001 17:09:48 >> >> >> >> Please respond to Joshua Tseng >> >> >> >> To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu >> >> cc: >> >> Subject: RE: iSCSI: Logout needs ISID >> >> >> >> Julian, >> >> >> >> In 2.14, you state that a logout command can be >> >> sent on a different connection to destroy a single- >> >> connection iSCSI session. If you have multiple >> >> sessions ongoing, and the logout command is sent >> >> on a different connection than the one used >> >> to support the session that is to be killed, then >> >> you will need ISID to distinguish the session that >> >> you want killed. >> >> >> >> Regards, >> >> Josh >> >> >> >> > -----Original Message----- >> >> > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] >> >> > Sent: Tuesday, March 13, 2001 9:57 PM >> >> > To: ips@ece.cmu.edu >> >> > Subject: Re: iSCSI: Logout needs ISID >> >> > >> >> > >> >> > >> >> > >> >> > Josh, >> >> > >> >> > There is something I am missing. The Logout can be issued >> >> > only after Login >> >> > (authentication and the rest). >> >> > The session is then implied. >> >> > >> >> > Regards, >> >> > Julo >> >> > >> >> > Joshua Tseng on 14/03/2001 03:56:45 >> >> > >> >> > Please respond to Joshua Tseng >> >> > >> >> > To: Julian Satran/Haifa/IBM@IBMIL >> >> > cc: ips@ece.cmu.edu >> >> > Subject: iSCSI: Logout needs ISID >> >> > >> >> > >> >> > >> >> > >> >> > Julian, >> >> > >> >> > Section 2.14 states that the logout command may >> >> > be sent on a second connection to clean up the >> >> > a single connection iSCSI session. If the reason >> >> > code is 0 (close the session), then ISID is needed >> >> > to identify the iSCSI session to close. >> >> > >> >> > Josh >> >> > >> >> > >> >> > >> > >> >-- >> >Mark A. Bakke >> >Cisco Systems >> >mbakke@cisco.com >> >763.398.1054 >> > >> > >> > >> > >> > >> > >> > >> > > >_________________________________________________________ >Do You Yahoo!? >Get your free @yahoo.com address at http://mail.yahoo.com > > From owner-ips@ece.cmu.edu Thu Mar 15 02:40:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA26756 for ; Thu, 15 Mar 2001 02:40:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2F5MdP24689 for ips-outgoing; Thu, 15 Mar 2001 00:22:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2F5MPr24679 for ; Thu, 15 Mar 2001 00:22:25 -0500 (EST) Received: from f2n22e. (westrelay01.boulder.ibm.com [9.99.140.22]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id AAA24964 for ; Thu, 15 Mar 2001 00:15:10 -0500 Received: from f4n49e (d03nm065h.boulder.ibm.com [9.99.140.49]) by f2n22e. (8.8.8m3/NCO v4.95) with ESMTP id WAA166190 for ; Wed, 14 Mar 2001 22:22:22 -0700 Importance: Normal Subject: RE: iSCSI: Need to Kill Session from Surviving Node To: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Wed, 14 Mar 2001 21:21:03 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/14/2001 10:22:21 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Wait a minute, I did not expect my Surviving Node question to somehow trigger the discussion of whether the Target Reset should be issued by iSCSI Initiators. That is a completely different Pit. My question was, when the Storage Controller fails over to a surviving NODE, how do we tell the Initiator that is waiting peacefully for the TCP/IP connection to time-out. To quickly give-up and let the initiator's SCSI layer, retry the operations, so that the iSCSI layer can start either sending the SCSI cmds to the Surviving Node, or starts a new Session with the Surviving Node. This has nothing to do with the Initiator issuing Target Resets. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com "Somesh Gupta" @ece.cmu.edu on 03/14/2001 08:25:58 PM Please respond to Sent by: owner-ips@ece.cmu.edu To: , cc: Subject: RE: iSCSI: Need to Kill Session from Surviving Node Mallikarjun, I am not disagreeing with the desire to do a graceful termination. However, when recovery states at the TCP level, and at the iSCSI level get intertwined with this, it is probably better to focus on a clean (possibly graceful) termination and a clean bringup. Somesh > -----Original Message----- > From: cbm@core.rose.hp.com [mailto:cbm@core.rose.hp.com]On Behalf Of > Mallikarjun C. > Sent: Wednesday, March 14, 2001 7:31 PM > To: someshg@yahoo.com > Cc: ips@ece.cmu.edu > Subject: RE: iSCSI: Need to Kill Session from Surviving Node > > > Somesh, > > Please refer to the following URL (and emails around that) for > a long discussion on this topic. Both flavors of resets have value, > and I agree with Julian that iSCSI should support both. > > http://ips.pdl.cs.cmu.edu/mail/msg00740.html > > -- > Mallikarjun > > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > > >Do you need both a "warm-reset" and a "cold-reset"? > >The "warm-reset" seems to be an over-optimization. > > > >Somesh > > > >> -----Original Message----- > >> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > >> Mallikarjun C. > >> Sent: Wednesday, March 14, 2001 12:16 PM > >> To: ips@ece.cmu.edu > >> Subject: Re: iSCSI: Need to Kill Session from Surviving Node > >> > >> > >> >There is a related problem that I think we need to address, and > >> that is -- > >> >the case where the Storage Controller is an Active-Active > Fail-Over unit > >> >with very little state shared between the Storage Controller > Nodes. (I > >> >think a number of you folks have told me that you are planning such a > >> >design.) This would mean that a specific initiator OS (WWUI) > would have > >> >separate sessions between the Nodes. At fail-over, the > >> Take-Over Node will > >> >want to cause the Initiator, to Stop messing around with the > Failing node > >> >ASAP, and start using the existing session, or start a new > >> session with the > >> >Take_Over Node. The Initiator, will normally just time-out the TCP/IP > >> >connection, and then turn the recovery over to the SCSI > Layer. Then, when > >> >SCSI retries, iSCSI will either use the Session to the > Surviving Node or > >> >Create a New Session to that Surviving Node. The problem is, > that this > >> >will take a longer time then a system might want. > >> > > >> >So the question: is there a way for a surviving Target Node > to tell the > >> >Initiator to "Kill" the Session to the Failing Target Node? > >> > > >> >Comments? > >> > >> Assuming that the failing Node is still able to execute commands, > >> a cold target reset from the Initiator to the failing node > should "kill" > >> all the sessions to it. > >> > >> The answer to two hosts configured in a failover configuration sharing > >> a single Storage Controller Node is also the same. > >> > >> If it is active-active configuration, it appears to me that there's a > >> need to somehow prompt the Initiator to doing the reset on a > third party > >> (Async PDU?). > >> > >> This is a reason why we'd still need target reset in iSCSI. > >> -- > >> Mallikarjun > >> > >> > >> Mallikarjun Chadalapaka > >> Networked Storage Architecture > >> Network Storage Solutions Organization > >> MS 5668 Hewlett-Packard, Roseville. > >> cbm@rose.hp.com > >> > >> > > >> >. > >> >. > >> >. > >> >John L. Hufferd > >> >Senior Technical Staff Member (STSM) > >> >IBM/SSG San Jose Ca > >> >(408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > >> >Internet address: hufferd@us.ibm.com > >> > > >> > > >> >Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/14/2001 08:31:19 AM > >> > > >> >Sent by: owner-ips@ece.cmu.edu > >> > > >> > > >> >To: ips@ece.cmu.edu > >> >cc: > >> >Subject: Re: iSCSI: Logout needs ISID > >> > > >> > > >> > > >> > > >> > > >> >Mark, > >> > > >> >If you have only one connection you are supposed to use a > Login with the > >> >restart bit one - and achieve a similar effect as a > Login/Logout - i.e. > >> >keep the session alive. Even this might be a problem for a > >> target that is > >> >so poor it has only one socket (even SNMP won't work there). > >> For this case > >> >the only way out is to drop the connection and hope the > target will hear > >> >you (it probably won't -:)). The comment is the draft is a memento for > >> >implementers to keep looking for new connections always (even > for a one > >> >connection target) but probably it does not come through clear enough. > >> > > >> >Regards, > >> >Julo > >> > > >> >Mark Bakke on 14/03/2001 18:17:14 > >> > > >> >Please respond to Mark Bakke > >> > > >> >To: Julian Satran/Haifa/IBM@IBMIL > >> >cc: ips@ece.cmu.edu > >> >Subject: Re: iSCSI: Logout needs ISID > >> > > >> > > >> > > >> > > >> >Julian- > >> > > >> >A target that does not support multiple connections per session > >> >will return "Initiator SID Error" when the login attempt is made. > >> >In this case, there is no way to log in, so there's no way to > >> >log out the old connection. The initiator would be stuck waiting > >> >until the target's side of the connection times out and goes > >> >away, which could be a very long time. > >> > > >> >There is an open question within the MIB team about whether anyone > >> >needs to be able to kill connections or sessions from SNMP; however, > >> >I don't think that anyone will want to use SNMP as part of error > >> >recovery. > >> > > >> >-- > >> >Mark > >> > > >> >julian_satran@il.ibm.com wrote: > >> >> > >> >> Josh, > >> >> > >> >> No command, including logout, can be sent without a login. > >> >> The complete scenario is: > >> >> > >> >> -open a new connection > >> >> -login in the same session as the old connection (this has > the ISID & > >> >TSID) > >> >> -logout the old connection > >> >> -recover commands > >> >> > >> >> You are not supposed to be able to kill a session from outside > >> (at least > >> >> not an iSCSI defined mode). > >> >> You will need management support for that (SNMP?) > >> >> > >> >> Regards, > >> >> Julo > >> >> > >> >> Joshua Tseng on 14/03/2001 17:09:48 > >> >> > >> >> Please respond to Joshua Tseng > >> >> > >> >> To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > >> >> cc: > >> >> Subject: RE: iSCSI: Logout needs ISID > >> >> > >> >> Julian, > >> >> > >> >> In 2.14, you state that a logout command can be > >> >> sent on a different connection to destroy a single- > >> >> connection iSCSI session. If you have multiple > >> >> sessions ongoing, and the logout command is sent > >> >> on a different connection than the one used > >> >> to support the session that is to be killed, then > >> >> you will need ISID to distinguish the session that > >> >> you want killed. > >> >> > >> >> Regards, > >> >> Josh > >> >> > >> >> > -----Original Message----- > >> >> > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > >> >> > Sent: Tuesday, March 13, 2001 9:57 PM > >> >> > To: ips@ece.cmu.edu > >> >> > Subject: Re: iSCSI: Logout needs ISID > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > Josh, > >> >> > > >> >> > There is something I am missing. The Logout can be issued > >> >> > only after Login > >> >> > (authentication and the rest). > >> >> > The session is then implied. > >> >> > > >> >> > Regards, > >> >> > Julo > >> >> > > >> >> > Joshua Tseng on 14/03/2001 03:56:45 > >> >> > > >> >> > Please respond to Joshua Tseng > >> >> > > >> >> > To: Julian Satran/Haifa/IBM@IBMIL > >> >> > cc: ips@ece.cmu.edu > >> >> > Subject: iSCSI: Logout needs ISID > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > Julian, > >> >> > > >> >> > Section 2.14 states that the logout command may > >> >> > be sent on a second connection to clean up the > >> >> > a single connection iSCSI session. If the reason > >> >> > code is 0 (close the session), then ISID is needed > >> >> > to identify the iSCSI session to close. > >> >> > > >> >> > Josh > >> >> > > >> >> > > >> >> > > >> > > >> >-- > >> >Mark A. Bakke > >> >Cisco Systems > >> >mbakke@cisco.com > >> >763.398.1054 > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > > > > >_________________________________________________________ > >Do You Yahoo!? > >Get your free @yahoo.com address at http://mail.yahoo.com > > > > > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Thu Mar 15 05:53:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA28161 for ; Thu, 15 Mar 2001 05:53:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2F8Ths05132 for ips-outgoing; Thu, 15 Mar 2001 03:29:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2F8TWr05124 for ; Thu, 15 Mar 2001 03:29:33 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id JAA155666 for ; Thu, 15 Mar 2001 09:29:25 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id JAA101708 for ; Thu, 15 Mar 2001 09:28:11 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A10.002E73E2 ; Thu, 15 Mar 2001 09:27:23 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 15 Mar 2001 10:29:49 +0200 Subject: Re: iSCSI MIB Object Model Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mark, iSCSI uses LUN almost transparently. Also the LUN may change while a session is active. I am not sure that the fact that there is no SCSI MIB (yet) doe justify including those items in the iSCSI MIB. I also recall (a rumor?) that somebody is working on a SCSI MIB (it will be needed anyhow there is more to SCSI than the LU). Julo Mark Bakke on 14/03/2001 22:48:47 Please respond to Mark Bakke To: Venkat Rangan cc: IPS , jmuchow@cisco.com Subject: Re: iSCSI MIB Object Model Venkat- You have captured many of our open issues in this message. Please see my comments below. Thanks, Mark Venkat Rangan wrote: > > Mark, > > The UML is very nice, and the new object model, covering both > Initiator and Target instances is also nice. > > Overall, we do have some concern over the need for maintaining per-LU and > per-LUN statistics. In some iSCSI implementations, the iSCSI entity is > merely an intermediate layer, with no knowledge of LUs and LUNs. Placing > this in the GeneralInfo category and making it mandatory would add undue > burden on these implementations. One of our open questions is "how much is too much?". We added the LU and LUN statistics for a few reasons: 1. The iSCSI layer does see the LUN; it is included in the SCSI Command message. 2. For implementations that include the actual target and LUs, such as storage controllers, there is no other way to get information on LUNs through SNMP, since there is no SCSI MIB. 3. We felt that in this case customers might require LU- or LUN-level accounting. However, keeping statistics at this level does create some work; perhaps there is a way to make it optional. > In Section 5.9: > "The iscsiLUTable contains a list of iSCSI Logical Units that are > accessible via the iSCSI target." > > Would this list be the same as what the "Report LUNs" command would give? It depends. A LUN is just the address of a logical unit, and in the general case, is only valid between a particular initiator and a particular target. If a second initiator issues a "Report LUNs" to the same target, it may get back a different list if the target is performing some sort of LUN mapping. Some targets will show the same set of LUNs regardless of the initiator; some won't. There is no reliable way given the current MIB to find out what "Report LUNs" would return; this is probably the domain of a SCSI MIB. > In Section 5.7: > "The iscsiLunTable contains an entry for each known LUN that is being > accessed using this session. Note that is may not contain all of the > LUNs that could be accessed; the only ones available to iSCSI are the > ones that have been addressed specifically by iSCSI commands. > Therefore, LUNs that have been discovered via mechanisms such as the > SCSI "report LUNs" will not appear in this table until they are > specifically addressed by the initiator. > > How long do these entries live in the table? If iSCSI sessions logout and > close, do these entries continue to remain? The current intent is to have entries live only as long as the sessions; these would be removed when the session is removed. This is the simplest solution and would put the smallest burden on the implementation. However, we still have an open "requirements" question on whether anyone wants these to hang around longer. > Thanks, > > Venkat Rangan > Rhapsody Networks Inc. > http://www.rhapsodynetworks.com > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Thu Mar 15 06:51:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA28413 for ; Thu, 15 Mar 2001 06:51:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2F9ijT09133 for ips-outgoing; Thu, 15 Mar 2001 04:44:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from bramg1.net.external.hp.com (bramg1.net.external.hp.com [192.6.126.73]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2F9hvr09109 for ; Thu, 15 Mar 2001 04:43:57 -0500 (EST) Received: from quasit.br.itc.hp.com (quasit.br.itc.hp.com [15.145.8.135]) by bramg1.net.external.hp.com (Postfix) with ESMTP id 74D429A for ; Thu, 15 Mar 2001 10:43:40 +0100 (MET) Received: from fowey.BR.ITC.HP.COM (fowey.br.itc.hp.com [15.145.8.186]) by quasit.br.itc.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id JAA15216 for ; Thu, 15 Mar 2001 09:43:35 GMT Received: from 15.145.8.186 by fowey.BR.ITC.HP.COM (InterScan E-Mail VirusWall NT); Thu, 15 Mar 2001 09:43:35 -0000 (GMT Standard Time) Received: by fowey.br.itc.hp.com with Internet Mail Service (5.5.2650.21) id ; Thu, 15 Mar 2001 09:43:35 -0000 Message-ID: <0B9A57FF1D57D411B47500D0B73E5CC101E7A5BB@dickens.bri.hp.com> From: "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" To: ips@ece.cmu.edu Subject: iSCSI Login Date: Thu, 15 Mar 2001 09:43:33 -0000 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Julian, A couple of items: 1) Section 4.1 (iSCSI Rev 5) states that: The target can answer in the following ways: -Login Response with Login Reject (and F bit 1). This is an immediate rejection from the target, that causes the session to terminate I agree that this could be the case for a security breach - authentication failed but what if the target only supports one connection per session and the initiator is attempting to set up another connection. Surely, the new login should be rejected but the session remains intact. 2) Still on the subject of login: In section 4, page 74, the spec states that: "The initiator and target MAY want to negotiate authentication and data integrity parameters. Once this negotiation is completed, the channel is considered secure." It is unclear as to the mandated handling of conflicting/differing authentication mechanisms negotiated on multiple connections participating in the same session. I propose that the spec should state that if authentication is required then the same authentication method MUST be used on all connections in a session. Cheers Matthew Burbridge Network Infrastructure Solutions Hewlett Packard Bristol +44 117 312 7010 E-mail: matthewb@bri.hp.com From owner-ips@ece.cmu.edu Thu Mar 15 06:51:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA28424 for ; Thu, 15 Mar 2001 06:51:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FAHlo20430 for ips-outgoing; Thu, 15 Mar 2001 05:17:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FAHCr19121 for ; Thu, 15 Mar 2001 05:17:12 -0500 (EST) Received: from f2n22e. (westrelay01.boulder.ibm.com [9.99.140.22]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id FAA82314; Thu, 15 Mar 2001 05:09:58 -0500 Received: from f4n49e (d03nm065h.boulder.ibm.com [9.99.140.49]) by f2n22e. (8.8.8m3/NCO v4.95) with ESMTP id DAA199896; Thu, 15 Mar 2001 03:17:07 -0700 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI: Need to Kill Session from Surviving Node To: cbm@rose.hp.com Cc: someshg@yahoo.com (), ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Thu, 15 Mar 2001 02:16:48 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/15/2001 03:17:10 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Target Reset have value, but not as an in-line command. It is a management function, not a recovery function from a single initiator, and as such should be up to the implementation to see how the accomplish the Target Reset via there own admin facility. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com "Mallikarjun C." @ece.cmu.edu on 03/14/2001 07:31:29 PM Please respond to cbm@rose.hp.com Sent by: owner-ips@ece.cmu.edu To: someshg@yahoo.com () cc: ips@ece.cmu.edu Subject: RE: iSCSI: Need to Kill Session from Surviving Node Somesh, Please refer to the following URL (and emails around that) for a long discussion on this topic. Both flavors of resets have value, and I agree with Julian that iSCSI should support both. http://ips.pdl.cs.cmu.edu/mail/msg00740.html -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com >Do you need both a "warm-reset" and a "cold-reset"? >The "warm-reset" seems to be an over-optimization. > >Somesh > >> -----Original Message----- >> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >> Mallikarjun C. >> Sent: Wednesday, March 14, 2001 12:16 PM >> To: ips@ece.cmu.edu >> Subject: Re: iSCSI: Need to Kill Session from Surviving Node >> >> >> >There is a related problem that I think we need to address, and >> that is -- >> >the case where the Storage Controller is an Active-Active Fail-Over unit >> >with very little state shared between the Storage Controller Nodes. (I >> >think a number of you folks have told me that you are planning such a >> >design.) This would mean that a specific initiator OS (WWUI) would have >> >separate sessions between the Nodes. At fail-over, the >> Take-Over Node will >> >want to cause the Initiator, to Stop messing around with the Failing node >> >ASAP, and start using the existing session, or start a new >> session with the >> >Take_Over Node. The Initiator, will normally just time-out the TCP/IP >> >connection, and then turn the recovery over to the SCSI Layer. Then, when >> >SCSI retries, iSCSI will either use the Session to the Surviving Node or >> >Create a New Session to that Surviving Node. The problem is, that this >> >will take a longer time then a system might want. >> > >> >So the question: is there a way for a surviving Target Node to tell the >> >Initiator to "Kill" the Session to the Failing Target Node? >> > >> >Comments? >> >> Assuming that the failing Node is still able to execute commands, >> a cold target reset from the Initiator to the failing node should "kill" >> all the sessions to it. >> >> The answer to two hosts configured in a failover configuration sharing >> a single Storage Controller Node is also the same. >> >> If it is active-active configuration, it appears to me that there's a >> need to somehow prompt the Initiator to doing the reset on a third party >> (Async PDU?). >> >> This is a reason why we'd still need target reset in iSCSI. >> -- >> Mallikarjun >> >> >> Mallikarjun Chadalapaka >> Networked Storage Architecture >> Network Storage Solutions Organization >> MS 5668 Hewlett-Packard, Roseville. >> cbm@rose.hp.com >> >> > >> >. >> >. >> >. >> >John L. Hufferd >> >Senior Technical Staff Member (STSM) >> >IBM/SSG San Jose Ca >> >(408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 >> >Internet address: hufferd@us.ibm.com >> > >> > >> >Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/14/2001 08:31:19 AM >> > >> >Sent by: owner-ips@ece.cmu.edu >> > >> > >> >To: ips@ece.cmu.edu >> >cc: >> >Subject: Re: iSCSI: Logout needs ISID >> > >> > >> > >> > >> > >> >Mark, >> > >> >If you have only one connection you are supposed to use a Login with the >> >restart bit one - and achieve a similar effect as a Login/Logout - i.e. >> >keep the session alive. Even this might be a problem for a >> target that is >> >so poor it has only one socket (even SNMP won't work there). >> For this case >> >the only way out is to drop the connection and hope the target will hear >> >you (it probably won't -:)). The comment is the draft is a memento for >> >implementers to keep looking for new connections always (even for a one >> >connection target) but probably it does not come through clear enough. >> > >> >Regards, >> >Julo >> > >> >Mark Bakke on 14/03/2001 18:17:14 >> > >> >Please respond to Mark Bakke >> > >> >To: Julian Satran/Haifa/IBM@IBMIL >> >cc: ips@ece.cmu.edu >> >Subject: Re: iSCSI: Logout needs ISID >> > >> > >> > >> > >> >Julian- >> > >> >A target that does not support multiple connections per session >> >will return "Initiator SID Error" when the login attempt is made. >> >In this case, there is no way to log in, so there's no way to >> >log out the old connection. The initiator would be stuck waiting >> >until the target's side of the connection times out and goes >> >away, which could be a very long time. >> > >> >There is an open question within the MIB team about whether anyone >> >needs to be able to kill connections or sessions from SNMP; however, >> >I don't think that anyone will want to use SNMP as part of error >> >recovery. >> > >> >-- >> >Mark >> > >> >julian_satran@il.ibm.com wrote: >> >> >> >> Josh, >> >> >> >> No command, including logout, can be sent without a login. >> >> The complete scenario is: >> >> >> >> -open a new connection >> >> -login in the same session as the old connection (this has the ISID & >> >TSID) >> >> -logout the old connection >> >> -recover commands >> >> >> >> You are not supposed to be able to kill a session from outside >> (at least >> >> not an iSCSI defined mode). >> >> You will need management support for that (SNMP?) >> >> >> >> Regards, >> >> Julo >> >> >> >> Joshua Tseng on 14/03/2001 17:09:48 >> >> >> >> Please respond to Joshua Tseng >> >> >> >> To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu >> >> cc: >> >> Subject: RE: iSCSI: Logout needs ISID >> >> >> >> Julian, >> >> >> >> In 2.14, you state that a logout command can be >> >> sent on a different connection to destroy a single- >> >> connection iSCSI session. If you have multiple >> >> sessions ongoing, and the logout command is sent >> >> on a different connection than the one used >> >> to support the session that is to be killed, then >> >> you will need ISID to distinguish the session that >> >> you want killed. >> >> >> >> Regards, >> >> Josh >> >> >> >> > -----Original Message----- >> >> > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] >> >> > Sent: Tuesday, March 13, 2001 9:57 PM >> >> > To: ips@ece.cmu.edu >> >> > Subject: Re: iSCSI: Logout needs ISID >> >> > >> >> > >> >> > >> >> > >> >> > Josh, >> >> > >> >> > There is something I am missing. The Logout can be issued >> >> > only after Login >> >> > (authentication and the rest). >> >> > The session is then implied. >> >> > >> >> > Regards, >> >> > Julo >> >> > >> >> > Joshua Tseng on 14/03/2001 03:56:45 >> >> > >> >> > Please respond to Joshua Tseng >> >> > >> >> > To: Julian Satran/Haifa/IBM@IBMIL >> >> > cc: ips@ece.cmu.edu >> >> > Subject: iSCSI: Logout needs ISID >> >> > >> >> > >> >> > >> >> > >> >> > Julian, >> >> > >> >> > Section 2.14 states that the logout command may >> >> > be sent on a second connection to clean up the >> >> > a single connection iSCSI session. If the reason >> >> > code is 0 (close the session), then ISID is needed >> >> > to identify the iSCSI session to close. >> >> > >> >> > Josh >> >> > >> >> > >> >> > >> > >> >-- >> >Mark A. Bakke >> >Cisco Systems >> >mbakke@cisco.com >> >763.398.1054 >> > >> > >> > >> > >> > >> > >> > >> > > >_________________________________________________________ >Do You Yahoo!? >Get your free @yahoo.com address at http://mail.yahoo.com > > From owner-ips@ece.cmu.edu Thu Mar 15 10:38:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA01387 for ; Thu, 15 Mar 2001 10:38:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FDGoV29892 for ips-outgoing; Thu, 15 Mar 2001 08:16:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from NL-AMS-05.interxion.net (gate.interxion.com [194.153.74.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FCKgr26824 for ; Thu, 15 Mar 2001 07:20:42 -0500 (EST) Subject: Some thoughts about draft 05 of iSCSI Date: Thu, 15 Mar 2001 13:20:34 +0100 Message-ID: <23B31A8C6CF0D411B9B800508BB8465C25C461@ETXBRE101.etx> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Thread-Topic: Some thoughts about draft 05 of iSCSI Thread-Index: AcCtSlVPmw/ri6HPTDmyPsjXIGchbA== content-class: urn:content-classes:message From: "Jeroen Ruigrok van der Werven" X-MimeOLE: Produced By Microsoft Exchange V6.0.4417.0 To: , Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id f2FCKls26833 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Hi all, first, I am (not yet) subscribed to the list, so if you people could keep me in the CC:-loop for this thread I would very much appreciate it. Second, I have some remarks I want to make and some questions to ask. Please bear with me that storage solutions aren't exactly my forte aside from a consumer/customer point of view, I am more of a networking/programmer type, so I might be asking things which might be normal stuff for you. I've read some large parts of the archives already, but I might have missed my answers. 1) Why was it decided to use a 16 number dotted decimal specifier for IPv6 addresses? As far as I know, and reading RFC 2373 again, it is not mentioned anywhere, and it is so different from what is currently standard practice in notation. Sure, it's simply basic convert-the-number-from-base-x-to-base-y type of stuff, but the current practice always settles on colon-delimited hexadecimal. 2) From reading the draft it seems logical to implement iSCSI in Operatnig System in a NFS like form, in which certain disk access gets exported and the kernel/low-level stuff/daemon(s) handle the actual requests/responses. What scares me though is that with NFS one knows he is accessing filesystems over a network, with iSCSI we want to make it as transparent as possible [at least that's how it struck me], but how is the blocking issue looked upon then? It seems hard to guarantee the same reponse over a network as with directly coupled cables. I am very much interested in this, since if we want to deploy this as a product our customers will have at least the following demands/questions: - data integrity - speed/latency 3) Julian, on page 80 of draft 05, section 6.2, last sentence: "it receives the a Data PDU with" should probably be: "it receives a DATA PDU with". 4) Am I correct in assuming, based 'pon my readnig of the draft, that bursting is limited to what FirstBurstSize is set to? So there is little/no chance of having the iSCSI protocol saturate the available networkbandwidth all of a sudden by bursting to the max. 5) The draft specifies that both data PDU and commands are being sent over the same TCP connection. This got me curious about the fact if iSCSI commands would get delayed if sufficiently large amounts of data are being sent over the TCP connection. For example, with some protocols they have a seperate control and data connection. IIRC, NNTP-streaming has it, ISDN is another prime example of this. Thanks in advance for any trouble and time taken to answer my questions, -- Jeroen Ruigrok van der Werven Systems Software Engineer / Connectivity Services Development InterXion, where the Internet lives From owner-ips@ece.cmu.edu Thu Mar 15 11:29:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA02446 for ; Thu, 15 Mar 2001 11:29:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FEYqr06872 for ips-outgoing; Thu, 15 Mar 2001 09:34:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FEYCr06840 for ; Thu, 15 Mar 2001 09:34:12 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id PAA59664 for ; Thu, 15 Mar 2001 15:34:04 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id PAA216068 for ; Thu, 15 Mar 2001 15:32:51 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A10.004FD330 ; Thu, 15 Mar 2001 15:31:54 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 15 Mar 2001 16:34:16 +0200 Subject: Re: iSCSI Login Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mathew, Answers in text - Julo "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" on 15/03/2001 11:43:33 Please respond to "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" To: ips@ece.cmu.edu cc: Subject: iSCSI Login Hi Julian, A couple of items: 1) Section 4.1 (iSCSI Rev 5) states that: The target can answer in the following ways: -Login Response with Login Reject (and F bit 1). This is an immediate rejection from the target, that causes the session to terminate I agree that this could be the case for a security breach - authentication failed but what if the target only supports one connection per session and the initiator is attempting to set up another connection. Surely, the new login should be rejected but the session remains intact. +++ will fix - the new text will be: Login Response with Login Reject (and F bit 1). This is an immediate rejection from the target, that causes the connection to terminate. If this connection is the leading or only connection of a session the session is also terminated. ++++ 2) Still on the subject of login: In section 4, page 74, the spec states that: "The initiator and target MAY want to negotiate authentication and data integrity parameters. Once this negotiation is completed, the channel is considered secure." It is unclear as to the mandated handling of conflicting/differing authentication mechanisms negotiated on multiple connections participating in the same session. I propose that the spec should state that if authentication is required then the same authentication method MUST be used on all connections in a session. +++ we are going to add the following clarification text to 8.1 (I hope it answers also your implied question). Each connection in a session may operate in a different security protection mode. Some connection may use private networks and require minimal protection while others may use public networks and require the highest level of protection. ++++ Cheers Matthew Burbridge Network Infrastructure Solutions Hewlett Packard Bristol +44 117 312 7010 E-mail: matthewb@bri.hp.com From owner-ips@ece.cmu.edu Thu Mar 15 11:30:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA02467 for ; Thu, 15 Mar 2001 11:30:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FEJt504067 for ips-outgoing; Thu, 15 Mar 2001 09:19:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sandmail.sandburst.com (sandburst-gw.bstn-gw02.ma.us.intelilink.net [216.57.129.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FEJ8r04026 for ; Thu, 15 Mar 2001 09:19:08 -0500 (EST) Received: from cs.uchicago.edu (dynamite-38.sandburst.com [172.16.5.38]) by sandmail.sandburst.com (Postfix) with ESMTP id E5F2E94006 for ; Thu, 15 Mar 2001 09:19:04 -0500 (EST) To: ips@ece.cmu.edu Subject: Re: FCIP iFCP encapsulation proposal In-Reply-To: Message from "Y P Cheng" of "Wed, 14 Mar 2001 18:23:19 PST." References: Date: Thu, 15 Mar 2001 09:18:02 -0500 From: Stephen Bailey Message-Id: <20010315141904.E5F2E94006@sandmail.sandburst.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk > It is NOT just these fields looking like a valid frame; they MUST > match the saved information inside a valid Exchange Control Block > (ECB) refer to by the OX_ID and RX_ID. Not if the frame begins a new exchange. Instead of concentrating on FCP_DATA, and the read (target->initiator) direction, look at other FC PDUs and the write direction (initiator->target). A single frame could be an FCP_CMD with a SCSI command, (e.g. RESERVE or START STOP UNIT with LOEJ=1), or a task management function (e.g. target reset) or an FC link service (e.g. LOGO). I'm sure there are more clever attacks too. The only thing you need to know is the FC IDs of a logged-in target/initiator pair which share the TCP connection you're using. Certainly, the easiest pair would be the attacking system's FC_ID and the target from which you are launching the attack (i.e. the target to which you are directing the stream of write data). However, if you know other logged-in FC_ID pairs, you can also do a 3rd party attack. It is not the case that FC_IDs are going to be well kept secrets which will require a 'til the sun grows cold parameter search. Steph From owner-ips@ece.cmu.edu Thu Mar 15 12:22:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA03714 for ; Thu, 15 Mar 2001 12:22:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FFJ4Q10000 for ips-outgoing; Thu, 15 Mar 2001 10:19:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FFInr09987 for ; Thu, 15 Mar 2001 10:18:49 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 15 Mar 2001 10:18:43 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07080152A0@corpmx9.isus.emc.com> From: Black_David@emc.com To: rsnively@Brocade.COM, ips@ece.cmu.edu Subject: RE: FCIP iFCP encapsulation proposal Date: Thu, 15 Mar 2001 10:18:42 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk I don't think Bob's understood the issue -- If a trace file containing all the headers and framing information from traced FCIP communications is stored through FCIP it must be the case that any framing mechanism used by FCIP does not get confused by the framing information in the trace file. If the framing mechanism gets confused and attempts to execute the (correctly formed) frames contained in the trace file, sooner or later, something visibly disastrous *will* happen (e.g., my FORMAT command example :-) ). This is a design constraint on the framing mechanism, and Vi Chau's email that started this thread proposed a framing mechanism that did not meet this design constraint. The initial proposal is appreciated, but the WG is obligated to do better. Beyond that, Bob's characterization of the difficulty of inserting traffic into a TCP session is way off the mark. In the hope of avoiding further expenditure of list bandwidth on the discussion of fundamentally inadequate security mechanisms and approaches, I want to remind people of the following: (1) TCP hijacking is a well-known attack in which the attacker takes over one end of the connection. Exploit code is readily available for this. Not only is it not difficult - it's actually a solved problem (unfortunately). The exploit code monitors the TCP connection to learn its state and then injects just the right packets to take it over. (2) Creation of a "properly defined TCP connection" is not in general a "secure action". Even in the specific cases where it is (e.g., in-band authentication is used), a hijack attack can be launched after the authentication if there is no cryptographic data integrity used on the connection. (3) The entire discussion about the difficulty of reproducing Fibre Channel related state makes the common mistake of confusing obscurity and/or complexity with actual security. TCP has already been hacked in this "darned unlikely" fashion -- it's just a matter of time before someone with nothing better to do figures out how to get just enough of FC-2 right in order to do serious damage to/via FCIP. In particular, I have some serious problems with: > In effect, the problem is not a "frame in data" problem. It is a > "frame in successfully delivered TCP/IP data across a legitimate > TCP/IP connection consistent with the fully defined states of > the Fibre Channel FC-2 layer and consistent with the states, > requirements, requests, and acknowledgements of the upper layer protocol" > problem, which is darned unlikely even if you are trying maliciously > to do it. As indicated above, the "frame in data" problem involves a framing mechanism that gets confused about where the frames are and makes the mistake of extracting a frame from the data. I hope everyone agrees that any framing mechanism design MUST prevent this. Second, "darned unlikely" is an inadequate level of security assurance. What's required is "can't happen" backed by a discussion of the cryptographic difficulty involved in making it happen when appropriate security mechanisms are used. I realize that all cryptographic assurance arguments come down to some version of "darned unlikely" (e.g., if the adversary correctly guesses the key(s) involved, it's all over), but the level of difficulty involved for reasonable security mechanisms is generally well beyond that of grabbing some FC-2 code and grafting it onto TCP hijack code. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Thu Mar 15 12:23:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA03738 for ; Thu, 15 Mar 2001 12:23:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FFj0111861 for ips-outgoing; Thu, 15 Mar 2001 10:45:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FFikr11850 for ; Thu, 15 Mar 2001 10:44:46 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAA190876 for ; Thu, 15 Mar 2001 16:44:40 +0100 From: julian_satran@il.ibm.com Received: from d12mta05.de.ibm.com (d12mta06_cs0 [9.165.222.255]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA28358 for ; Thu, 15 Mar 2001 16:42:35 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A10.00567A5C ; Thu, 15 Mar 2001 16:44:34 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: "Jeroen Ruigrok van der Werven" cc: ips@ece.cmu.edu Message-ID: Date: Thu, 15 Mar 2001 17:45:00 +0200 Subject: Re: Some thoughts about draft 05 of iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Responses in text. Julo "Jeroen Ruigrok van der Werven" on 15/03/2001 14:20:34 Please respond to "Jeroen Ruigrok van der Werven" To: ips@ece.cmu.edu, Julian Satran/Haifa/IBM@IBMIL cc: Subject: Some thoughts about draft 05 of iSCSI Hi all, first, I am (not yet) subscribed to the list, so if you people could keep me in the CC:-loop for this thread I would very much appreciate it. Second, I have some remarks I want to make and some questions to ask. Please bear with me that storage solutions aren't exactly my forte aside from a consumer/customer point of view, I am more of a networking/programmer type, so I might be asking things which might be normal stuff for you. I've read some large parts of the archives already, but I might have missed my answers. 1) Why was it decided to use a 16 number dotted decimal specifier for IPv6 addresses? As far as I know, and reading RFC 2373 again, it is not mentioned anywhere, and it is so different from what is currently standard practice in notation. Sure, it's simply basic convert-the-number-from-base-x-to-base-y type of stuff, but the current practice always settles on colon-delimited hexadecimal. +++ No reason other than convenience. At first I did not know and when I learned about I did not know how popular it is and then I forgot completely -:) If it is very popular we can change +++ 2) From reading the draft it seems logical to implement iSCSI in Operatnig System in a NFS like form, in which certain disk access gets exported and the kernel/low-level stuff/daemon(s) handle the actual requests/responses. What scares me though is that with NFS one knows he is accessing filesystems over a network, with iSCSI we want to make it as transparent as possible [at least that's how it struck me], but how is the blocking issue looked upon then? It seems hard to guarantee the same reponse over a network as with directly coupled cables. I am very much interested in this, since if we want to deploy this as a product our customers will have at least the following demands/questions: - data integrity - speed/latency +++ both should be better than NFS -:). For data integrity we have end-to-end digests - (does NFS?) for latency we have no wonders but we did our best to allow hiding it and not let it affect throughput +++ 3) Julian, on page 80 of draft 05, section 6.2, last sentence: "it receives the a Data PDU with" should probably be: "it receives a DATA PDU with". +++ will fix case +++ 4) Am I correct in assuming, based 'pon my readnig of the draft, that bursting is limited to what FirstBurstSize is set to? So there is little/no chance of having the iSCSI protocol saturate the available networkbandwidth all of a sudden by bursting to the max. +++ bursting limit (unsolicited data) is meant mainly to help targets but network could also benefit +++ 5) The draft specifies that both data PDU and commands are being sent over the same TCP connection. This got me curious about the fact if iSCSI commands would get delayed if sufficiently large amounts of data are being sent over the TCP connection. For example, with some protocols they have a seperate control and data connection. IIRC, NNTP-streaming has it, ISDN is another prime example of this. +++ don't get back into this - many of us have open wounds here -:) Thanks in advance for any trouble and time taken to answer my questions, -- Jeroen Ruigrok van der Werven Systems Software Engineer / Connectivity Services Development InterXion, where the Internet lives From owner-ips@ece.cmu.edu Thu Mar 15 14:56:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA06770 for ; Thu, 15 Mar 2001 14:56:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FHiLN20851 for ips-outgoing; Thu, 15 Mar 2001 12:44:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zmamail05.zma.compaq.com (zmamail05.zma.compaq.com [161.114.64.105]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FHhZr20817 for ; Thu, 15 Mar 2001 12:43:35 -0500 (EST) Received: by zmamail05.zma.compaq.com (Postfix, from userid 12345) id 325A16784; Thu, 15 Mar 2001 12:43:30 -0500 (EST) Received: from exchou-gh02.cca.cpqcorp.net (exchou-gh02.cca.cpqcorp.net [16.110.248.202]) by zmamail05.zma.compaq.com (Postfix) with ESMTP id D558D6496 for ; Thu, 15 Mar 2001 12:43:29 -0500 (EST) Received: by exchou-gh02.cca.cpqcorp.net with Internet Mail Service (5.5.2652.78) id ; Thu, 15 Mar 2001 11:43:29 -0600 Message-ID: <78AF3C342AEAEF4BA33B35A8A15668C659C10A@cceexc17.americas.cpqcorp.net> From: "Elliott, Robert" To: ips@ece.cmu.edu Subject: RE: iSCSI: Need to Kill Session from Surviving Node Date: Thu, 15 Mar 2001 11:43:28 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2652.78) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk There is a T10 proposal being developed by Ken Moe at Sun (originally by Bob Snively) titled Asymmetric SCSI Behavior (00-232r6) that deals with controller fail-over. It describes how a controller with multiple target ports connected to the same logical unit can report the state of those target ports (active, standy, or unavailable) with a new REPORT TARGET GROUPS command. It lets an initiator initiate state changes using a new SET TARGET GROUPS command. If the fabric path to a controller's target port fails, then I think you would be more concerned with getting the path running again than with reusing some now unusable target port resources. It's like a RAID system during a disk rebuild - you are operating without redundancy and are subject to a single point of failure until you fix the problem. During this time the logical unit resources can be recovered. CLEAR TASK SET and LOGICAL UNIT RESET task management functions work. If persistent reservations is being used, the PREEMPT AND ABORT service action can clear out the tasks from the nexus that is no longer available. I think it's a mistake to affect logical units you don't own, which a TARGET RESET would do. Some other operating system might be using them and expect a different error recovery procedure. > -----Original Message----- > From: John Hufferd [mailto:hufferd@us.ibm.com] > Sent: Wednesday, March 14, 2001 11:21 PM > To: ips@ece.cmu.edu > Subject: RE: iSCSI: Need to Kill Session from Surviving Node > > > > Wait a minute, I did not expect my Surviving Node question to somehow > trigger the discussion of whether the Target Reset should be issued by > iSCSI Initiators. That is a completely different Pit. > > My question was, when the Storage Controller fails over to a surviving > NODE, how do we tell the Initiator that is waiting peacefully for the > TCP/IP connection to time-out. To quickly give-up and let > the initiator's > SCSI layer, retry the operations, so that the iSCSI layer can > start either > sending the SCSI cmds to the Surviving Node, or starts a new > Session with > the Surviving Node. > > This has nothing to do with the Initiator issuing Target Resets. > > > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Internet address: hufferd@us.ibm.com > > > "Somesh Gupta" @ece.cmu.edu on 03/14/2001 > 08:25:58 PM > > Please respond to > > Sent by: owner-ips@ece.cmu.edu > > > To: , > cc: > Subject: RE: iSCSI: Need to Kill Session from Surviving Node > > > > Mallikarjun, > > I am not disagreeing with the desire to do a graceful > termination. However, when recovery states at the TCP > level, and at the iSCSI level get intertwined with this, > it is probably better to focus on a clean (possibly graceful) > termination and a clean bringup. > > Somesh > > > -----Original Message----- > > From: cbm@core.rose.hp.com [mailto:cbm@core.rose.hp.com]On Behalf Of > > Mallikarjun C. > > Sent: Wednesday, March 14, 2001 7:31 PM > > To: someshg@yahoo.com > > Cc: ips@ece.cmu.edu > > Subject: RE: iSCSI: Need to Kill Session from Surviving Node > > > > > > Somesh, > > > > Please refer to the following URL (and emails around that) for > > a long discussion on this topic. Both flavors of resets have value, > > and I agree with Julian that iSCSI should support both. > > > > http://ips.pdl.cs.cmu.edu/mail/msg00740.html > > > > -- > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > > > >Do you need both a "warm-reset" and a "cold-reset"? > > >The "warm-reset" seems to be an over-optimization. > > > > > >Somesh > > > > > >> -----Original Message----- > > >> From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > >> Mallikarjun C. > > >> Sent: Wednesday, March 14, 2001 12:16 PM > > >> To: ips@ece.cmu.edu > > >> Subject: Re: iSCSI: Need to Kill Session from Surviving Node > > >> > > >> > > >> >There is a related problem that I think we need to address, and > > >> that is -- > > >> >the case where the Storage Controller is an Active-Active > > Fail-Over unit > > >> >with very little state shared between the Storage Controller > > Nodes. (I > > >> >think a number of you folks have told me that you are > planning such a > > >> >design.) This would mean that a specific initiator OS (WWUI) > > would have > > >> >separate sessions between the Nodes. At fail-over, the > > >> Take-Over Node will > > >> >want to cause the Initiator, to Stop messing around with the > > Failing node > > >> >ASAP, and start using the existing session, or start a new > > >> session with the > > >> >Take_Over Node. The Initiator, will normally just time-out the > TCP/IP > > >> >connection, and then turn the recovery over to the SCSI > > Layer. Then, when > > >> >SCSI retries, iSCSI will either use the Session to the > > Surviving Node or > > >> >Create a New Session to that Surviving Node. The problem is, > > that this > > >> >will take a longer time then a system might want. > > >> > > > >> >So the question: is there a way for a surviving Target Node > > to tell the > > >> >Initiator to "Kill" the Session to the Failing Target Node? > > >> > > > >> >Comments? > > >> > > >> Assuming that the failing Node is still able to execute commands, > > >> a cold target reset from the Initiator to the failing node > > should "kill" > > >> all the sessions to it. > > >> > > >> The answer to two hosts configured in a failover > configuration sharing > > >> a single Storage Controller Node is also the same. > > >> > > >> If it is active-active configuration, it appears to me > that there's a > > >> need to somehow prompt the Initiator to doing the reset on a > > third party > > >> (Async PDU?). > > >> > > >> This is a reason why we'd still need target reset in iSCSI. > > >> -- > > >> Mallikarjun > > >> > > >> > > >> Mallikarjun Chadalapaka > > >> Networked Storage Architecture > > >> Network Storage Solutions Organization > > >> MS 5668 Hewlett-Packard, Roseville. > > >> cbm@rose.hp.com > > >> > > >> > > > >> >. > > >> >. > > >> >. > > >> >John L. Hufferd > > >> >Senior Technical Staff Member (STSM) > > >> >IBM/SSG San Jose Ca > > >> >(408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > > >> >Internet address: hufferd@us.ibm.com > > >> > > > >> > > > >> >Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/14/2001 > 08:31:19 AM > > >> > > > >> >Sent by: owner-ips@ece.cmu.edu > > >> > > > >> > > > >> >To: ips@ece.cmu.edu > > >> >cc: > > >> >Subject: Re: iSCSI: Logout needs ISID > > >> > > > >> > > > >> > > > >> > > > >> > > > >> >Mark, > > >> > > > >> >If you have only one connection you are supposed to use a > > Login with the > > >> >restart bit one - and achieve a similar effect as a > > Login/Logout - i.e. > > >> >keep the session alive. Even this might be a problem for a > > >> target that is > > >> >so poor it has only one socket (even SNMP won't work there). > > >> For this case > > >> >the only way out is to drop the connection and hope the > > target will hear > > >> >you (it probably won't -:)). The comment is the draft > is a memento > for > > >> >implementers to keep looking for new connections always (even > > for a one > > >> >connection target) but probably it does not come through clear > enough. > > >> > > > >> >Regards, > > >> >Julo > > >> > > > >> >Mark Bakke on 14/03/2001 18:17:14 > > >> > > > >> >Please respond to Mark Bakke > > >> > > > >> >To: Julian Satran/Haifa/IBM@IBMIL > > >> >cc: ips@ece.cmu.edu > > >> >Subject: Re: iSCSI: Logout needs ISID > > >> > > > >> > > > >> > > > >> > > > >> >Julian- > > >> > > > >> >A target that does not support multiple connections per session > > >> >will return "Initiator SID Error" when the login > attempt is made. > > >> >In this case, there is no way to log in, so there's no way to > > >> >log out the old connection. The initiator would be > stuck waiting > > >> >until the target's side of the connection times out and goes > > >> >away, which could be a very long time. > > >> > > > >> >There is an open question within the MIB team about > whether anyone > > >> >needs to be able to kill connections or sessions from > SNMP; however, > > >> >I don't think that anyone will want to use SNMP as part of error > > >> >recovery. > > >> > > > >> >-- > > >> >Mark > > >> > > > >> >julian_satran@il.ibm.com wrote: > > >> >> > > >> >> Josh, > > >> >> > > >> >> No command, including logout, can be sent without a login. > > >> >> The complete scenario is: > > >> >> > > >> >> -open a new connection > > >> >> -login in the same session as the old connection (this has > > the ISID & > > >> >TSID) > > >> >> -logout the old connection > > >> >> -recover commands > > >> >> > > >> >> You are not supposed to be able to kill a session from outside > > >> (at least > > >> >> not an iSCSI defined mode). > > >> >> You will need management support for that (SNMP?) > > >> >> > > >> >> Regards, > > >> >> Julo > > >> >> > > >> >> Joshua Tseng on 14/03/2001 17:09:48 > > >> >> > > >> >> Please respond to Joshua Tseng > > >> >> > > >> >> To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > >> >> cc: > > >> >> Subject: RE: iSCSI: Logout needs ISID > > >> >> > > >> >> Julian, > > >> >> > > >> >> In 2.14, you state that a logout command can be > > >> >> sent on a different connection to destroy a single- > > >> >> connection iSCSI session. If you have multiple > > >> >> sessions ongoing, and the logout command is sent > > >> >> on a different connection than the one used > > >> >> to support the session that is to be killed, then > > >> >> you will need ISID to distinguish the session that > > >> >> you want killed. > > >> >> > > >> >> Regards, > > >> >> Josh > > >> >> > > >> >> > -----Original Message----- > > >> >> > From: julian_satran@il.ibm.com > [mailto:julian_satran@il.ibm.com] > > >> >> > Sent: Tuesday, March 13, 2001 9:57 PM > > >> >> > To: ips@ece.cmu.edu > > >> >> > Subject: Re: iSCSI: Logout needs ISID > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > Josh, > > >> >> > > > >> >> > There is something I am missing. The Logout can be issued > > >> >> > only after Login > > >> >> > (authentication and the rest). > > >> >> > The session is then implied. > > >> >> > > > >> >> > Regards, > > >> >> > Julo > > >> >> > > > >> >> > Joshua Tseng on > 14/03/2001 03:56:45 > > >> >> > > > >> >> > Please respond to Joshua Tseng > > >> >> > > > >> >> > To: Julian Satran/Haifa/IBM@IBMIL > > >> >> > cc: ips@ece.cmu.edu > > >> >> > Subject: iSCSI: Logout needs ISID > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > Julian, > > >> >> > > > >> >> > Section 2.14 states that the logout command may > > >> >> > be sent on a second connection to clean up the > > >> >> > a single connection iSCSI session. If the reason > > >> >> > code is 0 (close the session), then ISID is needed > > >> >> > to identify the iSCSI session to close. > > >> >> > > > >> >> > Josh > > >> >> > > > >> >> > > > >> >> > > > >> > > > >> >-- > > >> >Mark A. Bakke > > >> >Cisco Systems > > >> >mbakke@cisco.com > > >> >763.398.1054 > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > > > > > > >_________________________________________________________ > > >Do You Yahoo!? > > >Get your free @yahoo.com address at http://mail.yahoo.com > > > > > > > > > > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com > > > > From owner-ips@ece.cmu.edu Thu Mar 15 14:56:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA06841 for ; Thu, 15 Mar 2001 14:56:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FIL0423619 for ips-outgoing; Thu, 15 Mar 2001 13:21:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e1.ny.us.ibm.com (e1.ny.us.ibm.com [32.97.182.101]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FIKEr23519 for ; Thu, 15 Mar 2001 13:20:14 -0500 (EST) Received: from northrelay02.pok.ibm.com (northrelay02.pok.ibm.com [9.117.200.22]) by e1.ny.us.ibm.com (8.9.3/8.9.3) with ESMTP id NAA54908 for ; Thu, 15 Mar 2001 13:18:58 -0500 From: julian_satran@il.ibm.com Received: from D51MTA05.pok.ibm.com (d51mta05.pok.ibm.com [9.117.200.33]) by northrelay02.pok.ibm.com (8.8.8m3/NCO v4.95) with SMTP id NAA127534 for ; Thu, 15 Mar 2001 13:16:00 -0500 Received: by D51MTA05.pok.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 85256A10.0064B552 ; Thu, 15 Mar 2001 13:20:00 -0500 X-Lotus-FromDomain: IBMIL@IBMDE@IBMUS To: hufferd@us.ibm.com cc: ips@ece.cmu.edu Message-ID: <85256A10.0064B496.00@D51MTA05.pok.ibm.com> Date: Thu, 15 Mar 2001 20:20:35 +0200 Subject: Re: iSCSI Login Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk John, Yes it is easier to do, it does not make sense to enforce similarity (no good infrastructure and no real benefit as sessions are long- very much unlike http where it makes sense to replicate security and avoid negotiation) and then it will be nice to have private channel with a backup public channel not having the same security. Regards, Julo John Hufferd@IBMUS 15/03/2001 19:53 To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: iSCSI Login (Document link: Julian Satran - Mail) Julian, Perhaps I missed something important here but I can not remember a reason why there would need to be different Security actions on each connection within a session. I can understand that being appropriate for different Sessions, but it seems strange to me that it make since within a Session. Perhaps, the code is easier if you did not check to see if they are consistent, so that your proposed wordage is applicable in all cases and therefore the Target does not have to care about, and not have to enforce the same Security process across different connections within the same Session. If that is what you are saying, then maybe it is alright, but it does seem strange. I can not think of a credible reason why any Session would have different Security processes on its different connections. Please let me know what I missed, or your thinking about the value of your proposed new wording. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/15/2001 06:34:16 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iSCSI Login Mathew, Answers in text - Julo "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" on 15/03/2001 11:43:33 Please respond to "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" To: ips@ece.cmu.edu cc: Subject: iSCSI Login Hi Julian, A couple of items: 1) Section 4.1 (iSCSI Rev 5) states that: The target can answer in the following ways: -Login Response with Login Reject (and F bit 1). This is an immediate rejection from the target, that causes the session to terminate I agree that this could be the case for a security breach - authentication failed but what if the target only supports one connection per session and the initiator is attempting to set up another connection. Surely, the new login should be rejected but the session remains intact. +++ will fix - the new text will be: Login Response with Login Reject (and F bit 1). This is an immediate rejection from the target, that causes the connection to terminate. If this connection is the leading or only connection of a session the session is also terminated. ++++ 2) Still on the subject of login: In section 4, page 74, the spec states that: "The initiator and target MAY want to negotiate authentication and data integrity parameters. Once this negotiation is completed, the channel is considered secure." It is unclear as to the mandated handling of conflicting/differing authentication mechanisms negotiated on multiple connections participating in the same session. I propose that the spec should state that if authentication is required then the same authentication method MUST be used on all connections in a session. +++ we are going to add the following clarification text to 8.1 (I hope it answers also your implied question). Each connection in a session may operate in a different security protection mode. Some connection may use private networks and require minimal protection while others may use public networks and require the highest level of protection. ++++ Cheers Matthew Burbridge Network Infrastructure Solutions Hewlett Packard Bristol +44 117 312 7010 E-mail: matthewb@bri.hp.com From owner-ips@ece.cmu.edu Thu Mar 15 14:57:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA06852 for ; Thu, 15 Mar 2001 14:57:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FI63o22451 for ips-outgoing; Thu, 15 Mar 2001 13:06:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FI5Xr22367 for ; Thu, 15 Mar 2001 13:05:33 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id MAA38340 for ; Thu, 15 Mar 2001 12:58:15 -0500 Received: from f4n49e (d03nm065h.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id LAA59198 for ; Thu, 15 Mar 2001 11:05:26 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI Login To: "Julian Satran" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Thu, 15 Mar 2001 09:53:37 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/15/2001 11:05:26 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, Perhaps I missed something important here but I can not remember a reason why there would need to be different Security actions on each connection within a session. I can understand that being appropriate for different Sessions, but it seems strange to me that it make since within a Session. Perhaps, the code is easier if you did not check to see if they are consistent, so that your proposed wordage is applicable in all cases and therefore the Target does not have to care about, and not have to enforce the same Security process across different connections within the same Session. If that is what you are saying, then maybe it is alright, but it does seem strange. I can not think of a credible reason why any Session would have different Security processes on its different connections. Please let me know what I missed, or your thinking about the value of your proposed new wording. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/15/2001 06:34:16 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iSCSI Login Mathew, Answers in text - Julo "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" on 15/03/2001 11:43:33 Please respond to "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" To: ips@ece.cmu.edu cc: Subject: iSCSI Login Hi Julian, A couple of items: 1) Section 4.1 (iSCSI Rev 5) states that: The target can answer in the following ways: -Login Response with Login Reject (and F bit 1). This is an immediate rejection from the target, that causes the session to terminate I agree that this could be the case for a security breach - authentication failed but what if the target only supports one connection per session and the initiator is attempting to set up another connection. Surely, the new login should be rejected but the session remains intact. +++ will fix - the new text will be: Login Response with Login Reject (and F bit 1). This is an immediate rejection from the target, that causes the connection to terminate. If this connection is the leading or only connection of a session the session is also terminated. ++++ 2) Still on the subject of login: In section 4, page 74, the spec states that: "The initiator and target MAY want to negotiate authentication and data integrity parameters. Once this negotiation is completed, the channel is considered secure." It is unclear as to the mandated handling of conflicting/differing authentication mechanisms negotiated on multiple connections participating in the same session. I propose that the spec should state that if authentication is required then the same authentication method MUST be used on all connections in a session. +++ we are going to add the following clarification text to 8.1 (I hope it answers also your implied question). Each connection in a session may operate in a different security protection mode. Some connection may use private networks and require minimal protection while others may use public networks and require the highest level of protection. ++++ Cheers Matthew Burbridge Network Infrastructure Solutions Hewlett Packard Bristol +44 117 312 7010 E-mail: matthewb@bri.hp.com From owner-ips@ece.cmu.edu Thu Mar 15 14:58:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA06879 for ; Thu, 15 Mar 2001 14:58:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FIZwN24693 for ips-outgoing; Thu, 15 Mar 2001 13:35:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from marlborough.cnchost.com (marlborough.concentric.net [207.155.248.14]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FIZKr24620 for ; Thu, 15 Mar 2001 13:35:20 -0500 (EST) Received: from BSNCBINFORD (ts002d27.wic-ks.concentric.net [206.173.163.87]) by marlborough.cnchost.com id NAA23968; Thu, 15 Mar 2001 13:35:15 -0500 (EST) [ConcentricHost SMTP Relay 1.10] From: "Charles Binford" To: "IPS Reflector" Subject: RE: iscsi: rev 05 2.5.1 requires ACA support Date: Thu, 15 Mar 2001 12:35:26 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <3AAD9F2D.9F9B8673@compuserve.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Ralph, you missed a reference in iSCSI where ACA is required. In 7.2 (p 86) it says: "ACA helps preserving ordered command execution in presence of errors. As iSCSI can have many commands in-flight between initiator and target iSCSI mandates support for ACA." I agree with Ralph that iSCSI should not be mandating ACA support. However, I don't really understand what the problem/scenario is that ACA is supposed to be solving. Can someone please explain that to me? Before I start throwing more darts at ACA I'd like to better understand the perceived value. Charles Binford Blue Spruce Networks office: (316) 315-0382 / cell: (316) 210-6404 e-fax: (509) 756-4425 -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ralph Weber Sent: Monday, March 12, 2001 10:17 PM To: IPS Reflector Subject: iscsi: rev 05 2.5.1 requires ACA support The following two 2.5.1 statements require the use of ACA and there is no way for an initiator that doesn't implement ACA to stop them. For the , ... The target MUST then enter the ACA state for any initiator for which it had pending tasks. In addition, for the the target enters the ACA state on all sessions and all LUs on which an AE was sent. Have you all signed the operating system vendors up for rewrites of their class drivers to support ACA? If not anticipate some serious deadlocks when iSCSI is first attempted with a non ACA class driver, since the driver will never clear these ACA conditions and no progress will be made after that. Alternatively, a way can be found to give the initiator control over the "MUST" statements. Thanks. Ralph... From owner-ips@ece.cmu.edu Thu Mar 15 15:58:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA08013 for ; Thu, 15 Mar 2001 15:58:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FJA2u27103 for ips-outgoing; Thu, 15 Mar 2001 14:10:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FGTdr15138 for ; Thu, 15 Mar 2001 11:29:39 -0500 (EST) Received: from ypportable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id GQ9BKBX9; Thu, 15 Mar 2001 08:29:38 -0800 From: "Y P Cheng" To: Subject: RE: FCIP iFCP encapsulation proposal Date: Thu, 15 Mar 2001 08:25:04 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <20010315141904.E5F2E94006@sandmail.sandburst.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Stephen Bailey > Not if the frame begins a new exchange. Instead of concentrating on > FCP_DATA, and the read (target->initiator) direction, look at other FC > PDUs and the write direction (initiator->target). > > A single frame could be an FCP_CMD with a SCSI command, (e.g. RESERVE > or START STOP UNIT with LOEJ=1), or a task management function > (e.g. target reset) or an FC link service (e.g. LOGO). I'm sure there > are more clever attacks too. > > The only thing you need to know is the FC IDs of a logged-in > target/initiator pair which share the TCP connection you're using. > Certainly, the easiest pair would be the attacking system's FC_ID ... To begin a new exchange in an Internet environment, I do hope we check stuff more than just FC_ID before allowing a START_STOP_UNIT or TARGET_RESET. After a successful login and text message exchanges, both initiator and target keep a Port/Connection Control Block to save all kinds of goodies resulting from the exchanges. The goodies WILL be checked before allowing an operation. If not, there is no security and all the discussions herein are somewhat academic. Furthermore, if the login and text messages are easily stolen, it is a security issue not an operation issue. Y.P. From owner-ips@ece.cmu.edu Thu Mar 15 15:58:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA08024 for ; Thu, 15 Mar 2001 15:58:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FJTA228594 for ips-outgoing; Thu, 15 Mar 2001 14:29:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FJSmr28561 for ; Thu, 15 Mar 2001 14:28:49 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id UAA149152 for ; Thu, 15 Mar 2001 20:28:41 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id UAA140988 for ; Thu, 15 Mar 2001 20:27:27 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A10.006ACD80 ; Thu, 15 Mar 2001 20:26:34 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 15 Mar 2001 21:29:02 +0200 Subject: RE: iscsi: rev 05 2.5.1 requires ACA support Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, After an error if you have commands in flight you want them all dropped until you specifically reset the ACA and restart the queue (prevent things to be executed out of order). Julo "Charles Binford" on 15/03/2001 20:35:26 Please respond to "Charles Binford" To: "IPS Reflector" cc: Subject: RE: iscsi: rev 05 2.5.1 requires ACA support Ralph, you missed a reference in iSCSI where ACA is required. In 7.2 (p 86) it says: "ACA helps preserving ordered command execution in presence of errors. As iSCSI can have many commands in-flight between initiator and target iSCSI mandates support for ACA." I agree with Ralph that iSCSI should not be mandating ACA support. However, I don't really understand what the problem/scenario is that ACA is supposed to be solving. Can someone please explain that to me? Before I start throwing more darts at ACA I'd like to better understand the perceived value. Charles Binford Blue Spruce Networks office: (316) 315-0382 / cell: (316) 210-6404 e-fax: (509) 756-4425 -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ralph Weber Sent: Monday, March 12, 2001 10:17 PM To: IPS Reflector Subject: iscsi: rev 05 2.5.1 requires ACA support The following two 2.5.1 statements require the use of ACA and there is no way for an initiator that doesn't implement ACA to stop them. For the , ... The target MUST then enter the ACA state for any initiator for which it had pending tasks. In addition, for the the target enters the ACA state on all sessions and all LUs on which an AE was sent. Have you all signed the operating system vendors up for rewrites of their class drivers to support ACA? If not anticipate some serious deadlocks when iSCSI is first attempted with a non ACA class driver, since the driver will never clear these ACA conditions and no progress will be made after that. Alternatively, a way can be found to give the initiator control over the "MUST" statements. Thanks. Ralph... From owner-ips@ece.cmu.edu Thu Mar 15 16:00:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA08084 for ; Thu, 15 Mar 2001 16:00:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FJLAW28030 for ips-outgoing; Thu, 15 Mar 2001 14:21:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FGTdr15138 for ; Thu, 15 Mar 2001 11:29:39 -0500 (EST) Received: from ypportable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id GQ9BKBX9; Thu, 15 Mar 2001 08:29:38 -0800 From: "Y P Cheng" To: Subject: RE: FCIP iFCP encapsulation proposal Date: Thu, 15 Mar 2001 08:25:04 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <20010315141904.E5F2E94006@sandmail.sandburst.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Stephen Bailey > Not if the frame begins a new exchange. Instead of concentrating on > FCP_DATA, and the read (target->initiator) direction, look at other FC > PDUs and the write direction (initiator->target). > > A single frame could be an FCP_CMD with a SCSI command, (e.g. RESERVE > or START STOP UNIT with LOEJ=1), or a task management function > (e.g. target reset) or an FC link service (e.g. LOGO). I'm sure there > are more clever attacks too. > > The only thing you need to know is the FC IDs of a logged-in > target/initiator pair which share the TCP connection you're using. > Certainly, the easiest pair would be the attacking system's FC_ID ... To begin a new exchange in an Internet environment, I do hope we check stuff more than just FC_ID before allowing a START_STOP_UNIT or TARGET_RESET. After a successful login and text message exchanges, both initiator and target keep a Port/Connection Control Block to save all kinds of goodies resulting from the exchanges. The goodies WILL be checked before allowing an operation. If not, there is no security and all the discussions herein are somewhat academic. Furthermore, if the login and text messages are easily stolen, it is a security issue not an operation issue. Y.P. From owner-ips@ece.cmu.edu Thu Mar 15 16:05:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA08178 for ; Thu, 15 Mar 2001 16:05:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FJLCr28033 for ips-outgoing; Thu, 15 Mar 2001 14:21:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.brocade.com (asbestos.brocade.com [63.121.140.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FHILr18879 for ; Thu, 15 Mar 2001 12:18:21 -0500 (EST) Received: from thor.brocade.com (thor [192.168.126.45]) by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id JAA03346; Thu, 15 Mar 2001 09:17:55 -0800 (PST) Received: by thor.brocade.com with Internet Mail Service (5.5.2653.19) id ; Thu, 15 Mar 2001 09:17:55 -0800 Message-ID: From: Robert Snively To: "'Douglas Otis'" , Robert Snively , Black_David@emc.com, ips@ece.cmu.edu Subject: RE: FCIP iFCP encapsulation proposal Date: Thu, 15 Mar 2001 09:17:51 -0800 X-Mailer: Internet Mail Service (5.5.2653.19) Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, The binary image is of ethernet frames. It requires the existence of a matching TCP/IP connection with matching TCP headers, including pdu sequencing information, which is not knowable from the spoofer. How are these delivered? The same question is true for each of the other layers of the transfer, and the same unlikely scenario must be played back for each. I just don't see such data being delivered by a responsible software layer. Bob > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Wednesday, March 14, 2001 1:15 PM > To: Robert Snively; Black_David@emc.com; ips@ece.cmu.edu > Subject: RE: FCIP iFCP encapsulation proposal > > > Bob, > > With out discussing spoofing where attackers successfully guess TCP > sequences (made too easy in some cases), a binary image is > stored and then > legitimately sent as a payload, with the example being > binary content of a > debug analyzer. In this case, headers contained within the > payload could be > seen as valid. The valid header within the payload may fool > a process that > attempts to recover header synchronization following a > dropped packet. This > header may carry the same information in current use and be > acted upon or > send the connection into error oblivion. It would appear to > represent a > weakness that can be exploited. Dropped packets happen. > > Doug > > > From owner-ips@ece.cmu.edu Thu Mar 15 16:07:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA08208 for ; Thu, 15 Mar 2001 16:07:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FIYxK24597 for ips-outgoing; Thu, 15 Mar 2001 13:34:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FIYar24576 for ; Thu, 15 Mar 2001 13:34:36 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id NAA41718; Thu, 15 Mar 2001 13:27:20 -0500 Received: from f4n49e (d03nm065h.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id LAA187832; Thu, 15 Mar 2001 11:34:05 -0700 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI: Need to Kill Session from Surviving Node To: "Elliott, Robert" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Thu, 15 Mar 2001 10:25:38 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/15/2001 11:34:05 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Elliott, I think what you said is right (Don't use Target Reset). That as I have stated should be an Admin action. OK, What I was trying to get at is --- trying to enable the Surviving Node (which has no TCP/IP state from the Failing node) to get the Initiator, which had a previous Session with the Failing Node to take actions right away, and not wait for its TCP/IP connections with the Failed Node to Time Out. (This requires some sort of a message from the Surviving node back to an Initiator.) This is not quite what we have defined into iSCSI. Since, I preceived that a number of companies were building such devices it seemed that a method for accomplishing such a Fail-Over, in a timely manor might be a common requirement. If so, did anyone have any bright ideas...... . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com "Elliott, Robert" @ece.cmu.edu on 03/15/2001 09:43:28 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: RE: iSCSI: Need to Kill Session from Surviving Node There is a T10 proposal being developed by Ken Moe at Sun (originally by Bob Snively) titled Asymmetric SCSI Behavior (00-232r6) that deals with controller fail-over. It describes how a controller with multiple target ports connected to the same logical unit can report the state of those target ports (active, standy, or unavailable) with a new REPORT TARGET GROUPS command. It lets an initiator initiate state changes using a new SET TARGET GROUPS command. If the fabric path to a controller's target port fails, then I think you would be more concerned with getting the path running again than with reusing some now unusable target port resources. It's like a RAID system during a disk rebuild - you are operating without redundancy and are subject to a single point of failure until you fix the problem. During this time the logical unit resources can be recovered. CLEAR TASK SET and LOGICAL UNIT RESET task management functions work. If persistent reservations is being used, the PREEMPT AND ABORT service action can clear out the tasks from the nexus that is no longer available. I think it's a mistake to affect logical units you don't own, which a TARGET RESET would do. Some other operating system might be using them and expect a different error recovery procedure. > -----Original Message----- > From: John Hufferd [mailto:hufferd@us.ibm.com] > Sent: Wednesday, March 14, 2001 11:21 PM > To: ips@ece.cmu.edu > Subject: RE: iSCSI: Need to Kill Session from Surviving Node > > > > Wait a minute, I did not expect my Surviving Node question to somehow > trigger the discussion of whether the Target Reset should be issued by > iSCSI Initiators. That is a completely different Pit. > > My question was, when the Storage Controller fails over to a surviving > NODE, how do we tell the Initiator that is waiting peacefully for the > TCP/IP connection to time-out. To quickly give-up and let > the initiator's > SCSI layer, retry the operations, so that the iSCSI layer can > start either > sending the SCSI cmds to the Surviving Node, or starts a new > Session with > the Surviving Node. > > This has nothing to do with the Initiator issuing Target Resets. > > > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Internet address: hufferd@us.ibm.com > > > "Somesh Gupta" @ece.cmu.edu on 03/14/2001 > 08:25:58 PM > > Please respond to > > Sent by: owner-ips@ece.cmu.edu > > > To: , > cc: > Subject: RE: iSCSI: Need to Kill Session from Surviving Node > > > > Mallikarjun, > > I am not disagreeing with the desire to do a graceful > termination. However, when recovery states at the TCP > level, and at the iSCSI level get intertwined with this, > it is probably better to focus on a clean (possibly graceful) > termination and a clean bringup. > > Somesh > > > -----Original Message----- > > From: cbm@core.rose.hp.com [mailto:cbm@core.rose.hp.com]On Behalf Of > > Mallikarjun C. > > Sent: Wednesday, March 14, 2001 7:31 PM > > To: someshg@yahoo.com > > Cc: ips@ece.cmu.edu > > Subject: RE: iSCSI: Need to Kill Session from Surviving Node > > > > > > Somesh, > > > > Please refer to the following URL (and emails around that) for > > a long discussion on this topic. Both flavors of resets have value, > > and I agree with Julian that iSCSI should support both. > > > > http://ips.pdl.cs.cmu.edu/mail/msg00740.html > > > > -- > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > > > >Do you need both a "warm-reset" and a "cold-reset"? > > >The "warm-reset" seems to be an over-optimization. > > > > > >Somesh > > > > > >> -----Original Message----- > > >> From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > >> Mallikarjun C. > > >> Sent: Wednesday, March 14, 2001 12:16 PM > > >> To: ips@ece.cmu.edu > > >> Subject: Re: iSCSI: Need to Kill Session from Surviving Node > > >> > > >> > > >> >There is a related problem that I think we need to address, and > > >> that is -- > > >> >the case where the Storage Controller is an Active-Active > > Fail-Over unit > > >> >with very little state shared between the Storage Controller > > Nodes. (I > > >> >think a number of you folks have told me that you are > planning such a > > >> >design.) This would mean that a specific initiator OS (WWUI) > > would have > > >> >separate sessions between the Nodes. At fail-over, the > > >> Take-Over Node will > > >> >want to cause the Initiator, to Stop messing around with the > > Failing node > > >> >ASAP, and start using the existing session, or start a new > > >> session with the > > >> >Take_Over Node. The Initiator, will normally just time-out the > TCP/IP > > >> >connection, and then turn the recovery over to the SCSI > > Layer. Then, when > > >> >SCSI retries, iSCSI will either use the Session to the > > Surviving Node or > > >> >Create a New Session to that Surviving Node. The problem is, > > that this > > >> >will take a longer time then a system might want. > > >> > > > >> >So the question: is there a way for a surviving Target Node > > to tell the > > >> >Initiator to "Kill" the Session to the Failing Target Node? > > >> > > > >> >Comments? > > >> > > >> Assuming that the failing Node is still able to execute commands, > > >> a cold target reset from the Initiator to the failing node > > should "kill" > > >> all the sessions to it. > > >> > > >> The answer to two hosts configured in a failover > configuration sharing > > >> a single Storage Controller Node is also the same. > > >> > > >> If it is active-active configuration, it appears to me > that there's a > > >> need to somehow prompt the Initiator to doing the reset on a > > third party > > >> (Async PDU?). > > >> > > >> This is a reason why we'd still need target reset in iSCSI. > > >> -- > > >> Mallikarjun > > >> > > >> > > >> Mallikarjun Chadalapaka > > >> Networked Storage Architecture > > >> Network Storage Solutions Organization > > >> MS 5668 Hewlett-Packard, Roseville. > > >> cbm@rose.hp.com > > >> > > >> > > > >> >. > > >> >. > > >> >. > > >> >John L. Hufferd > > >> >Senior Technical Staff Member (STSM) > > >> >IBM/SSG San Jose Ca > > >> >(408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > > >> >Internet address: hufferd@us.ibm.com > > >> > > > >> > > > >> >Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/14/2001 > 08:31:19 AM > > >> > > > >> >Sent by: owner-ips@ece.cmu.edu > > >> > > > >> > > > >> >To: ips@ece.cmu.edu > > >> >cc: > > >> >Subject: Re: iSCSI: Logout needs ISID > > >> > > > >> > > > >> > > > >> > > > >> > > > >> >Mark, > > >> > > > >> >If you have only one connection you are supposed to use a > > Login with the > > >> >restart bit one - and achieve a similar effect as a > > Login/Logout - i.e. > > >> >keep the session alive. Even this might be a problem for a > > >> target that is > > >> >so poor it has only one socket (even SNMP won't work there). > > >> For this case > > >> >the only way out is to drop the connection and hope the > > target will hear > > >> >you (it probably won't -:)). The comment is the draft > is a memento > for > > >> >implementers to keep looking for new connections always (even > > for a one > > >> >connection target) but probably it does not come through clear > enough. > > >> > > > >> >Regards, > > >> >Julo > > >> > > > >> >Mark Bakke on 14/03/2001 18:17:14 > > >> > > > >> >Please respond to Mark Bakke > > >> > > > >> >To: Julian Satran/Haifa/IBM@IBMIL > > >> >cc: ips@ece.cmu.edu > > >> >Subject: Re: iSCSI: Logout needs ISID > > >> > > > >> > > > >> > > > >> > > > >> >Julian- > > >> > > > >> >A target that does not support multiple connections per session > > >> >will return "Initiator SID Error" when the login > attempt is made. > > >> >In this case, there is no way to log in, so there's no way to > > >> >log out the old connection. The initiator would be > stuck waiting > > >> >until the target's side of the connection times out and goes > > >> >away, which could be a very long time. > > >> > > > >> >There is an open question within the MIB team about > whether anyone > > >> >needs to be able to kill connections or sessions from > SNMP; however, > > >> >I don't think that anyone will want to use SNMP as part of error > > >> >recovery. > > >> > > > >> >-- > > >> >Mark > > >> > > > >> >julian_satran@il.ibm.com wrote: > > >> >> > > >> >> Josh, > > >> >> > > >> >> No command, including logout, can be sent without a login. > > >> >> The complete scenario is: > > >> >> > > >> >> -open a new connection > > >> >> -login in the same session as the old connection (this has > > the ISID & > > >> >TSID) > > >> >> -logout the old connection > > >> >> -recover commands > > >> >> > > >> >> You are not supposed to be able to kill a session from outside > > >> (at least > > >> >> not an iSCSI defined mode). > > >> >> You will need management support for that (SNMP?) > > >> >> > > >> >> Regards, > > >> >> Julo > > >> >> > > >> >> Joshua Tseng on 14/03/2001 17:09:48 > > >> >> > > >> >> Please respond to Joshua Tseng > > >> >> > > >> >> To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > >> >> cc: > > >> >> Subject: RE: iSCSI: Logout needs ISID > > >> >> > > >> >> Julian, > > >> >> > > >> >> In 2.14, you state that a logout command can be > > >> >> sent on a different connection to destroy a single- > > >> >> connection iSCSI session. If you have multiple > > >> >> sessions ongoing, and the logout command is sent > > >> >> on a different connection than the one used > > >> >> to support the session that is to be killed, then > > >> >> you will need ISID to distinguish the session that > > >> >> you want killed. > > >> >> > > >> >> Regards, > > >> >> Josh > > >> >> > > >> >> > -----Original Message----- > > >> >> > From: julian_satran@il.ibm.com > [mailto:julian_satran@il.ibm.com] > > >> >> > Sent: Tuesday, March 13, 2001 9:57 PM > > >> >> > To: ips@ece.cmu.edu > > >> >> > Subject: Re: iSCSI: Logout needs ISID > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > Josh, > > >> >> > > > >> >> > There is something I am missing. The Logout can be issued > > >> >> > only after Login > > >> >> > (authentication and the rest). > > >> >> > The session is then implied. > > >> >> > > > >> >> > Regards, > > >> >> > Julo > > >> >> > > > >> >> > Joshua Tseng on > 14/03/2001 03:56:45 > > >> >> > > > >> >> > Please respond to Joshua Tseng > > >> >> > > > >> >> > To: Julian Satran/Haifa/IBM@IBMIL > > >> >> > cc: ips@ece.cmu.edu > > >> >> > Subject: iSCSI: Logout needs ISID > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > Julian, > > >> >> > > > >> >> > Section 2.14 states that the logout command may > > >> >> > be sent on a second connection to clean up the > > >> >> > a single connection iSCSI session. If the reason > > >> >> > code is 0 (close the session), then ISID is needed > > >> >> > to identify the iSCSI session to close. > > >> >> > > > >> >> > Josh > > >> >> > > > >> >> > > > >> >> > > > >> > > > >> >-- > > >> >Mark A. Bakke > > >> >Cisco Systems > > >> >mbakke@cisco.com > > >> >763.398.1054 > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > > > > > > >_________________________________________________________ > > >Do You Yahoo!? > > >Get your free @yahoo.com address at http://mail.yahoo.com > > > > > > > > > > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com > > > > From owner-ips@ece.cmu.edu Thu Mar 15 17:23:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA09347 for ; Thu, 15 Mar 2001 17:23:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FJJ3r27850 for ips-outgoing; Thu, 15 Mar 2001 14:19:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from NL-AMS-05.interxion.net (gate.interxion.com [194.153.74.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FCKgr26824 for ; Thu, 15 Mar 2001 07:20:42 -0500 (EST) Subject: Some thoughts about draft 05 of iSCSI Date: Thu, 15 Mar 2001 13:20:34 +0100 Message-ID: <23B31A8C6CF0D411B9B800508BB8465C25C461@ETXBRE101.etx> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Thread-Topic: Some thoughts about draft 05 of iSCSI Thread-Index: AcCtSlVPmw/ri6HPTDmyPsjXIGchbA== content-class: urn:content-classes:message From: "Jeroen Ruigrok van der Werven" X-MimeOLE: Produced By Microsoft Exchange V6.0.4417.0 To: , Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id f2FCKls26833 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Hi all, first, I am (not yet) subscribed to the list, so if you people could keep me in the CC:-loop for this thread I would very much appreciate it. Second, I have some remarks I want to make and some questions to ask. Please bear with me that storage solutions aren't exactly my forte aside from a consumer/customer point of view, I am more of a networking/programmer type, so I might be asking things which might be normal stuff for you. I've read some large parts of the archives already, but I might have missed my answers. 1) Why was it decided to use a 16 number dotted decimal specifier for IPv6 addresses? As far as I know, and reading RFC 2373 again, it is not mentioned anywhere, and it is so different from what is currently standard practice in notation. Sure, it's simply basic convert-the-number-from-base-x-to-base-y type of stuff, but the current practice always settles on colon-delimited hexadecimal. 2) From reading the draft it seems logical to implement iSCSI in Operatnig System in a NFS like form, in which certain disk access gets exported and the kernel/low-level stuff/daemon(s) handle the actual requests/responses. What scares me though is that with NFS one knows he is accessing filesystems over a network, with iSCSI we want to make it as transparent as possible [at least that's how it struck me], but how is the blocking issue looked upon then? It seems hard to guarantee the same reponse over a network as with directly coupled cables. I am very much interested in this, since if we want to deploy this as a product our customers will have at least the following demands/questions: - data integrity - speed/latency 3) Julian, on page 80 of draft 05, section 6.2, last sentence: "it receives the a Data PDU with" should probably be: "it receives a DATA PDU with". 4) Am I correct in assuming, based 'pon my readnig of the draft, that bursting is limited to what FirstBurstSize is set to? So there is little/no chance of having the iSCSI protocol saturate the available networkbandwidth all of a sudden by bursting to the max. 5) The draft specifies that both data PDU and commands are being sent over the same TCP connection. This got me curious about the fact if iSCSI commands would get delayed if sufficiently large amounts of data are being sent over the TCP connection. For example, with some protocols they have a seperate control and data connection. IIRC, NNTP-streaming has it, ISDN is another prime example of this. Thanks in advance for any trouble and time taken to answer my questions, -- Jeroen Ruigrok van der Werven Systems Software Engineer / Connectivity Services Development InterXion, where the Internet lives From owner-ips@ece.cmu.edu Thu Mar 15 17:24:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA09370 for ; Thu, 15 Mar 2001 17:24:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FKU0I03179 for ips-outgoing; Thu, 15 Mar 2001 15:30:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FKTNr03144 for ; Thu, 15 Mar 2001 15:29:23 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id PAA24042; Thu, 15 Mar 2001 15:29:14 -0500 (EST) Message-ID: <3AB12619.43B47567@cisco.com> Date: Thu, 15 Mar 2001 14:29:13 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: someshg@yahoo.com CC: Venkat Rangan , IPS , jmuchow@cisco.com Subject: Re: iSCSI MIB Object Model References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Somesh- We are looking into making the LU and LUN tables optional, which would help in this case. I will bring up the appropriateness of including them at IETF next week. I like your idea of keeping track of the last few LUN errors. I assume that you would like to see something like: 1. Each target includes a list of errors; the target would fill in something like the timestamp, LUN, and initiator WWUI that caused that error last. 2. Each session includes a list of errors, and does a similar thing. Does this sound reasonable? Do you have other ideas for doing this that would simplify finding out what went wrong with LUs and LUNs? -- Mark Somesh Gupta wrote: > > Mark, > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Mark Bakke > > Sent: Wednesday, March 14, 2001 12:49 PM > > To: Venkat Rangan > > Cc: IPS; jmuchow@cisco.com > > Subject: Re: iSCSI MIB Object Model > > > > > > Venkat- > > > > You have captured many of our open issues in this > > message. Please see my comments below. > > > > Thanks, > > > > Mark > > > > Venkat Rangan wrote: > > > > > > Mark, > > > > > > The UML is very nice, and the new object model, covering both > > > Initiator and Target instances is also nice. > > > > > > Overall, we do have some concern over the need for maintaining > > per-LU and > > > per-LUN statistics. In some iSCSI implementations, the iSCSI entity is > > > merely an intermediate layer, with no knowledge of LUs and LUNs. Placing > > > this in the GeneralInfo category and making it mandatory would add undue > > > burden on these implementations. > > > > One of our open questions is "how much is too much?". We added the LU > > and LUN statistics for a few reasons: > > > > 1. The iSCSI layer does see the LUN; it is included in the SCSI Command > > message. > > > > 2. For implementations that include the actual target and LUs, such > > as storage controllers, there is no other way to get information > > on LUNs through SNMP, since there is no SCSI MIB. > > > > 3. We felt that in this case customers might require LU- or LUN-level > > accounting. > > I just did a rough (very rough) calculation for the amount of data > required per lu and per lun - It is of the order of 256 bytes per lu > and 512 bytes (actually more like 350) per lun. The number of > Luns/lus can be in the 10s of thousands. > > Even though the iSCSI PDU header carries a LUN number, this value > is really transparent to the iSCSI layer. The iSCSI layer should > not even know how many LUNs there are in the system. The size of > the data set, and the cost of managing it are fairly significant. > The ability of systems to dynamically add & delete LUNs/LUs should > not be tied to the ability of the iSCSI layer to track the stats. > > I think reasons 1 & 2 are not very good reasons. An analogy would > be asking TCP to perpetually keep statistics on every 4-tuple that > was ever used to create a connection. > > I would recommend focusing on the error cases only for one - and > perhaps have a table per error type tracking the last "n" errors. > What does it matter if an R2T was received for a LUN or not? It > just becomes a bunch of probably unneeded information to weed > through to get to the real info. > > > > > However, keeping statistics at this level does create some work; perhaps > > there is a way to make it optional. > > > > > In Section 5.9: > > > "The iscsiLUTable contains a list of iSCSI Logical Units that are > > > accessible via the iSCSI target." > > > > > > Would this list be the same as what the "Report LUNs" command > > would give? > > > > It depends. A LUN is just the address of a logical unit, and in the > > general case, is only valid between a particular initiator and a > > particular > > target. If a second initiator issues a "Report LUNs" to the same target, > > it may get back a different list if the target is performing some sort of > > LUN mapping. Some targets will show the same set of LUNs > > regardless of the > > initiator; some won't. There is no reliable way given the current MIB to > > find out what "Report LUNs" would return; this is probably the domain of > > a SCSI MIB. > > > > > In Section 5.7: > > > "The iscsiLunTable contains an entry for each known LUN that is being > > > accessed using this session. Note that is may not contain all of the > > > LUNs that could be accessed; the only ones available to iSCSI are the > > > ones that have been addressed specifically by iSCSI commands. > > > Therefore, LUNs that have been discovered via mechanisms such as the > > > SCSI "report LUNs" will not appear in this table until they are > > > specifically addressed by the initiator. > > > > > > How long do these entries live in the table? If iSCSI sessions > > logout and > > > close, do these entries continue to remain? > > > > The current intent is to have entries live only as long as the sessions; > > these would be removed when the session is removed. This is the simplest > > solution and would put the smallest burden on the implementation. > > > > However, we still have an open "requirements" question on whether anyone > > wants these to hang around longer. > > > > > Thanks, > > > > > > Venkat Rangan > > > Rhapsody Networks Inc. > > > http://www.rhapsodynetworks.com > > > > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 > > Somesh > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Thu Mar 15 17:25:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA09389 for ; Thu, 15 Mar 2001 17:25:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FKN1R02642 for ips-outgoing; Thu, 15 Mar 2001 15:23:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from crufty.research.bell-labs.com (crufty.research.bell-labs.com [204.178.16.49]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2FKLwr02581 for ; Thu, 15 Mar 2001 15:21:58 -0500 (EST) Received: from scummy.research.bell-labs.com ([135.104.2.10]) by crufty; Thu Mar 15 15:19:45 EST 2001 Received: from aura.research.bell-labs.com ([135.104.46.10]) by scummy; Thu Mar 15 15:21:39 EST 2001 Received: from research.bell-labs.com (IDENT:sandeepj@sandeepj-pcmh.research.bell-labs.com [135.104.47.90]) by aura.research.bell-labs.com (8.9.1/8.9.1) with ESMTP id PAA23896 for ; Thu, 15 Mar 2001 15:21:39 -0500 (EST) Message-ID: <3AB12452.928F425C@research.bell-labs.com> Date: Thu, 15 Mar 2001 15:21:38 -0500 From: Sandeep Joshi X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.2.16-3 i686) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Section 2.4.2. Error Communication Latencies. References: <3AB01DE1.2DD3C455@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit your question has been asked once before. see http://ips.pdl.cs.cmu.edu/mail/msg03453.html scroll to where it says "Section 2.4.2". -sandeep Santosh Rao wrote: > > All, > > The rev 05 draft mandates targets to wait until all the Data PDUs of an > exchange have been received, prior to communicating any error with the > I/O that may have occurred on an earlier I/O. Why is this needed ? This > only adds additional latency to error communication from target to > initiator and can cause large delays in communicating an error with long > lasting I/Os like tape. > > The ability to communicate an error on its occurrence is something other > SCSI transports provide and it requires the target to be capable of > safeguarding against the arrival of stale frames of a PDU from an old > exchange, something which should be possible and required in any case. > > Regards, > Santosh > > Section 2.4.2, pg 36 > --------------- > "If a SCSI device error is detected while data from the initiator is > still expected (the command PDU did not contain all the data and the > target has not received a Data PDU with the final bit Set) the target > MUST wait until it receives the a Data PDU with the F bit set before > sending the Response PDU." From owner-ips@ece.cmu.edu Thu Mar 15 17:33:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA09504 for ; Thu, 15 Mar 2001 17:33:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FL52G05657 for ips-outgoing; Thu, 15 Mar 2001 16:05:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FL4wr05647 for ; Thu, 15 Mar 2001 16:04:58 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id QAA21661 for ; Thu, 15 Mar 2001 16:04:52 -0500 (EST) Message-ID: <3AB12E74.3A7E696D@cisco.com> Date: Thu, 15 Mar 2001 15:04:52 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: IPS Subject: iSCSI: Rationale behind the SLP draft Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David Black has requested that I send something to the list showing the rationale behind the SLP draft, including its fit in the big picture and its relationship with iSNS. Here's an attempt to do so. I believe it to be consistent with the Naming & Discovery, iSNS, and iSCSI/SLP drafts, and their IETF-50 presentations. The relevant drafts are: draft-ietf-ips-iSCSI-name-disc-00 draft-ietf-ips-isns-00 draft-bakke-iscsi-slp-00 Naming and Discovery Requirements These are the discovery and name services requirements from draft-ietf-ips-iscsi-name-disc-00. From a discovery requirements point-of-view, SLP and iSNS provide some similar services. SLP provides very simple discovery services for smaller networks; iSNS provides more comprehensive management services for larger networks. Where they overlap, SLP and iSNS can interoperate fairly easily. 1. Multicast method to discover targets, to allow zero- configuration of initiators and targets. SLP provides this. 2. A method to look up a WWUI, and get one or more addresses. "Where is target ?" SLP and iSNS both provide this. 3. A method to find targets that a given initiator should access. "I am initiator ; which targets should I access?" SLP and iSNS both provide this. 4. Ability to limit discovery to relevant initiators and targets. iSNS uses Discovery Domain, which are used to limit which initiators and targets can discover each other. Each target and initiator belong to Discovery Domain; the iSNS will not allow initiators to discover targets outside their domain. SLP uses a named Scope. Each target and initiator is configured with one or more scope names; a target will only respond to an initiator who has a matching scope. Note that this is not a security mechanism; an initiator can ask for any scope it wishes, and a target can respond to any scope it wishes. This provides a simple mechanism to limit discovery, but does not attempt to provide security. 5. Access Lists by initiator WWUI SLP and iSNS both provide the ability to store a list of initiator WWUIs that are allowed access to a given target. 6. iSCSI Object Model support SLP registers only the targets iSNS registers targets, initiators, network entities, portals 7. TLV Message Format SLP and iSNS both do this. However, SLP uses a text key and value for its attributes; iSNS uses a binary key value. 8. Authentication of discovery messages iSNS uses SLP's authentication mechanism 9. Registration and Query SLP registers targets only iSNS registers targets, inititors, network entities, portals 10. State Change Notification iSNS provides a state change service for which entities can register. SLP does not provide state change or event notification. There is a draft on doing some notification, but it is not adequate for propagating attribute changes, which are necessary for an initiator to discover that it has gained (or lost) access to a target, or that a target has added a new address. 11. Light-weight Protocol Neither protocol should be too much of a burden to implement. Adding authentication to the protocol can be done later, and will be the same effort for both. In the case that both SLP and iSNS are supported in the same product, they can share the code for this feature. 12. Compatibility with Boot Requirements Both the SLP template and the iSNS specification have worked to ensure that they fit in with the boot requirements. SLP has defined DHCP options that allow it to locate a directory agent, avoiding multicast traffic from initiators in a DHCP environment. 13. Compatibility with LDAP Although this is not a requirement, we have seen the desire for interworking with LDAP discussed on the list enough that it may merit a requirement at some point. Both SLP and iSNS are designed with LDAP compatibility in mind. 14. Entity Status Inquiry/Heartbeat iSNS provides an ESI/heartbeat ping message to monitor the availability of initiators and targets. The heartbeat is originated by the iSNS server. We have not defined this capability for SLP. It may be possible to do something similar if targets advertise themselves with a shorter lifetime, and initiators expire their discovered targets appropriately. Adding a directory agent (DA) may be required for this to work properly. 15. Distribution of X.509v3 Public Key certificates iSNS provides this capability. Our SLP template does not attempt to do this for two reasons: - SLP attributes are strings; the binary certificates would have to be escaped or uuencoded. - iSCSI only registers targets; there is no way for targets to discover an initiator's certificate. 16. Usage of Multicast SLP normally relies on multicast, for initiators to request advertisements from targets. Multicast use can be eliminated at the expense of adding a DA, and configuring its address on each of the initiators and targets. iSNS does not use multicast. We had originally started out looking for something to fulfill the multicast requirements that iSNS did not do. However, we found that if one already had SLP, it was just as easy to find individual targets, or at least canonical targets using it as it was to locate an iSNS service. SLP can then be used to handle discovery for simpler host and device environments, and iSNS could be added later to help manage these environments. SLP and iSNS both fulfill many of the requirements from the naming & discovery draft. However, they don't completely overlap, and they take two different approaches to discovery. SLP provides Discovery of Targets. SLP is used for discovery ONLY. It uses a distributed model, that can start with initiators and targets, and no directory servers. It can add Directory Agents to scale to medium-sized environments and reduce or eliminate the use of multicast. Some work is underway on mesh- enhanced SLP (mSLP), which provides peer-to-peer information exchange between DAs for larger environments. From a pure discovery point of view, SLP's chief weakness is that it does not yet have a notification mechanism. - Discovery of iSCSI targets - Discovery of iSNS services - Multicast-optional discovery mechanism for targets and other name services. - Open source implementations - Existing standard protocols iSNS provides Centralized Management of Initiators and Targets. In addition to target discovery, iSNS provides higher-level functions such as centralized access management, active monitoring, public-key certificate management, and state-change notifications. - Discovery of iSCSI targets - Centralized management of initiator and target access - Active monitoring of initiators and targets - State change notification Using SLP with iSNS Although SLP and iSNS solve several of the same problems, they are fairly well-suited to interoperating. They share a common authentication structure, which is not something one would want to code twice. SLP can be used to do basic discovery where configuring an iSNS server is not warranted, and can help discover iSNS servers in environments where centralized management is needed. It is relatively simple for an iSNS server to provide a proxy SLP service agent on behalf of the targets it manages, allowing initiators to participate that implement only the basic SLP. Similarly, an iSNS server can work with gateways to provide its services for targets that support only SLP. When initiators and targets support BOTH SLP and iSNS, there are a few rules that should be followed: 1. Initiators can use both SLP and iSNS concurrently to discover their targets. In fact, an initiator should be able to use more than one iSNS, if it is accessing storage from two different providers. This allows an initiator to discover its locally-managed devices using SLP, and its centrally- managed devices using iSNS. 2. Targets should be advertised using a single discovery method. A target should be advertised either with SLP, or with a single iSNS environment. Targets discovered multiple ways would probably not break anything, but a target discovered via multiple services could produce conflicting information to the initiator. 3. Gateways or iSCSI proxies can be used to provide local SLP discovery of targets that are managed using iSNS. Implementation Approach I recommend a three-phase approach to implementing iSCSI naming and discovery. The first phase is to simply support the WWUIs, and their use with the iSCSI protocol, and allow static configuration of hosts and targets. The second is to support SLP for simple discovery. This should be relatively quick to implement, as the source for SLP is readily available. The third would be to support iSNS as a storage management capability. 1. Simple naming and configuration - Admins configure targets with some form of access list, which may include the initiator WWUIs that are supposed to access them. - Admins configure initiators with the canonical target "iscsi" for targets that may provide them services. - Initiators use the SendTargets text command to discovery their targets. This eliminates configuring target WWUIs in the initiator. 2. SLP for multicast and simple discovery - Instead of configuring initiators with target addresses, admins enable SLP on the initiator. - Targets also enable SLP, and are discovered by the initiators. 3. iSNS for centralized management - Initiators and targets are configured to be managed by an iSNS service. - Admins configure both hosts and targets from the iSNS server. - Initiators and targets use the iSNS protcol to discover each other. Other Discovery Protocols There are several other discovery protocols; each has its strengths and weaknesses. Here are some of them. Take a look at "Building Networks on the Fly", IEEE Spectrum, March 2001 for more information on these. 1. DNS SRV Records - these seemed too limiting, and there did not seem to be much point in making DNS do more. The SLP folks started their work because DNS SRV records didn't meet their requirements. 2. LDAP, with an appropriate schema defined, could handle iSCSI registrations and queries, and distribute essentially the same information as SLP. There are three reasons we didn't go directly to LDAP: 1. LDAP would require a heavier implementation than SLP 2. LDAP would not solve the multicast requirements 3. SLP is built to be compatible with LDAP; SLP can be used by iniators and targets, with LDAP as a back-end database. This is left as an exercise for the implementor. 3. Jini is controlled by a single company, and is tied to a particular programming language, which is not appropriate for an internet standard. 4. UPnP from Microsoft is based on XML data with an HTTP transport. XML has a lot of merit for application interfaces like these, but the code for XML and HTTP could be a bit heavy for a really simple device. It's company-controlled anyway, so we can't use it. 5. Salutation - Have not explored this in depth. Salutation has been around longer than the other protocols. It is made to be more flexible; it does not assume IP. 6. Bluetooth, HAVi - These are done by controlled-membership organizations, and are designed for specific non-IP environments. -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Thu Mar 15 17:54:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA09838 for ; Thu, 15 Mar 2001 17:54:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FL9AL05986 for ips-outgoing; Thu, 15 Mar 2001 16:09:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FL8Lr05939 for ; Thu, 15 Mar 2001 16:08:26 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GA9002F4C1QT4@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Thu, 15 Mar 2001 13:08:14 -0800 (PST) Date: Thu, 15 Mar 2001 13:06:51 -0800 From: Douglas Otis Subject: RE: FCIP iFCP encapsulation proposal In-reply-to: To: Robert Snively , Black_David@emc.com, ips@ece.cmu.edu Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Bob, You are assuming that a debug analyzer would store the entire TCP frame. Perhaps, but just as likely only iFCP payload is stored as the analyzer may assume TCP does not require debugging and perhaps not seen depending on where the filter is placed. Even so, a binary image of an entire Ethernet frame is not likely to be contained completely within an Ethernet frame as a storage block. It is likely to be fragmented. Even if the definition of a valid header was extended to include a valid trailer from the previous PDU, you still will be confronted with the same problem. A TCP header placed anywhere before an apparently valid header will not be of any concern to a header search. The method of delivery is simply the image is contained within storage blocks delivered as legitimate payload. Once synchronization loss due to a packet drop, the ability to determine payload from headers is removed and that was the point that David was making. In other words, it may not even be a malicious act for this error to occur. Doug > -----Original Message----- > From: Robert Snively [mailto:rsnively@brocade.com] > Sent: Thursday, March 15, 2001 9:18 AM > To: 'Douglas Otis'; Robert Snively; Black_David@emc.com; ips@ece.cmu.edu > Subject: RE: FCIP iFCP encapsulation proposal > > > Doug, > > The binary image is of ethernet frames. It requires the existence > of a matching TCP/IP connection with matching TCP headers, > including pdu sequencing information, which is not knowable > from the spoofer. How are these delivered? > > The same question is true for each of the other layers of the > transfer, and the same unlikely scenario must be played back > for each. I just don't see such data being delivered by > a responsible software layer. > > Bob > > > -----Original Message----- > > From: Douglas Otis [mailto:dotis@sanlight.net] > > Sent: Wednesday, March 14, 2001 1:15 PM > > To: Robert Snively; Black_David@emc.com; ips@ece.cmu.edu > > Subject: RE: FCIP iFCP encapsulation proposal > > > > > > Bob, > > > > With out discussing spoofing where attackers successfully guess TCP > > sequences (made too easy in some cases), a binary image is > > stored and then > > legitimately sent as a payload, with the example being > > binary content of a > > debug analyzer. In this case, headers contained within the > > payload could be > > seen as valid. The valid header within the payload may fool > > a process that > > attempts to recover header synchronization following a > > dropped packet. This > > header may carry the same information in current use and be > > acted upon or > > send the connection into error oblivion. It would appear to > > represent a > > weakness that can be exploited. Dropped packets happen. > > > > Doug > > > > > > > From owner-ips@ece.cmu.edu Thu Mar 15 17:58:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA09904 for ; Thu, 15 Mar 2001 17:58:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FL9Rm06003 for ips-outgoing; Thu, 15 Mar 2001 16:09:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FL8Pr05944 for ; Thu, 15 Mar 2001 16:08:25 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GA9002F4C1QT4@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Thu, 15 Mar 2001 13:08:18 -0800 (PST) Date: Thu, 15 Mar 2001 13:06:53 -0800 From: Douglas Otis Subject: RE: iSCSI: Need to Kill Session from Surviving Node In-reply-to: To: John Hufferd , ips@ece.cmu.edu Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit John, It does bring up some interesting points. If there is loss of ability to process any further PDUs due to a loss of a single PDU due to a digest error, there can be no further communication on that connection other than a TCP graceful close with perhaps data sent past the point where a problem was detected. Rather than issuing a reject, a message indicating the closing of the connection would be appropriate but it would be without acknowledgement. There is a requirement in this case of always allowing multiple connections for the case of recovery if a close is not possible. There should be no single connection limitation in this case or take-over is not possible. Perhaps the minimum would be two connections and always a free connection allocation. There would seem to be a need for a clear definition of how this recovery takes place to ensure sequential integrity. There seems a reluctance in specifying these details as it limits some architectural freedom but these details are just as much a part of this protocol as are the PDU structures. Several have argued that it would be impractical to have this transport attend to LUNs, the issue of reset is also interesting. Presently there is no convention for differentiating systems with or without administrative authority to allow filtering such system wide commands. It could be a convention provided by the authority information stored within a standardized database. My vote would be to develop a schema for LDAP to address this issue or decide if command filter is even practical. Doug > Wait a minute, I did not expect my Surviving Node question to somehow > trigger the discussion of whether the Target Reset should be issued by > iSCSI Initiators. That is a completely different Pit. > > My question was, when the Storage Controller fails over to a surviving > NODE, how do we tell the Initiator that is waiting peacefully for the > TCP/IP connection to time-out. To quickly give-up and let the initiator's > SCSI layer, retry the operations, so that the iSCSI layer can start either > sending the SCSI cmds to the Surviving Node, or starts a new Session with > the Surviving Node. > > This has nothing to do with the Initiator issuing Target Resets. > > > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Internet address: hufferd@us.ibm.com > > > "Somesh Gupta" @ece.cmu.edu on 03/14/2001 08:25:58 PM > > Please respond to > > Sent by: owner-ips@ece.cmu.edu > > > To: , > cc: > Subject: RE: iSCSI: Need to Kill Session from Surviving Node > > > > Mallikarjun, > > I am not disagreeing with the desire to do a graceful > termination. However, when recovery states at the TCP > level, and at the iSCSI level get intertwined with this, > it is probably better to focus on a clean (possibly graceful) > termination and a clean bringup. > > Somesh > > > -----Original Message----- > > From: cbm@core.rose.hp.com [mailto:cbm@core.rose.hp.com]On Behalf Of > > Mallikarjun C. > > Sent: Wednesday, March 14, 2001 7:31 PM > > To: someshg@yahoo.com > > Cc: ips@ece.cmu.edu > > Subject: RE: iSCSI: Need to Kill Session from Surviving Node > > > > > > Somesh, > > > > Please refer to the following URL (and emails around that) for > > a long discussion on this topic. Both flavors of resets have value, > > and I agree with Julian that iSCSI should support both. > > > > http://ips.pdl.cs.cmu.edu/mail/msg00740.html > > > > -- > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > > > >Do you need both a "warm-reset" and a "cold-reset"? > > >The "warm-reset" seems to be an over-optimization. > > > > > >Somesh > > > > > >> -----Original Message----- > > >> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > Behalf Of > > >> Mallikarjun C. > > >> Sent: Wednesday, March 14, 2001 12:16 PM > > >> To: ips@ece.cmu.edu > > >> Subject: Re: iSCSI: Need to Kill Session from Surviving Node > > >> > > >> > > >> >There is a related problem that I think we need to address, and > > >> that is -- > > >> >the case where the Storage Controller is an Active-Active > > Fail-Over unit > > >> >with very little state shared between the Storage Controller > > Nodes. (I > > >> >think a number of you folks have told me that you are > planning such a > > >> >design.) This would mean that a specific initiator OS (WWUI) > > would have > > >> >separate sessions between the Nodes. At fail-over, the > > >> Take-Over Node will > > >> >want to cause the Initiator, to Stop messing around with the > > Failing node > > >> >ASAP, and start using the existing session, or start a new > > >> session with the > > >> >Take_Over Node. The Initiator, will normally just time-out the > TCP/IP > > >> >connection, and then turn the recovery over to the SCSI > > Layer. Then, when > > >> >SCSI retries, iSCSI will either use the Session to the > > Surviving Node or > > >> >Create a New Session to that Surviving Node. The problem is, > > that this > > >> >will take a longer time then a system might want. > > >> > > > >> >So the question: is there a way for a surviving Target Node > > to tell the > > >> >Initiator to "Kill" the Session to the Failing Target Node? > > >> > > > >> >Comments? > > >> > > >> Assuming that the failing Node is still able to execute commands, > > >> a cold target reset from the Initiator to the failing node > > should "kill" > > >> all the sessions to it. > > >> > > >> The answer to two hosts configured in a failover > configuration sharing > > >> a single Storage Controller Node is also the same. > > >> > > >> If it is active-active configuration, it appears to me that there's a > > >> need to somehow prompt the Initiator to doing the reset on a > > third party > > >> (Async PDU?). > > >> > > >> This is a reason why we'd still need target reset in iSCSI. > > >> -- > > >> Mallikarjun > > >> > > >> > > >> Mallikarjun Chadalapaka > > >> Networked Storage Architecture > > >> Network Storage Solutions Organization > > >> MS 5668 Hewlett-Packard, Roseville. > > >> cbm@rose.hp.com > > >> > > >> > > > >> >. > > >> >. > > >> >. > > >> >John L. Hufferd > > >> >Senior Technical Staff Member (STSM) > > >> >IBM/SSG San Jose Ca > > >> >(408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > > >> >Internet address: hufferd@us.ibm.com > > >> > > > >> > > > >> >Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/14/2001 08:31:19 AM > > >> > > > >> >Sent by: owner-ips@ece.cmu.edu > > >> > > > >> > > > >> >To: ips@ece.cmu.edu > > >> >cc: > > >> >Subject: Re: iSCSI: Logout needs ISID > > >> > > > >> > > > >> > > > >> > > > >> > > > >> >Mark, > > >> > > > >> >If you have only one connection you are supposed to use a > > Login with the > > >> >restart bit one - and achieve a similar effect as a > > Login/Logout - i.e. > > >> >keep the session alive. Even this might be a problem for a > > >> target that is > > >> >so poor it has only one socket (even SNMP won't work there). > > >> For this case > > >> >the only way out is to drop the connection and hope the > > target will hear > > >> >you (it probably won't -:)). The comment is the draft is a memento > for > > >> >implementers to keep looking for new connections always (even > > for a one > > >> >connection target) but probably it does not come through clear > enough. > > >> > > > >> >Regards, > > >> >Julo > > >> > > > >> >Mark Bakke on 14/03/2001 18:17:14 > > >> > > > >> >Please respond to Mark Bakke > > >> > > > >> >To: Julian Satran/Haifa/IBM@IBMIL > > >> >cc: ips@ece.cmu.edu > > >> >Subject: Re: iSCSI: Logout needs ISID > > >> > > > >> > > > >> > > > >> > > > >> >Julian- > > >> > > > >> >A target that does not support multiple connections per session > > >> >will return "Initiator SID Error" when the login attempt is made. > > >> >In this case, there is no way to log in, so there's no way to > > >> >log out the old connection. The initiator would be stuck waiting > > >> >until the target's side of the connection times out and goes > > >> >away, which could be a very long time. > > >> > > > >> >There is an open question within the MIB team about whether anyone > > >> >needs to be able to kill connections or sessions from SNMP; however, > > >> >I don't think that anyone will want to use SNMP as part of error > > >> >recovery. > > >> > > > >> >-- > > >> >Mark > > >> > > > >> >julian_satran@il.ibm.com wrote: > > >> >> > > >> >> Josh, > > >> >> > > >> >> No command, including logout, can be sent without a login. > > >> >> The complete scenario is: > > >> >> > > >> >> -open a new connection > > >> >> -login in the same session as the old connection (this has > > the ISID & > > >> >TSID) > > >> >> -logout the old connection > > >> >> -recover commands > > >> >> > > >> >> You are not supposed to be able to kill a session from outside > > >> (at least > > >> >> not an iSCSI defined mode). > > >> >> You will need management support for that (SNMP?) > > >> >> > > >> >> Regards, > > >> >> Julo > > >> >> > > >> >> Joshua Tseng on 14/03/2001 17:09:48 > > >> >> > > >> >> Please respond to Joshua Tseng > > >> >> > > >> >> To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > >> >> cc: > > >> >> Subject: RE: iSCSI: Logout needs ISID > > >> >> > > >> >> Julian, > > >> >> > > >> >> In 2.14, you state that a logout command can be > > >> >> sent on a different connection to destroy a single- > > >> >> connection iSCSI session. If you have multiple > > >> >> sessions ongoing, and the logout command is sent > > >> >> on a different connection than the one used > > >> >> to support the session that is to be killed, then > > >> >> you will need ISID to distinguish the session that > > >> >> you want killed. > > >> >> > > >> >> Regards, > > >> >> Josh > > >> >> > > >> >> > -----Original Message----- > > >> >> > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > > >> >> > Sent: Tuesday, March 13, 2001 9:57 PM > > >> >> > To: ips@ece.cmu.edu > > >> >> > Subject: Re: iSCSI: Logout needs ISID > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > Josh, > > >> >> > > > >> >> > There is something I am missing. The Logout can be issued > > >> >> > only after Login > > >> >> > (authentication and the rest). > > >> >> > The session is then implied. > > >> >> > > > >> >> > Regards, > > >> >> > Julo > > >> >> > > > >> >> > Joshua Tseng on 14/03/2001 03:56:45 > > >> >> > > > >> >> > Please respond to Joshua Tseng > > >> >> > > > >> >> > To: Julian Satran/Haifa/IBM@IBMIL > > >> >> > cc: ips@ece.cmu.edu > > >> >> > Subject: iSCSI: Logout needs ISID > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > Julian, > > >> >> > > > >> >> > Section 2.14 states that the logout command may > > >> >> > be sent on a second connection to clean up the > > >> >> > a single connection iSCSI session. If the reason > > >> >> > code is 0 (close the session), then ISID is needed > > >> >> > to identify the iSCSI session to close. > > >> >> > > > >> >> > Josh > > >> >> > > > >> >> > > > >> >> > > > >> > > > >> >-- > > >> >Mark A. Bakke > > >> >Cisco Systems > > >> >mbakke@cisco.com > > >> >763.398.1054 > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > > > > > > >_________________________________________________________ > > >Do You Yahoo!? > > >Get your free @yahoo.com address at http://mail.yahoo.com > > > > > > > > > > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com > > > > > From owner-ips@ece.cmu.edu Thu Mar 15 17:59:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA09915 for ; Thu, 15 Mar 2001 17:58:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FLi4i08515 for ips-outgoing; Thu, 15 Mar 2001 16:44:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp015.mail.yahoo.com (smtp015.mail.yahoo.com [216.136.173.59]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2FLhFr08473 for ; Thu, 15 Mar 2001 16:43:15 -0500 (EST) Received: from unknown (HELO somesh) (206.112.127.187) by smtp.mail.vip.sc5.yahoo.com with SMTP; 15 Mar 2001 21:43:12 -0000 X-Apparently-From: Reply-To: From: "Somesh Gupta" To: , Cc: "Venkat Rangan" , "IPS" , Subject: RE: iSCSI MIB Object Model Date: Thu, 15 Mar 2001 13:38:33 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <3AB12619.43B47567@cisco.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mark, What you propose is what I was thinking of. I would expect this info to be the most useful part. My preference would be to not have per LU and per LUN info at all, even optional (at the transport level). Just because it is visible is not a good reason. This issue is really something to be addressed at a higher level, and if they don't have it yet, maybe that is just fine. Regards, Somesh > -----Original Message----- > From: mbakke@cisco.com [mailto:mbakke@cisco.com] > Sent: Thursday, March 15, 2001 12:29 PM > To: someshg@yahoo.com > Cc: Venkat Rangan; IPS; jmuchow@cisco.com > Subject: Re: iSCSI MIB Object Model > > > Somesh- > > We are looking into making the LU and LUN tables optional, > which would help in this case. I will bring up the appropriateness > of including them at IETF next week. > > I like your idea of keeping track of the last few LUN errors. > > I assume that you would like to see something like: > > 1. Each target includes a list of errors; the target would fill > in something like the timestamp, LUN, and initiator WWUI that > caused that error last. > > 2. Each session includes a list of errors, and does a similar > thing. > > Does this sound reasonable? Do you have other ideas for doing > this that would simplify finding out what went wrong with LUs > and LUNs? > > -- > Mark > > Somesh Gupta wrote: > > > > Mark, > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Mark Bakke > > > Sent: Wednesday, March 14, 2001 12:49 PM > > > To: Venkat Rangan > > > Cc: IPS; jmuchow@cisco.com > > > Subject: Re: iSCSI MIB Object Model > > > > > > > > > Venkat- > > > > > > You have captured many of our open issues in this > > > message. Please see my comments below. > > > > > > Thanks, > > > > > > Mark > > > > > > Venkat Rangan wrote: > > > > > > > > Mark, > > > > > > > > The UML is very nice, and the new object model, covering both > > > > Initiator and Target instances is also nice. > > > > > > > > Overall, we do have some concern over the need for maintaining > > > per-LU and > > > > per-LUN statistics. In some iSCSI implementations, the > iSCSI entity is > > > > merely an intermediate layer, with no knowledge of LUs and > LUNs. Placing > > > > this in the GeneralInfo category and making it mandatory > would add undue > > > > burden on these implementations. > > > > > > One of our open questions is "how much is too much?". We added the LU > > > and LUN statistics for a few reasons: > > > > > > 1. The iSCSI layer does see the LUN; it is included in the > SCSI Command > > > message. > > > > > > 2. For implementations that include the actual target and LUs, such > > > as storage controllers, there is no other way to get information > > > on LUNs through SNMP, since there is no SCSI MIB. > > > > > > 3. We felt that in this case customers might require LU- or LUN-level > > > accounting. > > > > I just did a rough (very rough) calculation for the amount of data > > required per lu and per lun - It is of the order of 256 bytes per lu > > and 512 bytes (actually more like 350) per lun. The number of > > Luns/lus can be in the 10s of thousands. > > > > Even though the iSCSI PDU header carries a LUN number, this value > > is really transparent to the iSCSI layer. The iSCSI layer should > > not even know how many LUNs there are in the system. The size of > > the data set, and the cost of managing it are fairly significant. > > The ability of systems to dynamically add & delete LUNs/LUs should > > not be tied to the ability of the iSCSI layer to track the stats. > > > > I think reasons 1 & 2 are not very good reasons. An analogy would > > be asking TCP to perpetually keep statistics on every 4-tuple that > > was ever used to create a connection. > > > > I would recommend focusing on the error cases only for one - and > > perhaps have a table per error type tracking the last "n" errors. > > What does it matter if an R2T was received for a LUN or not? It > > just becomes a bunch of probably unneeded information to weed > > through to get to the real info. > > > > > > > > However, keeping statistics at this level does create some > work; perhaps > > > there is a way to make it optional. > > > > > > > In Section 5.9: > > > > "The iscsiLUTable contains a list of iSCSI Logical Units that are > > > > accessible via the iSCSI target." > > > > > > > > Would this list be the same as what the "Report LUNs" command > > > would give? > > > > > > It depends. A LUN is just the address of a logical unit, and in the > > > general case, is only valid between a particular initiator and a > > > particular > > > target. If a second initiator issues a "Report LUNs" to the > same target, > > > it may get back a different list if the target is performing > some sort of > > > LUN mapping. Some targets will show the same set of LUNs > > > regardless of the > > > initiator; some won't. There is no reliable way given the > current MIB to > > > find out what "Report LUNs" would return; this is probably > the domain of > > > a SCSI MIB. > > > > > > > In Section 5.7: > > > > "The iscsiLunTable contains an entry for each known LUN > that is being > > > > accessed using this session. Note that is may not > contain all of the > > > > LUNs that could be accessed; the only ones available to > iSCSI are the > > > > ones that have been addressed specifically by iSCSI commands. > > > > Therefore, LUNs that have been discovered via mechanisms > such as the > > > > SCSI "report LUNs" will not appear in this table until they are > > > > specifically addressed by the initiator. > > > > > > > > How long do these entries live in the table? If iSCSI sessions > > > logout and > > > > close, do these entries continue to remain? > > > > > > The current intent is to have entries live only as long as > the sessions; > > > these would be removed when the session is removed. This is > the simplest > > > solution and would put the smallest burden on the implementation. > > > > > > However, we still have an open "requirements" question on > whether anyone > > > wants these to hang around longer. > > > > > > > Thanks, > > > > > > > > Venkat Rangan > > > > Rhapsody Networks Inc. > > > > http://www.rhapsodynetworks.com > > > > > > > > > > -- > > > Mark A. Bakke > > > Cisco Systems > > > mbakke@cisco.com > > > 763.398.1054 > > > > Somesh > > > > _________________________________________________________ > > Do You Yahoo!? > > Get your free @yahoo.com address at http://mail.yahoo.com > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Thu Mar 15 17:59:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA09926 for ; Thu, 15 Mar 2001 17:59:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FLM5g07015 for ips-outgoing; Thu, 15 Mar 2001 16:22:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FLLNr06945 for ; Thu, 15 Mar 2001 16:21:23 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id C10B0D2F for ; Thu, 15 Mar 2001 14:21:22 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id F1ED1124 for ; Thu, 15 Mar 2001 16:21:21 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id NAA05630 for ; Thu, 15 Mar 2001 13:21:20 -0800 (PST) Message-ID: <3AB1333F.FB2E8F70@agilent.com> Date: Thu, 15 Mar 2001 13:25:19 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI MIB Object Model References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I agree that LUNs and LUs are (should be) transparent to iSCSI, and thus such LU/LUN specific information should not be included in the MIB. Mark, since your company is now a member of T10, perhaps you can repackage the information as a SCSI MIB draft? -Matt Wakeley julian_satran@il.ibm.com wrote: > > Mark, > > iSCSI uses LUN almost transparently. Also the LUN may change while a > session is active. > I am not sure that the fact that there is no SCSI MIB (yet) doe justify > including those items in the iSCSI MIB. > > I also recall (a rumor?) that somebody is working on a SCSI MIB (it will be > needed anyhow there is more to SCSI than the LU). > > Julo > > Mark Bakke on 14/03/2001 22:48:47 > > Please respond to Mark Bakke > > To: Venkat Rangan > cc: IPS , jmuchow@cisco.com > Subject: Re: iSCSI MIB Object Model > > Venkat- > > You have captured many of our open issues in this > message. Please see my comments below. > > Thanks, > > Mark > > Venkat Rangan wrote: > > > > Mark, > > > > The UML is very nice, and the new object model, covering both > > Initiator and Target instances is also nice. > > > > Overall, we do have some concern over the need for maintaining per-LU and > > per-LUN statistics. In some iSCSI implementations, the iSCSI entity is > > merely an intermediate layer, with no knowledge of LUs and LUNs. Placing > > this in the GeneralInfo category and making it mandatory would add undue > > burden on these implementations. > > One of our open questions is "how much is too much?". We added the LU > and LUN statistics for a few reasons: > > 1. The iSCSI layer does see the LUN; it is included in the SCSI Command > message. > > 2. For implementations that include the actual target and LUs, such > as storage controllers, there is no other way to get information > on LUNs through SNMP, since there is no SCSI MIB. > > 3. We felt that in this case customers might require LU- or LUN-level > accounting. > > However, keeping statistics at this level does create some work; perhaps > there is a way to make it optional. > > > In Section 5.9: > > "The iscsiLUTable contains a list of iSCSI Logical Units that are > > accessible via the iSCSI target." > > > > Would this list be the same as what the "Report LUNs" command would give? > > It depends. A LUN is just the address of a logical unit, and in the > general case, is only valid between a particular initiator and a particular > target. If a second initiator issues a "Report LUNs" to the same target, > it may get back a different list if the target is performing some sort of > LUN mapping. Some targets will show the same set of LUNs regardless of the > initiator; some won't. There is no reliable way given the current MIB to > find out what "Report LUNs" would return; this is probably the domain of > a SCSI MIB. > > > In Section 5.7: > > "The iscsiLunTable contains an entry for each known LUN that is being > > accessed using this session. Note that is may not contain all of the > > LUNs that could be accessed; the only ones available to iSCSI are the > > ones that have been addressed specifically by iSCSI commands. > > Therefore, LUNs that have been discovered via mechanisms such as the > > SCSI "report LUNs" will not appear in this table until they are > > specifically addressed by the initiator. > > > > How long do these entries live in the table? If iSCSI sessions logout and > > close, do these entries continue to remain? > > The current intent is to have entries live only as long as the sessions; > these would be removed when the session is removed. This is the simplest > solution and would put the smallest burden on the implementation. > > However, we still have an open "requirements" question on whether anyone > wants these to hang around longer. > > > Thanks, > > > > Venkat Rangan > > Rhapsody Networks Inc. > > http://www.rhapsodynetworks.com > > > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 From owner-ips@ece.cmu.edu Thu Mar 15 19:01:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA10618 for ; Thu, 15 Mar 2001 19:01:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FM25A09873 for ips-outgoing; Thu, 15 Mar 2001 17:02:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (mars.iol.unh.edu [132.177.121.222]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FM1dr09819 for ; Thu, 15 Mar 2001 17:01:39 -0500 (EST) Received: from localhost (rdr@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id RAA02388; Thu, 15 Mar 2001 17:06:20 -0500 (EST) Date: Thu, 15 Mar 2001 17:06:20 -0500 From: "Robert D. Russell" To: julian_satran@il.ibm.com cc: ips@ece.cmu.edu Subject: Re: iSCSI: Towards a more effective PDU format In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo: Even if we give up on the idea of having a second header digest, I still believe that some changes should to be made to the PDU Format proposed in version 5 to make it simpler, more efficient to process, and more robust. I would therefore propose eliminating the use of the current WN Next-Qualifier scheme and replacing it as follows (this is just a slight modification of my earlier proposal in order to eliminate the second header digest). 1. Every PDU must start with a 48-byte Basic Header (BHS) that can be read in a single "read" operation. The format of this is identical to that proposed in section 2.2.4 of version 5, but with the addition of 2 fields (4-bytes) as described next. 2. Every BHS contains 2 fields at fixed locations: a) AHS_length, containing the total length of all Additional Header Segments (AHS) that follow. This field is 0 if there are no AHSs. b) DATA_length, containing the total length of all data that follows the AHSs. This field is 0 if there is no data. 3. If the AHS_length field in the BHS is non-zero, all the AHSs immediately follow the BHS in the PDU. The AHS_length value gives the total number of bytes in all the AHSs, allowing a single "read" operation to read them all. (The total header consists of 48+AHS_length bytes.) 4. If header digests are in use, one header digest covering the BHS and all the AHSs (if any) immediately follows the last AHS. This digest does NOT require a separate read, since the first read (of the BHS) should be for 48+(length-of-header-digest) bytes. If the AHS_length field is zero, there are no more reads. If the AHS_length field is non-zero, exactly that many more bytes need to be read in a second read. These additional bytes are appended to those obtained in the first read. The header digest will always be the last (length-of-header-digest) bytes. 4. If the DATA-length field in the BHS is non-zero, all the data immediately follows the header digest. If data digests are in use, a data digest immediately follows the data. 5. To save space within the BHS, the AHS-length field can be restricted to a single byte, and can be in units of 4-byte words. This allows up to 1020 bytes of AHSs to follow a BHS, which seems to be more than enough for the uses foreseen (so far, only extended CDB and bidi-read info). Currently, byte 2 is unused (reserved) in all PDU types except Login and Login response, where byte 6 is unused. Therefore, the AHS-length field can be added without increasing the BHS size of 48 bytes. 6. Each AHS should start with a word containing a TYPE field and a LENGTH field. The TYPE field should be enumerated rather than bit-field encoded, for easier decoding and future expansion. The LENGTH field is the number of bytes of additional information in this AHS that follows this word. 7. The DATA_length field should be a 4-byte field added to the 44-byte BHS of the current section 2.2.4 to give a new BHS of 48-bytes. However, since the version 5 44-byte header is always preceded by a 4-byte WN Next-Qualifier that would no longer be needed, there is no net change in the effective BHS size of 48-bytes. Advantages of this proposal over the current WN Next-Qualifer of version 5. 1. Because the receiver gets the AHS-length field from the BHS, it can obtain the entire set of ALL AHSs that follow the BHS in a single "read" operation (the current WN Next-Qualifier scheme requires a separate "read" for EACH additional header segment after the BHS). 2. By limiting the total size of all AHSs to 1020 bytes, a receiver can preallocate a fixed-length buffer of (48 + 1020 + size-of-header_digest) bytes for header processing (the current WN Next-Qualifier scheme has no limit on either total header size nor individual AHS sizes). 3. Since each AHS begins with a word containing the TYPE and LENGTH in fixed positions, an unknown TYPE (i.e., a type introduced in the future that is received by a legacy receiver) does NOT result in loss of synchronization during header processing -- the receiver knows the length of this AHS in any case, and can just skip over it (the current WN Next-Qualifier type determines how the length field is to be interpreted -- there is no "general rule", so unknown types mean loss of synchronization). 4. The format of the AHS is the familiar Type/Length/Value (T/L/V) structure (the current WN Next-Qualifier is T+1/L+1/V, but this does not seem to provide any benefit and only adds confusion and complexity -- see e-mails from David Black and Barry Reinhold). 5. The AHS TYPE is enumerated (the current WN Next-Qualifier is bit encoded, again without seeming to provide any benefit -- see e-mail from David Black). 6. Because the BHS always contains a 4-byte DATA_length field, the maximum data segment size is 4 Gigabytes (the current WN Next-Qualifier scheme, with the long data header removed, limits the data segment size to 16 Megabytes). 7. The "Header Digest Present" and "Data Digest Present" bits have been completely eliminated (the purpose of these in the current WN Next-Qualifier scheme was never explained, but they appear to be useless -- the presence or absence of digests has to be negotiated, and once negotiated, all PDUs must obey the negotiated decision). I see no disadvantages of this proposal relative to the current WN scheme. Since there is only one header digest, both schemes suffer the disadvantage of using unreliable data to find the header digest, which can lead to unnecessary blocking on "reads". Proposed iSCSI PDU Format +------------------------+ | required BHS | > fixed length of 48 bytes +------------------------+ | optional AHS 1 |\ | - - - - - - - - - - - | \ | optional AHS 2 | \ | - - - - - - - - - - - | > total length in AHS_length field in BHS | . . . . | / | - - - - - - - - - - - | / | optional AHS n |/ +------------------------+ | optional header digest | -- covers preceding (48 + AHS_length) bytes +------------------------+ | |\ | optional data | > total length in DATA_length field in BHS | |/ +------------------------+ | optional data digest | -- covers preceding (DATA_length) bytes +------------------------+ Thanks, Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 On Tue, 13 Mar 2001 julian_satran@il.ibm.com wrote: > > > Bob, > > Interesting. This is close to one of the variants I had for IETF-50 (a > clear header). > The only adavantage it has is that you are able to read all the AHSs in one > read. > The (admitedly academic) disadvantage it has is that you are limited in the > size of extensions and have some redundancy. > The basic issue that was raised - and there is no simple way out - is that > once you have lost a block (BHS in your suggested layout) you are out of > synch. There is no simple way around it (at least not one that can be > solved by changing layout) and an added digest (only the code or silicon to > account for it) is IMHO not warranted by what you gain. If you are willing > to spend silicon or code on this the making header failures less probable > and recovering if you fail by dropping the connection is a better bet > (using the redundancy for a coding gain). But this involves some > complexity too. > > Julo > From owner-ips@ece.cmu.edu Thu Mar 15 19:23:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA10935 for ; Thu, 15 Mar 2001 19:23:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2FM43q10027 for ips-outgoing; Thu, 15 Mar 2001 17:04:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FKCXr01900 for ; Thu, 15 Mar 2001 15:12:33 -0500 (EST) Received: from ypportable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id GQ9BKC1N; Thu, 15 Mar 2001 12:12:28 -0800 From: "Y P Cheng" To: Subject: RE: iSCSI: Need to Kill Session from Surviving Node Date: Thu, 15 Mar 2001 12:07:52 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > Wait a minute, I did not expect my Surviving Node question to somehow > trigger the discussion of whether the Target Reset should be issued by > iSCSI Initiators. That is a completely different Pit. > > My question was, when the Storage Controller fails over to a surviving > NODE, how do we tell the Initiator that is waiting peacefully for the > TCP/IP connection to time-out. To quickly give-up and let the initiator's > SCSI layer, retry the operations, so that the iSCSI layer can start either > sending the SCSI cmds to the Surviving Node, or starts a new Session with > the Surviving Node. > > This has nothing to do with the Initiator issuing Target Resets. > . > John L. Hufferd John, I think the FC-PLDA is a very good document defining the recovery behavior of the SCSI initiator and target for fibre channel. In my experience of designing SCSI adapters, we never tried to do failover on the target side and then inform the initiator to go to a different connection. This is because no matter how hard a target tries, it can't detect a missing command lost in transit. Ultimately, the failover must be started by the initiator. I don't agree on optimizing a recovery by speeding up the restart because if connection failure happens often we have something much bigger to worry about than how fast we do failover. If a connection fails, the initiator on its own must detect the failure by either time out or the frequency of failure exceeding certain threshold. If a target could inform the initiator about the troubled connection using another connection, I am not sure it would worth the added complexity. After all, if the target knows about the problem, the initiator would know equally fast as well. Check out the FC-PLDA and you will see what I meant. Just my two cents. Y.P. From owner-ips@ece.cmu.edu Thu Mar 15 22:37:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA14574 for ; Thu, 15 Mar 2001 22:37:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2G1l8025198 for ips-outgoing; Thu, 15 Mar 2001 20:47:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com (42s3082.isus.emc.com [168.159.211.82] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2G1kcr25144 for ; Thu, 15 Mar 2001 20:46:38 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 15 Mar 2001 20:47:54 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07080152B3@corpmx9.isus.emc.com> From: Black_David@emc.com To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iscsi: rev 05 2.5.1 requires ACA support Date: Thu, 15 Mar 2001 20:46:29 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > After an error if you have commands in flight you want them all dropped > until you specifically reset the ACA and restart the queue (prevent things > to be executed out of order). The T10 folks will have to go after this one in more detail, but Julian's above statement is correct *only* if the commands in flight depend on the one that caused the error (i.e. executing them out of order will cause problems). This is generally not the case for disks where the usual practice is to enforce command execution order dependencies (e.g., database write ordering) in the operating system and applications by waiting for responses (yes it's possible to do better, but lots of existing software doesn't). The result is that commands in flight can be executed in arbitrary order with arbitrary ones of them failing without causing further difficulties. As Ralph has pointed out, most hosts do not use ACA for disk-based storage, and if iSCSI always does ACA, this will cause nasty integration issues. Just to stir the pot further, I believe Fibre Channel provides a negative example, because if FC drops a frame (which is not a good idea, but can still happen), the FC component that dropped the frame has no clue about what ACA is, or how to get the target (which is not the same piece of hardware) to enter ACA state. Both Class 2 and Class 3 are vulnerable to this. Tapes are another matter - do we still have a tape expert on the list? I thought where we were headed on ACA was something along the lines of: - Targets MUST implement. - Initiators MAY use. - Initiator support for ACA is NOT REQUIRED. which would imply a text key for Initiators to tell Targets whether ACA behavior is expected. Did I miss something? --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Thu Mar 15 22:37:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA14598 for ; Thu, 15 Mar 2001 22:37:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2G1Z6024353 for ips-outgoing; Thu, 15 Mar 2001 20:35:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2FKCXr01900 for ; Thu, 15 Mar 2001 15:12:33 -0500 (EST) Received: from ypportable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id GQ9BKC1N; Thu, 15 Mar 2001 12:12:28 -0800 From: "Y P Cheng" To: Subject: RE: iSCSI: Need to Kill Session from Surviving Node Date: Thu, 15 Mar 2001 12:07:52 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > Wait a minute, I did not expect my Surviving Node question to somehow > trigger the discussion of whether the Target Reset should be issued by > iSCSI Initiators. That is a completely different Pit. > > My question was, when the Storage Controller fails over to a surviving > NODE, how do we tell the Initiator that is waiting peacefully for the > TCP/IP connection to time-out. To quickly give-up and let the initiator's > SCSI layer, retry the operations, so that the iSCSI layer can start either > sending the SCSI cmds to the Surviving Node, or starts a new Session with > the Surviving Node. > > This has nothing to do with the Initiator issuing Target Resets. > . > John L. Hufferd John, I think the FC-PLDA is a very good document defining the recovery behavior of the SCSI initiator and target for fibre channel. In my experience of designing SCSI adapters, we never tried to do failover on the target side and then inform the initiator to go to a different connection. This is because no matter how hard a target tries, it can't detect a missing command lost in transit. Ultimately, the failover must be started by the initiator. I don't agree on optimizing a recovery by speeding up the restart because if connection failure happens often we have something much bigger to worry about than how fast we do failover. If a connection fails, the initiator on its own must detect the failure by either time out or the frequency of failure exceeding certain threshold. If a target could inform the initiator about the troubled connection using another connection, I am not sure it would worth the added complexity. After all, if the target knows about the problem, the initiator would know equally fast as well. Check out the FC-PLDA and you will see what I meant. Just my two cents. Y.P. From owner-ips@ece.cmu.edu Thu Mar 15 22:40:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA14675 for ; Thu, 15 Mar 2001 22:40:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2G1oAp25384 for ips-outgoing; Thu, 15 Mar 2001 20:50:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2G1nKr25350 for ; Thu, 15 Mar 2001 20:49:20 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 74C82506 for ; Thu, 15 Mar 2001 17:49:19 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id RAA15784; Thu, 15 Mar 2001 17:49:14 -0800 (PST) From: Santosh Rao Message-Id: <200103160149.RAA15784@hpcuhe.cup.hp.com> Subject: iSCSI : Rev 05 Review Comments To: ips@ece.cmu.edu (ips) Date: Thu, 15 Mar 2001 17:49:13 -0800 (PST) Cc: alok_gupta@hp.com (Alok Gupta), santoshr@hpcuhe.cup.hp.com (Santosh Rao) X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian & All, Enclosed is some review comments on rev 05 of the iSCSI draft. Regards, Santosh Comments on iSCSI Rev 05 : ========================== 1) Section 2.10.7, pg 52. Login ------------------------------- "Is significant only if TSID is zero and indicates the starting Command Sequence Number for this session; it SHOULD be zero for all other instances." The above SHOULD to be replaced by MUST. 2) Section 2.11, pg 55. Login Response -------------------------------------- Section 2.11.5 refers to the use of a 0 value for TSID to indicate Login Reject. This can be removed since a status of 0206 indicates the same. Also, are targets expected to populate all the fields of a login response when rejecting a login (i.e. non-0 status class) ? Needs to be called out explicitly one way or the other. 3) Section 2.12.4, pg 58. PingMaxReplyLength -------------------------------------------- "Ping data is reflected in the Ping Response. Note that the length of the reflected data is limited by a negotiated parameter and the initiator SHOULD avoid sending more than the negotiated limit. " Replace SHOULD by MUST. 4) Section 2.6, pg 41. SCSI Task Mgmt Response ---------------------------------------------- The Response Codes in the SCSI Task Mgmt Response are insufficient. The following additional response codes should be considered for addition to the existing list : a) Command Not Supported (ex : target does not support LUN Reset) b) Invalid LUN (ex : LUN Reset sent to a non-existent LUN) c) Busy (ex : Task Mgmt command cannot be executed since the task manager is busy.) 5) Section 2.12, pg 57. NOP-OUT -------------------------------- When does ExpStatSN can take a value of 0. Needs to be called out in the spec. 6) Section 2.13, pg 59. NOP-IN requires iSCSI to be LUN aware. --------------------------------------------------------------- In the NOP-IN PDU, when the P bit is 1, the spec requires the LUN field to contain a valid LUN (should not be 0 ?). This requires LUN knowledge at the iSCSI layer, since NOP-IN is generated at the iSCSI layer which is LUN-list unaware. The LUN field is not required for the NOP-IN PDU and should be removed. (As a side note, the figure states LUN or reserved, while the description states the field MUST contain a valid LUN.) 7) Section 2.12, pg 60. StatSN not required in NOP-IN. ------------------------------------------------------- "Whenever the NOP-In is not issued in response to a NOP-Out the StatSN field will contain as usual the next StatSN but StatSN for this connection is not advanced." Why is StatSN required in the NOP-IN PDU ? The above is adding extra code paths in StatSN assignment in the performance path.(conditional StatSN assignment based on opcode.). It is un-necessary complexity. 8) Section 2.16, pg 64. SACK should be included in "connection allegiance" --------------------------------------------------------------------------- The "connection allegiance" definition should be expanded to include the SACK in order to avoid spanning I/O state information across connections.(Needs to be explicitly stated in the spec.) 9) Section 2.16. Initiator Task Tag in SACK PDU. ------------------------------------------------- The spec needs to document when I.T.T can be reserved in a SACK. (A general comment is that all fields in all PDUs need describing and should be described in the order of the PDU layout.) 10) The layout of the Additional Runs for a SACK should be described in Section 2.16. 11) Per the spec, "The SACK also implicitly acknowledges data or status PDUs." How is this ? The SACK is meant as a request to re-transmit a missing PDU. How is it an implicit acknowledgement (?) 12) Section 2.17, pg 66, 67. Protocol rule regarding Target Transfer Tag. ------------------------------------------------------------------------- Section 2.17 contradicts itself by stating that "there is no protocol rule about Target Transfer Tag" and then, stating : "All outstanding R2Ts should have different Target Transfer Tags". The usage of the Target Transfer Tags is device dependent and the draft need not elaborate on this. 13) Section 2.18.2., pg 69 Asynch Message SCSI Event. ----------------------------------------------------- The draft refers to a non-existent "Length parameter" as : "The Length Parameter is set to the length of the sense data." 14) Section 2.20. pg 71. Reject PDU ----------------------------------- The draft must state that a reject PDU is not to be transmitted when the target detects any of the reasons to reject on a NOP-OUT PDU with the Target Transfer Tag set. (i.e. a NOP-OUT from initiator in response to the target having sent a NOP-IN.) 15) Section 6.1. pg 80. Format Errors ------------------------------------- "When an initiator receives an iSCSI PDU with a format error, for which it has an outstanding task, it MUST abort the target task and report the error through an appropriate service response" How can an initiator know which task to abort when it may not be able to extract the Initiator Task Tag from the PDU due to the format error ? Suggest re-wording the above to indicate that the initiator discard a PDU with a format error and recover the I/O through the timeout path. It should specify that on an I/O timeout, the initiator MUST abort the I/O. (As discussed during rev 03 review, a section on Abort handling would be useful.) 16) Section 6.3. pg 81. Sequence Errors --------------------------------------- "As a side effect of receiving the missing responses, the initiator may discover missing data PDUs. The initiator MUST NOT acknowledge (either explicitly through ExpStatRN or implicitly through a status SACK) the received responses until it has completed receiving all the data PDUs of a SCSI command. " The above is not clear. Does this imply it should not acknowledge the status until the I/O is "retried" and all the data PDUs are received ? 17) Section 6.6. pg 82. Session Errors. --------------------------------------- This section is not clear on the recovery action being prescribed. Can the initiator perform a logout with a reason of "closes the session" as session recovery ? 18) Section 6.7.1.1. pg 84. Recovery within a connection. --------------------------------------------------------- "(1)Status/Response not acknowledged for a long time. The target MAY issue a NOP-IN, with the P bit set to 1 or 0, which indicates in the StatSN field the next status number it is going to issue. This helps the initiator detect missing StatSN and issue a SACK-status. " StatSN is not required to be sent in NOP-IN. When the target sends a NOP-IN with P bit set, the initiator sends back ExpStatSN thereby acknowledging Status. 19) Section 1.2.2.1, pg 12 -------------------------- "If immediate delivery is used with task management commands, these commands may reach the SCSI target task management before the tasks they are supposed to act upon. Whenever those effects are undesirable, connection allegiance or ordered delivery MAY be used." The term "connection allegiance" is used above before its definition in Section 1.2.5. Recommend adding a list of acronyms and definitions to the document. 20) Section 2.3.2, pg 33 ------------------------ The use of "SCSI Execute Command" must be additionally qualified with [SAM2] reference or better still, a brief explanation of the SAM2 CRN to iSCSI CmdRN mapping added to this section. -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Thu Mar 15 23:33:26 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA15552 for ; Thu, 15 Mar 2001 23:33:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2G2R7N27732 for ips-outgoing; Thu, 15 Mar 2001 21:27:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2G2Qtr27719 for ; Thu, 15 Mar 2001 21:26:55 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 15 Mar 2001 21:26:46 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07080152B4@corpmx9.isus.emc.com> From: Black_David@emc.com To: hufferd@us.ibm.com, Robert.Elliott@compaq.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI: Need to Kill Session from Surviving Node Date: Thu, 15 Mar 2001 21:26:44 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > What I was trying to get at is --- trying to enable the > Surviving Node (which has no TCP/IP state from the Failing > node) to get the Initiator, which had a previous Session with > the Failing Node to take actions right away, and not wait > for its TCP/IP connections with the Failed Node to Time Out. > (This requires some sort of a message from the Surviving > node back to an Initiator.) This is not quite what we have > defined into iSCSI. Since this is a TCP problem (get the connection to close), iSCSI may not need to do anything ... what about failing over the IP address and responding with ICMP Destination Unreachable to anything that arrives on that failed-over IP address? Implementations of this probably need to support multiple IP addresses on a single network interface, which is not a big deal. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Mar 16 00:53:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA16518 for ; Fri, 16 Mar 2001 00:53:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2G3n9h02695 for ips-outgoing; Thu, 15 Mar 2001 22:49:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2G3m1r02650 for ; Thu, 15 Mar 2001 22:48:01 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GA900D8QUJB6A@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Thu, 15 Mar 2001 19:47:38 -0800 (PST) Date: Thu, 15 Mar 2001 19:45:57 -0800 From: Douglas Otis Subject: RE: iSCSI: Rationale behind the SLP draft In-reply-to: <3AB12E74.3A7E696D@cisco.com> To: Mark Bakke , IPS Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mark, If I have two servers with similar information to promulgate and expectation of multi-casts reaching the target, I may find SLP efforts greater than desired over standard methods found using DHCP and LDAP. DHCP can provide information of the location of the LDAP server as already defined in existing boot specifications. The BOOTP-DHCP multi-cast request already have relay provisions in routers and switches because of its common use and has defined entries for LDAP server discovery. You could have defined an LDAP schema that would have covered both iSNS and SLP without adding additional standards efforts for building yet another protocol that does the same function as protocols already in common use. For an overview see: http://www.umich.edu/~dirsvcs/ldap/doc/guides/slapd/1.html#RTFToC1 Could you explain why LDAP is unsuitable for this application? There are many conventions within LDAP in terms of extensibility that seem to make it more suitable than SLP and iSNS. You will also find security extensions already in place for LDAP merged with BOOTP-DHCP. It seems you have problems to overcome that have already been solved. Not being clever, I tend to ask dumb questions. Doug > David Black has requested that I send something to the > list showing the rationale behind the SLP draft, including > its fit in the big picture and its relationship with iSNS. > Here's an attempt to do so. I believe it to be consistent > with the Naming & Discovery, iSNS, and iSCSI/SLP drafts, > and their IETF-50 presentations. > > The relevant drafts are: > > draft-ietf-ips-iSCSI-name-disc-00 > draft-ietf-ips-isns-00 > draft-bakke-iscsi-slp-00 > > Naming and Discovery Requirements > > These are the discovery and name services requirements > from draft-ietf-ips-iscsi-name-disc-00. From a discovery > requirements point-of-view, SLP and iSNS provide some > similar services. SLP provides very simple discovery services > for smaller networks; iSNS provides more comprehensive > management services for larger networks. Where they overlap, > SLP and iSNS can interoperate fairly easily. > > 1. Multicast method to discover targets, to allow zero- > configuration of initiators and targets. > > SLP provides this. > > 2. A method to look up a WWUI, and get one or more addresses. > "Where is target ?" > > SLP and iSNS both provide this. > > 3. A method to find targets that a given initiator should access. > "I am initiator ; which targets should I access?" > > SLP and iSNS both provide this. > > 4. Ability to limit discovery to relevant initiators and targets. > > iSNS uses Discovery Domain, which are used to limit which initiators > and targets can discover each other. Each target and initiator > belong to Discovery Domain; the iSNS will not allow initiators to > discover targets outside their domain. > > SLP uses a named Scope. Each target and initiator is configured > with one or more scope names; a target will only respond to an > initiator who has a matching scope. Note that this is not a > security mechanism; an initiator can ask for any scope it wishes, > and a target can respond to any scope it wishes. This provides > a simple mechanism to limit discovery, but does not attempt to > provide security. > > 5. Access Lists by initiator WWUI > > SLP and iSNS both provide the ability to store a list of initiator > WWUIs that are allowed access to a given target. > > 6. iSCSI Object Model support > > SLP registers only the targets > iSNS registers targets, initiators, network entities, portals > > 7. TLV Message Format > > SLP and iSNS both do this. However, SLP uses a text key and > value for its attributes; iSNS uses a binary key value. > > 8. Authentication of discovery messages > > iSNS uses SLP's authentication mechanism > > 9. Registration and Query > > SLP registers targets only > iSNS registers targets, inititors, network entities, portals > > 10. State Change Notification > > iSNS provides a state change service for which entities can > register. > > SLP does not provide state change or event notification. There > is a draft on doing some notification, but it is not adequate for > propagating attribute changes, which are necessary for an > initiator to discover that it has gained (or lost) access to > a target, or that a target has added a new address. > > 11. Light-weight Protocol > > Neither protocol should be too much of a burden to implement. > > Adding authentication to the protocol can be done later, and will be > the same effort for both. In the case that both SLP and iSNS are > supported in the same product, they can share the code for this > feature. > > 12. Compatibility with Boot Requirements > > Both the SLP template and the iSNS specification have worked to > ensure that they fit in with the boot requirements. > > SLP has defined DHCP options that allow it to locate a directory > agent, avoiding multicast traffic from initiators in a DHCP > environment. > > 13. Compatibility with LDAP > > Although this is not a requirement, we have seen the desire for > interworking with LDAP discussed on the list enough that it may > merit a requirement at some point. Both SLP and iSNS are designed > with LDAP compatibility in mind. > > 14. Entity Status Inquiry/Heartbeat > > iSNS provides an ESI/heartbeat ping message to monitor the > availability of initiators and targets. The heartbeat is > originated by the iSNS server. > > We have not defined this capability for SLP. It may be possible > to do something similar if targets advertise themselves with a > shorter lifetime, and initiators expire their discovered targets > appropriately. Adding a directory agent (DA) may be required for > this to work properly. > > 15. Distribution of X.509v3 Public Key certificates > > iSNS provides this capability. > > Our SLP template does not attempt to do this for two reasons: > > - SLP attributes are strings; the binary certificates would have > to be escaped or uuencoded. > > - iSCSI only registers targets; there is no way for targets > to discover an initiator's certificate. > > 16. Usage of Multicast > > SLP normally relies on multicast, for initiators to request > advertisements from targets. Multicast use can be eliminated > at the expense of adding a DA, and configuring its address on > each of the initiators and targets. > > iSNS does not use multicast. > > > We had originally started out looking for something to fulfill the > multicast requirements that iSNS did not do. However, we found that > if one already had SLP, it was just as easy to find individual > targets, or at least canonical targets using it as it was to locate > an iSNS service. SLP can then be used to handle discovery for > simpler host and device environments, and iSNS could be added later > to help manage these environments. > > SLP and iSNS both fulfill many of the requirements from > the naming & discovery draft. However, they don't completely > overlap, and they take two different approaches to discovery. > > SLP provides Discovery of Targets. > > SLP is used for discovery ONLY. It uses a distributed > model, that can start with initiators and targets, and > no directory servers. It can add Directory Agents to > scale to medium-sized environments and reduce or eliminate > the use of multicast. Some work is underway on mesh- > enhanced SLP (mSLP), which provides peer-to-peer information > exchange between DAs for larger environments. > > From a pure discovery point of view, SLP's chief weakness > is that it does not yet have a notification mechanism. > > - Discovery of iSCSI targets > - Discovery of iSNS services > - Multicast-optional discovery mechanism for targets and > other name services. > - Open source implementations > - Existing standard protocols > > iSNS provides Centralized Management of Initiators and Targets. > > In addition to target discovery, iSNS provides higher-level > functions such as centralized access management, active > monitoring, public-key certificate management, and state-change > notifications. > > - Discovery of iSCSI targets > - Centralized management of initiator and target access > - Active monitoring of initiators and targets > - State change notification > > Using SLP with iSNS > > Although SLP and iSNS solve several of the same problems, they > are fairly well-suited to interoperating. They share a common > authentication structure, which is not something one would want > to code twice. SLP can be used to do basic discovery where > configuring an iSNS server is not warranted, and can help > discover iSNS servers in environments where centralized management > is needed. It is relatively simple for an iSNS server to > provide a proxy SLP service agent on behalf of the targets it > manages, allowing initiators to participate that implement only > the basic SLP. Similarly, an iSNS server can work with gateways > to provide its services for targets that support only SLP. > > When initiators and targets support BOTH SLP and iSNS, there > are a few rules that should be followed: > > 1. Initiators can use both SLP and iSNS concurrently to discover > their targets. In fact, an initiator should be able to use > more than one iSNS, if it is accessing storage from two > different providers. This allows an initiator to discover > its locally-managed devices using SLP, and its centrally- > managed devices using iSNS. > > 2. Targets should be advertised using a single discovery method. > A target should be advertised either with SLP, or with a > single iSNS environment. Targets discovered multiple ways > would probably not break anything, but a target discovered > via multiple services could produce conflicting information > to the initiator. > > 3. Gateways or iSCSI proxies can be used to provide local SLP > discovery of targets that are managed using iSNS. > > Implementation Approach > > I recommend a three-phase approach to implementing iSCSI > naming and discovery. The first phase is to simply support the > WWUIs, and their use with the iSCSI protocol, and allow static > configuration of hosts and targets. The second is to support > SLP for simple discovery. This should be relatively quick to > implement, as the source for SLP is readily available. The third > would be to support iSNS as a storage management capability. > > 1. Simple naming and configuration > > - Admins configure targets with some form of access list, which > may include the initiator WWUIs that are supposed to access > them. > > - Admins configure initiators with the canonical target "iscsi" > for targets that may provide them services. > > - Initiators use the SendTargets text command to discovery their > targets. This eliminates configuring target WWUIs in the > initiator. > > 2. SLP for multicast and simple discovery > > - Instead of configuring initiators with target addresses, admins > enable SLP on the initiator. > > - Targets also enable SLP, and are discovered by the initiators. > > 3. iSNS for centralized management > > - Initiators and targets are configured to be managed by an iSNS > service. > > - Admins configure both hosts and targets from the iSNS server. > > - Initiators and targets use the iSNS protcol to discover each other. > > Other Discovery Protocols > > There are several other discovery protocols; each has its > strengths and weaknesses. Here are some of them. Take a look > at "Building Networks on the Fly", IEEE Spectrum, March 2001 > for more information on these. > > 1. DNS SRV Records - these seemed too limiting, and there did > not seem to be much point in making DNS do more. The SLP > folks started their work because DNS SRV records didn't meet > their requirements. > > 2. LDAP, with an appropriate schema defined, could handle iSCSI > registrations and queries, and distribute essentially the > same information as SLP. There are three reasons we didn't > go directly to LDAP: > > 1. LDAP would require a heavier implementation than SLP > 2. LDAP would not solve the multicast requirements > 3. SLP is built to be compatible with LDAP; SLP can be used > by iniators and targets, with LDAP as a back-end database. > This is left as an exercise for the implementor. > > 3. Jini is controlled by a single company, and is tied to a > particular programming language, which is not appropriate > for an internet standard. > > 4. UPnP from Microsoft is based on XML data with an HTTP transport. > XML has a lot of merit for application interfaces like these, but > the code for XML and HTTP could be a bit heavy for a really simple > device. It's company-controlled anyway, so we can't use it. > > 5. Salutation - Have not explored this in depth. Salutation has > been around longer than the other protocols. It is made to be > more flexible; it does not assume IP. > > 6. Bluetooth, HAVi - These are done by controlled-membership > organizations, and are designed for specific non-IP environments. > > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 > From owner-ips@ece.cmu.edu Fri Mar 16 01:34:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA19968 for ; Fri, 16 Mar 2001 01:34:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2G4gHq05894 for ips-outgoing; Thu, 15 Mar 2001 23:42:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2G4gDr05888 for ; Thu, 15 Mar 2001 23:42:13 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GA900JWQX0R0J@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Thu, 15 Mar 2001 20:41:17 -0800 (PST) Date: Thu, 15 Mar 2001 20:39:31 -0800 From: Douglas Otis Subject: RE: iscsi: rev 05 2.5.1 requires ACA support In-reply-to: <0F31E5C394DAD311B60C00E029101A07080152B3@corpmx9.isus.emc.com> To: Black_David@emc.com, julian_satran@il.ibm.com, ips@ece.cmu.edu Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, In the good old days Tape did not offer ACA as a solution as it did not queue commands, it depended on buffering. Oddly neither do drives use ACA for the reason David mentions. Tape does use relative addressing within linked commands where these are defined as being sent over multiple connections in iSCSI (ye ha). We have run in circles about placing ACA into a tape application that never expected to see ACA with Ralph being the proponent of bit fiddling ACA in and yes it can be done as now required for those devices bridged over FC as someone pointed out. I thought the simple solution would be to not return Sense data within the AutoSense requirement of iSCSI. This would force the Tape application to do the right thing and request Sense specifically upon seeing bad status. Handing commands received pending resolution of an error relies on ACA so we seem suck with it. Should latency become a greater problem when assuring sequence at the initiator, there may be a future where a more stringent view would be desired and ACA is the option that is intended to provide this mechanism. Who knows, some day someone may actually use ACA. Doug > > After an error if you have commands in flight you want them all dropped > > until you specifically reset the ACA and restart the queue > (prevent things > > to be executed out of order). > > The T10 folks will have to go after this one in more detail, but Julian's > above statement is correct *only* if the commands in flight depend on > the one that caused the error (i.e. executing them out of order will cause > problems). This is generally not the case for disks where the usual > practice is to enforce command execution order dependencies > (e.g., database write ordering) in the operating system and applications > by waiting for responses (yes it's possible to do better, but lots of > existing software doesn't). The result is that commands in > flight can be executed in arbitrary order with arbitrary ones of them > failing without causing further difficulties. As Ralph has pointed out, > most hosts do not use ACA for disk-based storage, and if iSCSI > always does ACA, this will cause nasty integration issues. > > Just to stir the pot further, I believe Fibre Channel provides a negative > example, because if FC drops a frame (which is not a good idea, > but can still happen), the FC component that dropped the frame has no > clue about what ACA is, or how to get the target (which is not the > same piece of hardware) to enter ACA state. Both Class 2 and Class 3 > are vulnerable to this. > > Tapes are another matter - do we still have a tape expert on the list? > > I thought where we were headed on ACA was something along the > lines of: > - Targets MUST implement. > - Initiators MAY use. > - Initiator support for ACA is NOT REQUIRED. > which would imply a text key for Initiators to tell Targets > whether ACA behavior is expected. Did I miss something? > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Fri Mar 16 01:36:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA20262 for ; Fri, 16 Mar 2001 01:36:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2G4kFR06137 for ips-outgoing; Thu, 15 Mar 2001 23:46:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2G4jvr06119 for ; Thu, 15 Mar 2001 23:45:57 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GA900KCGX55CI@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Thu, 15 Mar 2001 20:43:59 -0800 (PST) Date: Thu, 15 Mar 2001 20:42:09 -0800 From: Douglas Otis Subject: RE: iSCSI: Need to Kill Session from Surviving Node In-reply-to: <0F31E5C394DAD311B60C00E029101A07080152B4@corpmx9.isus.emc.com> To: Black_David@emc.com, hufferd@us.ibm.com, Robert.Elliott@compaq.com Cc: ips@ece.cmu.edu Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, That would seem to make the problem worse. Due to a single digest error, you dump the entire lot connected to an IP? Doug > > What I was trying to get at is --- trying to enable the > > Surviving Node (which has no TCP/IP state from the Failing > > node) to get the Initiator, which had a previous Session with > > the Failing Node to take actions right away, and not wait > > for its TCP/IP connections with the Failed Node to Time Out. > > (This requires some sort of a message from the Surviving > > node back to an Initiator.) This is not quite what we have > > defined into iSCSI. > > Since this is a TCP problem (get the connection to close), > iSCSI may not need to do anything ... what about failing over > the IP address and responding with ICMP Destination Unreachable > to anything that arrives on that failed-over IP address? > Implementations of this probably need to support multiple > IP addresses on a single network interface, which is not a > big deal. > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Fri Mar 16 03:54:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA00855 for ; Fri, 16 Mar 2001 03:54:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2G6RIW12422 for ips-outgoing; Fri, 16 Mar 2001 01:27:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2G6QZr12358 for ; Fri, 16 Mar 2001 01:26:35 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id BAA60166; Fri, 16 Mar 2001 01:19:14 -0500 Received: from f4n49e (d03nm065h.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id XAA246906; Thu, 15 Mar 2001 23:26:27 -0700 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI: Need to Kill Session from Surviving Node To: Black_David@emc.com Cc: Robert.Elliott@compaq.com, ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Thu, 15 Mar 2001 22:26:10 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/15/2001 11:26:27 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk OK, let me see if I can stretch it a bit more. The key to what you said, is for the Initiator needs to be sending something to the target, not just waiting. So if the Initiator always sent a Nop-Out after some period of time of inactivity from the target, (hopefully a lot smaller time then the TCP/IP Time Out) then the Fail-Over node would be able to cause the failure reasonably quickly by returning ICMP Destination Unreachable. If I have it right, it seems straight forward, but of course you need to take over the IP Address. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com Black_David@emc.com@ece.cmu.edu on 03/15/2001 06:26:44 PM Sent by: owner-ips@ece.cmu.edu To: John Hufferd/San Jose/IBM@IBMUS, Robert.Elliott@compaq.com cc: ips@ece.cmu.edu Subject: RE: iSCSI: Need to Kill Session from Surviving Node > What I was trying to get at is --- trying to enable the > Surviving Node (which has no TCP/IP state from the Failing > node) to get the Initiator, which had a previous Session with > the Failing Node to take actions right away, and not wait > for its TCP/IP connections with the Failed Node to Time Out. > (This requires some sort of a message from the Surviving > node back to an Initiator.) This is not quite what we have > defined into iSCSI. Since this is a TCP problem (get the connection to close), iSCSI may not need to do anything ... what about failing over the IP address and responding with ICMP Destination Unreachable to anything that arrives on that failed-over IP address? Implementations of this probably need to support multiple IP addresses on a single network interface, which is not a big deal. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Mar 16 04:50:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA01191 for ; Fri, 16 Mar 2001 04:50:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2G7YI916169 for ips-outgoing; Fri, 16 Mar 2001 02:34:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2G7Y2r16157 for ; Fri, 16 Mar 2001 02:34:02 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id IAA118512 for ; Fri, 16 Mar 2001 08:33:54 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA148600 for ; Fri, 16 Mar 2001 08:32:39 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A11.00295E38 ; Fri, 16 Mar 2001 08:31:50 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 16 Mar 2001 09:34:18 +0200 Subject: RE: iscsi: rev 05 2.5.1 requires ACA support Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk We are aware of the support for ACA missing from some drivers. The situation is even exacerbated by the fact that certain exceptions that are not errors per-se will require ACA to be fired to accomodate for commands in flight (like reservations, busy, task-set-full). However actions at the initiator can be fine-tuned with the NACA bit in CDB and the ACA atrribute for the task. Julo Black_David@emc.com on 16/03/2001 03:46:29 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iscsi: rev 05 2.5.1 requires ACA support > After an error if you have commands in flight you want them all dropped > until you specifically reset the ACA and restart the queue (prevent things > to be executed out of order). The T10 folks will have to go after this one in more detail, but Julian's above statement is correct *only* if the commands in flight depend on the one that caused the error (i.e. executing them out of order will cause problems). This is generally not the case for disks where the usual practice is to enforce command execution order dependencies (e.g., database write ordering) in the operating system and applications by waiting for responses (yes it's possible to do better, but lots of existing software doesn't). The result is that commands in flight can be executed in arbitrary order with arbitrary ones of them failing without causing further difficulties. As Ralph has pointed out, most hosts do not use ACA for disk-based storage, and if iSCSI always does ACA, this will cause nasty integration issues. Just to stir the pot further, I believe Fibre Channel provides a negative example, because if FC drops a frame (which is not a good idea, but can still happen), the FC component that dropped the frame has no clue about what ACA is, or how to get the target (which is not the same piece of hardware) to enter ACA state. Both Class 2 and Class 3 are vulnerable to this. Tapes are another matter - do we still have a tape expert on the list? I thought where we were headed on ACA was something along the lines of: - Targets MUST implement. - Initiators MAY use. - Initiator support for ACA is NOT REQUIRED. which would imply a text key for Initiators to tell Targets whether ACA behavior is expected. Did I miss something? --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Mar 16 09:49:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id JAA04465 for ; Fri, 16 Mar 2001 09:49:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GCSMp11630 for ips-outgoing; Fri, 16 Mar 2001 07:28:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zoetermeer.tripace.com (mail.tripace.com [212.136.147.1]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2GCRZr11572 for ; Fri, 16 Mar 2001 07:27:35 -0500 (EST) Received: by ZOETERMEER with Internet Mail Service (5.5.2650.21) id ; Fri, 16 Mar 2001 13:35:35 +0100 Message-ID: <8C59010722BBD31194640050DA6EC69711F28B@ZOETERMEER> From: "Sanjeev Bhagat (TRIPACE/Zoetermeer)" To: "'IPS Reflector'" , "'julian_satran@il.ibm.com'" Subject: iSCSI: Some gramatical/spelling mistakes in iSCSI draft ver 5 Date: Fri, 16 Mar 2001 13:35:34 +0100 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Sec 1.2.2.1 on page 11 The sentence Not covered in this document are *he means by which the SCSI Layer may request immediate delivery for command or by which iSCSI will decide by itself to mark a PDU for immediate delivery I guess he should be changed to the. Section 1.2.5 on page 16 It is assumed that these tags are *be used by the target to tag (alone or in combination with the LUN) the solicited data "be" should be removed. Section 2.4.2 on page 36 If a SCSI device error is detected while data from the initiator is still expected (the commadn PDU did not contain all the data and the target has not received the *a Data PDU with the F bit set before sending the Response PDU "a" should be removed Section 2.4.6 on page 37 One past the largest DataSN in an input (read) data PDU the target has sent for the command. 0 means no data PDUs *where sent. "Where" should be replace with "were" Section 2.4.7 on page 37 One past the largest DataSN in an R2T PDU the target has sent for the command. 0 means no R2T PDUs *where sent. "Where" should be replace with "were" I will report more as I find more.. Regards, Sanjeev Bhagat From owner-ips@ece.cmu.edu Fri Mar 16 11:30:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA06332 for ; Fri, 16 Mar 2001 11:30:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GEMQc18249 for ips-outgoing; Fri, 16 Mar 2001 09:22:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com (42s3082.isus.emc.com [168.159.211.82] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2GEM1r18223 for ; Fri, 16 Mar 2001 09:22:01 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 16 Mar 2001 09:23:18 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07080152B6@corpmx9.isus.emc.com> From: Black_David@emc.com To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iscsi: rev 05 2.5.1 requires ACA support Date: Fri, 16 Mar 2001 09:21:53 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk If that's the case, then the wording that Ralph pointed out needs to be modified to indicate that ACA is used only when appropriate. --David > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Friday, March 16, 2001 2:34 AM > To: ips@ece.cmu.edu > Subject: RE: iscsi: rev 05 2.5.1 requires ACA support > > > > We are aware of the support for ACA missing from some drivers. > The situation is even exacerbated by the fact that certain exceptions that > are not errors per-se > will require ACA to be fired to accomodate for commands in flight (like > reservations, busy, task-set-full). > > However actions at the initiator can be fine-tuned with the NACA bit in > CDB > and the ACA atrribute for the task. > > Julo > > Black_David@emc.com on 16/03/2001 03:46:29 > > Please respond to Black_David@emc.com > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: iscsi: rev 05 2.5.1 requires ACA support > > > > > > After an error if you have commands in flight you want them all dropped > > until you specifically reset the ACA and restart the queue (prevent > things > > to be executed out of order). > > The T10 folks will have to go after this one in more detail, but Julian's > above statement is correct *only* if the commands in flight depend on > the one that caused the error (i.e. executing them out of order will cause > problems). This is generally not the case for disks where the usual > practice is to enforce command execution order dependencies > (e.g., database write ordering) in the operating system and applications > by waiting for responses (yes it's possible to do better, but lots of > existing software doesn't). The result is that commands in > flight can be executed in arbitrary order with arbitrary ones of them > failing without causing further difficulties. As Ralph has pointed out, > most hosts do not use ACA for disk-based storage, and if iSCSI > always does ACA, this will cause nasty integration issues. > > Just to stir the pot further, I believe Fibre Channel provides a negative > example, because if FC drops a frame (which is not a good idea, > but can still happen), the FC component that dropped the frame has no > clue about what ACA is, or how to get the target (which is not the > same piece of hardware) to enter ACA state. Both Class 2 and Class 3 > are vulnerable to this. > > Tapes are another matter - do we still have a tape expert on the list? > > I thought where we were headed on ACA was something along the > lines of: > - Targets MUST implement. > - Initiators MAY use. > - Initiator support for ACA is NOT REQUIRED. > which would imply a text key for Initiators to tell Targets > whether ACA behavior is expected. Did I miss something? > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > > From owner-ips@ece.cmu.edu Fri Mar 16 11:30:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA06347 for ; Fri, 16 Mar 2001 11:30:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GFMOR22262 for ips-outgoing; Fri, 16 Mar 2001 10:22:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lub1028.lss.emc.com ([168.159.39.28]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2GFLrr22238 for ; Fri, 16 Mar 2001 10:21:53 -0500 (EST) Received: from emc.com (IDENT:jhall@localhost.localdomain [127.0.0.1]) by lub1028.lss.emc.com (8.9.3/8.9.3) with ESMTP id LAA28606 for ; Fri, 16 Mar 2001 11:21:49 -0500 Message-Id: <200103161621.LAA28606@lub1028.lss.emc.com> To: ips@ece.cmu.edu Subject: iSCSI: too much recovery? Date: Fri, 16 Mar 2001 11:21:48 -0500 From: "Jon Hall" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi, I have some questions... 1) Why is there a header digest, is the TCP cksum not sufficient for iSCSI headers? 2) Why is the StatSN per connection and not per session? Why not adopt the model that once the target sends a StatSN it may release the associated resources and need not retain that task's state? If the initiator does not see the StatSN it must recover and retry. 3) Similarly, why is there a SACK Request pdu? 4) And, why is there a Logout Response pdu? Why not send a Logout and wait for the peer to close the connection. If the connection does not close send another Logout and close the connection. Implicit is an overarching question -- when errors happen will it be possible for iSCSI peers to communicate in the manner that the spec seems to assume? Thanks, Jon From owner-ips@ece.cmu.edu Fri Mar 16 12:16:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA07426 for ; Fri, 16 Mar 2001 12:16:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GFEPX21639 for ips-outgoing; Fri, 16 Mar 2001 10:14:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2GFDnr21615 for ; Fri, 16 Mar 2001 10:13:49 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA51256 for ; Fri, 16 Mar 2001 16:13:40 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA169742 for ; Fri, 16 Mar 2001 16:12:25 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A11.0053764C ; Fri, 16 Mar 2001 16:11:37 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 16 Mar 2001 17:14:06 +0200 Subject: Re: iSCSI: Some gramatical/spelling mistakes in iSCSI draft ver 5 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Thanks, Julo "Sanjeev Bhagat (TRIPACE/Zoetermeer)" on 16/03/2001 14:35:34 Please respond to "Sanjeev Bhagat (TRIPACE/Zoetermeer)" To: "'IPS Reflector'" , Julian Satran/Haifa/IBM@IBMIL cc: Subject: iSCSI: Some gramatical/spelling mistakes in iSCSI draft ver 5 Sec 1.2.2.1 on page 11 The sentence Not covered in this document are *he means by which the SCSI Layer may request immediate delivery for command or by which iSCSI will decide by itself to mark a PDU for immediate delivery I guess he should be changed to the. Section 1.2.5 on page 16 It is assumed that these tags are *be used by the target to tag (alone or in combination with the LUN) the solicited data "be" should be removed. Section 2.4.2 on page 36 If a SCSI device error is detected while data from the initiator is still expected (the commadn PDU did not contain all the data and the target has not received the *a Data PDU with the F bit set before sending the Response PDU "a" should be removed Section 2.4.6 on page 37 One past the largest DataSN in an input (read) data PDU the target has sent for the command. 0 means no data PDUs *where sent. "Where" should be replace with "were" Section 2.4.7 on page 37 One past the largest DataSN in an R2T PDU the target has sent for the command. 0 means no R2T PDUs *where sent. "Where" should be replace with "were" I will report more as I find more.. Regards, Sanjeev Bhagat From owner-ips@ece.cmu.edu Fri Mar 16 12:18:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA07450 for ; Fri, 16 Mar 2001 12:18:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GFCQh21519 for ips-outgoing; Fri, 16 Mar 2001 10:12:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2GFBbr21448 for ; Fri, 16 Mar 2001 10:11:37 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id KAA10167; Fri, 16 Mar 2001 10:00:06 -0500 (EST) Message-ID: <3AB22A73.3DFFF56B@cisco.com> Date: Fri, 16 Mar 2001 09:00:03 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: someshg@yahoo.com CC: Venkat Rangan , IPS , jmuchow@cisco.com Subject: Re: iSCSI MIB Object Model References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt brought up the same thing, about not having LU and LUN info. If we can get a SCSI MIB done quickly enough, that might give us the visibility we would need for these, and perhaps we could take them out. I am hoping to get a strong WG consensus on this next week. -- Mark Somesh Gupta wrote: > > Mark, > > What you propose is what I was thinking of. I would > expect this info to be the most useful part. > > My preference would be to not have per LU and per LUN > info at all, even optional (at the transport level). > Just because it is visible is not a good reason. > > This issue is really something to be addressed at a > higher level, and if they don't have it yet, maybe that > is just fine. > > Regards, > Somesh > > > -----Original Message----- > > From: mbakke@cisco.com [mailto:mbakke@cisco.com] > > Sent: Thursday, March 15, 2001 12:29 PM > > To: someshg@yahoo.com > > Cc: Venkat Rangan; IPS; jmuchow@cisco.com > > Subject: Re: iSCSI MIB Object Model > > > > > > Somesh- > > > > We are looking into making the LU and LUN tables optional, > > which would help in this case. I will bring up the appropriateness > > of including them at IETF next week. > > > > I like your idea of keeping track of the last few LUN errors. > > > > I assume that you would like to see something like: > > > > 1. Each target includes a list of errors; the target would fill > > in something like the timestamp, LUN, and initiator WWUI that > > caused that error last. > > > > 2. Each session includes a list of errors, and does a similar > > thing. > > > > Does this sound reasonable? Do you have other ideas for doing > > this that would simplify finding out what went wrong with LUs > > and LUNs? > > > > -- > > Mark > > > > Somesh Gupta wrote: > > > > > > Mark, > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > Mark Bakke > > > > Sent: Wednesday, March 14, 2001 12:49 PM > > > > To: Venkat Rangan > > > > Cc: IPS; jmuchow@cisco.com > > > > Subject: Re: iSCSI MIB Object Model > > > > > > > > > > > > Venkat- > > > > > > > > You have captured many of our open issues in this > > > > message. Please see my comments below. > > > > > > > > Thanks, > > > > > > > > Mark > > > > > > > > Venkat Rangan wrote: > > > > > > > > > > Mark, > > > > > > > > > > The UML is very nice, and the new object model, covering both > > > > > Initiator and Target instances is also nice. > > > > > > > > > > Overall, we do have some concern over the need for maintaining > > > > per-LU and > > > > > per-LUN statistics. In some iSCSI implementations, the > > iSCSI entity is > > > > > merely an intermediate layer, with no knowledge of LUs and > > LUNs. Placing > > > > > this in the GeneralInfo category and making it mandatory > > would add undue > > > > > burden on these implementations. > > > > > > > > One of our open questions is "how much is too much?". We added the LU > > > > and LUN statistics for a few reasons: > > > > > > > > 1. The iSCSI layer does see the LUN; it is included in the > > SCSI Command > > > > message. > > > > > > > > 2. For implementations that include the actual target and LUs, such > > > > as storage controllers, there is no other way to get information > > > > on LUNs through SNMP, since there is no SCSI MIB. > > > > > > > > 3. We felt that in this case customers might require LU- or LUN-level > > > > accounting. > > > > > > I just did a rough (very rough) calculation for the amount of data > > > required per lu and per lun - It is of the order of 256 bytes per lu > > > and 512 bytes (actually more like 350) per lun. The number of > > > Luns/lus can be in the 10s of thousands. > > > > > > Even though the iSCSI PDU header carries a LUN number, this value > > > is really transparent to the iSCSI layer. The iSCSI layer should > > > not even know how many LUNs there are in the system. The size of > > > the data set, and the cost of managing it are fairly significant. > > > The ability of systems to dynamically add & delete LUNs/LUs should > > > not be tied to the ability of the iSCSI layer to track the stats. > > > > > > I think reasons 1 & 2 are not very good reasons. An analogy would > > > be asking TCP to perpetually keep statistics on every 4-tuple that > > > was ever used to create a connection. > > > > > > I would recommend focusing on the error cases only for one - and > > > perhaps have a table per error type tracking the last "n" errors. > > > What does it matter if an R2T was received for a LUN or not? It > > > just becomes a bunch of probably unneeded information to weed > > > through to get to the real info. > > > > > > > > > > > However, keeping statistics at this level does create some > > work; perhaps > > > > there is a way to make it optional. > > > > > > > > > In Section 5.9: > > > > > "The iscsiLUTable contains a list of iSCSI Logical Units that are > > > > > accessible via the iSCSI target." > > > > > > > > > > Would this list be the same as what the "Report LUNs" command > > > > would give? > > > > > > > > It depends. A LUN is just the address of a logical unit, and in the > > > > general case, is only valid between a particular initiator and a > > > > particular > > > > target. If a second initiator issues a "Report LUNs" to the > > same target, > > > > it may get back a different list if the target is performing > > some sort of > > > > LUN mapping. Some targets will show the same set of LUNs > > > > regardless of the > > > > initiator; some won't. There is no reliable way given the > > current MIB to > > > > find out what "Report LUNs" would return; this is probably > > the domain of > > > > a SCSI MIB. > > > > > > > > > In Section 5.7: > > > > > "The iscsiLunTable contains an entry for each known LUN > > that is being > > > > > accessed using this session. Note that is may not > > contain all of the > > > > > LUNs that could be accessed; the only ones available to > > iSCSI are the > > > > > ones that have been addressed specifically by iSCSI commands. > > > > > Therefore, LUNs that have been discovered via mechanisms > > such as the > > > > > SCSI "report LUNs" will not appear in this table until they are > > > > > specifically addressed by the initiator. > > > > > > > > > > How long do these entries live in the table? If iSCSI sessions > > > > logout and > > > > > close, do these entries continue to remain? > > > > > > > > The current intent is to have entries live only as long as > > the sessions; > > > > these would be removed when the session is removed. This is > > the simplest > > > > solution and would put the smallest burden on the implementation. > > > > > > > > However, we still have an open "requirements" question on > > whether anyone > > > > wants these to hang around longer. > > > > > > > > > Thanks, > > > > > > > > > > Venkat Rangan > > > > > Rhapsody Networks Inc. > > > > > http://www.rhapsodynetworks.com > > > > > > > > > > > > > -- > > > > Mark A. Bakke > > > > Cisco Systems > > > > mbakke@cisco.com > > > > 763.398.1054 > > > > > > Somesh > > > > > > _________________________________________________________ > > > Do You Yahoo!? > > > Get your free @yahoo.com address at http://mail.yahoo.com > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 > > _________________________________________________________ > Do You Yahoo!? > Get your free @yahoo.com address at http://mail.yahoo.com -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Fri Mar 16 12:53:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA07792 for ; Fri, 16 Mar 2001 12:53:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GFXhn23076 for ips-outgoing; Fri, 16 Mar 2001 10:33:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2GFX7r23047 for ; Fri, 16 Mar 2001 10:33:07 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA116458 for ; Fri, 16 Mar 2001 16:32:54 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA42294 for ; Fri, 16 Mar 2001 16:31:39 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A11.0055383C ; Fri, 16 Mar 2001 16:30:49 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 16 Mar 2001 17:33:16 +0200 Subject: RE: iscsi: rev 05 2.5.1 requires ACA support Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, As SCSI can liberally set or reset the bits in CDB and we don't want iSCSI to have to look into it (we managed to avoid this until now) I though that it will serve us well to request ACA support so that we get it at least from all the new drivers. As with many of other tiny inconsistencies we are bound to encounter this stems in part from the fact that SAM considers the specific protocol (FCP, SBP etc.) as a blending of transport and SCSI (see the many protocol specific parameters and behaviors) without specifically addressing the "separability" issue that the layering purists among us demand. Regards, Julo Black_David@emc.com on 16/03/2001 16:21:53 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iscsi: rev 05 2.5.1 requires ACA support If that's the case, then the wording that Ralph pointed out needs to be modified to indicate that ACA is used only when appropriate. --David > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Friday, March 16, 2001 2:34 AM > To: ips@ece.cmu.edu > Subject: RE: iscsi: rev 05 2.5.1 requires ACA support > > > > We are aware of the support for ACA missing from some drivers. > The situation is even exacerbated by the fact that certain exceptions that > are not errors per-se > will require ACA to be fired to accomodate for commands in flight (like > reservations, busy, task-set-full). > > However actions at the initiator can be fine-tuned with the NACA bit in > CDB > and the ACA atrribute for the task. > > Julo > > Black_David@emc.com on 16/03/2001 03:46:29 > > Please respond to Black_David@emc.com > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: iscsi: rev 05 2.5.1 requires ACA support > > > > > > After an error if you have commands in flight you want them all dropped > > until you specifically reset the ACA and restart the queue (prevent > things > > to be executed out of order). > > The T10 folks will have to go after this one in more detail, but Julian's > above statement is correct *only* if the commands in flight depend on > the one that caused the error (i.e. executing them out of order will cause > problems). This is generally not the case for disks where the usual > practice is to enforce command execution order dependencies > (e.g., database write ordering) in the operating system and applications > by waiting for responses (yes it's possible to do better, but lots of > existing software doesn't). The result is that commands in > flight can be executed in arbitrary order with arbitrary ones of them > failing without causing further difficulties. As Ralph has pointed out, > most hosts do not use ACA for disk-based storage, and if iSCSI > always does ACA, this will cause nasty integration issues. > > Just to stir the pot further, I believe Fibre Channel provides a negative > example, because if FC drops a frame (which is not a good idea, > but can still happen), the FC component that dropped the frame has no > clue about what ACA is, or how to get the target (which is not the > same piece of hardware) to enter ACA state. Both Class 2 and Class 3 > are vulnerable to this. > > Tapes are another matter - do we still have a tape expert on the list? > > I thought where we were headed on ACA was something along the > lines of: > - Targets MUST implement. > - Initiators MAY use. > - Initiator support for ACA is NOT REQUIRED. > which would imply a text key for Initiators to tell Targets > whether ACA behavior is expected. Did I miss something? > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > > From owner-ips@ece.cmu.edu Fri Mar 16 12:54:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA07815 for ; Fri, 16 Mar 2001 12:54:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GErSq20308 for ips-outgoing; Fri, 16 Mar 2001 09:53:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2GEqur20283 for ; Fri, 16 Mar 2001 09:52:56 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id JAA02104; Fri, 16 Mar 2001 09:52:43 -0500 (EST) Message-ID: <3AB228B8.E22CEC1B@cisco.com> Date: Fri, 16 Mar 2001 08:52:40 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Matt Wakeley CC: ips@ece.cmu.edu, Marjorie Krueger , Dave Peterson Subject: Re: iSCSI MIB Object Model References: <3AB1333F.FB2E8F70@agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt- LUs are not visible in iSCSI, but LUNs are visible within an iSCSI session. Once the session is gone, the LUN is gone, too. Since the information was avaiable, we wanted to see what value it might add. Matt- If we get a SCSI MIB going, perhaps we can avoid at least some of this. Without a SCSI MIB, we still have customer requirements to see LUs through a MIB, and if we had to break layers to do it, ... Somesh's proposal would give users the information they need to determine if access to a particular LUN is causing trouble, without keeping track of individual LUNs. Marjorie Krueger (HP) has been looking into doing a SCSI MIB draft, starting with object models derived from iSCSI. There are several other members of the MIB team that are interested in working on this as well. One of our open issues for next week is whether to pursue a SCSI MIB, and if so, how to do it within T10. Dave Peterson will bring it up with T10 as well. At any rate, I hope to get some clear direction from the IPS WG in general on this by next week. Most of the comments from the WG so far support minimizing or eliminating the LU and LUN information from the iSCSI MIB. -- Mark Matt Wakeley wrote: > > I agree that LUNs and LUs are (should be) transparent to iSCSI, and thus such > LU/LUN specific information should not be included in the MIB. > > Mark, since your company is now a member of T10, perhaps you can repackage the > information as a SCSI MIB draft? > > -Matt Wakeley > > julian_satran@il.ibm.com wrote: > > > > Mark, > > > > iSCSI uses LUN almost transparently. Also the LUN may change while a > > session is active. > > I am not sure that the fact that there is no SCSI MIB (yet) doe justify > > including those items in the iSCSI MIB. > > > > I also recall (a rumor?) that somebody is working on a SCSI MIB (it will be > > needed anyhow there is more to SCSI than the LU). > > > > Julo > > > > Mark Bakke on 14/03/2001 22:48:47 > > > > Please respond to Mark Bakke > > > > To: Venkat Rangan > > cc: IPS , jmuchow@cisco.com > > Subject: Re: iSCSI MIB Object Model > > > > Venkat- > > > > You have captured many of our open issues in this > > message. Please see my comments below. > > > > Thanks, > > > > Mark > > > > Venkat Rangan wrote: > > > > > > Mark, > > > > > > The UML is very nice, and the new object model, covering both > > > Initiator and Target instances is also nice. > > > > > > Overall, we do have some concern over the need for maintaining per-LU and > > > per-LUN statistics. In some iSCSI implementations, the iSCSI entity is > > > merely an intermediate layer, with no knowledge of LUs and LUNs. Placing > > > this in the GeneralInfo category and making it mandatory would add undue > > > burden on these implementations. > > > > One of our open questions is "how much is too much?". We added the LU > > and LUN statistics for a few reasons: > > > > 1. The iSCSI layer does see the LUN; it is included in the SCSI Command > > message. > > > > 2. For implementations that include the actual target and LUs, such > > as storage controllers, there is no other way to get information > > on LUNs through SNMP, since there is no SCSI MIB. > > > > 3. We felt that in this case customers might require LU- or LUN-level > > accounting. > > > > However, keeping statistics at this level does create some work; perhaps > > there is a way to make it optional. > > > > > In Section 5.9: > > > "The iscsiLUTable contains a list of iSCSI Logical Units that are > > > accessible via the iSCSI target." > > > > > > Would this list be the same as what the "Report LUNs" command would give? > > > > It depends. A LUN is just the address of a logical unit, and in the > > general case, is only valid between a particular initiator and a particular > > target. If a second initiator issues a "Report LUNs" to the same target, > > it may get back a different list if the target is performing some sort of > > LUN mapping. Some targets will show the same set of LUNs regardless of the > > initiator; some won't. There is no reliable way given the current MIB to > > find out what "Report LUNs" would return; this is probably the domain of > > a SCSI MIB. > > > > > In Section 5.7: > > > "The iscsiLunTable contains an entry for each known LUN that is being > > > accessed using this session. Note that is may not contain all of the > > > LUNs that could be accessed; the only ones available to iSCSI are the > > > ones that have been addressed specifically by iSCSI commands. > > > Therefore, LUNs that have been discovered via mechanisms such as the > > > SCSI "report LUNs" will not appear in this table until they are > > > specifically addressed by the initiator. > > > > > > How long do these entries live in the table? If iSCSI sessions logout and > > > close, do these entries continue to remain? > > > > The current intent is to have entries live only as long as the sessions; > > these would be removed when the session is removed. This is the simplest > > solution and would put the smallest burden on the implementation. > > > > However, we still have an open "requirements" question on whether anyone > > wants these to hang around longer. > > > > > Thanks, > > > > > > Venkat Rangan > > > Rhapsody Networks Inc. > > > http://www.rhapsodynetworks.com > > > > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Fri Mar 16 12:54:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA07827 for ; Fri, 16 Mar 2001 12:54:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GFkRI24035 for ips-outgoing; Fri, 16 Mar 2001 10:46:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2GFkDr24023 for ; Fri, 16 Mar 2001 10:46:13 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAA191036 for ; Fri, 16 Mar 2001 16:46:06 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA110208 for ; Fri, 16 Mar 2001 16:44:50 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A11.00566C7A ; Fri, 16 Mar 2001 16:43:58 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 16 Mar 2001 17:46:27 +0200 Subject: Re: iSCSI: too much recovery? Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Jon, 1)Read digest as CRC. Sometimes you may want a stronger one with authentication but the you will not use CRC. 2) You don't need ordering only simple acknowledgement to free resources. As such per connection is simpler to implement and maintain. You may choose to do what you suggested and recover by retrying but most of this group consider this too drastic. 3)SACK is there to recover lost (digest error) responses or data blocks. 4)Logout response is there to make sure that you had an orderly close (not necessarily a shutdown - e.g., take out an adapter for maintenance) and no other recovery is needed. Recall that it can be sent on a healthy connection for a bad one. Julo "Jon Hall" on 16/03/2001 18:21:48 Please respond to "Jon Hall" To: ips@ece.cmu.edu cc: Subject: iSCSI: too much recovery? Hi, I have some questions... 1) Why is there a header digest, is the TCP cksum not sufficient for iSCSI headers? 2) Why is the StatSN per connection and not per session? Why not adopt the model that once the target sends a StatSN it may release the associated resources and need not retain that task's state? If the initiator does not see the StatSN it must recover and retry. 3) Similarly, why is there a SACK Request pdu? 4) And, why is there a Logout Response pdu? Why not send a Logout and wait for the peer to close the connection. If the connection does not close send another Logout and close the connection. Implicit is an overarching question -- when errors happen will it be possible for iSCSI peers to communicate in the manner that the spec seems to assume? Thanks, Jon From owner-ips@ece.cmu.edu Fri Mar 16 14:32:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA09003 for ; Fri, 16 Mar 2001 14:32:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GHblK02471 for ips-outgoing; Fri, 16 Mar 2001 12:37:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2GHZgr02237 for ; Fri, 16 Mar 2001 12:35:42 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id MAA00344 for ; Fri, 16 Mar 2001 12:35:37 -0500 (EST) Message-ID: <3AB24EE6.A9EE60FE@cisco.com> Date: Fri, 16 Mar 2001 11:35:34 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: IPS Subject: iSCSI: Rationale behind the iSCSI WWUI URN Draft Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David has asked for a rationale behind this draft as well. Basically, the naming & discovery team set out to solve the question "what's in a name?". We produced a set of requirements for naming initiators and targets, which resulted in the world- wide unique identifier (WWUI). During our research on what constitutes good naming requirements, we looked that the requirements that had been previously done in RFC 1737, "Functional Requirements for Uniform Resource Names". These requirements were very similar to ours, and were well-specified. We have been careful to keep separate the concept of a name and an address. A name is an identifier that is location-independent; I can move a target to another address, or have multiple addresses for it, but it is still the same target. An address is also an identifier, but it identifies a particular place, or access point, at which the target may currently be found. When configuring an initiator to talk to a target, either type of identifier could be used. An administrator that has the name (WWUI) of the target can configure the name, and the initiator can use one or more of the discovery procedures outlined in the naming and discovery draft to resolve this name into one or more addresses. An administrator that has the address of the target can configure this address, and the discovery step can be skipped, unless the address has changed. The WWW and internet folks have already defined a similar set of structures. A Uniform Resource Identifier (URI; RFC 1630, 2396) is used to identify something; this identifier can be either a Uniform Resource Name (URN; RFC 2141), or the more familiar Uniform Resource Locator (URL; RFC 1738). These address formats are familiar to users and administrators, are well-defined, and include standard methods for handling non-ASCII characters, comparisons, etc. Since iSCSI implementations require the use and configuration of both names and addresses, it makes sense to allow a user to specify these in terms of normal, everyday strings, using URNs for names, and URLs for addresses. This draft (draft-bakke-issi-wwui-urn-00) is very simple; all it really does is register the string "iscsi" as the top of iSCSI's URN name space, to avoid other protocols or vendors using it and causing confusion. For more information on URIs, URLs, and URNs, please see: http://www.w3.org/Addressing/ Hope this helps, -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Fri Mar 16 14:32:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA09019 for ; Fri, 16 Mar 2001 14:32:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GGjww28440 for ips-outgoing; Fri, 16 Mar 2001 11:45:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lub1028.lss.emc.com ([168.159.39.28]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2GGj4r28335 for ; Fri, 16 Mar 2001 11:45:04 -0500 (EST) Received: from emc.com (IDENT:jhall@localhost.localdomain [127.0.0.1]) by lub1028.lss.emc.com (8.9.3/8.9.3) with ESMTP id MAA28879 for ; Fri, 16 Mar 2001 12:44:59 -0500 Message-Id: <200103161744.MAA28879@lub1028.lss.emc.com> To: ips@ece.cmu.edu Subject: Re: iSCSI: too much recovery? Date: Fri, 16 Mar 2001 12:44:59 -0500 From: "Jon Hall" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, OK, Thanks. But... :-) julian_satran@il.ibm.com writes: >1)Read digest as CRC. Sometimes you may want a stronger one with >authentication but the you will not use CRC. For data sure, why on iSCSI headers? Does not the StatSN mechanism derive from the notion that a header digest/CRC fails? Is it possible that this header digest/CRC is not providing much over and above the TCP cksum. While retry based on StatSN is causing a good bit of complexity in the protocol? >2) You don't need ordering only simple acknowledgement to free resources. >As such per connection is simpler to implement and maintain. You may >choose to do what you suggested >and recover by retrying but most of this group consider this too drastic. I get confused, why is per connection easier than per session? The target's task state must be maintained at the session level. Pushing StatSN into the connection makes per connection state more complicated; it now contains a subset of the session's task state. Stepping back, if my naive model is too drastic, what specifically are the error cases that can be recovered, and is their frequency such that the complexity of all the retained state is warranted? If the error in question is a pdu that passes the TCP cksum but fails an iSCSI header CRC, that may happen but is that going to be a frequent event? Since initiator recover and retry must be implemented in any case... >3)SACK is there to recover lost (digest error) responses or data blocks. Sorry to be dense, but the questions above... >4)Logout response is there to make sure that you had an orderly close (not >necessarily a shutdown - e.g., take out an adapter for maintenance) and no >other recovery is needed. Recall that it can be sent on a healthy >connection for a bad one. Assuming a healthy connection, what additional information is in the Logout Response pdu that is not seen by the Logout sender when the Logout receiver closes the connection? Or, asked another way, if response=1 in the Logout Response pdu, what happens... Thanks, Jon >"Jon Hall" on 16/03/2001 18:21:48 > >Please respond to "Jon Hall" > >To: ips@ece.cmu.edu >cc: >Subject: iSCSI: too much recovery? > > > > >Hi, > >I have some questions... > >1) Why is there a header digest, is the TCP cksum not sufficient > for iSCSI headers? > >2) Why is the StatSN per connection and not per session? Why not > adopt the model that once the target sends a StatSN it may release > the associated resources and need not retain that task's state? > If the initiator does not see the StatSN it must recover and retry. > >3) Similarly, why is there a SACK Request pdu? > >4) And, why is there a Logout Response pdu? Why not send a Logout > and wait for the peer to close the connection. If the connection > does not close send another Logout and close the connection. > >Implicit is an overarching question -- when errors happen will it be >possible for iSCSI peers to communicate in the manner that the spec >seems to assume? > >Thanks, >Jon > From owner-ips@ece.cmu.edu Fri Mar 16 14:34:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA09055 for ; Fri, 16 Mar 2001 14:34:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GHPWo01514 for ips-outgoing; Fri, 16 Mar 2001 12:25:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2GHPBr01487; Fri, 16 Mar 2001 12:25:11 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id MAA54850; Fri, 16 Mar 2001 12:17:54 -0500 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id KAA62766; Fri, 16 Mar 2001 10:25:02 -0700 Subject: RE: iscsi: rev 05 2.5.1 requires ACA support To: Black_David@emc.com Cc: ips@ece.cmu.edu, "Julian Satran" , owner-ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.5 September 22, 2000 Message-ID: From: "Prasenjit Sarkar" Date: Fri, 16 Mar 2001 09:24:59 -0800 X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/16/2001 09:25:02 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi David and Julian, To impart some finality into the discussion, can you summarize for the benefit of the mailing list what is distinctive about iSCSI that ACA must be made mandatory for targets. I'm not trying to start an argument here, but a few words of motivation might help us to understand as to why iSCSI-ACA must be made manadatory (required in most situations) versus optional (implement where needed). Regards, Prasenjit Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose Black_David@em c.com To: Julian Satran/Haifa/IBM@IBMIL, Sent by: ips@ece.cmu.edu owner-ips@ece. cc: cmu.edu Subject: RE: iscsi: rev 05 2.5.1 requires ACA support 03/16/2001 06:21 AM If that's the case, then the wording that Ralph pointed out needs to be modified to indicate that ACA is used only when appropriate. --David > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Friday, March 16, 2001 2:34 AM > To: ips@ece.cmu.edu > Subject: RE: iscsi: rev 05 2.5.1 requires ACA support > > > > We are aware of the support for ACA missing from some drivers. > The situation is even exacerbated by the fact that certain exceptions that > are not errors per-se > will require ACA to be fired to accomodate for commands in flight (like > reservations, busy, task-set-full). > > However actions at the initiator can be fine-tuned with the NACA bit in > CDB > and the ACA atrribute for the task. > > Julo > > Black_David@emc.com on 16/03/2001 03:46:29 > > Please respond to Black_David@emc.com > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: iscsi: rev 05 2.5.1 requires ACA support > > > > > > After an error if you have commands in flight you want them all dropped > > until you specifically reset the ACA and restart the queue (prevent > things > > to be executed out of order). > > The T10 folks will have to go after this one in more detail, but Julian's > above statement is correct *only* if the commands in flight depend on > the one that caused the error (i.e. executing them out of order will cause > problems). This is generally not the case for disks where the usual > practice is to enforce command execution order dependencies > (e.g., database write ordering) in the operating system and applications > by waiting for responses (yes it's possible to do better, but lots of > existing software doesn't). The result is that commands in > flight can be executed in arbitrary order with arbitrary ones of them > failing without causing further difficulties. As Ralph has pointed out, > most hosts do not use ACA for disk-based storage, and if iSCSI > always does ACA, this will cause nasty integration issues. > > Just to stir the pot further, I believe Fibre Channel provides a negative > example, because if FC drops a frame (which is not a good idea, > but can still happen), the FC component that dropped the frame has no > clue about what ACA is, or how to get the target (which is not the > same piece of hardware) to enter ACA state. Both Class 2 and Class 3 > are vulnerable to this. > > Tapes are another matter - do we still have a tape expert on the list? > > I thought where we were headed on ACA was something along the > lines of: > - Targets MUST implement. > - Initiators MAY use. > - Initiator support for ACA is NOT REQUIRED. > which would imply a text key for Initiators to tell Targets > whether ACA behavior is expected. Did I miss something? > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > > From owner-ips@ece.cmu.edu Fri Mar 16 16:58:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA13786 for ; Fri, 16 Mar 2001 16:58:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GJvX913600 for ips-outgoing; Fri, 16 Mar 2001 14:57:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from goliath.cnchost.com (goliath.cnchost.com [207.155.252.47]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2GJutr13564 for ; Fri, 16 Mar 2001 14:56:55 -0500 (EST) Received: from BSNCBINFORD (ts002d10.wic-ks.concentric.net [206.173.163.70]) by goliath.cnchost.com id OAA16632; Fri, 16 Mar 2001 14:56:49 -0500 (EST) [ConcentricHost SMTP Relay 1.10] From: "Charles Binford" To: Subject: RE: iscsi: rev 05 2.5.1 requires ACA support Date: Fri, 16 Mar 2001 13:57:11 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit (sorry this got kind of lengthy - but I do have proposed solution to what I believe are the problems below) Julian, I don't think iSCSI needs to look at the NACA bit in the CDB - I don't think iSCSI should care about ACA at all. In my 10+ years of working with SCSI disk products I've yet to see a host driver that cared about ordering. As David said below, if there is an ordering dependency then the application waits for completion of the first I/O before releasing the second. One gets the best performance from a SCSI block I/O target (from a single disk to a large RAID controller) by issuing queued commands with simple tags and allowing the target to freely re-order the commands internally. When an error happens it doesn't matter what commands are in-flight because they are all independent I/Os. This is the way high performance SCSI disks and disk subsystems work. It would be a mistake (in my opinion) for iSCSI to come in and mandate the extra complexity of ACA to handle a corner case most of the SCSI world it is trying to penetrate doesn't care about. Granted, tape and other sequential devices are a different story. However, I participated heavily in T11 for 2 to 3 years helping develop a error recovery scheme that would allow SCSI tape devices to work with FC. The bottom line, however, was that while we talked about it over and over, no FC tape vendor had a device that supported queuing. If you don't queue, then nothing is in-flight when an error occurs. (streaming is accomplished by caching in the target). After this drug on for a couple of years and the complexities of queuing to sequential devices became evident, a new project was started in T10 to develop a new tape command set that included relative block addresses in the CDB - making the new tape model very similar to disk - each command can be interpreted independently and precise delivery order is not a fundamental requirement. =================== OK, enough of my preaching. Here is my understanding of the issues, and what I believe the solution set should include: Problem Statements: -------------------------- - problem 1) When an initiator's I/Os are cleared by it own actions (e.g. sending a task management function of abort task set) how does it know which of issued commands were aborted and which were still in-flight and had not yet made it to the target. - problem 2) (closely related to 1) How are task management functions delivered - with the immediate delivery option (CmdSN=0) or not? (reference 1.2.2.1, top of page 12) If immediate, then problem 1 is aggravated if multiple connections are used. If not immediate, then we can have a problem with the task management function never being delivered because the I/O we are trying to abort never made it to the target and the target's iSCSI layer wont' pass up the task management function because the CmdSN number is wrong. - problem 3) When an initiators I/Os are cleared by the actions of another initiator (e.g. the other imitator sent clear task set or lun reset) how does it know which of its outstanding commands were cleared and which will finish. Proposed Solutions: ------------------------ + solution to 1 and 2) I propose all task management functions be required to use the following: - immediate delivery option - utilize a new task management function sequence number field (call it TskMgmtSN) - all task management functions shall be sent on all active connections with the same value for TskMgmtSN - the target must receive the same task management function (identified by TskMgmtSN) on all active connections before it acts on it. (Use a timer (TBD) to weed out and close down defective connections.) - the target must send the task management response on all active connections, again identified by the TskMgmtSN assigned by the initiator. + solution to problem 3) Two currently defined mechanism in SCSI already take care of this. Neither are widely used today, but they are already defined. A) initiator may set NACA on all commands and the target will enter ACA when it sends the unit attention that commands were cleared by another initiator. (Note, I see no need for the async event notice.) OR B) enable the newly defined Task Aborted status. This option doesn't give any guarantees about ordering, but as I argued above the vast majority of SCSI devices don't care. Charles Binford Blue Spruce Networks office: (316) 315-0382 / cell: (316) 210-6404 e-fax: (509) 756-4425 -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Friday, March 16, 2001 9:33 AM To: ips@ece.cmu.edu Subject: RE: iscsi: rev 05 2.5.1 requires ACA support David, As SCSI can liberally set or reset the bits in CDB and we don't want iSCSI to have to look into it (we managed to avoid this until now) I though that it will serve us well to request ACA support so that we get it at least from all the new drivers. As with many of other tiny inconsistencies we are bound to encounter this stems in part from the fact that SAM considers the specific protocol (FCP, SBP etc.) as a blending of transport and SCSI (see the many protocol specific parameters and behaviors) without specifically addressing the "separability" issue that the layering purists among us demand. Regards, Julo Black_David@emc.com on 16/03/2001 16:21:53 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iscsi: rev 05 2.5.1 requires ACA support If that's the case, then the wording that Ralph pointed out needs to be modified to indicate that ACA is used only when appropriate. --David > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Friday, March 16, 2001 2:34 AM > To: ips@ece.cmu.edu > Subject: RE: iscsi: rev 05 2.5.1 requires ACA support > > > > We are aware of the support for ACA missing from some drivers. > The situation is even exacerbated by the fact that certain exceptions that > are not errors per-se > will require ACA to be fired to accomodate for commands in flight (like > reservations, busy, task-set-full). > > However actions at the initiator can be fine-tuned with the NACA bit in > CDB > and the ACA atrribute for the task. > > Julo > > Black_David@emc.com on 16/03/2001 03:46:29 > > Please respond to Black_David@emc.com > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: iscsi: rev 05 2.5.1 requires ACA support > > > > > > After an error if you have commands in flight you want them all dropped > > until you specifically reset the ACA and restart the queue (prevent > things > > to be executed out of order). > > The T10 folks will have to go after this one in more detail, but Julian's > above statement is correct *only* if the commands in flight depend on > the one that caused the error (i.e. executing them out of order will cause > problems). This is generally not the case for disks where the usual > practice is to enforce command execution order dependencies > (e.g., database write ordering) in the operating system and applications > by waiting for responses (yes it's possible to do better, but lots of > existing software doesn't). The result is that commands in > flight can be executed in arbitrary order with arbitrary ones of them > failing without causing further difficulties. As Ralph has pointed out, > most hosts do not use ACA for disk-based storage, and if iSCSI > always does ACA, this will cause nasty integration issues. > > Just to stir the pot further, I believe Fibre Channel provides a negative > example, because if FC drops a frame (which is not a good idea, > but can still happen), the FC component that dropped the frame has no > clue about what ACA is, or how to get the target (which is not the > same piece of hardware) to enter ACA state. Both Class 2 and Class 3 > are vulnerable to this. > > Tapes are another matter - do we still have a tape expert on the list? > > I thought where we were headed on ACA was something along the > lines of: > - Targets MUST implement. > - Initiators MAY use. > - Initiator support for ACA is NOT REQUIRED. > which would imply a text key for Initiators to tell Targets > whether ACA behavior is expected. Did I miss something? > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > > From owner-ips@ece.cmu.edu Fri Mar 16 16:58:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA13813 for ; Fri, 16 Mar 2001 16:58:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GJgWY12425 for ips-outgoing; Fri, 16 Mar 2001 14:42:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dirty.research.bell-labs.com (dirty.research.bell-labs.com [204.178.16.6]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2GJg4r12400 for ; Fri, 16 Mar 2001 14:42:04 -0500 (EST) Received: from grubby.research.bell-labs.com ([135.104.2.9]) by dirty; Fri Mar 16 14:41:13 EST 2001 Received: from aura.research.bell-labs.com ([135.104.46.10]) by grubby; Fri Mar 16 14:41:13 EST 2001 Received: (from jkf@localhost) by aura.research.bell-labs.com (8.9.1/8.9.1) id OAA20106 for ips@ece.cmu.edu; Fri, 16 Mar 2001 14:41:12 -0500 (EST) Date: Fri, 16 Mar 2001 14:41:12 -0500 (EST) From: Jeff Fellin Message-Id: <200103161941.OAA20106@aura.research.bell-labs.com> To: ips@ece.cmu.edu Subject: iSCSI: comments on rev 05 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, I have several comments about revision 5. I have grouped them into concerns, questions/comments, and editorial/grammar categories. Jeff Fellin fellin@lucent.com Member Technical Staff Lucent Technologies/Bell Laboratories 600 Mountain Ave, Murray Hill NJ ====== Concerns ====== These concerns may be due to not comprending some of the mail on the reflector, or lack of understanding the SAM2 specification. If they really aren't issues with the specification please excuse my naivete. Concern 1: Login processing and PDU descriptions. The login processing descriptions and details are spread throughout the document, section 1.2.3 iSCSI Login, 1.2.4 Text Mode Negotiation, 2.8 Text Command, 2.9 Text Response, 2.10 Login Command, 2.11 Login Response, and Section 4 Login Phase. As a result of this separation there are inconsistencies in statements about how login processing proceeds. For example in Section 1.2.3 the statement is a login Response concludes the Login Phase. In Section 2.11, Login Response, the statement is made "The Login Response indicates the progress and/or end of the login phase". The last paragraph of section 4 states the Login Response with the Final bit set to 1 can only be sent in response to a Login command or Text command with the F bit set to 1. Section 2.11.4 until reading Section 4, it appears there is no way to send a Login Response with the Final bit set to 1, unless the iSCSI initiator sends a Login command with the Final bit set to 1. However, the iSCSI initiator can only send one Login command. Section 4.1 states a Login Response indicating a rejection must have the F bit set to 1, but if it can only be sent when a Login or Text is received with an F bit set to 1, than this is a protocol error. The above statements could confuse an implementor reading the various sections. All the sections should be made consistent and/or provide pointers to the sections in the document where the other details are found. Concern 2: Text mode negotiation The understanding of text mode negotiation is also complicated by being described in several places in the document: section 1.2.4 Text Mode Negotiation section 2.8 Text Command, 2.9 Text Response, and section 4 Login Phase. For example in Section 1.2.4 the description states the initiator can provide a list that the target can select from. In Section 4.2 the initiator provides an ordered list where the target must select the first one it supports. This section also specifies that the target must select one of the list entries, although Section 1.2.4 implies that not returning the keyword in a response results in selected the value "none" by omission. Concern 3: Parameter Negotiation outside login There are several places in the document where the statement of text parameter negotiation outside the login phase occurs. Usually followed by a statement that this is after a login phase. My suggestion is to change the statements to be text negotiation after login phase. This occurs in Sections 4 and 5. Concern 4: Text parameters There is no one place where all the possible text parameters are listed. One has to read the entire document to make sure they didn't miss any. It would be helpful for all the defined text parameters mentioned for login processing and security processing to be added to Appendix D changing the title to iSCSI Text parameter List. Concern 5: Error/Exception procession It would be nice to have references to specific parts of Section 6 when error processing concerns may arise. For example, referring to Section 6.1, in Section 2 when describing the values of reserved fields and undefined values. Concern 6: Data transfer sizes Since iSCSI is designed to work with the SCSI-3 specification. There is an inconsistency of data sizes with SCSI-3 WRITE(12)/READ(12) data transfer sizes. The WRITE(12) and READ(12) have a 32-bit length field that is in units of the device sector size, usually 512 bytes. This results in an maximum size in bytes of 41 bits. Whereas, the maximum size of a data transfer byte size in iSCSI is 32 bits. How, is iSCSI going to handle a SCSI-3 CDB requesting a data transfer size in bytes greater than 32 bits? Concern 7: Text parameter negotiation Section 2.9.3 states that if the Text Response does not contain a requested key the initiator must assume the the target doesn't understand the key or the answer is 'none'. However, in section 4.2 the statement concerning negotiation is the target MUST reply with the first option in the list. So, if the initiator doesn't provide a selection the target supports the target must pick an element from the list and cannot respond as section 2.9.3 states. The various sections should be revised to be more consistent, which I am willing to help with. Concern 8: Section 2.18.2 Asynchronous Messages The description of the various SCSI events in section 2.18.2 appears to imply the iSCSI target has knowledge of events concerning the SCSI initiator on the target system, or perhaps that the iSCSI target actually is the SCSI initiator on the target system. I believe this violates the concepts of transferring SCSI CDB's across a network into actually interpreting them, or having a strong interaction between the two protocol layers that doesn't appear in many other places. Why is there the need to have this type of interaction between the clients of the iSCSI layer and the iSCSI layer? Concern 9: Section 3, Mode pages I am not sure from reading the descriptions, but it sounds as if the iSCSI target modifies the SCSI mode pages sent by it's SCSI targets on return to the iSCSI initiator. This is due to the reference to the reference to the disconnect-reconnect mode page parameters in Section 1.2.5 and Section 3. Although I cannot find a SCSI document defining a Logical Unit Control Mode Page and Port Control Mode Page. Is this a reuse of a SCSI mode page name in iSCSI or actually the SCSI mode pages being modified by the iSCSI target? This appears to be similar to the issue about SCSI events in Asynchronous Messages. ======= Comments/Questions ====== Question 1: Section 1.2.3 iSCSI processing. What is the value in sending an iSCSI command reject on a connection that isn't in the Full Feature Phase. Since the TCP connection is going to be terminated. Why not just terminate the TCP connection. Question 2: Section 1.2.5 I don't find any description of what an initiator is to do when it receives an R2T with is not valid, or requests data outside the command bounds. Question 3: Section 1.2.5 The description about an initiator sending too much unsolicited data to a target mentions this is controlled by the FirstBurstSize parameter, and is available from the disconnect-reconnect mode page. I don't see a iSCSI PDU that would allow an initiator to request this information. Is the assumption the iSCSI initiator would send a SCSI command to the iSCSI target to retrieve a SCSI mode page. This appears to violate the concept of packaging SCSI commands across a network, and instead actually looking inside the SCSI command which should just be an opaque SCSI CDB and SCSI data responses. Comment 4: Section 1.2.5 There is a statement "A target receiving data out of order SHOULD terminate the session." Since this is not mandatory, I expected to find something in Section 6 stating how an iSCSI target would deal with receiving data out or order. Otherwise I suggest changing this SHOULD to a MUST. Comment 5: Section 1.2.8.2 The statement is made the iSCSI considers the information it delivers as a contiguous stream of bytes. Although in Section 1.2.8.1 the description implies iSCSI deals with messages and has found a way to maintain message boundaries across the TCP bytestream. In addition throughout the rest of the section it is mentioned iSCSI sends and receives PDU's (which are message oriented). Perhaps, I don't understand to what iSCSI is delivering a bytestream to in the Synch and Steering Model. It would be helpful to clarify where iSCSI delivers a bytestream or remove the statements concerning bytestream data. Comment 6: Section 2.2 The text refers to a 4-byte Next-Qualifier. However there is no description of 'Next-Qualifier'. It appears this 'Next-Qualifier' is a different name of the 'What's Next' field. I suggest replacing the term 'Next-Qualifier' with 'What's Next'. This also appears in section 2.2.2.3 Question 7: Section 2.2.1 The field description describes two bytes, although the orientation of the text doesn't make it clear. Request changing this to have a header for each byte, and make sure all text description for a bit(s) be indented past the bit position. Question 8: Section 2.2.2.1 I interpret the description of the AddCDB to be a multiplier of the expected number of additional CDB words to follow. Hence a value of zero would indicate no additional CDB words instead of 4 additional CDB words. Starting the addCBD at one allows determination of the additional header segment size by a simple shift. This could be considered a hardware type optimization. Question 9: Section 2.2.2.3 I'm confused as to which data length is ignored and which one is used by the description in this section. Is it the amount of data received or the length in a WN field, or something else? Question 10: Section 2.3.1 and 2.3.5 I'm not sure where the Length field referred in the description of b7 is? I think the Expected Length is the value of 'Expected Data Transfer Length', but don't know where 'Length' field is? Question 11: Section 2.4.1 The layout of Byte 1 shows bits 5 and 6 has reserved with bit 7 having the value of 1. In 2.4.1 it states bits 5-7 are reserved. Although the description in Section 2 would allow for reserved fields to have a non-zero value, this could be missed in implementation. Request changing the description to say bits 5 and 6 are reserved, and stating bit 7 must have the value of 1. Question 12: Section 2.4.2 Why is there no successful iSCSI Response code in the Status/Response field? Question 13: Section 2.5 I'm not sure what service the Task Managment Command and Response are providing. Is it how a SCSI-3 application client requests the iSCSI layer to provide the task management remote procedure call as defined in Section 6.7 of the SAM2? If so, could you please state this in the beginning of Section 2.5. Question 14: Sections 2.5 and 2.6 Throughout the section I'm not sure what entities Initiator and Target refer to. Is it the actual SCSI initiator on the iSCSI initiator system or the iSCSI initiator? Likewise for the term target. Comment 15: Section 2.6 It appears the information on and is more appropriate in the paragraphs in section 2.5 Question 16: Section 2.6 The statement about the 'target MUST then close all of its TCP connections' appears to imply the receipient of the task management command is the iSCSI target. Or must the iSCSI target examine the information in the SAM2 remote procedure call to determine specific actions on the iSCSI target's part? Comment 17: Section 2.6 There is no description of the value in the Referenced Task Tag when the Response field has a value of zero. Since the field only appears to have a value when Response has a value of one(1). Question 18: Section 2.10 Should there be a section describing the value of the TSID? Comment 19: Section 2.11.6 The statement about TSID being the same on partial and final responses seems to fit better in section 2.11.5. Comment 20: Section 2.12 The description of the NOP-Out refers to a Ping Response. There is no such command. So Ping Response should be changed to 'NOP-In Response'. There is a reference to a Ping bit, but it isn't defined yet. the term should be replaced with the 'P bit' Question 21: Section 2.12 The statement is made that the initiator may conclude there is a problem with the connection if the NOP-in data is different than the NOP-Out data. It than describes what to do if the conclusion is made. If the initiator doesn't conclude a problem exists what would it do? Is there any insight from IP ICMP echo messages that could help with determining what an iSCSI initiator could do? Comment 22: Section 2.12 The last sentence about how an initiator sends a NOP-Out in response to a NOP-in is confusing to me. Suggest changing the sentence to: An initiator sends a NOP-Out when it receives a NOP-In with the P bit set, in which case the initiator copies the Target Tag from NOP-In and the P bit in the NOP-Out MUST be zero. Comment 23: Section 2.16 The description of AddRun in Section 2.16.2 and RunLength in Section 2.16.4 appear to be inconsistent. Since a run is described by a starting sequence and a length, it appears the fields BegRun and RunLength describe such a sequence. I interpret the description of RunLength to be the number of additional runs missing. Is this really the definition or am I confused? Question 24: Section 2.17 The statement is made that an initiator can chose to return only a portion of a R2T request. What is the expection of how the iSCSI target will handle this situation? Question 25: Section 2.18.2 The section states the 'Length parameter is set to the ...'. However there is no field in the message. Perhaps the 'Length parameter' should be named 'Parameter1 field'? Question 26: Section 4.2 There is a statement the 'security authenticates the user and the target'. Is the user in this situation the user of the iSCSI initiator or is it the iSCSI initiator? Question 27: Section 4.2 The first item in the list of negotiable security mechanisms refers to 'the host and target'. Is the host the iSCSI initiator or the user of the iSCSI initiator? Question 28: Section 4.2 The first item in the negotiation procedure mentions the options are listed in the initiator's reverse order of preference. This is a different type of list than described in section 1.2.4. To me this means the most preferred entry will appear last in the last. The second item states the iSCSI target selects the first item from the list it supports. Doesn't this always result in the least preferred item from the iSCSI target's perspective? Why would the negotiation automatically select the least preferred entry even if better entries would exist? Question 29: Section 4.3 The statement is made a 'target MUST NOT send more than one Login Response with the F bit set to zero. However the description of Login Response, section 2.11, doesn't state this restriction, and implies more than one F bit set to zero could be sent. Why is this restriction necessary? And if it is necessary would it be possible to Question 30: Section 6 I am not sure what the term 'target' means in this section. Is it the iSCSI target implementation or the SCSI target devices attached to the iSCSI target system? Question 31: Section 6.1 If an iSCSI initiator receives a PDU with a format error. How does the iSCSI initiator determine an outstanding task for the PDU? The reason for this is I thought a session can have multiple tasks. If there are multiple tasks how does the iSCSI initiator determine which iSCSI target task to abort, since the task id information in the PDU cannot be relied on? ======= Editorial ====== Section 1.2.8.3 First sentence: "We recognize that in many environments the following isa more..." Change 'isa' to 'is a'. Section 2.3.1 The description of b7 (F). 'set to 1 when the immediate data that accompany the command are all' change to 'set to 1 when the immediate data that accompanies the comman is all' Section 2.4.1 This is the only description of bits in a byte numbered from b0 instead of b7. It should be changed to start with b7 to be consistent with the other PDU descriptions. Section 2.7.5. Second paragraph, sentence stating 'if this bit is 1 the target MUST deliver packets'. The word deliver implies to me the target is guaran- teeing the initiator will get the packets in the exact order the target sends them. Given the many discussions about TCP dropping packets and data arriving out of order, Perhaps the only constraint on the target is that it 'SEND' the packets in increasing buffer offset order, not 'DELIVER' the packets to the initiator after TCP and the Synch and steering layer process the packets. Section 2.10.1 Change 'CID does not change and this command is performs' to 'The CID does not change and the target performs' Section 2.11.4 Change 'the initiator may proceed to negote operational parameters' to 'the initiator may proceed to negotiate operational parameters' Section 2.12 Change 'duplicating the data that was provided in the NOP-Out' to 'duplicating the data provided in the NOP-Out' Section 4.1 List of how target can answer in the following ways, item 2. Change '... authentication mechanism and starts with the session immediately (enters full feature phase).' To '... authentication mechanism and immediately enters the full feature phase.' Section 4.2 The first paragraph has: Change ' algorithms that were chosen in the negotiaton phase and is conducted by the text commands ...' To ' algorithms chosen in the login negotiaton phase done by the text commands ...' From owner-ips@ece.cmu.edu Fri Mar 16 17:51:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA14411 for ; Fri, 16 Mar 2001 17:51:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GKnbh17561 for ips-outgoing; Fri, 16 Mar 2001 15:49:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2GKmqr17518 for ; Fri, 16 Mar 2001 15:48:52 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id PAA00757; Fri, 16 Mar 2001 15:48:45 -0500 (EST) Message-ID: <3AB27C2A.881DFE25@cisco.com> Date: Fri, 16 Mar 2001 14:48:42 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: julian_satran@il.ibm.com CC: ips@ece.cmu.edu Subject: Re: iSCSI MIB Object Model References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian- You are right, it is "almost transparently". We still have enough visibility to keep track of who's-doing-what to a LUN, or at least pointing at LUNs that have been the subject of an error. Who changes LUNs during an active session? This sounds pretty strange to me. I could see adding new LUNs, but changing the addresses of existing ones sounds really odd. Is there a reference I can look at for something that does this? I have the SCSI MIB topic in my presentation for Monday; hopefully we will get some consensus on what to do. Thanks, Mark julian_satran@il.ibm.com wrote: > > Mark, > > iSCSI uses LUN almost transparently. Also the LUN may change while a > session is active. > I am not sure that the fact that there is no SCSI MIB (yet) doe justify > including those items in the iSCSI MIB. > > I also recall (a rumor?) that somebody is working on a SCSI MIB (it will be > needed anyhow there is more to SCSI than the LU). > > Julo > > Mark Bakke on 14/03/2001 22:48:47 > > Please respond to Mark Bakke > > To: Venkat Rangan > cc: IPS , jmuchow@cisco.com > Subject: Re: iSCSI MIB Object Model > > Venkat- > > You have captured many of our open issues in this > message. Please see my comments below. > > Thanks, > > Mark > > Venkat Rangan wrote: > > > > Mark, > > > > The UML is very nice, and the new object model, covering both > > Initiator and Target instances is also nice. > > > > Overall, we do have some concern over the need for maintaining per-LU and > > per-LUN statistics. In some iSCSI implementations, the iSCSI entity is > > merely an intermediate layer, with no knowledge of LUs and LUNs. Placing > > this in the GeneralInfo category and making it mandatory would add undue > > burden on these implementations. > > One of our open questions is "how much is too much?". We added the LU > and LUN statistics for a few reasons: > > 1. The iSCSI layer does see the LUN; it is included in the SCSI Command > message. > > 2. For implementations that include the actual target and LUs, such > as storage controllers, there is no other way to get information > on LUNs through SNMP, since there is no SCSI MIB. > > 3. We felt that in this case customers might require LU- or LUN-level > accounting. > > However, keeping statistics at this level does create some work; perhaps > there is a way to make it optional. > > > In Section 5.9: > > "The iscsiLUTable contains a list of iSCSI Logical Units that are > > accessible via the iSCSI target." > > > > Would this list be the same as what the "Report LUNs" command would give? > > It depends. A LUN is just the address of a logical unit, and in the > general case, is only valid between a particular initiator and a particular > target. If a second initiator issues a "Report LUNs" to the same target, > it may get back a different list if the target is performing some sort of > LUN mapping. Some targets will show the same set of LUNs regardless of the > initiator; some won't. There is no reliable way given the current MIB to > find out what "Report LUNs" would return; this is probably the domain of > a SCSI MIB. > > > In Section 5.7: > > "The iscsiLunTable contains an entry for each known LUN that is being > > accessed using this session. Note that is may not contain all of the > > LUNs that could be accessed; the only ones available to iSCSI are the > > ones that have been addressed specifically by iSCSI commands. > > Therefore, LUNs that have been discovered via mechanisms such as the > > SCSI "report LUNs" will not appear in this table until they are > > specifically addressed by the initiator. > > > > How long do these entries live in the table? If iSCSI sessions logout and > > close, do these entries continue to remain? > > The current intent is to have entries live only as long as the sessions; > these would be removed when the session is removed. This is the simplest > solution and would put the smallest burden on the implementation. > > However, we still have an open "requirements" question on whether anyone > wants these to hang around longer. > > > Thanks, > > > > Venkat Rangan > > Rhapsody Networks Inc. > > http://www.rhapsodynetworks.com > > > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Fri Mar 16 17:52:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA14442 for ; Fri, 16 Mar 2001 17:52:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GLMX020080 for ips-outgoing; Fri, 16 Mar 2001 16:22:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2GLLor20013 for ; Fri, 16 Mar 2001 16:21:50 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id QAA14242; Fri, 16 Mar 2001 16:21:38 -0500 (EST) Message-ID: <3AB283DE.D67F4C41@cisco.com> Date: Fri, 16 Mar 2001 15:21:34 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Douglas Otis CC: IPS Subject: Re: iSCSI: Rationale behind the SLP draft References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Doug- SLP provides the multicast discovery we were looking for, and does not require DHCP, although it can work with it. LDAP does not provide service location; it provides database access. SLP also looks like a simpler protocol than LDAP, and a simpler way to specify the template (schema) for our attributes. LDAP provides database access; iSNS provides active management services. I could not see using LDAP to do what iSNS does. Josh could comment further on this one. Both SLP and iSNS are designed to be used with LDAP; an SLP DA can use LDAP as its back-end database, and clients can be served using both protocols. We felt that this capability gives us a way for those who want to set up LDAP to make use of it, without forcing it to be implemented in our devices. Anyway, LDAP does meet several of our requirements, but it stands right in the middle of what SLP and iSNS are used for. It doesn't quite give us what we want on the low end, and it doesn't do the management on the high end. It could, however, be utilized as an excellent database access protocol for these other services to be built on, but that would not have to be visible to hosts and devices. SLP already has DHCP extensions, security extensions, etc., iSNS borrows them. All we had to do for SLP was to define a template, which is just a simple version of a schema. -- Mark Douglas Otis wrote: > > Mark, > > If I have two servers with similar information to promulgate and expectation > of multi-casts reaching the target, I may find SLP efforts greater than > desired over standard methods found using DHCP and LDAP. DHCP can provide > information of the location of the LDAP server as already defined in > existing boot specifications. The BOOTP-DHCP multi-cast request already > have relay provisions in routers and switches because of its common use and > has defined entries for LDAP server discovery. You could have defined an > LDAP schema that would have covered both iSNS and SLP without adding > additional standards efforts for building yet another protocol that does the > same function as protocols already in common use. > > For an overview see: > http://www.umich.edu/~dirsvcs/ldap/doc/guides/slapd/1.html#RTFToC1 > > Could you explain why LDAP is unsuitable for this application? There are > many conventions within LDAP in terms of extensibility that seem to make it > more suitable than SLP and iSNS. You will also find security extensions > already in place for LDAP merged with BOOTP-DHCP. It seems you have > problems to overcome that have already been solved. Not being clever, I > tend to ask dumb questions. > > Doug > > > David Black has requested that I send something to the > > list showing the rationale behind the SLP draft, including > > its fit in the big picture and its relationship with iSNS. > > Here's an attempt to do so. I believe it to be consistent > > with the Naming & Discovery, iSNS, and iSCSI/SLP drafts, > > and their IETF-50 presentations. > > > > The relevant drafts are: > > > > draft-ietf-ips-iSCSI-name-disc-00 > > draft-ietf-ips-isns-00 > > draft-bakke-iscsi-slp-00 > > > > Naming and Discovery Requirements > > > > These are the discovery and name services requirements > > from draft-ietf-ips-iscsi-name-disc-00. From a discovery > > requirements point-of-view, SLP and iSNS provide some > > similar services. SLP provides very simple discovery services > > for smaller networks; iSNS provides more comprehensive > > management services for larger networks. Where they overlap, > > SLP and iSNS can interoperate fairly easily. > > > > 1. Multicast method to discover targets, to allow zero- > > configuration of initiators and targets. > > > > SLP provides this. > > > > 2. A method to look up a WWUI, and get one or more addresses. > > "Where is target ?" > > > > SLP and iSNS both provide this. > > > > 3. A method to find targets that a given initiator should access. > > "I am initiator ; which targets should I access?" > > > > SLP and iSNS both provide this. > > > > 4. Ability to limit discovery to relevant initiators and targets. > > > > iSNS uses Discovery Domain, which are used to limit which initiators > > and targets can discover each other. Each target and initiator > > belong to Discovery Domain; the iSNS will not allow initiators to > > discover targets outside their domain. > > > > SLP uses a named Scope. Each target and initiator is configured > > with one or more scope names; a target will only respond to an > > initiator who has a matching scope. Note that this is not a > > security mechanism; an initiator can ask for any scope it wishes, > > and a target can respond to any scope it wishes. This provides > > a simple mechanism to limit discovery, but does not attempt to > > provide security. > > > > 5. Access Lists by initiator WWUI > > > > SLP and iSNS both provide the ability to store a list of initiator > > WWUIs that are allowed access to a given target. > > > > 6. iSCSI Object Model support > > > > SLP registers only the targets > > iSNS registers targets, initiators, network entities, portals > > > > 7. TLV Message Format > > > > SLP and iSNS both do this. However, SLP uses a text key and > > value for its attributes; iSNS uses a binary key value. > > > > 8. Authentication of discovery messages > > > > iSNS uses SLP's authentication mechanism > > > > 9. Registration and Query > > > > SLP registers targets only > > iSNS registers targets, inititors, network entities, portals > > > > 10. State Change Notification > > > > iSNS provides a state change service for which entities can > > register. > > > > SLP does not provide state change or event notification. There > > is a draft on doing some notification, but it is not adequate for > > propagating attribute changes, which are necessary for an > > initiator to discover that it has gained (or lost) access to > > a target, or that a target has added a new address. > > > > 11. Light-weight Protocol > > > > Neither protocol should be too much of a burden to implement. > > > > Adding authentication to the protocol can be done later, and will be > > the same effort for both. In the case that both SLP and iSNS are > > supported in the same product, they can share the code for this > > feature. > > > > 12. Compatibility with Boot Requirements > > > > Both the SLP template and the iSNS specification have worked to > > ensure that they fit in with the boot requirements. > > > > SLP has defined DHCP options that allow it to locate a directory > > agent, avoiding multicast traffic from initiators in a DHCP > > environment. > > > > 13. Compatibility with LDAP > > > > Although this is not a requirement, we have seen the desire for > > interworking with LDAP discussed on the list enough that it may > > merit a requirement at some point. Both SLP and iSNS are designed > > with LDAP compatibility in mind. > > > > 14. Entity Status Inquiry/Heartbeat > > > > iSNS provides an ESI/heartbeat ping message to monitor the > > availability of initiators and targets. The heartbeat is > > originated by the iSNS server. > > > > We have not defined this capability for SLP. It may be possible > > to do something similar if targets advertise themselves with a > > shorter lifetime, and initiators expire their discovered targets > > appropriately. Adding a directory agent (DA) may be required for > > this to work properly. > > > > 15. Distribution of X.509v3 Public Key certificates > > > > iSNS provides this capability. > > > > Our SLP template does not attempt to do this for two reasons: > > > > - SLP attributes are strings; the binary certificates would have > > to be escaped or uuencoded. > > > > - iSCSI only registers targets; there is no way for targets > > to discover an initiator's certificate. > > > > 16. Usage of Multicast > > > > SLP normally relies on multicast, for initiators to request > > advertisements from targets. Multicast use can be eliminated > > at the expense of adding a DA, and configuring its address on > > each of the initiators and targets. > > > > iSNS does not use multicast. > > > > > > We had originally started out looking for something to fulfill the > > multicast requirements that iSNS did not do. However, we found that > > if one already had SLP, it was just as easy to find individual > > targets, or at least canonical targets using it as it was to locate > > an iSNS service. SLP can then be used to handle discovery for > > simpler host and device environments, and iSNS could be added later > > to help manage these environments. > > > > SLP and iSNS both fulfill many of the requirements from > > the naming & discovery draft. However, they don't completely > > overlap, and they take two different approaches to discovery. > > > > SLP provides Discovery of Targets. > > > > SLP is used for discovery ONLY. It uses a distributed > > model, that can start with initiators and targets, and > > no directory servers. It can add Directory Agents to > > scale to medium-sized environments and reduce or eliminate > > the use of multicast. Some work is underway on mesh- > > enhanced SLP (mSLP), which provides peer-to-peer information > > exchange between DAs for larger environments. > > > > From a pure discovery point of view, SLP's chief weakness > > is that it does not yet have a notification mechanism. > > > > - Discovery of iSCSI targets > > - Discovery of iSNS services > > - Multicast-optional discovery mechanism for targets and > > other name services. > > - Open source implementations > > - Existing standard protocols > > > > iSNS provides Centralized Management of Initiators and Targets. > > > > In addition to target discovery, iSNS provides higher-level > > functions such as centralized access management, active > > monitoring, public-key certificate management, and state-change > > notifications. > > > > - Discovery of iSCSI targets > > - Centralized management of initiator and target access > > - Active monitoring of initiators and targets > > - State change notification > > > > Using SLP with iSNS > > > > Although SLP and iSNS solve several of the same problems, they > > are fairly well-suited to interoperating. They share a common > > authentication structure, which is not something one would want > > to code twice. SLP can be used to do basic discovery where > > configuring an iSNS server is not warranted, and can help > > discover iSNS servers in environments where centralized management > > is needed. It is relatively simple for an iSNS server to > > provide a proxy SLP service agent on behalf of the targets it > > manages, allowing initiators to participate that implement only > > the basic SLP. Similarly, an iSNS server can work with gateways > > to provide its services for targets that support only SLP. > > > > When initiators and targets support BOTH SLP and iSNS, there > > are a few rules that should be followed: > > > > 1. Initiators can use both SLP and iSNS concurrently to discover > > their targets. In fact, an initiator should be able to use > > more than one iSNS, if it is accessing storage from two > > different providers. This allows an initiator to discover > > its locally-managed devices using SLP, and its centrally- > > managed devices using iSNS. > > > > 2. Targets should be advertised using a single discovery method. > > A target should be advertised either with SLP, or with a > > single iSNS environment. Targets discovered multiple ways > > would probably not break anything, but a target discovered > > via multiple services could produce conflicting information > > to the initiator. > > > > 3. Gateways or iSCSI proxies can be used to provide local SLP > > discovery of targets that are managed using iSNS. > > > > Implementation Approach > > > > I recommend a three-phase approach to implementing iSCSI > > naming and discovery. The first phase is to simply support the > > WWUIs, and their use with the iSCSI protocol, and allow static > > configuration of hosts and targets. The second is to support > > SLP for simple discovery. This should be relatively quick to > > implement, as the source for SLP is readily available. The third > > would be to support iSNS as a storage management capability. > > > > 1. Simple naming and configuration > > > > - Admins configure targets with some form of access list, which > > may include the initiator WWUIs that are supposed to access > > them. > > > > - Admins configure initiators with the canonical target "iscsi" > > for targets that may provide them services. > > > > - Initiators use the SendTargets text command to discovery their > > targets. This eliminates configuring target WWUIs in the > > initiator. > > > > 2. SLP for multicast and simple discovery > > > > - Instead of configuring initiators with target addresses, admins > > enable SLP on the initiator. > > > > - Targets also enable SLP, and are discovered by the initiators. > > > > 3. iSNS for centralized management > > > > - Initiators and targets are configured to be managed by an iSNS > > service. > > > > - Admins configure both hosts and targets from the iSNS server. > > > > - Initiators and targets use the iSNS protcol to discover each other. > > > > Other Discovery Protocols > > > > There are several other discovery protocols; each has its > > strengths and weaknesses. Here are some of them. Take a look > > at "Building Networks on the Fly", IEEE Spectrum, March 2001 > > for more information on these. > > > > 1. DNS SRV Records - these seemed too limiting, and there did > > not seem to be much point in making DNS do more. The SLP > > folks started their work because DNS SRV records didn't meet > > their requirements. > > > > 2. LDAP, with an appropriate schema defined, could handle iSCSI > > registrations and queries, and distribute essentially the > > same information as SLP. There are three reasons we didn't > > go directly to LDAP: > > > > 1. LDAP would require a heavier implementation than SLP > > 2. LDAP would not solve the multicast requirements > > 3. SLP is built to be compatible with LDAP; SLP can be used > > by iniators and targets, with LDAP as a back-end database. > > This is left as an exercise for the implementor. > > > > 3. Jini is controlled by a single company, and is tied to a > > particular programming language, which is not appropriate > > for an internet standard. > > > > 4. UPnP from Microsoft is based on XML data with an HTTP transport. > > XML has a lot of merit for application interfaces like these, but > > the code for XML and HTTP could be a bit heavy for a really simple > > device. It's company-controlled anyway, so we can't use it. > > > > 5. Salutation - Have not explored this in depth. Salutation has > > been around longer than the other protocols. It is made to be > > more flexible; it does not assume IP. > > > > 6. Bluetooth, HAVi - These are done by controlled-membership > > organizations, and are designed for specific non-IP environments. > > > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 > > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Fri Mar 16 18:27:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA14904 for ; Fri, 16 Mar 2001 18:27:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2GLeZY21420 for ips-outgoing; Fri, 16 Mar 2001 16:40:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.wrs.com (unknown-1-11.wrs.com [147.11.1.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2GLdjr21380 for ; Fri, 16 Mar 2001 16:39:45 -0500 (EST) Received: from london ([192.168.1.48]) by mail.wrs.com (8.9.3/8.9.1) with SMTP id NAA29246 for ; Fri, 16 Mar 2001 13:39:14 -0800 (PST) From: "Rod Harrison" To: Subject: RE: iSCSI: Towards a more effective PDU format Date: Fri, 16 Mar 2001 21:42:00 -0000 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I rather like the new header format proposed here, although I would prefer to see a separate digest for the AHS segments. I like that the new format allows the next BHS to be found if anything except the current BHS is damaged, which is not the case with the current format since a damaged AHS length causes loss of sync. If a BHS is damaged I think recovery is going to be expensive and including more bytes in the digest covering the BHS increases the likelihood of having to perform that recovery, even though the BHS itself might not be damaged. How expensive is it to add the second digest covering all the AHS segments? I'm guessing that the second digest could be calculated using the first as a starting point. I don't see a down side of mandating that any AHS digest will be of the same type as the BHS digest. On a related point, if a BHS is damaged I would prefer to see the connection immediately dumped rather than any re-sync activity performed. I don't want to get into the debate about the merits versus risks of re-syncing but I think the fact that it has been talked about for so long shows that the risks are not obvious. Either way I think there is a more fundamental reason to prefer connection dumping. In order to re-sync there must be a data stream, if a BHS is damaged during a period of light activity there may be a significant delay before any more data is sent. In the limit the next PDU may be a retry of the damaged PDU by the iSCSI or host SCSI layers. Those timeouts are going to be in seconds or more likely tens of seconds, whereas dumping the connection will immediately cause the recovery to begin. The time to make a new connection will be significantly shorter than the worst case re-sync time. Further, unless we can guarantee that re-syncing will always succeed there will have to be a point where we give up and dump the connection anyway. Also, even if we do manage to re-sync we still incur a timeout for the damaged PDU to be resent. Again, if the connection is dumped that timeout doesn't need to happen. I should point out that when I say dumped here I mean a graceful TCP close. - Rod Harrison -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Robert D. Russell Sent: Thursday, March 15, 2001 10:06 PM To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI: Towards a more effective PDU format Julo: Even if we give up on the idea of having a second header digest, I still believe that some changes should to be made to the PDU Format proposed in version 5 to make it simpler, more efficient to process, and more robust. I would therefore propose eliminating the use of the current WN Next-Qualifier scheme and replacing it as follows (this is just a slight modification of my earlier proposal in order to eliminate the second header digest). 1. Every PDU must start with a 48-byte Basic Header (BHS) that can be read in a single "read" operation. The format of this is identical to that proposed in section 2.2.4 of version 5, but with the addition of 2 fields (4-bytes) as described next. 2. Every BHS contains 2 fields at fixed locations: a) AHS_length, containing the total length of all Additional Header Segments (AHS) that follow. This field is 0 if there are no AHSs. b) DATA_length, containing the total length of all data that follows the AHSs. This field is 0 if there is no data. 3. If the AHS_length field in the BHS is non-zero, all the AHSs immediately follow the BHS in the PDU. The AHS_length value gives the total number of bytes in all the AHSs, allowing a single "read" operation to read them all. (The total header consists of 48+AHS_length bytes.) 4. If header digests are in use, one header digest covering the BHS and all the AHSs (if any) immediately follows the last AHS. This digest does NOT require a separate read, since the first read (of the BHS) should be for 48+(length-of-header-digest) bytes. If the AHS_length field is zero, there are no more reads. If the AHS_length field is non-zero, exactly that many more bytes need to be read in a second read. These additional bytes are appended to those obtained in the first read. The header digest will always be the last (length-of-header-digest) bytes. 4. If the DATA-length field in the BHS is non-zero, all the data immediately follows the header digest. If data digests are in use, a data digest immediately follows the data. 5. To save space within the BHS, the AHS-length field can be restricted to a single byte, and can be in units of 4-byte words. This allows up to 1020 bytes of AHSs to follow a BHS, which seems to be more than enough for the uses foreseen (so far, only extended CDB and bidi-read info). Currently, byte 2 is unused (reserved) in all PDU types except Login and Login response, where byte 6 is unused. Therefore, the AHS-length field can be added without increasing the BHS size of 48 bytes. 6. Each AHS should start with a word containing a TYPE field and a LENGTH field. The TYPE field should be enumerated rather than bit-field encoded, for easier decoding and future expansion. The LENGTH field is the number of bytes of additional information in this AHS that follows this word. 7. The DATA_length field should be a 4-byte field added to the 44-byte BHS of the current section 2.2.4 to give a new BHS of 48-bytes. However, since the version 5 44-byte header is always preceded by a 4-byte WN Next-Qualifier that would no longer be needed, there is no net change in the effective BHS size of 48-bytes. Advantages of this proposal over the current WN Next-Qualifer of version 5. 1. Because the receiver gets the AHS-length field from the BHS, it can obtain the entire set of ALL AHSs that follow the BHS in a single "read" operation (the current WN Next-Qualifier scheme requires a separate "read" for EACH additional header segment after the BHS). 2. By limiting the total size of all AHSs to 1020 bytes, a receiver can preallocate a fixed-length buffer of (48 + 1020 + size-of-header_digest) bytes for header processing (the current WN Next-Qualifier scheme has no limit on either total header size nor individual AHS sizes). 3. Since each AHS begins with a word containing the TYPE and LENGTH in fixed positions, an unknown TYPE (i.e., a type introduced in the future that is received by a legacy receiver) does NOT result in loss of synchronization during header processing -- the receiver knows the length of this AHS in any case, and can just skip over it (the current WN Next-Qualifier type determines how the length field is to be interpreted -- there is no "general rule", so unknown types mean loss of synchronization). 4. The format of the AHS is the familiar Type/Length/Value (T/L/V) structure (the current WN Next-Qualifier is T+1/L+1/V, but this does not seem to provide any benefit and only adds confusion and complexity -- see e-mails from David Black and Barry Reinhold). 5. The AHS TYPE is enumerated (the current WN Next-Qualifier is bit encoded, again without seeming to provide any benefit -- see e-mail from David Black). 6. Because the BHS always contains a 4-byte DATA_length field, the maximum data segment size is 4 Gigabytes (the current WN Next-Qualifier scheme, with the long data header removed, limits the data segment size to 16 Megabytes). 7. The "Header Digest Present" and "Data Digest Present" bits have been completely eliminated (the purpose of these in the current WN Next-Qualifier scheme was never explained, but they appear to be useless -- the presence or absence of digests has to be negotiated, and once negotiated, all PDUs must obey the negotiated decision). I see no disadvantages of this proposal relative to the current WN scheme. Since there is only one header digest, both schemes suffer the disadvantage of using unreliable data to find the header digest, which can lead to unnecessary blocking on "reads". Proposed iSCSI PDU Format +------------------------+ | required BHS | > fixed length of 48 bytes +------------------------+ | optional AHS 1 |\ | - - - - - - - - - - - | \ | optional AHS 2 | \ | - - - - - - - - - - - | > total length in AHS_length field in BHS | . . . . | / | - - - - - - - - - - - | / | optional AHS n |/ +------------------------+ | optional header digest | -- covers preceding (48 + AHS_length) bytes +------------------------+ | |\ | optional data | > total length in DATA_length field in BHS | |/ +------------------------+ | optional data digest | -- covers preceding (DATA_length) bytes +------------------------+ Thanks, Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 On Tue, 13 Mar 2001 julian_satran@il.ibm.com wrote: > > > Bob, > > Interesting. This is close to one of the variants I had for IETF-50 (a > clear header). > The only adavantage it has is that you are able to read all the AHSs in one > read. > The (admitedly academic) disadvantage it has is that you are limited in the > size of extensions and have some redundancy. > The basic issue that was raised - and there is no simple way out - is that > once you have lost a block (BHS in your suggested layout) you are out of > synch. There is no simple way around it (at least not one that can be > solved by changing layout) and an added digest (only the code or silicon to > account for it) is IMHO not warranted by what you gain. If you are willing > to spend silicon or code on this the making header failures less probable > and recovering if you fail by dropping the connection is a better bet > (using the redundancy for a coding gain). But this involves some > complexity too. > > Julo > From owner-ips@ece.cmu.edu Fri Mar 16 22:02:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA18488 for ; Fri, 16 Mar 2001 22:02:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2H0hZ704059 for ips-outgoing; Fri, 16 Mar 2001 19:43:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2H0hTr04048 for ; Fri, 16 Mar 2001 19:43:29 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 16 Mar 2001 19:43:23 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07080152C4@corpmx9.isus.emc.com> From: Black_David@emc.com To: psarkar@almaden.ibm.com, ips@ece.cmu.edu Subject: RE: iscsi: rev 05 2.5.1 requires ACA support Date: Fri, 16 Mar 2001 19:43:21 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > To impart some finality into the discussion, can you summarize for > the benefit of the mailing list what is distinctive about iSCSI that ACA > must be made mandatory for targets. I'm not trying to start an argument > here, but a few words of motivation might help us to understand > as to why iSCSI-ACA must be made manadatory (required in most > situations) versus optional (implement where needed). Actually there are (at least) two issues underlying Prasenjit's simple question -- this whole area of target response to errors feels like the old Adventure "maze of little twisty passages, all different" :-). Issue 1: Should ACA be REQUIRED, RECOMMENDED, or OPTIONAL to implement? - Independently for Initiators and Targets. Issue 2: Should ACA be REQUIRED, RECOMMENDED, or OPTIONAL to use? - Basically an Initiator issue, as SCSI puts the Initiator in charge of deciding whether to use ACA. I'm not sure Julian and I see eye-to-eye on this, so here are my current opinions: Issue 1, Initiators - ACA should be OPTIONAL to implement Issue 1, Targets - ACA should be REQUIRED or RECOMMENDED to be implemented (note, this is a slight change from my earlier email). Issue 2 - ACA should OPTIONAL to use, Initiator makes the choice in each case. So, Prasenjit's question to me is "Why should ACA be REQUIRED or RECOMMENDED to implement in targets?". ACA is specified in SAM, and T10 therefore requires that any transport mapping of SAM specify how ACA is implemented. Hence it has to be specified for iSCSI, lest we get dinged by T10 for failing to handle all SCSI features. This leaves the choice of REQUIRED, RECOMMENDED or OPTIONAL for target implementation. My concern is based on looking forward to the point in time where we work on moving iSCSI from Proposed Standard to Draft Standard. At that juncture, features that aren't implemented will be removed from the specification (retaining feature X requires at least two independently developed interoperable implementations of X). If ACA is removed at that point, iSCSI will get dinged by T10 for no longer being a complete mapping of SCSI, which would be bad. Hence, the word we choose should lead to multiple implementations of ACA that can be tested for interoperability, even if ACA is rarely used in practice. The removal of features for which interoperability cannot be demonstrated as part of advancing from Proposed Standard to Draft Standard is distinctive to iSCSI by comparison to other SCSI transport mappings. OPTIONAL strikes me as the wrong choice. I'd like to avoid a situation in which iSCSI can't move to Draft Standard because nobody's implemented an OPTIONAL to implement feature -- IMHO, that sounds really silly. REQUIRED seems most likely to lead to implementations. RECOMMENDED would also be ok. Sorry to have to give such a long answer to a short question, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Mar 16 23:22:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA19838 for ; Fri, 16 Mar 2001 23:22:36 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2H1ub308545 for ips-outgoing; Fri, 16 Mar 2001 20:56:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2H1u2r08523 for ; Fri, 16 Mar 2001 20:56:02 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 09FB01260; Fri, 16 Mar 2001 17:56:01 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA13756; Fri, 16 Mar 2001 17:55:56 -0800 (PST) Message-ID: <3AB2C52D.3AB80B01@cup.hp.com> Date: Fri, 16 Mar 2001 18:00:13 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Jon Hall , ips@ece.cmu.edu Subject: Re: iSCSI: too much recovery? References: <200103161744.MAA28879@lub1028.lss.emc.com> Content-Type: multipart/mixed; boundary="------------2AF68DCAD7654A12D6AFD818" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------2AF68DCAD7654A12D6AFD818 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Jon Hall wrote: > >2) You don't need ordering only simple acknowledgement to free resources. > >As such per connection is simpler to implement and maintain. You may > >choose to do what you suggested > >and recover by retrying but most of this group consider this too drastic. > > I get confused, why is per connection easier than per > session? The target's task state must be maintained at the > session level. Pushing StatSN into the connection makes per > connection state more complicated; it now contains a subset > of the session's task state. StatSN needs to be on a per-connection basis. If StatSNs were on a per-session basis, consider the following problem : - StatSN 1 is sent out on connection 1 - StatSN 2 is sent out on connection 2 Since these 2 status' are sent on different connections, StatSN 2 could arrive prior to StatSN 1. The initiator detects a hole in StatSN (StatSN 1 is missing) and issues a Status SACK to request StatSN 1, whereas StatSN 1 is still on its way to the initiator. Also, StatSN per connection avoids sharing StatSN state across connections [and thereby, NICs]. > >4)Logout response is there to make sure that you had an orderly close (not > >necessarily a shutdown - e.g., take out an adapter for maintenance) and no > >other recovery is needed. Recall that it can be sent on a healthy > >connection for a bad one. > > Assuming a healthy connection, what additional information is in > the Logout Response pdu that is not seen by the Logout sender when > the Logout receiver closes the connection? Or, asked another way, > if response=1 in the Logout Response pdu, what happens... I agree with Jon on the logout response issue and this is the same position I have stated earlier on this issue. The logout response adds little value and logout can be a 1-way process with the implicit response being the target closing the TCP connection. Initiators typically do not care about the logout response. IOW, Initiators typically don't care if logout response indicates "cleanup failed". Regards, Santosh --------------2AF68DCAD7654A12D6AFD818 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------2AF68DCAD7654A12D6AFD818-- From owner-ips@ece.cmu.edu Sat Mar 17 00:47:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA20776 for ; Sat, 17 Mar 2001 00:47:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2H3Occ13955 for ips-outgoing; Fri, 16 Mar 2001 22:24:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2H2imr11476 for ; Fri, 16 Mar 2001 21:44:48 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 16 Mar 2001 21:44:42 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07080152CE@corpmx9.isus.emc.com> From: Black_David@emc.com To: Charles.Binford@BlueSpruceNet.com, ips@ece.cmu.edu Subject: RE: iscsi: rev 05 2.5.1 requires ACA support Date: Fri, 16 Mar 2001 21:44:40 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Considering Charles's first two problems: > - problem 1) When an initiator's I/Os are cleared by it own actions (e.g. > sending a task management function of abort task set) how does it know which > of issued commands were aborted and which were still in-flight and had not > yet made it to the target. Ordered delivery solves this one, but can cause: > - problem 2) (closely related to 1) How are task management functions > delivered - with the immediate delivery option (CmdSN=0) or not? (reference > 1.2.2.1, top of page 12) If immediate, then problem 1 is aggravated if > multiple connections are used. If not immediate, then we can have a problem > with the task management function never being delivered because the I/O we > are trying to abort never made it to the target and the target's iSCSI layer > wont' pass up the task management function because the CmdSN number is > wrong. And this could be even worse if the I/O to be aborted made it to the target, but some other I/O is stuck in a connection and hence holding up the task management function, even though one or more commands to be aborted are merrily executing. > + solution to 1 and 2) I propose all task management functions be required > to use the following: > - immediate delivery option > > - utilize a new task management function sequence number field (call it > TskMgmtSN) > > - all task management functions shall be sent on all active connections > with the same value for TskMgmtSN > > - the target must receive the same task management function (identified by > TskMgmtSN) on all active connections before it acts on it. (Use a timer > (TBD) to weed out and close down defective connections.) The target still has to receive the task management command on the slow connection, or time out and close the slow connection before it can execute the task management command. This is vulnerable to the "arbitrary delay by some unrelated command" version of problem 2). If both problems need to be solved, the task management command may have to be sent at least twice when there are multiple connections active: - With immediate delivery to take effect as soon as possible. - With ordered delivery to ensure that everything it is supposed to affect has made it to the target. My reaction to this entire area that having iSCSI do any duplication of task management commands transparently to SCSI is introducing a bunch of extra complexity that doesn't seem to deliver a lot of extra value. Any user of iSCSI already has to decide between immediate and ordered delivery, and the less we muck with the task management commands, the more likely the user will be able to obtain desired/expected/understood behavior. Comments? --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Sat Mar 17 00:48:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA20798 for ; Sat, 17 Mar 2001 00:48:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2H43eM16318 for ips-outgoing; Fri, 16 Mar 2001 23:03:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2H43Pr16304 for ; Fri, 16 Mar 2001 23:03:25 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Fri, 16 Mar 2001 20:03:18 -0800 Message-ID: From: Charles Monia To: ips@ece.cmu.edu Cc: Charles Binford , Charles Monia Subject: RE: iscsi: rev 05 2.5.1 requires ACA support Date: Fri, 16 Mar 2001 20:03:14 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi: I though the action items in the following extracts from the T10 CAP minutes were intended to address this issue. What happened? From T10 SCSI Commands, Architecture, & Protocol Working Group Meeting Minutes -- January 17, 2001 T10/01-007r1 "4.4 Queue Locking for Busy, Task-Queue-Full, & Reservation Active [Satran] "Julian Satran presented a discussion regarding the lack of ACA-like interlocks on the BUSY, TASK SET FULL, and RESERVATION CONFLICT status codes (01-048r0). The group tentative agreed to resolve the issue by providing an initiator selectable way to make BUSY, TASK SET FULL, and RESERVATION CONFLICT status codes translate to CHECK CONDITION status with informative sense data. Ed Gardner agreed incorporate the tentative resolution in the next revision of his proposal (see agenda item 5.5)." "5.5 Unit attention issue (00-359) [Gardner] "Ed Gardner described a problem where in autosense clears a unit attention condition (00-359r1). Basically, the combination of commands in flight and autosense results in a condition where the initiator may not be able to stop commands that should be stopped based on unit attention conditions. Ed asked for recommendations on how the initiator should indicate that the new feature was being requested (recognizing that there must be a way to maintain the current behavior). Options discussed were Control byte (plus INQUIRY bit), Control mode page, and protocol specific. The group recommended using the Control mode page and Ed agreed to bring a specific proposal based\ on that recommendation to the next meeting." Charles > -----Original Message----- > From: Charles Binford [mailto:Charles.Binford@BlueSpruceNet.com] > Sent: Friday, March 16, 2001 11:57 AM > To: ips@ece.cmu.edu > Subject: RE: iscsi: rev 05 2.5.1 requires ACA support > > > (sorry this got kind of lengthy - but I do have proposed > solution to what I > believe are the problems below) > > Julian, > I don't think iSCSI needs to look at the NACA bit in the CDB > - I don't think > iSCSI should care about ACA at all. > > In my 10+ years of working with SCSI disk products I've yet > to see a host > driver that cared about ordering. As David said below, if there is an > ordering dependency then the application waits for completion > of the first > I/O before releasing the second. One gets the best > performance from a SCSI > block I/O target (from a single disk to a large RAID > controller) by issuing > queued commands with simple tags and allowing the target to > freely re-order > the commands internally. When an error happens it doesn't matter what > commands are in-flight because they are all independent I/Os. > This is the > way high performance SCSI disks and disk subsystems work. It > would be a > mistake (in my opinion) for iSCSI to come in and mandate the extra > complexity of ACA to handle a corner case most of the SCSI world it is > trying to penetrate doesn't care about. > > Granted, tape and other sequential devices are a different > story. However, > I participated heavily in T11 for 2 to 3 years helping develop a error > recovery scheme that would allow SCSI tape devices to work > with FC. The > bottom line, however, was that while we talked about it over > and over, no FC > tape vendor had a device that supported queuing. If you > don't queue, then > nothing is in-flight when an error occurs. (streaming is > accomplished by > caching in the target). After this drug on for a couple of > years and the > complexities of queuing to sequential devices became evident, > a new project > was started in T10 to develop a new tape command set that > included relative > block addresses in the CDB - making the new tape model very similar to > disk - each command can be interpreted independently and > precise delivery > order is not a fundamental requirement. > > =================== From owner-ips@ece.cmu.edu Sat Mar 17 03:31:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA05379 for ; Sat, 17 Mar 2001 03:31:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2H6IgM24194 for ips-outgoing; Sat, 17 Mar 2001 01:18:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta6.snfc21.pbi.net (mta6.snfc21.pbi.net [206.13.28.240]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2H6I4r24165 for ; Sat, 17 Mar 2001 01:18:04 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta6.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GAB003UEW5POE@mta6.snfc21.pbi.net> for ips@ece.cmu.edu; Fri, 16 Mar 2001 22:17:53 -0800 (PST) Date: Fri, 16 Mar 2001 22:15:46 -0800 From: Douglas Otis Subject: RE: iSCSI: Rationale behind the SLP draft In-reply-to: <3AB283DE.D67F4C41@cisco.com> To: Mark Bakke Cc: IPS Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mark, The SLP multi-cast will not work to the same level as DHCP due to a lack of co-operation with networking equipment. LDAP can provide information based on client attributes within 100k of code if embedded. As LDAP can be structured, the size of LDAP data can be constrained to fit within a switch without much wasted code and with an ability to extend other devices. For small local domain only versions, see slapd and slurpd. The principle aspect provided by iSNS is signaling of fabric configuration changes. Such could be done by configuring IPS servers to run with a slave to then signal affected clients based on login information. iSCSI does not but should provide signaling done by this external system that now relays information out-of-band. The iSNS describes LDAP as a possible source of information so the same signaling between iSNS and LDAP could exist between various IPS servers rather than all of their clients. It would become the task of the IPS server to signal clients already connected without a keep-alive heart-beat needed for persistent iSNS connections otherwise. As LDAP is extensible, provides needed features if a schema is defined, and does not represent a burden for small to big configurations, the justification seems weak. From what could have been a simple schema definition, there are now two inter-related beta servers with vendor unique support to inter-operate with a beta protocol. Do you see an advantage of defining the schema as an alternative to these LDAP front-ends? Such a schema would help those wishing to make these front-ends and for the tools to service them. Without a schema, there can only be vendor unique solutions to support iSNS and there will be no vendor inter-operability with respect to configuration exchanges. Doug > Doug- > > SLP provides the multicast discovery we were looking for, and does > not require DHCP, although it can work with it. LDAP does not provide > service location; it provides database access. SLP also looks like a > simpler protocol than LDAP, and a simpler way to specify the template > (schema) for our attributes. > > LDAP provides database access; iSNS provides active management > services. I could not see using LDAP to do what iSNS does. Josh > could comment further on this one. > > Both SLP and iSNS are designed to be used with LDAP; an SLP DA > can use LDAP as its back-end database, and clients can be served > using both protocols. We felt that this capability gives us a > way for those who want to set up LDAP to make use of it, without > forcing it to be implemented in our devices. > > Anyway, LDAP does meet several of our requirements, but it stands right > in the middle of what SLP and iSNS are used for. It doesn't quite > give us what we want on the low end, and it doesn't do the management > on the high end. It could, however, be utilized as an excellent > database access protocol for these other services to be built on, > but that would not have to be visible to hosts and devices. > > SLP already has DHCP extensions, security extensions, etc., iSNS > borrows them. All we had to do for SLP was to define a template, > which is just a simple version of a schema. > > -- > Mark > > Douglas Otis wrote: > > > > Mark, > > > > If I have two servers with similar information to promulgate > and expectation > > of multi-casts reaching the target, I may find SLP efforts greater than > > desired over standard methods found using DHCP and LDAP. DHCP > can provide > > information of the location of the LDAP server as already defined in > > existing boot specifications. The BOOTP-DHCP multi-cast request already > > have relay provisions in routers and switches because of its > common use and > > has defined entries for LDAP server discovery. You could have > defined an > > LDAP schema that would have covered both iSNS and SLP without adding > > additional standards efforts for building yet another protocol > that does the > > same function as protocols already in common use. > > > > For an overview see: > > http://www.umich.edu/~dirsvcs/ldap/doc/guides/slapd/1.html#RTFToC1 > > > > Could you explain why LDAP is unsuitable for this application? > There are > > many conventions within LDAP in terms of extensibility that > seem to make it > > more suitable than SLP and iSNS. You will also find security extensions > > already in place for LDAP merged with BOOTP-DHCP. It seems you have > > problems to overcome that have already been solved. Not being clever, I > > tend to ask dumb questions. > > > > Doug > > > > > David Black has requested that I send something to the > > > list showing the rationale behind the SLP draft, including > > > its fit in the big picture and its relationship with iSNS. > > > Here's an attempt to do so. I believe it to be consistent > > > with the Naming & Discovery, iSNS, and iSCSI/SLP drafts, > > > and their IETF-50 presentations. > > > > > > The relevant drafts are: > > > > > > draft-ietf-ips-iSCSI-name-disc-00 > > > draft-ietf-ips-isns-00 > > > draft-bakke-iscsi-slp-00 > > > > > > Naming and Discovery Requirements > > > > > > These are the discovery and name services requirements > > > from draft-ietf-ips-iscsi-name-disc-00. From a discovery > > > requirements point-of-view, SLP and iSNS provide some > > > similar services. SLP provides very simple discovery services > > > for smaller networks; iSNS provides more comprehensive > > > management services for larger networks. Where they overlap, > > > SLP and iSNS can interoperate fairly easily. > > > > > > 1. Multicast method to discover targets, to allow zero- > > > configuration of initiators and targets. > > > > > > SLP provides this. > > > > > > 2. A method to look up a WWUI, and get one or more addresses. > > > "Where is target ?" > > > > > > SLP and iSNS both provide this. > > > > > > 3. A method to find targets that a given initiator should access. > > > "I am initiator ; which targets should I access?" > > > > > > SLP and iSNS both provide this. > > > > > > 4. Ability to limit discovery to relevant initiators and targets. > > > > > > iSNS uses Discovery Domain, which are used to limit which > initiators > > > and targets can discover each other. Each target and initiator > > > belong to Discovery Domain; the iSNS will not allow initiators to > > > discover targets outside their domain. > > > > > > SLP uses a named Scope. Each target and initiator is configured > > > with one or more scope names; a target will only respond to an > > > initiator who has a matching scope. Note that this is not a > > > security mechanism; an initiator can ask for any scope it wishes, > > > and a target can respond to any scope it wishes. This provides > > > a simple mechanism to limit discovery, but does not attempt to > > > provide security. > > > > > > 5. Access Lists by initiator WWUI > > > > > > SLP and iSNS both provide the ability to store a list of initiator > > > WWUIs that are allowed access to a given target. > > > > > > 6. iSCSI Object Model support > > > > > > SLP registers only the targets > > > iSNS registers targets, initiators, network entities, portals > > > > > > 7. TLV Message Format > > > > > > SLP and iSNS both do this. However, SLP uses a text key and > > > value for its attributes; iSNS uses a binary key value. > > > > > > 8. Authentication of discovery messages > > > > > > iSNS uses SLP's authentication mechanism > > > > > > 9. Registration and Query > > > > > > SLP registers targets only > > > iSNS registers targets, inititors, network entities, portals > > > > > > 10. State Change Notification > > > > > > iSNS provides a state change service for which entities can > > > register. > > > > > > SLP does not provide state change or event notification. There > > > is a draft on doing some notification, but it is not adequate for > > > propagating attribute changes, which are necessary for an > > > initiator to discover that it has gained (or lost) access to > > > a target, or that a target has added a new address. > > > > > > 11. Light-weight Protocol > > > > > > Neither protocol should be too much of a burden to implement. > > > > > > Adding authentication to the protocol can be done later, > and will be > > > the same effort for both. In the case that both SLP and iSNS are > > > supported in the same product, they can share the code for this > > > feature. > > > > > > 12. Compatibility with Boot Requirements > > > > > > Both the SLP template and the iSNS specification have worked to > > > ensure that they fit in with the boot requirements. > > > > > > SLP has defined DHCP options that allow it to locate a directory > > > agent, avoiding multicast traffic from initiators in a DHCP > > > environment. > > > > > > 13. Compatibility with LDAP > > > > > > Although this is not a requirement, we have seen the desire for > > > interworking with LDAP discussed on the list enough that it may > > > merit a requirement at some point. Both SLP and iSNS are designed > > > with LDAP compatibility in mind. > > > > > > 14. Entity Status Inquiry/Heartbeat > > > > > > iSNS provides an ESI/heartbeat ping message to monitor the > > > availability of initiators and targets. The heartbeat is > > > originated by the iSNS server. > > > > > > We have not defined this capability for SLP. It may be possible > > > to do something similar if targets advertise themselves with a > > > shorter lifetime, and initiators expire their discovered targets > > > appropriately. Adding a directory agent (DA) may be required for > > > this to work properly. > > > > > > 15. Distribution of X.509v3 Public Key certificates > > > > > > iSNS provides this capability. > > > > > > Our SLP template does not attempt to do this for two reasons: > > > > > > - SLP attributes are strings; the binary certificates would have > > > to be escaped or uuencoded. > > > > > > - iSCSI only registers targets; there is no way for targets > > > to discover an initiator's certificate. > > > > > > 16. Usage of Multicast > > > > > > SLP normally relies on multicast, for initiators to request > > > advertisements from targets. Multicast use can be eliminated > > > at the expense of adding a DA, and configuring its address on > > > each of the initiators and targets. > > > > > > iSNS does not use multicast. > > > > > > > > > We had originally started out looking for something to fulfill the > > > multicast requirements that iSNS did not do. However, we found that > > > if one already had SLP, it was just as easy to find individual > > > targets, or at least canonical targets using it as it was to locate > > > an iSNS service. SLP can then be used to handle discovery for > > > simpler host and device environments, and iSNS could be added later > > > to help manage these environments. > > > > > > SLP and iSNS both fulfill many of the requirements from > > > the naming & discovery draft. However, they don't completely > > > overlap, and they take two different approaches to discovery. > > > > > > SLP provides Discovery of Targets. > > > > > > SLP is used for discovery ONLY. It uses a distributed > > > model, that can start with initiators and targets, and > > > no directory servers. It can add Directory Agents to > > > scale to medium-sized environments and reduce or eliminate > > > the use of multicast. Some work is underway on mesh- > > > enhanced SLP (mSLP), which provides peer-to-peer information > > > exchange between DAs for larger environments. > > > > > > From a pure discovery point of view, SLP's chief weakness > > > is that it does not yet have a notification mechanism. > > > > > > - Discovery of iSCSI targets > > > - Discovery of iSNS services > > > - Multicast-optional discovery mechanism for targets and > > > other name services. > > > - Open source implementations > > > - Existing standard protocols > > > > > > iSNS provides Centralized Management of Initiators and Targets. > > > > > > In addition to target discovery, iSNS provides higher-level > > > functions such as centralized access management, active > > > monitoring, public-key certificate management, and state-change > > > notifications. > > > > > > - Discovery of iSCSI targets > > > - Centralized management of initiator and target access > > > - Active monitoring of initiators and targets > > > - State change notification > > > > > > Using SLP with iSNS > > > > > > Although SLP and iSNS solve several of the same problems, they > > > are fairly well-suited to interoperating. They share a common > > > authentication structure, which is not something one would want > > > to code twice. SLP can be used to do basic discovery where > > > configuring an iSNS server is not warranted, and can help > > > discover iSNS servers in environments where centralized management > > > is needed. It is relatively simple for an iSNS server to > > > provide a proxy SLP service agent on behalf of the targets it > > > manages, allowing initiators to participate that implement only > > > the basic SLP. Similarly, an iSNS server can work with gateways > > > to provide its services for targets that support only SLP. > > > > > > When initiators and targets support BOTH SLP and iSNS, there > > > are a few rules that should be followed: > > > > > > 1. Initiators can use both SLP and iSNS concurrently to discover > > > their targets. In fact, an initiator should be able to use > > > more than one iSNS, if it is accessing storage from two > > > different providers. This allows an initiator to discover > > > its locally-managed devices using SLP, and its centrally- > > > managed devices using iSNS. > > > > > > 2. Targets should be advertised using a single discovery method. > > > A target should be advertised either with SLP, or with a > > > single iSNS environment. Targets discovered multiple ways > > > would probably not break anything, but a target discovered > > > via multiple services could produce conflicting information > > > to the initiator. > > > > > > 3. Gateways or iSCSI proxies can be used to provide local SLP > > > discovery of targets that are managed using iSNS. > > > > > > Implementation Approach > > > > > > I recommend a three-phase approach to implementing iSCSI > > > naming and discovery. The first phase is to simply support the > > > WWUIs, and their use with the iSCSI protocol, and allow static > > > configuration of hosts and targets. The second is to support > > > SLP for simple discovery. This should be relatively quick to > > > implement, as the source for SLP is readily available. The third > > > would be to support iSNS as a storage management capability. > > > > > > 1. Simple naming and configuration > > > > > > - Admins configure targets with some form of access list, which > > > may include the initiator WWUIs that are supposed to access > > > them. > > > > > > - Admins configure initiators with the canonical target "iscsi" > > > for targets that may provide them services. > > > > > > - Initiators use the SendTargets text command to discovery their > > > targets. This eliminates configuring target WWUIs in the > > > initiator. > > > > > > 2. SLP for multicast and simple discovery > > > > > > - Instead of configuring initiators with target addresses, admins > > > enable SLP on the initiator. > > > > > > - Targets also enable SLP, and are discovered by the initiators. > > > > > > 3. iSNS for centralized management > > > > > > - Initiators and targets are configured to be managed by an iSNS > > > service. > > > > > > - Admins configure both hosts and targets from the iSNS server. > > > > > > - Initiators and targets use the iSNS protcol to discover > each other. > > > > > > Other Discovery Protocols > > > > > > There are several other discovery protocols; each has its > > > strengths and weaknesses. Here are some of them. Take a look > > > at "Building Networks on the Fly", IEEE Spectrum, March 2001 > > > for more information on these. > > > > > > 1. DNS SRV Records - these seemed too limiting, and there did > > > not seem to be much point in making DNS do more. The SLP > > > folks started their work because DNS SRV records didn't meet > > > their requirements. > > > > > > 2. LDAP, with an appropriate schema defined, could handle iSCSI > > > registrations and queries, and distribute essentially the > > > same information as SLP. There are three reasons we didn't > > > go directly to LDAP: > > > > > > 1. LDAP would require a heavier implementation than SLP > > > 2. LDAP would not solve the multicast requirements > > > 3. SLP is built to be compatible with LDAP; SLP can be used > > > by iniators and targets, with LDAP as a back-end database. > > > This is left as an exercise for the implementor. > > > > > > 3. Jini is controlled by a single company, and is tied to a > > > particular programming language, which is not appropriate > > > for an internet standard. > > > > > > 4. UPnP from Microsoft is based on XML data with an HTTP transport. > > > XML has a lot of merit for application interfaces like these, but > > > the code for XML and HTTP could be a bit heavy for a really simple > > > device. It's company-controlled anyway, so we can't use it. > > > > > > 5. Salutation - Have not explored this in depth. Salutation has > > > been around longer than the other protocols. It is made to be > > > more flexible; it does not assume IP. > > > > > > 6. Bluetooth, HAVi - These are done by controlled-membership > > > organizations, and are designed for specific non-IP environments. > > > > > > > > > -- > > > Mark A. Bakke > > > Cisco Systems > > > mbakke@cisco.com > > > 763.398.1054 > > > > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 > From owner-ips@ece.cmu.edu Sat Mar 17 03:33:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA05390 for ; Sat, 17 Mar 2001 03:33:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2H6XfE25032 for ips-outgoing; Sat, 17 Mar 2001 01:33:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta6.snfc21.pbi.net (mta6.snfc21.pbi.net [206.13.28.240]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2H6Wer24970 for ; Sat, 17 Mar 2001 01:32:41 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta6.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GAB005FIWU0T1@mta6.snfc21.pbi.net> for ips@ece.cmu.edu; Fri, 16 Mar 2001 22:32:26 -0800 (PST) Date: Fri, 16 Mar 2001 22:30:19 -0800 From: Douglas Otis Subject: RE: iscsi: rev 05 2.5.1 requires ACA support In-reply-to: <0F31E5C394DAD311B60C00E029101A07080152CE@corpmx9.isus.emc.com> To: Black_David@emc.com, Charles.Binford@BlueSpruceNet.com, ips@ece.cmu.edu Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, Even with management commands, position within the command stream is assumed. Should there be commands sent without an ability to verify relative position and with the potential for excessive skews between connections, the use of a zero seems clumsy. It could apply to command not yet received on different connections. Even though a command may have a sequence value, it could be 'Marked' for immediate use without strict adherence to sequence. The sequence value could be used to filter late comers, perhaps the cause for the management command in the first place. The task attribute could be used to understand if there is a need for immediate application or an iSCSI specific bit if that is too difficult. Doug > Considering Charles's first two problems: > > > - problem 1) When an initiator's I/Os are cleared by it own > actions (e.g. > > sending a task management function of abort task set) how does it know > which > > of issued commands were aborted and which were still in-flight > and had not > > yet made it to the target. > > Ordered delivery solves this one, but can cause: > > > - problem 2) (closely related to 1) How are task management functions > > delivered - with the immediate delivery option (CmdSN=0) or not? > (reference > > 1.2.2.1, top of page 12) If immediate, then problem 1 is aggravated if > > multiple connections are used. If not immediate, then we can have a > problem > > with the task management function never being delivered because > the I/O we > > are trying to abort never made it to the target and the target's iSCSI > layer > > wont' pass up the task management function because the CmdSN number is > > wrong. > > And this could be even worse if the I/O to be aborted made it to > the target, > but some other I/O is stuck in a connection and hence holding up the task > management function, even though one or more commands to be aborted > are merrily executing. > > > + solution to 1 and 2) I propose all task management functions > be required > > to use the following: > > - immediate delivery option > > > > - utilize a new task management function sequence number > field (call it > > TskMgmtSN) > > > > - all task management functions shall be sent on all active > connections > > with the same value for TskMgmtSN > > > > - the target must receive the same task management function > (identified > by > > TskMgmtSN) on all active connections before it acts on it. (Use a timer > > (TBD) to weed out and close down defective connections.) > > The target still has to receive the task management > command on the slow connection, or time out and close > the slow connection before it can execute the task > management command. This is vulnerable to the > "arbitrary delay by some unrelated command" version > of problem 2). > > If both problems need to be solved, the task management > command may have to be sent at least twice when there > are multiple connections active: > - With immediate delivery to take effect as soon as > possible. > - With ordered delivery to ensure that everything it > is supposed to affect has made it to the target. > > My reaction to this entire area that having iSCSI do > any duplication of task management commands > transparently to SCSI is introducing a bunch of extra > complexity that doesn't seem to deliver a lot of > extra value. Any user of iSCSI already has to decide > between immediate and ordered delivery, and the > less we muck with the task management commands, > the more likely the user will be able to obtain > desired/expected/understood behavior. > > Comments? > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Sat Mar 17 05:46:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA05833 for ; Sat, 17 Mar 2001 05:46:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2H8Vhx01456 for ips-outgoing; Sat, 17 Mar 2001 03:31:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2H8Uvr01430 for ; Sat, 17 Mar 2001 03:30:57 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA51554 for ; Sat, 17 Mar 2001 09:30:49 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id JAA52676 for ; Sat, 17 Mar 2001 09:28:39 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A12.002E9270 ; Sat, 17 Mar 2001 09:28:41 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 17 Mar 2001 10:31:10 +0200 Subject: Re: iSCSI: Towards a more effective PDU format Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, I will present at Minneapolis a survey of possibilities. Both your and Barry Reynhold's proposal will be listed. You will have a chance to say what you have to say. I have also a modified structure that enables one digest and a total AHSs length (as I mentioned earlier). I am not ignoring your proposal but I have a flight ahead and some more things to do during my remaining weekend hours. Regards, Julo "Robert D. Russell" on 16/03/2001 00:06:20 Please respond to "Robert D. Russell" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI: Towards a more effective PDU format Julo: Even if we give up on the idea of having a second header digest, I still believe that some changes should to be made to the PDU Format proposed in version 5 to make it simpler, more efficient to process, and more robust. I would therefore propose eliminating the use of the current WN Next-Qualifier scheme and replacing it as follows (this is just a slight modification of my earlier proposal in order to eliminate the second header digest). 1. Every PDU must start with a 48-byte Basic Header (BHS) that can be read in a single "read" operation. The format of this is identical to that proposed in section 2.2.4 of version 5, but with the addition of 2 fields (4-bytes) as described next. 2. Every BHS contains 2 fields at fixed locations: a) AHS_length, containing the total length of all Additional Header Segments (AHS) that follow. This field is 0 if there are no AHSs. b) DATA_length, containing the total length of all data that follows the AHSs. This field is 0 if there is no data. 3. If the AHS_length field in the BHS is non-zero, all the AHSs immediately follow the BHS in the PDU. The AHS_length value gives the total number of bytes in all the AHSs, allowing a single "read" operation to read them all. (The total header consists of 48+AHS_length bytes.) 4. If header digests are in use, one header digest covering the BHS and all the AHSs (if any) immediately follows the last AHS. This digest does NOT require a separate read, since the first read (of the BHS) should be for 48+(length-of-header-digest) bytes. If the AHS_length field is zero, there are no more reads. If the AHS_length field is non-zero, exactly that many more bytes need to be read in a second read. These additional bytes are appended to those obtained in the first read. The header digest will always be the last (length-of-header-digest) bytes. 4. If the DATA-length field in the BHS is non-zero, all the data immediately follows the header digest. If data digests are in use, a data digest immediately follows the data. 5. To save space within the BHS, the AHS-length field can be restricted to a single byte, and can be in units of 4-byte words. This allows up to 1020 bytes of AHSs to follow a BHS, which seems to be more than enough for the uses foreseen (so far, only extended CDB and bidi-read info). Currently, byte 2 is unused (reserved) in all PDU types except Login and Login response, where byte 6 is unused. Therefore, the AHS-length field can be added without increasing the BHS size of 48 bytes. 6. Each AHS should start with a word containing a TYPE field and a LENGTH field. The TYPE field should be enumerated rather than bit-field encoded, for easier decoding and future expansion. The LENGTH field is the number of bytes of additional information in this AHS that follows this word. 7. The DATA_length field should be a 4-byte field added to the 44-byte BHS of the current section 2.2.4 to give a new BHS of 48-bytes. However, since the version 5 44-byte header is always preceded by a 4-byte WN Next-Qualifier that would no longer be needed, there is no net change in the effective BHS size of 48-bytes. Advantages of this proposal over the current WN Next-Qualifer of version 5. 1. Because the receiver gets the AHS-length field from the BHS, it can obtain the entire set of ALL AHSs that follow the BHS in a single "read" operation (the current WN Next-Qualifier scheme requires a separate "read" for EACH additional header segment after the BHS). 2. By limiting the total size of all AHSs to 1020 bytes, a receiver can preallocate a fixed-length buffer of (48 + 1020 + size-of-header_digest) bytes for header processing (the current WN Next-Qualifier scheme has no limit on either total header size nor individual AHS sizes). 3. Since each AHS begins with a word containing the TYPE and LENGTH in fixed positions, an unknown TYPE (i.e., a type introduced in the future that is received by a legacy receiver) does NOT result in loss of synchronization during header processing -- the receiver knows the length of this AHS in any case, and can just skip over it (the current WN Next-Qualifier type determines how the length field is to be interpreted -- there is no "general rule", so unknown types mean loss of synchronization). 4. The format of the AHS is the familiar Type/Length/Value (T/L/V) structure (the current WN Next-Qualifier is T+1/L+1/V, but this does not seem to provide any benefit and only adds confusion and complexity -- see e-mails from David Black and Barry Reinhold). 5. The AHS TYPE is enumerated (the current WN Next-Qualifier is bit encoded, again without seeming to provide any benefit -- see e-mail from David Black). 6. Because the BHS always contains a 4-byte DATA_length field, the maximum data segment size is 4 Gigabytes (the current WN Next-Qualifier scheme, with the long data header removed, limits the data segment size to 16 Megabytes). 7. The "Header Digest Present" and "Data Digest Present" bits have been completely eliminated (the purpose of these in the current WN Next-Qualifier scheme was never explained, but they appear to be useless -- the presence or absence of digests has to be negotiated, and once negotiated, all PDUs must obey the negotiated decision). I see no disadvantages of this proposal relative to the current WN scheme. Since there is only one header digest, both schemes suffer the disadvantage of using unreliable data to find the header digest, which can lead to unnecessary blocking on "reads". Proposed iSCSI PDU Format +------------------------+ | required BHS | > fixed length of 48 bytes +------------------------+ | optional AHS 1 |\ | - - - - - - - - - - - | \ | optional AHS 2 | \ | - - - - - - - - - - - | > total length in AHS_length field in BHS | . . . . | / | - - - - - - - - - - - | / | optional AHS n |/ +------------------------+ | optional header digest | -- covers preceding (48 + AHS_length) bytes +------------------------+ | |\ | optional data | > total length in DATA_length field in BHS | |/ +------------------------+ | optional data digest | -- covers preceding (DATA_length) bytes +------------------------+ Thanks, Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 On Tue, 13 Mar 2001 julian_satran@il.ibm.com wrote: > > > Bob, > > Interesting. This is close to one of the variants I had for IETF-50 (a > clear header). > The only adavantage it has is that you are able to read all the AHSs in one > read. > The (admitedly academic) disadvantage it has is that you are limited in the > size of extensions and have some redundancy. > The basic issue that was raised - and there is no simple way out - is that > once you have lost a block (BHS in your suggested layout) you are out of > synch. There is no simple way around it (at least not one that can be > solved by changing layout) and an added digest (only the code or silicon to > account for it) is IMHO not warranted by what you gain. If you are willing > to spend silicon or code on this the making header failures less probable > and recovering if you fail by dropping the connection is a better bet > (using the redundancy for a coding gain). But this involves some > complexity too. > > Julo > From owner-ips@ece.cmu.edu Sat Mar 17 14:04:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA08628 for ; Sat, 17 Mar 2001 14:04:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2HGgsh09400 for ips-outgoing; Sat, 17 Mar 2001 11:42:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e1.ny.us.ibm.com (e1.ny.us.ibm.com [32.97.182.101]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2HGglr09357; Sat, 17 Mar 2001 11:42:47 -0500 (EST) Received: from northrelay02.pok.ibm.com (northrelay02.pok.ibm.com [9.117.200.22]) by e1.ny.us.ibm.com (8.9.3/8.9.3) with ESMTP id LAA83796; Sat, 17 Mar 2001 11:41:25 -0500 From: julian_satran@il.ibm.com Received: from D51MTA05.pok.ibm.com (d51mta05.pok.ibm.com [9.117.200.33]) by northrelay02.pok.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA133846; Sat, 17 Mar 2001 11:38:26 -0500 Received: by D51MTA05.pok.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 85256A12.005BCA08 ; Sat, 17 Mar 2001 11:42:34 -0500 X-Lotus-FromDomain: IBMIL@IBMDE@IBMUS To: psarkar@almaden.ibm.com cc: Black_David@emc.com, ips@ece.cmu.edu, owner-ips@ece.cmu.edu, ralphoweber@compuserve.com Message-ID: <85256A12.005BC7AF.00@D51MTA05.pok.ibm.com> Date: Sat, 17 Mar 2001 18:42:52 +0200 Subject: RE: iscsi: rev 05 2.5.1 requires ACA support Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Prasenjit, I think that David gave you a fair summary of the situation. The only think that I am uneasy about is that we (iSCSI) have direct way of controlling it. We require a target to support ACA because we know that on iSCSI connections there may be commands in flight. We have nothing to do with the way the SCSI initiator is going to use this (if at all). An iSCSI initiator is unaware of the ACA support. It would have been cleaner if SAM had something to say about high latency delivery subsystems and the requirements those create on targets and initiators. Regards, Julo From: Prasenjit Sarkar@IBMUS on 16/03/2001 19:24 To: Black_David@emc.com cc: ips@ece.cmu.edu, Julian Satran/Haifa/IBM@IBMIL, owner-ips@ece.cmu.edu From: Prasenjit Sarkar/Almaden/IBM@IBMUS Subject: RE: iscsi: rev 05 2.5.1 requires ACA support (Document link: Julian Satran - Mail) Hi David and Julian, To impart some finality into the discussion, can you summarize for the benefit of the mailing list what is distinctive about iSCSI that ACA must be made mandatory for targets. I'm not trying to start an argument here, but a few words of motivation might help us to understand as to why iSCSI-ACA must be made manadatory (required in most situations) versus optional (implement where needed). Regards, Prasenjit Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose |--------+------------------------> | | Black_David@em| | | c.com | | | Sent by: | | | owner-ips@ece.| | | cmu.edu | | | | | | | | | 03/16/2001 | | | 06:21 AM | | | | |--------+------------------------> >----------------------------------------------------| | | | To: Julian Satran/Haifa/IBM@IBMIL, | | ips@ece.cmu.edu | | cc: | | Subject: RE: iscsi: rev 05 2.5.1 requires | | ACA support | | | | | >----------------------------------------------------| If that's the case, then the wording that Ralph pointed out needs to be modified to indicate that ACA is used only when appropriate. --David > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Friday, March 16, 2001 2:34 AM > To: ips@ece.cmu.edu > Subject: RE: iscsi: rev 05 2.5.1 requires ACA support > > > > We are aware of the support for ACA missing from some drivers. > The situation is even exacerbated by the fact that certain exceptions that > are not errors per-se > will require ACA to be fired to accomodate for commands in flight (like > reservations, busy, task-set-full). > > However actions at the initiator can be fine-tuned with the NACA bit in > CDB > and the ACA atrribute for the task. > > Julo > > Black_David@emc.com on 16/03/2001 03:46:29 > > Please respond to Black_David@emc.com > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: iscsi: rev 05 2.5.1 requires ACA support > > > > > > After an error if you have commands in flight you want them all dropped > > until you specifically reset the ACA and restart the queue (prevent > things > > to be executed out of order). > > The T10 folks will have to go after this one in more detail, but Julian's > above statement is correct *only* if the commands in flight depend on > the one that caused the error (i.e. executing them out of order will cause > problems). This is generally not the case for disks where the usual > practice is to enforce command execution order dependencies > (e.g., database write ordering) in the operating system and applications > by waiting for responses (yes it's possible to do better, but lots of > existing software doesn't). The result is that commands in > flight can be executed in arbitrary order with arbitrary ones of them > failing without causing further difficulties. As Ralph has pointed out, > most hosts do not use ACA for disk-based storage, and if iSCSI > always does ACA, this will cause nasty integration issues. > > Just to stir the pot further, I believe Fibre Channel provides a negative > example, because if FC drops a frame (which is not a good idea, > but can still happen), the FC component that dropped the frame has no > clue about what ACA is, or how to get the target (which is not the > same piece of hardware) to enter ACA state. Both Class 2 and Class 3 > are vulnerable to this. > > Tapes are another matter - do we still have a tape expert on the list? > > I thought where we were headed on ACA was something along the > lines of: > - Targets MUST implement. > - Initiators MAY use. > - Initiator support for ACA is NOT REQUIRED. > which would imply a text key for Initiators to tell Targets > whether ACA behavior is expected. Did I miss something? > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > > From owner-ips@ece.cmu.edu Sat Mar 17 22:02:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA12284 for ; Sat, 17 Mar 2001 22:02:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2I0dwQ07324 for ips-outgoing; Sat, 17 Mar 2001 19:39:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com (mxic1.isus.emc.com [168.159.211.82]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2I0dmr07314 for ; Sat, 17 Mar 2001 19:39:48 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Sat, 17 Mar 2001 19:41:05 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07080152D2@corpmx9.isus.emc.com> From: Black_David@emc.com To: dotis@sanlight.net, ips@ece.cmu.edu Subject: RE: iscsi: rev 05 2.5.1 requires ACA support Date: Sat, 17 Mar 2001 19:39:40 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > Even with management commands, position within the command stream is > assumed. Should there be commands sent without an ability to verify > relative position and with the potential for excessive skews between > connections, the use of a zero seems clumsy. It could apply to command not > yet received on different connections. Even though a command may have a > sequence value, it could be 'Marked' for immediate use without strict > adherence to sequence. The sequence value could be used to filter late > comers, perhaps the cause for the management command in the first place. > The task attribute could be used to understand if there is a need for > immediate application or an iSCSI specific bit if that is too difficult. This sounds like what ordered delivery is currently specified to do, possibly coupled with a note to implementers suggesting that task management commands be executed promptly after delivery to SCSI. An example of a scenario that may work with immediate delivery ("use of a zero" above) is one in which some person or piece of software has determined or suspects that something undesirable is going on in/at the target device and wants it stopped NOW!!! (e.g., "What on earth is that tape robot doing???"). --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Sun Mar 18 00:47:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA14126 for ; Sun, 18 Mar 2001 00:47:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2I351x15822 for ips-outgoing; Sat, 17 Mar 2001 22:05:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2I340r15785 for ; Sat, 17 Mar 2001 22:04:00 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Sat, 17 Mar 2001 19:03:58 -0800 Message-ID: From: Joshua Tseng To: Douglas Otis , Mark Bakke Cc: IPS Subject: RE: iSCSI: Rationale behind the SLP draft Date: Sat, 17 Mar 2001 19:03:53 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug & Mark, My earlier response to Doug's message didn't seem to have gotten posted. (Is there something up with the reflector server or something?) The gist of my lost message is that LDAP is merely a network-based information repository where applications can store and retrieve their own application-specific data. Similarly, database engines such as a oracle and db2 are completely useless without an application (e.g., peoplesoft or SAP) to drive them. For IP storage, iSNS is the application which can be layered on top of LDAP. iSNS is the storage-aware application that can monitor and manage access and availability of IP storage devices. And it MAY use LDAP to store its data. For a more detailed comparison of LDAP and iSNS, see section 3.4 of the current -01 iSNS draft. I wrote this specifically with Doug's concerns in mind, and I don't think I need to repeat the material here. The main point is that LDAP is a general purpose tool. If you build naming and discovery services directly on the LDAP protocol, then you will live and die by its weaknesses. There will be just as much work defining and implementing the schema, and plus there will be many functions that LDAP will not be able to perform, such as iSNS's ESI, which will have to be incorporated into iSCSI or some other application. Regards, Josh > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Friday, March 16, 2001 10:16 PM > To: Mark Bakke > Cc: IPS > Subject: RE: iSCSI: Rationale behind the SLP draft > > > Mark, > > The SLP multi-cast will not work to the same level as DHCP > due to a lack of > co-operation with networking equipment. LDAP can provide > information based > on client attributes within 100k of code if embedded. As LDAP can be > structured, the size of LDAP data can be constrained to fit > within a switch > without much wasted code and with an ability to extend other > devices. For > small local domain only versions, see slapd and slurpd. > > The principle aspect provided by iSNS is signaling of fabric > configuration > changes. Such could be done by configuring IPS servers to run > with a slave > to then signal affected clients based on login information. > iSCSI does not > but should provide signaling done by this external system > that now relays > information out-of-band. The iSNS describes LDAP as a > possible source of > information so the same signaling between iSNS and LDAP could > exist between > various IPS servers rather than all of their clients. It > would become the > task of the IPS server to signal clients already connected without a > keep-alive heart-beat needed for persistent iSNS connections > otherwise. > > As LDAP is extensible, provides needed features if a schema > is defined, and > does not represent a burden for small to big configurations, the > justification seems weak. From what could have been a simple schema > definition, there are now two inter-related beta servers with > vendor unique > support to inter-operate with a beta protocol. Do you see an > advantage of > defining the schema as an alternative to these LDAP > front-ends? Such a > schema would help those wishing to make these front-ends and > for the tools > to service them. Without a schema, there can only be vendor unique > solutions to support iSNS and there will be no vendor > inter-operability with > respect to configuration exchanges. > > Doug > > > Doug- > > > > SLP provides the multicast discovery we were looking for, and does > > not require DHCP, although it can work with it. LDAP does > not provide > > service location; it provides database access. SLP also > looks like a > > simpler protocol than LDAP, and a simpler way to specify > the template > > (schema) for our attributes. > > > > LDAP provides database access; iSNS provides active management > > services. I could not see using LDAP to do what iSNS does. Josh > > could comment further on this one. > > > > Both SLP and iSNS are designed to be used with LDAP; an SLP DA > > can use LDAP as its back-end database, and clients can be served > > using both protocols. We felt that this capability gives us a > > way for those who want to set up LDAP to make use of it, without > > forcing it to be implemented in our devices. > > > > Anyway, LDAP does meet several of our requirements, but it > stands right > > in the middle of what SLP and iSNS are used for. It doesn't quite > > give us what we want on the low end, and it doesn't do the > management > > on the high end. It could, however, be utilized as an excellent > > database access protocol for these other services to be built on, > > but that would not have to be visible to hosts and devices. > > > > SLP already has DHCP extensions, security extensions, etc., iSNS > > borrows them. All we had to do for SLP was to define a template, > > which is just a simple version of a schema. > > > > -- > > Mark > > > > Douglas Otis wrote: > > > > > > Mark, > > > > > > If I have two servers with similar information to promulgate > > and expectation > > > of multi-casts reaching the target, I may find SLP > efforts greater than > > > desired over standard methods found using DHCP and LDAP. DHCP > > can provide > > > information of the location of the LDAP server as already > defined in > > > existing boot specifications. The BOOTP-DHCP multi-cast > request already > > > have relay provisions in routers and switches because of its > > common use and > > > has defined entries for LDAP server discovery. You could have > > defined an > > > LDAP schema that would have covered both iSNS and SLP > without adding > > > additional standards efforts for building yet another protocol > > that does the > > > same function as protocols already in common use. > > > > > > For an overview see: > > > http://www.umich.edu/~dirsvcs/ldap/doc/guides/slapd/1.html#RTFToC1 > > > > > > Could you explain why LDAP is unsuitable for this application? > > There are > > > many conventions within LDAP in terms of extensibility that > > seem to make it > > > more suitable than SLP and iSNS. You will also find > security extensions > > > already in place for LDAP merged with BOOTP-DHCP. It > seems you have > > > problems to overcome that have already been solved. Not > being clever, I > > > tend to ask dumb questions. > > > > > > Doug > > > > > > > David Black has requested that I send something to the > > > > list showing the rationale behind the SLP draft, including > > > > its fit in the big picture and its relationship with iSNS. > > > > Here's an attempt to do so. I believe it to be consistent > > > > with the Naming & Discovery, iSNS, and iSCSI/SLP drafts, > > > > and their IETF-50 presentations. > > > > > > > > The relevant drafts are: > > > > > > > > draft-ietf-ips-iSCSI-name-disc-00 > > > > draft-ietf-ips-isns-00 > > > > draft-bakke-iscsi-slp-00 > > > > > > > > Naming and Discovery Requirements > > > > > > > > These are the discovery and name services requirements > > > > from draft-ietf-ips-iscsi-name-disc-00. From a discovery > > > > requirements point-of-view, SLP and iSNS provide some > > > > similar services. SLP provides very simple discovery services > > > > for smaller networks; iSNS provides more comprehensive > > > > management services for larger networks. Where they overlap, > > > > SLP and iSNS can interoperate fairly easily. > > > > > > > > 1. Multicast method to discover targets, to allow zero- > > > > configuration of initiators and targets. > > > > > > > > SLP provides this. > > > > > > > > 2. A method to look up a WWUI, and get one or more addresses. > > > > "Where is target ?" > > > > > > > > SLP and iSNS both provide this. > > > > > > > > 3. A method to find targets that a given initiator > should access. > > > > "I am initiator ; which targets should I access?" > > > > > > > > SLP and iSNS both provide this. > > > > > > > > 4. Ability to limit discovery to relevant initiators > and targets. > > > > > > > > iSNS uses Discovery Domain, which are used to limit which > > initiators > > > > and targets can discover each other. Each target > and initiator > > > > belong to Discovery Domain; the iSNS will not allow > initiators to > > > > discover targets outside their domain. > > > > > > > > SLP uses a named Scope. Each target and initiator > is configured > > > > with one or more scope names; a target will only > respond to an > > > > initiator who has a matching scope. Note that this is not a > > > > security mechanism; an initiator can ask for any > scope it wishes, > > > > and a target can respond to any scope it wishes. > This provides > > > > a simple mechanism to limit discovery, but does not > attempt to > > > > provide security. > > > > > > > > 5. Access Lists by initiator WWUI > > > > > > > > SLP and iSNS both provide the ability to store a > list of initiator > > > > WWUIs that are allowed access to a given target. > > > > > > > > 6. iSCSI Object Model support > > > > > > > > SLP registers only the targets > > > > iSNS registers targets, initiators, network entities, portals > > > > > > > > 7. TLV Message Format > > > > > > > > SLP and iSNS both do this. However, SLP uses a text key and > > > > value for its attributes; iSNS uses a binary key value. > > > > > > > > 8. Authentication of discovery messages > > > > > > > > iSNS uses SLP's authentication mechanism > > > > > > > > 9. Registration and Query > > > > > > > > SLP registers targets only > > > > iSNS registers targets, inititors, network entities, portals > > > > > > > > 10. State Change Notification > > > > > > > > iSNS provides a state change service for which entities can > > > > register. > > > > > > > > SLP does not provide state change or event > notification. There > > > > is a draft on doing some notification, but it is not > adequate for > > > > propagating attribute changes, which are necessary for an > > > > initiator to discover that it has gained (or lost) access to > > > > a target, or that a target has added a new address. > > > > > > > > 11. Light-weight Protocol > > > > > > > > Neither protocol should be too much of a burden to implement. > > > > > > > > Adding authentication to the protocol can be done later, > > and will be > > > > the same effort for both. In the case that both SLP > and iSNS are > > > > supported in the same product, they can share the > code for this > > > > feature. > > > > > > > > 12. Compatibility with Boot Requirements > > > > > > > > Both the SLP template and the iSNS specification > have worked to > > > > ensure that they fit in with the boot requirements. > > > > > > > > SLP has defined DHCP options that allow it to locate > a directory > > > > agent, avoiding multicast traffic from initiators in a DHCP > > > > environment. > > > > > > > > 13. Compatibility with LDAP > > > > > > > > Although this is not a requirement, we have seen the > desire for > > > > interworking with LDAP discussed on the list enough > that it may > > > > merit a requirement at some point. Both SLP and > iSNS are designed > > > > with LDAP compatibility in mind. > > > > > > > > 14. Entity Status Inquiry/Heartbeat > > > > > > > > iSNS provides an ESI/heartbeat ping message to monitor the > > > > availability of initiators and targets. The heartbeat is > > > > originated by the iSNS server. > > > > > > > > We have not defined this capability for SLP. It may > be possible > > > > to do something similar if targets advertise themselves with a > > > > shorter lifetime, and initiators expire their > discovered targets > > > > appropriately. Adding a directory agent (DA) may be > required for > > > > this to work properly. > > > > > > > > 15. Distribution of X.509v3 Public Key certificates > > > > > > > > iSNS provides this capability. > > > > > > > > Our SLP template does not attempt to do this for two reasons: > > > > > > > > - SLP attributes are strings; the binary > certificates would have > > > > to be escaped or uuencoded. > > > > > > > > - iSCSI only registers targets; there is no way for targets > > > > to discover an initiator's certificate. > > > > > > > > 16. Usage of Multicast > > > > > > > > SLP normally relies on multicast, for initiators to request > > > > advertisements from targets. Multicast use can be eliminated > > > > at the expense of adding a DA, and configuring its address on > > > > each of the initiators and targets. > > > > > > > > iSNS does not use multicast. > > > > > > > > > > > > We had originally started out looking for something to > fulfill the > > > > multicast requirements that iSNS did not do. However, > we found that > > > > if one already had SLP, it was just as easy to find individual > > > > targets, or at least canonical targets using it as it > was to locate > > > > an iSNS service. SLP can then be used to handle discovery for > > > > simpler host and device environments, and iSNS could be > added later > > > > to help manage these environments. > > > > > > > > SLP and iSNS both fulfill many of the requirements from > > > > the naming & discovery draft. However, they don't completely > > > > overlap, and they take two different approaches to discovery. > > > > > > > > SLP provides Discovery of Targets. > > > > > > > > SLP is used for discovery ONLY. It uses a distributed > > > > model, that can start with initiators and targets, and > > > > no directory servers. It can add Directory Agents to > > > > scale to medium-sized environments and reduce or eliminate > > > > the use of multicast. Some work is underway on mesh- > > > > enhanced SLP (mSLP), which provides peer-to-peer information > > > > exchange between DAs for larger environments. > > > > > > > > From a pure discovery point of view, SLP's chief weakness > > > > is that it does not yet have a notification mechanism. > > > > > > > > - Discovery of iSCSI targets > > > > - Discovery of iSNS services > > > > - Multicast-optional discovery mechanism for targets and > > > > other name services. > > > > - Open source implementations > > > > - Existing standard protocols > > > > > > > > iSNS provides Centralized Management of Initiators and Targets. > > > > > > > > In addition to target discovery, iSNS provides higher-level > > > > functions such as centralized access management, active > > > > monitoring, public-key certificate management, and > state-change > > > > notifications. > > > > > > > > - Discovery of iSCSI targets > > > > - Centralized management of initiator and target access > > > > - Active monitoring of initiators and targets > > > > - State change notification > > > > > > > > Using SLP with iSNS > > > > > > > > Although SLP and iSNS solve several of the same problems, they > > > > are fairly well-suited to interoperating. They share a common > > > > authentication structure, which is not something one would want > > > > to code twice. SLP can be used to do basic discovery where > > > > configuring an iSNS server is not warranted, and can help > > > > discover iSNS servers in environments where centralized > management > > > > is needed. It is relatively simple for an iSNS server to > > > > provide a proxy SLP service agent on behalf of the targets it > > > > manages, allowing initiators to participate that implement only > > > > the basic SLP. Similarly, an iSNS server can work with gateways > > > > to provide its services for targets that support only SLP. > > > > > > > > When initiators and targets support BOTH SLP and iSNS, there > > > > are a few rules that should be followed: > > > > > > > > 1. Initiators can use both SLP and iSNS concurrently to discover > > > > their targets. In fact, an initiator should be able to use > > > > more than one iSNS, if it is accessing storage from two > > > > different providers. This allows an initiator to discover > > > > its locally-managed devices using SLP, and its centrally- > > > > managed devices using iSNS. > > > > > > > > 2. Targets should be advertised using a single discovery method. > > > > A target should be advertised either with SLP, or with a > > > > single iSNS environment. Targets discovered multiple ways > > > > would probably not break anything, but a target discovered > > > > via multiple services could produce conflicting information > > > > to the initiator. > > > > > > > > 3. Gateways or iSCSI proxies can be used to provide local SLP > > > > discovery of targets that are managed using iSNS. > > > > > > > > Implementation Approach > > > > > > > > I recommend a three-phase approach to implementing iSCSI > > > > naming and discovery. The first phase is to simply support the > > > > WWUIs, and their use with the iSCSI protocol, and allow static > > > > configuration of hosts and targets. The second is to support > > > > SLP for simple discovery. This should be relatively quick to > > > > implement, as the source for SLP is readily available. > The third > > > > would be to support iSNS as a storage management capability. > > > > > > > > 1. Simple naming and configuration > > > > > > > > - Admins configure targets with some form of access > list, which > > > > may include the initiator WWUIs that are supposed to access > > > > them. > > > > > > > > - Admins configure initiators with the canonical > target "iscsi" > > > > for targets that may provide them services. > > > > > > > > - Initiators use the SendTargets text command to > discovery their > > > > targets. This eliminates configuring target WWUIs in the > > > > initiator. > > > > > > > > 2. SLP for multicast and simple discovery > > > > > > > > - Instead of configuring initiators with target > addresses, admins > > > > enable SLP on the initiator. > > > > > > > > - Targets also enable SLP, and are discovered by the > initiators. > > > > > > > > 3. iSNS for centralized management > > > > > > > > - Initiators and targets are configured to be managed > by an iSNS > > > > service. > > > > > > > > - Admins configure both hosts and targets from the > iSNS server. > > > > > > > > - Initiators and targets use the iSNS protcol to discover > > each other. > > > > > > > > Other Discovery Protocols > > > > > > > > There are several other discovery protocols; each has its > > > > strengths and weaknesses. Here are some of them. Take a look > > > > at "Building Networks on the Fly", IEEE Spectrum, March 2001 > > > > for more information on these. > > > > > > > > 1. DNS SRV Records - these seemed too limiting, and there did > > > > not seem to be much point in making DNS do more. The SLP > > > > folks started their work because DNS SRV records didn't meet > > > > their requirements. > > > > > > > > 2. LDAP, with an appropriate schema defined, could handle iSCSI > > > > registrations and queries, and distribute essentially the > > > > same information as SLP. There are three reasons we didn't > > > > go directly to LDAP: > > > > > > > > 1. LDAP would require a heavier implementation than SLP > > > > 2. LDAP would not solve the multicast requirements > > > > 3. SLP is built to be compatible with LDAP; SLP can be used > > > > by iniators and targets, with LDAP as a back-end database. > > > > This is left as an exercise for the implementor. > > > > > > > > 3. Jini is controlled by a single company, and is tied to a > > > > particular programming language, which is not appropriate > > > > for an internet standard. > > > > > > > > 4. UPnP from Microsoft is based on XML data with an > HTTP transport. > > > > XML has a lot of merit for application interfaces > like these, but > > > > the code for XML and HTTP could be a bit heavy for a > really simple > > > > device. It's company-controlled anyway, so we can't use it. > > > > > > > > 5. Salutation - Have not explored this in depth. Salutation has > > > > been around longer than the other protocols. It is > made to be > > > > more flexible; it does not assume IP. > > > > > > > > 6. Bluetooth, HAVi - These are done by controlled-membership > > > > organizations, and are designed for specific non-IP > environments. > > > > > > > > > > > > -- > > > > Mark A. Bakke > > > > Cisco Systems > > > > mbakke@cisco.com > > > > 763.398.1054 > > > > > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 > > > From owner-ips@ece.cmu.edu Mon Mar 19 11:29:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA14477 for ; Mon, 19 Mar 2001 11:29:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2JE4Rc26364 for ips-outgoing; Mon, 19 Mar 2001 09:04:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lub1028.lss.emc.com ([168.159.39.28]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2JE3sr26343 for ; Mon, 19 Mar 2001 09:03:54 -0500 (EST) Received: from emc.com (IDENT:jhall@localhost.localdomain [127.0.0.1]) by lub1028.lss.emc.com (8.9.3/8.9.3) with ESMTP id KAA15519 for ; Mon, 19 Mar 2001 10:03:48 -0500 Message-Id: <200103191503.KAA15519@lub1028.lss.emc.com> To: ips@ece.cmu.edu Subject: Re: iSCSI: too much recovery? Date: Mon, 19 Mar 2001 10:03:48 -0500 From: "Jon Hall" Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh Rao writes: >StatSN needs to be on a per-connection basis. If StatSNs were on a per-session >basis, consider the following problem : > >- StatSN 1 is sent out on connection 1 >- StatSN 2 is sent out on connection 2 > >Since these 2 status' are sent on different connections, StatSN 2 could arrive >prior to StatSN 1. The initiator detects a hole in StatSN (StatSN 1 is missing) >and issues a Status SACK to request StatSN 1, whereas StatSN 1 is still on its >way to the initiator. > >Also, StatSN per connection avoids sharing StatSN state across connections [and >thereby, NICs]. Oh, that's what the SACK pdu is for :-). Seriously, I'm wanting to understand why the target is retaining the state necessary to support the SACK pdu. Because, it seems that the complexity that follows is large (for example, what about the potential for task tag collision?). Unless I'm missing something, the problem StatSN solves is a lost/corrupt iSCSI header. Since this is in the context of a TCP "reliable" byte stream, it happens when a TCP cksum fails to see corruption that an iSCSI CRC/digest does see. Assuming that the greater part of a iSCSI/TCP stream is data, this will happen mostly to data packets (which presumably are covered by a separate iSCSI data CRC). If that thinking holds, then it seems important to answer the question, what is the frequency of an iSCSI header digest/CRC failure? From this comes, is this event sufficiently rare that initiator recovery and retry is acceptable? Although some of my earlier questions presume the answer, I don't know the answer. -Jon From owner-ips@ece.cmu.edu Mon Mar 19 14:18:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA18914 for ; Mon, 19 Mar 2001 14:18:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2JGnTP07948 for ips-outgoing; Mon, 19 Mar 2001 11:49:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2JGmxr07922 for ; Mon, 19 Mar 2001 11:48:59 -0500 (EST) Received: from f2n23e. (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id LAA35496; Mon, 19 Mar 2001 11:41:42 -0500 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by f2n23e. (8.8.8m3/NCO v4.95) with ESMTP id JAA60144; Mon, 19 Mar 2001 09:48:56 -0700 Importance: Normal Subject: Re: iSCSI MIB Object Model To: Mark Bakke , ips@ece.cmu.edu From: "Jim Hafner" Date: Mon, 19 Mar 2001 08:48:54 -0800 Message-ID: X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/19/2001 08:48:55 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, Just to fill in a couple of things: 1) There is a move afoot in T10 to add some functions related to these LU inventory topics. There is a move to work on a bridge controller command set (George Penokie (Tivoli) and Robert Elliot (Compaq)). Such a command set would have ways of asking the bridge controller "what it is bridging". Additional George Penokie is pushing a set of proposals for getting other device level configuration information from a target (stuff like Logical unit inventory). 2) If your SCSI Target supports the Access Controls (approved for inclusion in SPC-3), there is a service action called "REPORT LU DESCRIPTORS" which essentially is a mechanism to get an inventory of all the logical units in the target device (technically as managed by the access controlls coordinator in the device). This is as a management type command, and is not subject to LUN Mapping constraints. This inventory information includes "default LUN value", VPD page83 LU WWUnique identifier, "Device Identifer" (as set by command) and, for disks the read capacity information. Jim Hafner From owner-ips@ece.cmu.edu Mon Mar 19 14:55:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA19931 for ; Mon, 19 Mar 2001 14:55:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2JHqWq12499 for ips-outgoing; Mon, 19 Mar 2001 12:52:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from goliath.cnchost.com (goliath.cnchost.com [207.155.252.47]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2JHpsr12456 for ; Mon, 19 Mar 2001 12:51:54 -0500 (EST) Received: from BSNCBINFORD (ts001d47.wic-ks.concentric.net [206.173.163.59]) by goliath.cnchost.com id MAA26578; Mon, 19 Mar 2001 12:51:50 -0500 (EST) [ConcentricHost SMTP Relay 1.10] From: "Charles Binford" To: Subject: RE: iscsi: rev 05 2.5.1 requires ACA support Date: Mon, 19 Mar 2001 11:52:05 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <0F31E5C394DAD311B60C00E029101A07080152D2@corpmx9.isus.emc.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, I interpreted Doug's idea different. I heard him saying we should give the task management function a sequence number, but the targets iSCSI layer pass it up as if it were immediate. Additionally, the iSCSI layer would be aware of, and watch for late coming I/Os on other connections using the knowledge of the CmdSN in the task management function. This idea can only work if the iSCSI layer is knowledgeable of the semantics of the task management function and has enough smarts to evaluate whether or not the late comer was effected by the task management function. I believe this puts more intelligence at the iSCSI layer than is necessary. I also don't think the iSCSI layer on the initiator side should/can have the intelligence to know whether a given task management function should be delivered with or without the immediate option as defined today. The prompt delivery, yet correct ordering of task management functions is a problem specifically tied to the way we have architected iSCSI with the multiple connections under one session. I believe the iSCSI should be responsible for solving this problem without any help knowledge from layers above. In response to my proposal to send the task management function on all connections to solve this problem, David Black had the response below (in another email, but copied here). My responses are in-line, marked with [cb]. ====From David Black======== The target still has to receive the task management command on the slow connection, or time out and close the slow connection before it can execute the task management command. This is vulnerable to the "arbitrary delay by some unrelated command" version of problem 2). [cb] I propose we have a relatively short timer defined for this. The target starts it as soon as it receives the first task management function. This puts a known upper bound on the problem. [cb] If both problems need to be solved, the task management command may have to be sent at least twice when there are multiple connections active: - With immediate delivery to take effect as soon as possible. - With ordered delivery to ensure that everything it is supposed to affect has made it to the target. My reaction to this entire area that having iSCSI do any duplication of task management commands transparently to SCSI is introducing a bunch of extra complexity that doesn't seem to deliver a lot of extra value. [cb]I agree, we don't want the iSCSI layer to duplicate task management commands as suggested here. That is not what I'm proposing. I'm saying send the task management function down all connections to synchronize the session, but the target SCSI layer only sees a single task movement function. [cb] Any user of iSCSI already has to decide between immediate and ordered delivery, and the less we muck with the task management commands, the more likely the user will be able to obtain desired/expected/understood behavior. [cb] Regardless of whether or not commands are being sent with ordered or immediate delivery we still have the problem of the task management function delivery. You need to know for sure which issued commands were caught by the task management function, yet you can't wait for an *unspecified* timer to pop on a slow connection. [cb] Charles Binford Blue Spruce Networks office: (316) 315-0382 / cell: (316) 210-6404 e-fax: (509) 756-4425 -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Black_David@emc.com Sent: Saturday, March 17, 2001 6:40 PM To: dotis@sanlight.net; ips@ece.cmu.edu Subject: RE: iscsi: rev 05 2.5.1 requires ACA support > Even with management commands, position within the command stream is > assumed. Should there be commands sent without an ability to verify > relative position and with the potential for excessive skews between > connections, the use of a zero seems clumsy. It could apply to command not > yet received on different connections. Even though a command may have a > sequence value, it could be 'Marked' for immediate use without strict > adherence to sequence. The sequence value could be used to filter late > comers, perhaps the cause for the management command in the first place. > The task attribute could be used to understand if there is a need for > immediate application or an iSCSI specific bit if that is too difficult. This sounds like what ordered delivery is currently specified to do, possibly coupled with a note to implementers suggesting that task management commands be executed promptly after delivery to SCSI. An example of a scenario that may work with immediate delivery ("use of a zero" above) is one in which some person or piece of software has determined or suspects that something undesirable is going on in/at the target device and wants it stopped NOW!!! (e.g., "What on earth is that tape robot doing???"). --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Mar 19 19:36:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA27523 for ; Mon, 19 Mar 2001 19:36:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2JMSZW03189 for ips-outgoing; Mon, 19 Mar 2001 17:28:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2JMSWr03183 for ; Mon, 19 Mar 2001 17:28:33 -0500 (EST) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id RAA20273; Mon, 19 Mar 2001 17:28:19 -0500 (EST) Date: Mon, 19 Mar 2001 17:28:19 -0500 (EST) From: Scott Bradner Message-Id: <200103192228.RAA20273@newdev.harvard.edu> To: ips@ece.cmu.edu, mbakke@cisco.com Subject: Re: iSCSI: Naming & Discovery Presention - Middle box taxonomy Sender: owner-ips@ece.cmu.edu Precedence: bulk > midtax is a BOF that is meeting this week. it is meeting right now Scott From owner-ips@ece.cmu.edu Mon Mar 19 19:38:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA27586 for ; Mon, 19 Mar 2001 19:38:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2JMKa602624 for ips-outgoing; Mon, 19 Mar 2001 17:20:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2JMJsr02593 for ; Mon, 19 Mar 2001 17:19:54 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id RAA00374 for ; Mon, 19 Mar 2001 17:19:49 -0500 (EST) Message-ID: <3AB685FA.C7E847EB@cisco.com> Date: Mon, 19 Mar 2001 16:19:38 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: IPS Subject: iSCSI: Naming & Discovery Presention - Middle box taxonomy Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit The middle box taxonomy referenced in today's naming and discovery draft is draft-carpenter-midtax-00.txt. midtax is a BOF that is meeting this week. -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Tue Mar 20 15:36:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA23631 for ; Tue, 20 Mar 2001 15:36:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2KI67625902 for ips-outgoing; Tue, 20 Mar 2001 13:06:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail07a.vwh1.net (mail07a.vwh1.net [209.238.9.57]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2KI0vr25528 for ; Tue, 20 Mar 2001 13:00:57 -0500 (EST) Received: from www.confluencenetworks.com (209.238.198.43) by mail07a.vwh1.net (RS ver 1.0.58s) with SMTP id 043802431 for ; Tue, 20 Mar 2001 13:00:29 -0500 (EST) Received: from kaveri [192.168.0.194] by confluencenetworks.com [127.0.0.1] with SMTP (MDaemon.v2.7.SP4.R) for ; Tue, 20 Mar 2001 10:04:25 -0800 From: "Nimesh Ray" To: Cc: Subject: RE: iSCSI spec qs. Date: Tue, 20 Mar 2001 09:58:46 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 X-MDaemon-Deliver-To: ips@ece.cmu.edu X-Return-Path: nimesh@confluencenetworks.com X-Loop-Detect: 1 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hello Julian, I am still not sure what value add is provided by the Synch and Steering model. 1. Does it avoid allocating any memory for assembly at the TCP layer ? If so, how would TCP recover lost packets ? 2. Does this require modifications to the "legacy TCP implementations" ? If so, what are the modifications required ? 3. How does the synch and steering layer help according to the proposal ? You would still need to allocate the buffers at the Synch and steering layer to provide redundant functionality that is already provided by TCP. The way I think of TCP is a dedicated, reliable connection. What you send at one end is what you get at the other end with all packet recovery taken care of at the TCP layer. So, in that case, iSCSI message framing is not an issue. Thanks for your input.. Regards, Nimesh -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Friday, March 16, 2001 2:29 AM To: Nimesh Ray Subject: Re: iSCSI spec qs. There issue is for both TCP and iSCSI. If you are building adapter that have to place the data somewhere and data arrives out of order you have to place it somewhere (even if you don't deliver it). That may be the final destination or some temporary memory. In the later case you will have: 1- to provide large amounts of temporary memory (possibly on the card) 2-copy data later Both are expensive and should be avoided. Regards, Julo "Nimesh Ray" on 15/03/2001 22:47:57 Please respond to "Nimesh Ray" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: iSCSI spec qs. Hello Julian, This is in reference to the iSCSI spec section 1.2.8 Message Synchronization and Steering. The following statement in section 1.2.8.2 page 21. "On the incoming path, the Synch and Steering layer extracts the Synch & Steering information from the TCP stream. Then it helps deliver (steer) the data stream to its final location and/or recover iSCSI PDU boundaries when some TCP packets are lost or received out of order" Seems like the assumption above is that you can receive packets out of order or lose packets if you are a client of TCP. Now as far as my understanding goes, TCP guarantees in-order and in-sequence guaranteed delivery of packets to the clients of TCP, in this case iSCSI. So, the receiver of the message should see exactly what the sender sent, with all the error recovery of lost packets done by the TCP layer. I believe that is exactly the rationale that went into selecting iSCSI over TCP. So, unless there is something else I do not understand, the "Synch and Steering" layer above TCP is not needed. Could you please clarify on this ? Thanks, Nimesh From owner-ips@ece.cmu.edu Tue Mar 20 16:45:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA26817 for ; Tue, 20 Mar 2001 16:45:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2KJc0t02740 for ips-outgoing; Tue, 20 Mar 2001 14:38:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from alemail1.firewall.lucent.com (alemail1.lucent.com [192.11.221.161]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2KJbgr02687 for ; Tue, 20 Mar 2001 14:37:42 -0500 (EST) Received: from alemail1.firewall.lucent.com (localhost [127.0.0.1]) by alemail1.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id OAA24247 for ; Tue, 20 Mar 2001 14:37:41 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by alemail1.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id OAA24241 for ; Tue, 20 Mar 2001 14:37:41 -0500 (EST) content-class: urn:content-classes:message Subject: IPS Interim meeting April 30 & May 1 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Date: Tue, 20 Mar 2001 14:37:40 -0500 Message-ID: Disposition-Notification-To: "Elizabeth Rodriguez" X-MimeOLE: Produced By Microsoft Exchange V6.0.4417.0 Thread-Topic: IPS Interim meeting April 30 & May 1 Thread-Index: AcCxXnVEKqVFJAdiR6yMCgRu+xPKtw== From: "Elizabeth Rodriguez" To: "IPS Mailing List (E-mail)" Cc: "Zane Daggett" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id f2KJbrs02701 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit IPS Interim Meeting An interim meeting of the IPS working group will be held on April 30 and May 1, 2001. This meeting will co-locate with the May T10 meeting in Nashua, NH. Monday, April 30 - FC related drafts Tuesday, May 1 - SCSI related drafts (Agenda for this meeting will be published early April) In addition to the above, a discussion of the development of a SCSI MIB will be held in the T10 CAP meeting on Wed, May 2. (Planned for morning) Please send RSVP (to insure we have enough room for attendees) to egrodriguez@lucent.com A block of rooms has been reserved at the Sheraton Nashua at a room rate of $133. Please ask for the group Hitachi Sheraton Nashua Hotel 11 Tara Boulevard Nashua, NH Phone: (603) 888-9970 Fax (603) 891-4179 e-mail: angie.forbes@sheraton.com Room cutoff date is April 9, 2001. The host requests that rooms be reserved as soon as possible to insure that we enough rooms reserved. More information may be found at From owner-ips@ece.cmu.edu Tue Mar 20 16:45:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA26818 for ; Tue, 20 Mar 2001 16:45:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2KJOtc01687 for ips-outgoing; Tue, 20 Mar 2001 14:24:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2KJOYr01658 for ; Tue, 20 Mar 2001 14:24:34 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id UAA04876; Tue, 20 Mar 2001 20:24:26 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id UAA79514; Tue, 20 Mar 2001 20:23:03 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A15.006A64A9 ; Tue, 20 Mar 2001 20:22:05 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: "Nimesh Ray" cc: ips@ece.cmu.edu Message-ID: Date: Tue, 20 Mar 2001 21:24:46 +0200 Subject: RE: iSCSI spec qs. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Nimesh, There where long and hot debates about why this is needed and how it should be achieved. On the mailing archives you will find docs describing in some detail the rational (there is a ID by Randy Haagens and an older memo I wrote on www.haifa.ibm.com/satran/ips) I'll certainly do a bad job trying to summarize all that went on in half a page but I am sure you will find searching through the archive rewarding. Look for the words framing and RDMA. If you still feel that you need explanations please give me a call next week. Regards, Julo "Nimesh Ray" on 20/03/2001 19:58:46 Please respond to "Nimesh Ray" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: RE: iSCSI spec qs. Hello Julian, I am still not sure what value add is provided by the Synch and Steering model. 1. Does it avoid allocating any memory for assembly at the TCP layer ? If so, how would TCP recover lost packets ? 2. Does this require modifications to the "legacy TCP implementations" ? If so, what are the modifications required ? 3. How does the synch and steering layer help according to the proposal ? You would still need to allocate the buffers at the Synch and steering layer to provide redundant functionality that is already provided by TCP. The way I think of TCP is a dedicated, reliable connection. What you send at one end is what you get at the other end with all packet recovery taken care of at the TCP layer. So, in that case, iSCSI message framing is not an issue. Thanks for your input.. Regards, Nimesh -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Friday, March 16, 2001 2:29 AM To: Nimesh Ray Subject: Re: iSCSI spec qs. There issue is for both TCP and iSCSI. If you are building adapter that have to place the data somewhere and data arrives out of order you have to place it somewhere (even if you don't deliver it). That may be the final destination or some temporary memory. In the later case you will have: 1- to provide large amounts of temporary memory (possibly on the card) 2-copy data later Both are expensive and should be avoided. Regards, Julo "Nimesh Ray" on 15/03/2001 22:47:57 Please respond to "Nimesh Ray" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: iSCSI spec qs. Hello Julian, This is in reference to the iSCSI spec section 1.2.8 Message Synchronization and Steering. The following statement in section 1.2.8.2 page 21. "On the incoming path, the Synch and Steering layer extracts the Synch & Steering information from the TCP stream. Then it helps deliver (steer) the data stream to its final location and/or recover iSCSI PDU boundaries when some TCP packets are lost or received out of order" Seems like the assumption above is that you can receive packets out of order or lose packets if you are a client of TCP. Now as far as my understanding goes, TCP guarantees in-order and in-sequence guaranteed delivery of packets to the clients of TCP, in this case iSCSI. So, the receiver of the message should see exactly what the sender sent, with all the error recovery of lost packets done by the TCP layer. I believe that is exactly the rationale that went into selecting iSCSI over TCP. So, unless there is something else I do not understand, the "Synch and Steering" layer above TCP is not needed. Could you please clarify on this ? Thanks, Nimesh From owner-ips@ece.cmu.edu Tue Mar 20 18:07:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA00320 for ; Tue, 20 Mar 2001 18:07:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2KKrwG08492 for ips-outgoing; Tue, 20 Mar 2001 15:53:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcamail04.zca.compaq.com (zcamail04.zca.compaq.com [161.114.32.104]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2KKrAr08418 for ; Tue, 20 Mar 2001 15:53:10 -0500 (EST) Received: by zcamail04.zca.compaq.com (Postfix, from userid 12345) id 1C3B2327; Tue, 20 Mar 2001 12:54:26 -0800 (PST) Received: from exchou-gh02.cca.cpqcorp.net (exchou-gh02.cca.cpqcorp.net [16.110.248.202]) by zcamail04.zca.compaq.com (Postfix) with ESMTP id 2D65A355; Tue, 20 Mar 2001 12:54:26 -0800 (PST) Received: by exchou-gh02.cca.cpqcorp.net with Internet Mail Service (5.5.2652.78) id ; Tue, 20 Mar 2001 14:53:03 -0600 Message-ID: <78AF3C342AEAEF4BA33B35A8A15668C659C115@cceexc17.americas.cpqcorp.net> From: "Elliott, Robert" To: "'ips@ece.cmu.edu'" , "'Jeff Fellin'" Subject: RE: iSCSI: comments on rev 05 Date: Tue, 20 Mar 2001 14:52:52 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2652.78) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk A few comments on general SCSI issues are embedded... > -----Original Message----- > From: Jeff Fellin [mailto:jkf@research.bell-labs.com] > Sent: Friday, March 16, 2001 1:41 PM > To: ips@ece.cmu.edu > Subject: iSCSI: comments on rev 05 > ... > Concern 6: Data transfer sizes > Since iSCSI is designed to work with the SCSI-3 > specification. There is an inconsistency of data sizes with > SCSI-3 WRITE(12)/READ(12) data transfer sizes. The WRITE(12) > and READ(12) have a 32-bit length field that is in units > of the device sector size, usually 512 bytes. This results > in an maximum size in bytes of 41 bits. Whereas, the > maximum size of a data transfer byte size in iSCSI is 32 > bits. How is iSCSI going to handle a SCSI-3 CDB requesting > a data transfer size in bytes greater than 32 bits? The initiator has to break up very long commands into multiple commands to fit its protocol. FCP-2 (SCSI over Fibre Channel) has a 32-bit DL field that imposes the same restriction. > ... > Concern 8: Section 2.18.2 Asynchronous Messages > The description of the various SCSI events in section > 2.18.2 appears to imply the iSCSI target has knowledge > of events concerning the SCSI initiator on the target > system, or perhaps that the iSCSI target actually > is the SCSI initiator on the target system. I believe this > violates the concepts of transferring SCSI CDB's across a > network into actually interpreting them, or having a strong > interaction between the two protocol layers that doesn't > appear in many other places. > > Why is there the need to have this type of interaction > between the clients of the iSCSI layer and the iSCSI layer? Julian agreed to remove this. The field will simply indicate whether or not a SCSI Event exists. > Concern 9: Section 3, Mode pages > I am not sure from reading the descriptions, but it sounds as if > the iSCSI target modifies the SCSI mode pages sent by it's SCSI > targets on return to the iSCSI initiator. This is due to > the reference to the reference to the disconnect-reconnect mode > page parameters in Section 1.2.5 and Section 3. Although I > cannot find a SCSI document defining a Logical Unit Control > Mode Page and Port Control Mode Page. > > Is this a reuse of a SCSI mode page name in iSCSI or > actually the SCSI mode pages being modified by the iSCSI > target? This appears to be similar > to the issue about SCSI events in Asynchronous Messages. See SPC-2 revision 19: section 8.3.7 Disconnect-reconnect page (02h) section 8.3.10 Protocol specific LUN page (18h) section 8.3.11 Protocol specific port page (19h) Page 19h was previously FCP-specific, but T10 redefined it as protocol specific and clarified that it affects the target port no matter which logical unit is being accessed. Page 18h was added to hold protocol specific parameters that can differ across logical units. Page 02h has protocol related properties, too. The device server providing these mode pages has to include information from the iSCSI target stack. SPC-2 includes this paragraph when describing page 02h: "The device server communicates the parameter values in this mode page to the service delivery subsystem. Similarly the application client may also communicate parameter values to the service delivery subsystem. This communication is internal to the initiator or target device and is outside the scope of SCSI." > ... > Question 3: Section 1.2.5 > The description about an initiator sending too much > unsolicited data to a target mentions this is > controlled by the FirstBurstSize parameter, > and is available from the disconnect-reconnect > mode page. I don't see a iSCSI PDU that would allow > an initiator to request this information. > Is the assumption the iSCSI initiator would send a SCSI > command to the iSCSI target to retrieve a SCSI mode > page. This appears to violate the concept of packaging > SCSI commands across a network, and instead actually > looking inside the SCSI command which should just be > an opaque SCSI CDB and SCSI data responses. iSCSI login can negotiate FirstBurstSize. See iSCSI-05 page 110 in Appendix D, Login/Text Miscellaneous Keys. If it supports this feature, an iSCSI initiator stack needs to track changes to the mode page value. An application client may choose to change the burst amount (within bounds set by the logical unit) using MODE SELECT and MODE SENSE commands. The iSCSI initiator stack either has to snoop MODE SELECT and MODE SENSE commands, or require the application client to use an API to set the FirstBurstSize used by the stack. The application client would program this when it makes a change. Only application clients that understand the protocol should change these pages, so it's reasonable to assume they can make special API calls to register their changes. A bridge between iSCSI and Fibre Channel might need to snoop the MODE SELECT and MODE SENSE commands as they pass through in each direction (possibly modifying data as it flows through) so the values represent its own capabilities correctly. > ... > Question 14: Sections 2.5 and 2.6 > Throughout the section I'm not sure what entities > Initiator and Target refer to. Is it the actual SCSI > initiator on the iSCSI initiator system > or the iSCSI initiator? Likewise for the term target. I think "SCSI initiator" and "iSCSI initiator" are the same. An iSCSI initiator is a SCSI initiator that uses iSCSI protocol. The SAM-2 term "application client" might be appropriate in some instances where initiator is used. That might map to the "SCSI initiator" you have in mind. > Comment 15: Section 2.6 > It appears the information on and Reset> is more appropriate in the paragraphs in section 2.5 I hope it will disappear altogether :-) > Question 16: Section 2.6 > The statement about the 'target MUST then close all of its TCP > connections' appears to imply the receipient of the task management > command is the iSCSI target. Or must the iSCSI target examine the > information in the SAM2 remote procedure call to determine specific > actions on the iSCSI target's part? A task management function is run by the "task manager" entity described in SAM-2. The task manager is part of the target (target device). It would have to communicate to the iSCSI target stack software or hardware to make this happen. > ... > Question 30: Section 6 > I am not sure what the term 'target' means in this > section. Is it the iSCSI target implementation or the > SCSI target devices attached to the iSCSI target system? See my previous initiator comments. --- Rob Elliott, Compaq Server Storage Robert.Elliott@compaq.com From owner-ips@ece.cmu.edu Wed Mar 21 15:07:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA11072 for ; Wed, 21 Mar 2001 15:07:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2LH4Ek04077 for ips-outgoing; Wed, 21 Mar 2001 12:04:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail01.san.yahoo.com (mail01.san.yahoo.com [209.132.1.35]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2LH3Xr04046 for ; Wed, 21 Mar 2001 12:03:33 -0500 (EST) Received: from bursar.muttsnuts.com (135.222.64.218) by mail01.san.yahoo.com (5.1.056) id 3AB773E40008A927 for ips@ece.cmu.edu; Wed, 21 Mar 2001 08:56:53 -0800 Message-Id: <5.0.2.1.2.20010321165727.00a65000@mail.muttsnuts.com> X-Sender: markemuttsnuts@mail.muttsnuts.com X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Wed, 21 Mar 2001 17:00:44 +0000 To: "'ips@ece.cmu.edu'" From: "Mark S. Edwards" Subject: FCP framing -- checksum/CRC Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk Considering the group has just voted to try and cripple software implementations of iFCP/FCIP by the use CRCs on the header, should we not go the same way as iSCSI and make the use of these CRCs as optional ??? Mark. From owner-ips@ece.cmu.edu Wed Mar 21 17:50:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA20180 for ; Wed, 21 Mar 2001 17:50:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2LKWKg20198 for ips-outgoing; Wed, 21 Mar 2001 15:32:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from auemlsrv.firewall.lucent.com (auemail1.lucent.com [192.11.223.161]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2LKVfr20121 for ; Wed, 21 Mar 2001 15:31:41 -0500 (EST) Received: from auemlsrv.firewall.lucent.com (localhost [127.0.0.1]) by auemlsrv.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id PAA04622 for ; Wed, 21 Mar 2001 15:31:40 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by auemlsrv.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id PAA04608 for ; Wed, 21 Mar 2001 15:31:40 -0500 (EST) Subject: Fibre Channel Design Team forming Date: Wed, 21 Mar 2001 15:31:39 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Message-ID: Disposition-Notification-To: "Elizabeth Rodriguez" content-class: urn:content-classes:message X-MimeOLE: Produced By Microsoft Exchange V6.0.4417.0 Thread-Topic: Fibre Channel Design Team forming Thread-Index: AcCyIXTM7RS5lRKkQy2+VhiBe7HCCw== From: "Elizabeth Rodriguez" To: "IPS Mailing List (E-mail)" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id f2LKWHs20190 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Hello all, A small Fibre Channel common encapsulation design team is being formed, to define a common header encapsulation for the Fibre Channel drafts -- FCIP and iFCP. Anyone interested in participating on this design team should notify either myself or David Black by Friday, March 23. Thank you, Elizabeth Rodriguez From owner-ips@ece.cmu.edu Wed Mar 21 17:50:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA20200 for ; Wed, 21 Mar 2001 17:50:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2LKYIt20374 for ips-outgoing; Wed, 21 Mar 2001 15:34:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2LKXtr20342 for ; Wed, 21 Mar 2001 15:33:56 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id VAA55724 for ; Wed, 21 Mar 2001 21:33:48 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id VAA82854 for ; Wed, 21 Mar 2001 21:32:22 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A16.0070BBE2 ; Wed, 21 Mar 2001 21:31:21 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 21 Mar 2001 22:34:06 +0200 Subject: foils Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, The slides that I presented at IETF-50 and the New-Format proposal are up at: http://www.haifa.il.ibm.com/satran/ips Barry Reinhold is with Trebia Networks not with IBM but I did not master Power Point good enough to remove the IBM Logo and leave the slides unharmed in 5 minutes (nor did I have the patience to try harder -:)). Regards, Julo From owner-ips@ece.cmu.edu Thu Mar 22 13:21:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA16653 for ; Thu, 22 Mar 2001 13:21:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2MG8bP12843 for ips-outgoing; Thu, 22 Mar 2001 11:08:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2MG8Xr12835 for ; Thu, 22 Mar 2001 11:08:33 -0500 (EST) Received: from ahganemw2k (dhcp-161-44-68-139.cisco.com [161.44.68.139]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with SMTP id LAA24440 for ; Thu, 22 Mar 2001 11:08:27 -0500 (EST) From: "Ayman Ghanem" To: Subject: RE: foils Date: Thu, 22 Mar 2001 10:07:15 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Bullet 3 of slide 6 (new PDU format) says that first read is 48 bytes + header digest size. I assume this is a typo as the header digest comes after the AHS. -Ayman From owner-ips@ece.cmu.edu Thu Mar 22 14:04:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA17842 for ; Thu, 22 Mar 2001 14:04:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2MH8f916910 for ips-outgoing; Thu, 22 Mar 2001 12:08:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2MH7lr16824 for ; Thu, 22 Mar 2001 12:07:48 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA110736 for ; Thu, 22 Mar 2001 18:07:33 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA189582 for ; Thu, 22 Mar 2001 18:06:05 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A17.005DDAB2 ; Thu, 22 Mar 2001 18:05:08 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 22 Mar 2001 19:06:47 +0200 Subject: RE: foils Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Correct - I there are several typos (all hopefully obvious). Barry (who has the source) might correct them and send me an update. Julo "Ayman Ghanem" on 22/03/2001 18:07:15 Please respond to "Ayman Ghanem" To: ips@ece.cmu.edu cc: Subject: RE: foils Bullet 3 of slide 6 (new PDU format) says that first read is 48 bytes + header digest size. I assume this is a typo as the header digest comes after the AHS. -Ayman From owner-ips@ece.cmu.edu Thu Mar 22 15:11:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA19963 for ; Thu, 22 Mar 2001 15:11:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2MHxkL20515 for ips-outgoing; Thu, 22 Mar 2001 12:59:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2MHxMr20491 for ; Thu, 22 Mar 2001 12:59:23 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Thu, 22 Mar 2001 09:59:16 -0800 Message-ID: From: Joshua Tseng To: ips@ece.cmu.edu Subject: iSNS/SLP Comparison Date: Thu, 22 Mar 2001 09:59:15 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, As promised, here is the text of the detailed analysis completed several weeks ago on the feasibility of using the SLP protocol to support the storage name service functionality for iSCSI. This issue was hashed and rehashed multiple times within the Naming & Discovery Team. Mark Bakke and I had numerous online and offline discussions on SLP and iSNS, and in the end we both concluded that SLP would not provide the needed functionality for a storage name service. Rather, we agreed SLP can be used to do simple discovery of iSCSI targets in a device-centric management model, without the aid of the iSNS. On the other hand, iSNS would provide the centralized management function that would allow iSCSI to scale to enterprise-class networks. The bottom line is that iSNS is a storage-aware application that is aware of client attributes, and can take appropriate action when events occur in the network. SLP is a generic discovery protocol and has no intelligence of storage devices and their attributes. LDAP alone is inappropriate for a similar reason in that it is a generic protocol, not an application. However, both LDAP and SLP can be used to support the iSNS application. LDAP to store and distribute attribute values in a directory-enabled iSNS implementation, and SLP to discover the iSNS service itself. Josh --------------------------------------------- Comparisons between iSNS and SLPv2: 1) State Change Notification: SLPv2 does not currently support it. This is an important function that exists in enterprise-scale Fibre Channel SANs. For SLPv2 to be feasible for large enterprise-scale storage networks, it would have to incorporate an asynchronous messaging mechanism that is closely tied to entities identified by service attributes. Specifically, this means SLPv2 must develop the intelligence to recognize a significant event, identify the entities (i.e., WWUI's) that are affected by that event by examining the DD/Zoning of each entity, and send a directed unicast message to the entities represented by those WWUI's. Such functionality seems to be beyond the intent and scope of SLP. WWUI would be stored as an attribute, and in SLP, attributes are simple values which carry no significance to SLP. It is beyond the purpose of SLP to interpret the contents of any attribute stored for a service. However, in order to support SCN's, SLP would need interpret the WWUI attribute values and map it to a destination IP address (i.e., extract the URL attribute and query for an IP address) for the State Change Notification. Therefore, we cannot count on SLP's ability to fulfill the State Change Notification requirement, even with future upgrades to the SLP protocol. 2) Multicast RFC 2608 contains "MUST" language that requires SA's to listen to multicast requests from UA's and DA's. However, in order to fulfill the SNS role, SA's do not need this capability, and multicast should not be used in many cases. An implementer will be faced with the choice of either compliance with the RFC 2608 and implementing a capability that is not needed, or being non-compliant and implementing only what is needed for the SNS. There are other areas in SLP which describe capabilities that are not needed for SNS. Multicast is not required for SNS because it may not be available in some networks, particularly the Public Internet or in a business partner's network. I shall assume no multicast functionality, since SNS must be able to work over the Public Internet per IETF Charter. 3) Zoning/Discovery Domains: The scoping functionality cannot be used for "Zoning/Discovery Domain" capability, because common scoping is a requirement for even DA's and SA's to communicate with each other. SLP assumes SA's, UA's, and DA's are already configured with the appropriate scoping, and SLP itself cannot be used to create, delete, or modify a scoping of an SA, UA, or DA, because they can't even communicate unless they already have common scoping. Therefore, a new attribute such as "Zone" in Mark's template is needed. But is the Zone an attribute of the WWUI or iSCSI instance? At the current time, it's unclear if the service represents a single WWUI, or an "iSCSI Instance" which may have multiple WWUI's. Either way, there is a disadvantage: a. If the service registers an iSCSI Instance, then the Zone/Discovery Domain is an attribute of the service (i.e., iSCSI instance), and not of the WWUI (i.e., iSCSI device), ALL WWUI's belonging to the iSCSI instance must be assigned to the same Zone/Discovery Domain. That means, if an iSCSI target device has 50 WWUI's, then all 50 must be assigned to the same Discovery Domain, and initiators must perform an iSCSI login to each of the 50 devices. While a native iSCSI device with 50 controllers may be rare, the real impact is on iSCSI-FC gateways. The gateway must proxy for a network of Fibre Channel devices, and the gateway will likely represent each FC device with a WWUI (using the "eui.xxx" WWUI format). But that means the user has no means of limiting access and discovery to a subset of devices in the FC network. b. If the "service" registers a single WWUI, then each "iSCSI instance" will have to register multiple times if it supports more than one WWUI. Once again, an iSCSI-FC gateway will have a large number of individual registration messages to send for any change to its attributes, and updates to receive for any change in status to other members in a common Zoning/Discovery Domain. For example, if the gateway supports 50 FC devices that are in a common Zone with initiator I, and initiator I is somehow removed from the Zone then the iSCSI-FC gateway will need to query the DA 50 times in order to make this simple update. And the last significant issue I'm aware of with Zones/Discovery Domains is that since the Zone is already an attribute, additional attributes cannot be associated with the Zone. This means a VLAN tag, multicast group address, or user-assigned symbolic name cannot be stored as an attribute of the Zone, in the DA. 4) Heartbeat/Client Status Inquiry: SLP accomplishes this function through a multicast DAAdvert. However, SLP does not specify an alternative if multicast is not available. Rather, SLP relies on the SA's and the UA's to continue refreshing their registrations in the DA at regular intervals. The refresh interval is dependent on the SA and UA settings, and is an interval between 0 and 18 hours. Since the DA has no control over the interval at which SA's send in registrations, it could get overwhelmed if there are a large number of SA's and UA's query for and register data. And the only way that the DA can discover (in the absence of multicast) that an SA has been removed from the network, is to wait until the timeout interval expires for SA registration refresh. Set the expiration timer too low, and the SA's will be re-registering with the DA too often and overwhelm it. Set the timer too high, and it could be hours or days before the DA discovers that an SA (i.e., iSCSI target) has disappeared. That would be unacceptable for an enterprise-class user. This is the original reason why SLP relied on the multicast advertisement. There is no unicast advertisement originated by the DA that I am aware of that is not in response to a multicast request by an SA. On the other hand, iSNS unicasts heartbeat messages to each iSNS client, and can adjust the interval at which it is comfortable sending out heartbeats, depending on its processing load. 5) Non-string-based attributes: SLP is optimized to store and transmit string-based attributes. However, the SNS must be able to store non-string attributes, such as an X.509 public key certificate. The only way that SLP can do this is if it escapes each byte of binary data. This "escaped" SLP formatting will effectively double the size of the original X.509 certificate. This is an additional burden for iSCSI devices, that will need to take the X.509 "blob" and place an "/" between every byte of binary data. Processing requirements for the DA host (which are already significantly more than iSNS server from the above issues 1-4), will also be increased as the a companion application on the DA host may need to extract the original certificate so that it can verify the signature and authenticate its origin. Also, doubling the size of the X.509 certificate in the SLP protocol message may overflow the size of the multicast datagram, which is a no-no. 6) Security: Because the DA, SA, and UA are all using the same scope ("SNS"), and Zone is now an attribute of the iSCSI device (either the WWUI or the iSCSI Instance), the DA (i.e., the SNS) must trust each of the SA's as far as access authorization. Although initiators are not supposed to change the attributes of a target, nothing prevents a wayward initiator from doing it anyway. Once again, attribute values have no significance to SLP, and there is no way for SLP to disallow a query or registration command by examining the value stored in an attribute field for the service that is the object of the query or request. This will become an issue when sharing the DA database with a business partner. Even if a new special "Zone" or "Discovery Domain" is created, and only devices that an enterprise intends to share with the business partner is placed into that "Discovery Domain", there is nothing which prevents the business partner from doing an SLP query with no "Discovery Domain" attribute. This will result in ALL devices registered in the DA being returned. The business partner now has access to all devices stored in the DA, and can do unlimited damage if they desire. CONCLUSION: I believe there may be other issues which have yet to be discovered, regarding use of SLP in the SNS role for a large enterprise-class storage network. My suggestion is that we limit the use of SLP in iSCSI to discovery only. We can recomend and use SLP for simple discovery of iSCSI devices, as well as discovery of the iSNS. But let's not encourage SLP to be used in a role for which it was not designed. My concern is that if we talk or hint about the possibility of using SLP to fulfill the SNS role, those that take this path will invest considerable engineering effort only to find in the end that there are insurmountable issues in supporting large, enterprise-class storage area networks. SLP simply was not designed for the SNS role. Attempting to force SLP to do what it wasn't designed for will result in consequences down the road if anyone takes our errant advice to do so. Josh From owner-ips@ece.cmu.edu Thu Mar 22 15:12:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA19992 for ; Thu, 22 Mar 2001 15:12:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2MITcO22582 for ips-outgoing; Thu, 22 Mar 2001 13:29:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com (mxic1.isus.emc.com [168.159.211.82]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2MISar22529 for ; Thu, 22 Mar 2001 13:28:37 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 22 Mar 2001 13:29:52 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0708015304@corpmx9.isus.emc.com> From: Black_David@emc.com To: vince_cavanna@agilent.com, ips@ece.cmu.edu Subject: Slides! Date: Thu, 22 Mar 2001 13:28:24 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Two points: (1) Please do not send slides to the list. If you want to make them immediately available, put them on a web site and send a URL. (2) Everybody who presented in Minneapolis should send their slides to me so I can get them rolled up into the proceedings. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Thu Mar 22 15:12:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA20022 for ; Thu, 22 Mar 2001 15:12:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2MIPfR22273 for ips-outgoing; Thu, 22 Mar 2001 13:25:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2MIP8r22243 for ; Thu, 22 Mar 2001 13:25:08 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 09BD71F6 for ; Thu, 22 Mar 2001 11:25:08 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 30A6D1E3 for ; Thu, 22 Mar 2001 13:25:06 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id KAA14602 for ; Thu, 22 Mar 2001 10:25:04 -0800 (PST) Message-ID: <3ABA439F.4CDB2EC9@agilent.com> Date: Thu, 22 Mar 2001 10:25:35 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: foils References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Why don't you submit the new format in an I-D format instead of slides. julian_satran@il.ibm.com wrote: > > Correct - I there are several typos (all hopefully obvious). > Barry (who has the source) might correct them and send me an update. > > Julo > > "Ayman Ghanem" on 22/03/2001 18:07:15 > > Please respond to "Ayman Ghanem" > > To: ips@ece.cmu.edu > cc: > Subject: RE: foils > > Bullet 3 of slide 6 (new PDU format) says that first read is 48 bytes + > header digest size. I assume this is a typo as the header digest comes > after > the AHS. > > -Ayman From owner-ips@ece.cmu.edu Thu Mar 22 15:28:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA20523 for ; Thu, 22 Mar 2001 15:28:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2MHqvG20086 for ips-outgoing; Thu, 22 Mar 2001 12:52:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2MHqTr20024 for ; Thu, 22 Mar 2001 12:52:29 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id AA60BF15 for ; Thu, 22 Mar 2001 10:52:18 -0700 (MST) Received: from axandbh3.and.agilent.com (axandbh3.and.agilent.com [130.30.32.200]) by msgrel1.and.agilent.com (Postfix) with SMTP id 38C4F19A for ; Thu, 22 Mar 2001 12:52:18 -0500 (EST) Received: from 130.30.32.200 by axandbh3.and.agilent.com (InterScan E-Mail VirusWall NT); Thu, 22 Mar 2001 12:52:18 -0500 (Eastern Standard Time) Received: by axandbh3.and.agilent.com with Internet Mail Service (5.5.2653.19) id ; Thu, 22 Mar 2001 12:52:18 -0500 Message-ID: From: "CAVANNA,VICENTE V (A-Roseville,ex1)" To: "'ips@ece.cmu.edu'" Cc: "CAVANNA,VICENTE V (A-Roseville,ex1)" , "'ipsan@rtl.rose.agilent.com'" Subject: CRC vs CHKSUM presentation slides Date: Thu, 22 Mar 2001 12:51:59 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01C0B2F8.CAFB3C30" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01C0B2F8.CAFB3C30 Content-Type: text/plain; charset="iso-8859-1" Here are the slides I presented at the IETF-50 in Minneapolis. Vicente Cavanna Agilent Technologies <> ------_=_NextPart_000_01C0B2F8.CAFB3C30 Content-Type: application/octet-stream; name="CRCorChksm.pdf" Content-Disposition: attachment; filename="CRCorChksm.pdf" Content-Transfer-Encoding: base64 JVBERi0xLjINJeLjz9MNCjg2IDAgb2JqDTw8IA0vTGluZWFyaXplZCAxIA0vTyA4OCANL0ggWyA4 NDMgNTQ3IF0gDS9MIDg4ODAwIA0vRSAyMzczMiANL04gMjIgDS9UIDg2OTYyIA0+PiANZW5kb2Jq DSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICB4cmVmDTg2IDIyIA0wMDAwMDAwMDE2IDAwMDAwIG4NCjAwMDAwMDA3ODggMDAwMDAgbg0KMDAw MDAwMTM5MCAwMDAwMCBuDQowMDAwMDAxNTQ1IDAwMDAwIG4NCjAwMDAwMDE3ODggMDAwMDAgbg0K MDAwMDAwMTgzOCAwMDAwMCBuDQowMDAwMDAyMDE4IDAwMDAwIG4NCjAwMDAwMDIwNjggMDAwMDAg bg0KMDAwMDAwMjExOCAwMDAwMCBuDQowMDAwMDAyNTExIDAwMDAwIG4NCjAwMDAwMDI5ODggMDAw MDAgbg0KMDAwMDAwMzUxNyAwMDAwMCBuDQowMDAwMDEwMDc0IDAwMDAwIG4NCjAwMDAwMTA1MDUg MDAwMDAgbg0KMDAwMDAxMTg5NCAwMDAwMCBuDQowMDAwMDExOTczIDAwMDAwIG4NCjAwMDAwMTIx NzIgMDAwMDAgbg0KMDAwMDAxMjI4OSAwMDAwMCBuDQowMDAwMDEyNDI2IDAwMDAwIG4NCjAwMDAw MjAxNjcgMDAwMDAgbg0KMDAwMDAwMDg0MyAwMDAwMCBuDQowMDAwMDAxMzY4IDAwMDAwIG4NCnRy YWlsZXINPDwNL1NpemUgMTA4DS9JbmZvIDg1IDAgUiANL1Jvb3QgODcgMCBSIA0vUHJldiA4Njk1 MiANL0lEWzxjYmFkYmYwMmYyODZkMTY0ZDc4N2I5MGNiNDcxYzU5Zj48Y2JhZGJmMDJmMjg2ZDE2 NGQ3ODdiOTBjYjQ3MWM1OWY+XQ0+Pg1zdGFydHhyZWYNMA0lJUVPRg0gICAgIA04NyAwIG9iag08 PCANL1R5cGUgL0NhdGFsb2cgDS9QYWdlcyA4MiAwIFIgDT4+IA1lbmRvYmoNMTA2IDAgb2JqDTw8 IC9TIDc1OCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEwNyAwIFIgPj4gDXN0cmVhbQ0K SIliYGBgBiJvBlYguYWBnwEB+IFibAwsDBwVDMhg6RfGF3waHAHCTk+aFCdtYdJguMHAEL+gJoH5 gGLDe8YUFgOmD0wNIg7uB3LXZmdHbV2bHXUNSETADWCcMrvAYEGmeWjlp3aFqdyWyTKl3x5kqsVx hhyWaeoM2NwTqebMuUK09tg930vsKku5LY7OQVewZU7tM0+NtgWyF6aHGjxMUuU2eDrT8/gslUUK shum8prf7Wx9GLCZL1ItvPBZp0YHe67BVN7viZ2tQGUYRqHbNSnFYFZj+J3cJ3cVirxACqbMyV3k pcR4jzdgfqkpt2HyTM9jt0AWPbjK+T04YLOYJUETjsfnsU1KfTDVx/CExlRs3kFXgGECwQBBN6GB gcHYuKKjARLiSkrlMKagIIQpKCiIJIrEZGBgDUXioMiwpeEzB8g0NkZSjcFEUoCqFrtLkKwgThvx KROow4GB4Z4gkBYCYmWwGYbAFH+FmYOBg8OAQYHBYQ3jGuYDqRu4PwQfYD6g2qDEEGEjYTKB+wUv Z7gDMA0zWDJUsR3YUcCgwOzA4ODEUMc0jVmDSwFsEkCAAQDfmf8oDWVuZHN0cmVhbQ1lbmRvYmoN MTA3IDAgb2JqDTQzOCANZW5kb2JqDTg4IDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA4 MSAwIFIgDS9SZXNvdXJjZXMgODkgMCBSIA0vQ29udGVudHMgOTQgMCBSIA0vUm90YXRlIDkwIA0v TWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1l bmRvYmoNODkgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvVGV4dCAvSW1hZ2VDIC9JbWFnZUkg XSANL0ZvbnQgPDwgL0YxIDk1IDAgUiA+PiANL1hPYmplY3QgPDwgL0ltMSAxMDQgMCBSIC9JbTIg OTkgMCBSIC9JbTMgMTA1IDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDEwMCAwIFIgPj4gDS9D b2xvclNwYWNlIDw8IC9DczUgOTEgMCBSIC9DczkgOTIgMCBSIC9DczEwIDkwIDAgUiAvQ3MxMSA5 MyAwIFIgPj4gDT4+IA1lbmRvYmoNOTAgMCBvYmoNWyANL0luZGV4ZWQgOTEgMCBSIDI1NSAxMDIg MCBSIA1dDWVuZG9iag05MSAwIG9iag1bIA0vQ2FsUkdCIDw8IC9XaGl0ZVBvaW50IFsgMC45NTA1 IDEgMS4wODkgXSAvR2FtbWEgWyAyLjIyMjIxIDIuMjIyMjEgMi4yMjIyMSBdIA0vTWF0cml4IFsg MC40MTI0IDAuMjEyNiAwLjAxOTMgMC4zNTc2IDAuNzE1MTkgMC4xMTkyIDAuMTgwNSAwLjA3MjIg MC45NTA1IF0gPj4gDQ1dDWVuZG9iag05MiAwIG9iag1bIA0vSW5kZXhlZCA5MSAwIFIgMjU1IDEw MyAwIFIgDV0NZW5kb2JqDTkzIDAgb2JqDVsgDS9JbmRleGVkIDkxIDAgUiAyNTUgMTAxIDAgUiAN XQ1lbmRvYmoNOTQgMCBvYmoNPDwgL0xlbmd0aCAzMTkgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4g DXN0cmVhbQ0KSIlMkElPwzAQRtUkzWIq2rIUyjrH5JCpZ2xnuaICghvCt8KpohVIFQIO/H2cBFEu tux5z/ONCdZidvtIsP4SBK8gVIHaQGkKVAqMqrGqIC+p3T9fxEp8CAnNmd29JkZTgASta5QEnbzc iNndhmD+Lh5anJSBvJBYskOJS6QSCim3LG9ZxVXzZF5WWGmHG10i16Dqf7jq8CsrZjcEBHblRNaQ c2PUhIWbQDar3bjKWuQSpXT57NId7bdIe54f9MMgipMnSZz0+0lm31wtJ2Sw81/BNMIi7e0MHGWi 3aE/8sIg01ikfuYQnYbZs73vPKrN1lSNmY73ov3ESQfR4WQy7jooN5tqvowrxloBadeS2qB/GRep N4i7nkfecTANp9FJlrNRyGl8GkdnwygeBf759OIS2gDXVvwIMAAj6VFYCmVuZHN0cmVhbQ1lbmRv YmoNOTUgMCBvYmoNPDwgDS9UeXBlIC9Gb250IA0vU3VidHlwZSAvVHlwZTEgDS9GaXJzdENoYXIg MSANL0xhc3RDaGFyIDczIA0vV2lkdGhzIFsgNTQ1IDI2MCA0NzUgNDc1IDUwMCAyOTUgMjUwIDU2 MCA0OTAgNDQwIDQ2NSA1NTUgMjg1IDYwNSAzNDUgNTAwIDQ2NSANNDY1IDg0NSAyMzAgNDY1IDQ2 NSA2MDYgNTAzIDQ0MyA1MDAgNTczIDQ4MCA0ODMgNTAwIDc0NyA0NjcgNTUzIA00OTAgNDg5IDQ4 OSAyNzkgMjMyIDQ0MiA3NTUgMjU3IDQ3MiA0ODkgNDc3IDQ2MyA4OTIgNjc0IDM3MiAzNzIgDTU5 NyA0NzMgNDY1IDU5NyA2MDAgMzc2IDQ4MyA1ODcgNjgzIDQ2MyA2ODMgNjgzIDQ2MyA1NTEgNDY0 IDM4NiANMjMyIDY4MyA2MDEgNDYzIDc3NyAyMzAgNjE4IDczMCBdIA0vRW5jb2RpbmcgOTggMCBS IA0vQmFzZUZvbnQgL0ZFTUtDRCtBZ2lsZW50LlRULkNvbmQuQm9sZDAyMDAgDS9Gb250RGVzY3Jp cHRvciA5NiAwIFIgDT4+IA1lbmRvYmoNOTYgMCBvYmoNPDwgDS9UeXBlIC9Gb250RGVzY3JpcHRv ciANL0FzY2VudCAwIA0vQ2FwSGVpZ2h0IDAgDS9EZXNjZW50IDAgDS9GbGFncyA0IA0vRm9udEJC b3ggWyAtMjc2IC0yMTAgMTA1NSA5MjAgXSANL0ZvbnROYW1lIC9GRU1LQ0QrQWdpbGVudC5UVC5D b25kLkJvbGQwMjAwIA0vSXRhbGljQW5nbGUgMCANL1N0ZW1WIDAgDS9DaGFyU2V0ICgvRzI5L0c5 MS9HNTQvRzE3OC9HNzkvRzQyL0czMC9HOTIvRzU1L0c4MC9HMTQvRzQzL0c2OC9HMzEvRzkzL0c4 MS9HNDQvRzFcDTUvRzY5L0czL0czMi9HNTcvRzgyL0cxNi9HNzAvRzMzL0c1OC9HODMvRzE3L0c3 MS9HMzQvRzU5L0c4NC9HNDcvRzE4L0c3MlwNL0cxNzEvRzIzL0c4NS9HMTkvRzQ4L0c3My9HMzYv RzI0L0c4Ni9HMjAvRzQ5L0c3NC9HMzcvRzgvRzI1L0c4Ny9HMjEvRzUwXA0vRzc1L0czOC9HMjYv Rzg4L0c1MS9HMjIvRzc2L0czOS9HODkvRzUyL0cxMS9HNDAvRzY1L0cyOC9HOTAvRzUzL0c3OC9H MTJcDS9HNDEpDS9Gb250RmlsZTMgOTcgMCBSIA0+PiANZW5kb2JqDTk3IDAgb2JqDTw8IC9GaWx0 ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNjQ2NSAvU3VidHlwZSAvVHlwZTFDID4+IA1zdHJlYW0N CkiJdJJ7UNTXFcfvxd3f/kRYRF3A325+i0BVoqz7YGHX1lEQvRVQXApCq0icgCiPBQwqyhtRkwVf ICS8BCPvBRXBRwBjgosaZJWOryFpHKepIZ3GV8b0LtzV9m6b6XQ60z/umd+c37mf7/eccyEQOAEI offaNesjVoctCUnZmZ5szFHExChWZxqTFKGZ6UlKtVLpqAm1JTnZkmfY5gnINpcZr1wEtu0uHvZR +2oZmy0TbRHyzeC3pSX/KC//z4eLyDY82/bYbSrbvVImtcwBkVQMCIAIOAMxcAfzgCfgwDvAG/iC hcAfLAXLgBpogQ78GqwAq8BqsBasA5EgCkSDWBAPNoOtYBtIAikgFWSAbJAD9oL9oAAUgwPgEPgI VIBjoBJUg49BHWgEzeAMaAOdoBucAxfARXAFDILPwZfAAm6CUWAF4+AeeAgmwJ8Agc7QB6n1SK9C 2kCkCtahYD0KVCONEunVSKtFOiVSBaJADQrSIY0K6TVIp0KBtFSLgvRIgzS0Khjp1EgVhIKVSKNB Wh3SaSgKBauQJhBp9UhHAcFIRdm0jGbVFKJFKipEUxQRhNSBSEejEgXqUXAg0lAiUlPxYKSmxpQo WIs0NBOEdDqkpQA1Cg5CGkrWIy1lUkdKFKRFah3SK5FWg2gfKjUKVK3PNGbm7MtKTgxI/GXBiTEx iY4FJzoW/D+5//sGAHSdAVgwC4pdZIwfu4zVO4WAMHaDm5nOF8A6eAXehHfgCfgJPAXPwE54FvbD QTgMu+EF+Bm8BkfhYXgEnoT1sBm2wl54GX4OLfAWvAsrYCX8GHbAc/AiHIJfwhtwDH4Ej8EGeBq2 wa9gFayFTbAFdsHz8BK8Cq9DKyyHx2ENbISfwnbYA/vgAPwCjsDb8EN4FFZDMzSB8F+eGxTDVPrr KbQ7uTrJnIKcJpzIDKPAT7BJ8I2wTjjA7GUqmUvMBPOzyEvUJHrEbmdtM+Nm7pl50vlXzmud45zT nAedf5qVP+uRS56L3XXA9YHYTXxEbBFPulXNnj27wn2me5X7N3OOznkw59Vc6dylc3vn2ueVSBhJ ucTqUeJx23Oep87zqOd1z9denl4rvaLe9ovt/WKbWSYskQn7JG8bZNMaIYljphtkQqsQxzFvT3hM jmfHDcoH4s74Ez+O+O72T4jlt8auzQqo8DWFtieOsInX9/wNcxx+98zLq1f5oaHx1qfScpHJTiRd pqaGmm62xnzC+oKbPDhabObNxbkNRmn67v1pRfLi1IPh/tySExE16XxNWnNeZwVbLpq6Qdwkk6bx 7MF4djDuU4footKQ3Hg+Nz4j+f1d7AdJm4rWSX1C2y2J8veu7/63csuLIap8dbzlqQNhtwu6muq7 a+Q13cetz7lvD98s7eVLe/c0plakm/bkFaWyRf+lnFbTnN9FDYvJgG3SPl9CNjGrcFg01iVjBbsd K3KxH57PYa9T2Pc8VvC9WGHB+od4DYs3MQYS9h7RZxAFm0kU+cSPeHGEqyN+7SSA7ySKS0Q3QsLY f7HxG5nwouQhWWMh+l56gZ5G4uu4MH8f8UsmCp6eaKJbRS9QCyM47DLWdVALHTigllqgbXoVYD8j tUBPItYbcJjDglgmeGK7K7En2BNIrI1GvMmWYEvAsXYaKUhs/9a2EHdIyFJ7pJBsZp7YIoX4FCMm vVMeNoXEfgRH2o6QSFo3gV97kFLcgktJCyO+N10pE/ZK3uyTTeuEJIKZ2icT3hLiCEbsa8uRCc9K sNj0l8zhGEtMz8raJewnS06QBWQDRzYcJN6FS/jCd3esiIlmDb9TG4mblIhVHSPR8mhLytcFz9iC ZwewNw7n8Ooq/E79c77+xfkJy4hl5Lt2PMuxPxmRCC6bWs0nb7PVo8ewD87mcMZh7FM6xpdas81b K7aYMtOLI9mSyEN0grs4knWcLKiO4GvC29MGHACx71SoB7k6/WfmN6v0vw/axeImEZ4z9P29L/7K 4lbZ9HbaxLSbx5ub+EMGb8Q7sCfW38fO7H088xx2w57cxTJzwWm+oHl3jbEizrRzf9FOtjClLDaU I339DJZ0/HhjjB8Ze2HGCyqwzPQqY2wja40yK4krRwRGAqMI5A0E0oFEkCxSxdpaSRoj9p4KnDJJ bhcMpfREsT1RDSsVHFmRQURR4Xz4Bp90slhKAnzaR2LkMSOZP+HFHL7EHA07ubkhuyGrpdBc0W/q qK87y9Z3V47+wH1fcMvYyXcY19cFSHGKyHrlWGWrvK2q6VRtJ1vbWTVg5cYOXMlr41vys2uTpRk5 RTn58vycA6mJXBq5JMKitslrd/m7156dxb5SvOh1+sZR+e2oLh/iz60tjMqO57Pi/5AUkuUYpfe0 At+REBW5Q/rwHXscPsDgv08LhbZ5b72xiuxg7HSAkik3Ecl94ybEuYzYD/88dUdyf+Jx3w8VL033 U66EsZ+FnSazyHKOLM5fkLGON65L2BJpZDPXL89fKCWyRV2j4fLwr9KeOx55L2Oo3tqYyWecai/o lV421/c0yZu6T36N3bkfy6z55/hz+cbG96UpWXnp++V5acVbywws6WfOTLsKpxLJB/bNTIndSyj2 tgmnHkoeDd+98Fj67EFyyJB8aFULEdAGybZ0Il8fwodG+qcQqZS4L+v5o0FuGE95UvicLXxehmU4 lMPtndhw8xX/8hYWXMPLpdjluwzDsHzY0KYkbhy5xtw41FfSxDeV7K3OkGbtKdmXJ8/PLUou+yeb ZR4VxZkE8ERnetoYh6hpVnrG7hHXCIQXxcgZIuAICAQ8AoigICA3yHAqKOCdZDJvvcGRwyPIfYMX ikeQQ0UFgwiux8pbzRrXxKxZq2dqht1vMG/37Xv7x/f6+/qr+vqr6qpf1XJ6x7Kv7PB99lsJTDDO ZBrU3xVrG+jDdfuGYDILE3YN5jVxTXmqsihZtGpTbB5fELvT/1MW5YccSuK40rjKnCbNW69PlItI Vk+Vo50YhyhhrlzUITbUgkp3D8NNOfhAAOazQO9oJ5mVR+NQKB86mA4TTVGzpgbk1+9y1wdfNoKF Bj5QP0voWtrl1eiktaYPW+3DP6Aji6dSMNjfirMKQHEoushwyqdV3YF8UFfqc2BY6KCW7Y0uzOYO ZR3bWqOpVh8/WFxWUlbUcqCbPtC95ymYEfPmvGBOqStKDzfShxv29/6F0Gn7o5w6LqcutThWE6dO z81LpPPjdy7Bd1mrfV5F0VzR+sqNZ8eN+1/S6GJE/6XO/4dVhODP4DdQBd9glQl4z2EKM/ZKiBaD p0RvemKixDiMT3VkiDFJYhLRdRARCNK/wiDKECv6z9y0JzwgewainiDR3YOnRjLEkEyOIu/QUyKd C2b6YUYDlhUvOjvoC9+PlsNkEgJPM8Ov8dfCa5zI//UqWJYRymWEhHkstt4QnZ24NZ0uSNuVHMui NcbBZknr/tYjrZy2teJSc19d5/EzRfV0Yf3etvNsw66K/DIuv3TTwXRNqnpTwbbMbZm7wtQE+icp Yf3YMIM74CTsICsp+JEYMMdblD08ExtCKHt8JsZmSvCUk15BkFLCfIIysWG10MPgAp0l+hktxfCc 6LnoDLodzJMtN2PaA+hzASdcP2bRPHHuUm/Oa6ltAkplhn+RI3TvUK/Vo4m3fel+nwYHS9YxS7lu ORe5wj/GNddURd1/ZQjuL54uqa/n6+tK2nvY3twzCSe5k4lR2jAZrsAxymgmgkiq+0xJYzVf3aC9 coe9lXsxrp6Lrw8u8dTQ6Gawp6TP9K1CGIMucThlsTvnpsRJMeggQzPbsmZv3rtldV/yj3Tyj5th AWSycKkRwgZ+4QZ+hqkd4K2B+WqQxP+gpAeVjSjGeWwi9jZIvnqQ153ZlNkQX7pW46FeE7XRmc52 2YY8BrOo2DerUMkVLi4N/U5VrqrPP6vpVLeeOH7uxLmifs3faOjZTBwUADOJAxghlYIlamBCHn9E P57ThkvxMIuHonC5A07icLIjmpMCLzOkEFkKeuTozoAZNXqt7mYL39xXBhYQwUL8RpBH9XDdkX7V 9jKUjOVQ0oV6sblxi8nDcZhqWIhjggI3GBQGe8EHN1BSSx3IRRcYg4tBJNgQYtkY38Uek3QKppO7 eeiHa5gEdESnHHSW4RxFVZcn79kVPpzzE73pp23AgD0Lywvho6O/ccdetz+81013DwFdDXYycIfJ sR59fJ97LUpxEYs9ddT9/KtpjVx6Y2TpCo2vOiwx2ZdO9t2MtridRckeuwN+3AG/o6tq19PRtRc3 9ssazu8vvsZDF9rBBiHYuFYiBaV+yBx7dSIcNYpMF9VCj2EYRoVh6JXr40jOJgtfmI9bQHz0trZ+ Ym6sfrtGbyEbsyipzlFOfW6OLXLREJWEinTFpsW4c8xmBt6kcKvedUtP1qsU4C3k5GJqoilESyDU 2F6F7x1TFNpaHHlz/PXJX2G97soMIWn8GzAiFz0yN1hR+FA3tgrk8f/MHiLaq/GEXFwqeErgtFHb i06t+EGJg4VclAsn5PpoYoyL8Ga8eVHCtFCbEX7EugVXYhOLZ9ZjmKuCU7jitGD0kKEbTr38xpV3 fRMNYXCGhaYWWDnykvv7MOkjYLHM1KTMEN1sb3tcyVc80gIPq1jwy4cPVXc51eCaJm+Nl3pNTLo7 neaWj9PRm8VgLfIVjpxj5bmoPlO3Ofx7CO6m4GM1iBOGvOi7Xk0KnMOiXQyKFym5RUqcHIdOJI1c DFfH/UuC0PhHBmRUb3tpM0nKpuIrP7B3tlxJIjUlaV1ZsIxw04f4JpYwzpwQJH7J7u2z+e3OgV+r ZOiO0y7DO29NWgPtpNK2QODIz9zLEZhuMgmWwPRQa+ISmxYMxGYWLfPwfZU1l2YdvSg4qPP4Be2p /fS+U5f33JJhuMQogyBm4Pypx1V81WMtaedCWPDPAyZ1iBtMDW1eIlOujnFL49Pc8pBBHxZXaVFR 5cg5VF6IvG0yH97T3zChbXvL7t/AgwXptpGsai67Jv5I2Dg3no0bLAxRMEk9mtIdQHctq5pPuGud 5bougIvwD4j5LPdpzo2YlpV084oyW7RgceHeWQdDuG8lRjPdZQZmUldPlzZV8pVN2s4htrvgbGY1 V50RX0zI5WyYS0kf6FbpC5hmSf/WjsxaLrs6tjhEE6aOK8jIpDPSCxLWskFHI9riuLbYwfB/BIPU /4nfRR/6kk+5E+kgZkfNdvLhfJyRSUKlDF1QXD8QwAfcTn4NPHtIU/ynoxx0Us5a/4ooriKqbeNl zSV1q7a6kq6qPNJyiWC0PbGWS6j9svHzVpx3f/bdwDt04IBqZJSFKW0v/jzAPegHugFsNDBXDROS BvzoO74NC1DCblWnfr2Ww05KukDvIxe1Mdgvxy+wl0yxH3L07kjo06N/bgosg5mpOOBzivQQpFzA LQooQ0Q/1pfPKp9VbHfArRxc2iC0E3LBXDg2A2pIjhbKJdNI+jD4hCSLGH+hUKGL3PzXDPgkEVQW KvTdgPNS8EN0M0bMwFEKrxNhMdyg7huC27G+FCeWoqjQdo9rEUx7KPyb8XIPaurK47izS26ufWDV BvVevFfrC11dB8eqXdepBayPolYZTQghBHmJkAAhARKeAnY13toVEAKBQEIgJKBAAkl4BN8PcKor Uq2vadd23HW0VLfunpucxNkTxv1jd//p3+feM7/n53y/74BVHgxnOaiDvvunBPVrmuB78xrhe3Ur T+5wgh7kp6Zr71SdBvM90+YC9KyDpaFBzSEoYPgVexRW+Y5y4HWMDffk+HDPHg54hgGlLxlI2WQO qq1vmv9zNtQv5wRv9M7xLOMhHQrj2ONQ5DvOgWkY+MaDUt6KgTv+Bk4wH2l9eBo7Zj7Wd+xcZlxq WiLqcaKodD/54Q7TOWQjR5T3XhHgra7HNy5SNy6A6e3gjwx+9a/YmfIOlYEqaJHXZiBJk11akKdS lB3SSHCox15Y746epUZHnraBmQx4X/NMfl2IXxd2RiwnfEZu5Fc7TsagWYR2diVvLJ+7SbOvVKrA s+RFCdFEXHWqLofS5RhLzEjVdxpMLrzVWTf2EKmq4H1suPdtnuOHCecLZlJzO9X1CT6w2QCXwmwC blHC2eKPqY8TIlPDSKGo1pBAiw0ZloJefAVsC3DZ4XNwX/nzeP0W00gj3TRS/RgEE2BW5d+KL1LF F2XmBCZBI5UXCfAiQUU45BFwftXqujgqrt6scAaWcoOXA+7zIJ6+dtseau/WpTJIkHDOiu6JaHrn RAZySTEESO8EH4zdosZuTZ4BoQxYqHmRObobv7bLugZOR/YIO1fRX9RJWYukOjEpkcpjkTAWVKyG BAGn1/yhMZ5qEnUobFNmptGCN1mrh/9CADtyR/u8vw2Bpn9ieZLMDIkCV0hEZQIyLKr7TjwtnlCC ZQETGGUAs1zfUgN3b/dOMj9rJhKdG3C2GzjQM8T1hYHlvAGLyd1A69xVT0AIAcIqnhY5qMJ+aUs8 k6TJVhTF40WiijUwhIB09aqGWEqgM8ttbxJno3m+GhUWXrNNL6b0YrOyh7FrzFpde6Oppv/EJRzY sUqIpa3fHoPH7PhdFpxJwuWL7Y/20jEPZYAEuwmw1QJCLj2gLj94aQdLGbBA8/LgaBR+LcqK3uJP iKVoFsAG3g3uhXKHykypzDn1B5lkjTQ/PwnPTz68fzuhBkqkjVXsVSRd4UK/Gzo8bjgDRCNH+iU2 3nze3kN1913VPyS/Hjty+CoN+dzkk4ovK5lcjepIaSleWnxElUvAPWAhjOYGy70LhDz4jnTdTj4l 2LVIDpeTMGRVSx+Ktz/r5lNiCNiyuQnVUq2aUtcbyztJe0eDtYFutFZdf06AadzEdZGSJUyYJuq0 +CaecFMBFoJDBNjQAt7t/5a61/+N7RdyaLBM6aRdyk5Zcxr+M9C/buX6FoE83rC1dVhLa4dOfgfe IgB95EmJkypxyAwSRoKmr1CIq2Mr1sO3CUhUhTckUOIGs7Iv0IQM77wQaPfwMUdlV6GRKmzJrctg 0jXZRfmZeEFmeUIM4V8B7QFZYfeNcs89ezQEgpjvNKM5NiFu5+s3wWmE/3NUwWb29+gerxLYXyux 0CBXECp8HrDDPCx4uzc1BOETNrFDHN/Qm8tec7mgjv2MA85gsM63Bth8E5xgXxsbHgIdU1s1pXJY pAKPBOKLxlwVZ1QmSm3K1cqYA5rM0vxcPF9+OFVI+Leh80+5A+U9qnaqoC1XKw10uKRg6jwFnX+K 4pIDPZzBPT3Y2+1sxVsd7rrz5MWhErmNtsmtKYZYq7u3d8CEm1wj2ivko+s5/EF6kG+MCif8Sb+2 RCh1PSS5lpEem6vN2XZWe4X5XnMtxy7Ebfzm/5Qpgi3zfsEDi/+RFjlKj0VYIAmFBBRJ4fydkVTk rkVpcAkSn4u7b++kd45nglAgIsB87SvzONU+7jh7uRPvvHxP9xPqm/9x0JCmy6Bz4Y2O6h/BXOQE K35UOymn+pBRRAozckVqWh1XsRbOIuDs6rW6eEon6sp1TzlBPjvPi/Fa+jqtDiNucA7WniMnJ1Ij BmhXhBEugVICRijgTPEn1GZxRMoyki851ZxKp+llFqUd90kC/duGUp3GBZQ/m2fvMAyg5XehIOYE qPcEUa/ogswcz4g1mfKi2AD11sD3p6hXL6JE9Z0Kd2DuEHmgjwcdbCqmFMsy0pS4Mk0SgE9k9x0x go/iv+DjmoLPsGaw1Kl0KTql+hR8EphCvU5EIPbk/xBoUeXfi11UsSsLDf8BjTRHHYur+eUfwnf/ j0DwpSfCY+KdSMxlKsmtsQpxOp0uVm8NJ3z1qOHPuWePDlTaKNCF1fH/LEkjWtyW3l6TzTSgPc8M HZsLC9ANYazbE83TuCu6C42FRlltIvKXygq1GlerypVZhKwut1VFGVVn5Q+kYIbwUUJfEt6X1LJ7 I7Ho4KYtAkqw5YMcuIAM31zfvpfe057cne/KHyj7+tgPOOhKxOJr0psUlFJvLuslezpOGQx0S0ut xUZYS9pyWyh58/62jVY4c+gjV3I/ntKXf2mC+P7MzVtuavjWcxOYQf50tzj7In0hqzddn4Q3HTi1 N4KAXcg6CNgXIcVCfuU+Ev5mY7Mtlo615Yw9Jtj7b8D3ERfYQRAHWsAqJHRW+U5wBVVp2jxKq2w7 3MtYNB0nzNU4jHizGXowB5MKUlLEclwu5pfsItdFm0aS6CR3/v1/EZ7dwO7nc8XV6bo8SqEzHe4i u80N1ma62VLjvEx4ViASyFEZV3tnepU82PNayYHjGGzzKjnAhoGVfgZ85mE4oAqxoDQUCwp8ZAh8 dAlbzK7cDRj1YzWS7Mchz7ceh1cwyAT+7MbAfL+0fmIc/AnMZsPngWFsEr6+Ar9ohjNqrxh/+feA zbx3P4wkvveDCqdAcOFk//31b5ffr1l/L2f7fvjHwt+zfqSyfj/C9l0VGArq31eyfl8OdNWfNGA5 tYCV77f2L3GQS2b83RLx2zj0t0/Q7zTJ31PYvjN///P7JsgFW9m+2/7l/O78k5P1+3Q2vt/zfpaI /XX73sb2Q+zHcdY/7N99fpr/TmT7o//nDesP/d8+QO95y7CGiv1u/r70ezOwF/GnDTsbGOYrf3iK /tv7XevX3t9abHwyrGUiv7aIAgCjkkRpCmVuZHN0cmVhbQ1lbmRvYmoNOTggMCBvYmoNPDwgDS9U eXBlIC9FbmNvZGluZyANL0RpZmZlcmVuY2VzIFsgMSAvRzU3IC9HNzYgL0c3MCAvRzcyIC9HODEg L0c4NyAvRzMgL0czOCAvRzY4IC9HODkgL0c0NyAvRzU0IC9HNDQgDS9HMzYgL0c4NSAvRzc1IC9H MjAgL0cyOCAvRzQ4IC9HMTUgL0cyMSAvRzE5IC9HMzkgL0c3NCAvRzg2IC9HMTc4IA0vRzUzIC9H ODIgL0c3OCAvRzg4IC9HODAgL0czNCAvRzUxIC9HODMgL0c2OSAvRzcxIC9HNzMgL0cxNyAvRzky IA0vRzkwIC9HNzkgL0c1NSAvRzg0IC9HNjUgL0cyMiAvRzU4IC9HNDkgL0cxMSAvRzEyIC9HMzcg L0c0MCAvRzI0IA0vRzUwIC9HNDIgL0cxNiAvRzkxIC9HNTkgL0czMiAvRzIzIC9HMzEgL0czMyAv RzI1IC9HMTggL0c0MSAvRzkzIA0vRzI5IC9HMTQgL0c0MyAvRzI2IC9HOCAvRzMwIC9HNTIgL0cx NzEgXSANPj4gDWVuZG9iag05OSAwIG9iag08PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvSW1h Z2UgL1dpZHRoIDE4MCAvSGVpZ2h0IDgxIC9CaXRzUGVyQ29tcG9uZW50IDQgDS9Db2xvclNwYWNl IDkwIDAgUiAvTGVuZ3RoIDEyMjEgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSIns l81q5DgQgIWM8XnfQBCCryai8QsM7NV0H/wIe/XIa/z6q/pVqd3dyUw6DAspQlqWqz5VlUo/3vdv +ZZv+aBc4nzs3GIcP2a+xlv2IJNzDiCDQyHeBZovbArt5qRjOueNB4PRXAjR0VOCtr8iJ2q3hexc Lz7mdiEHq1mTBzEz5M05Q2OyOLpoYBxw0azJambIqu8NWXCT8V9dIM2KnM18xGgKGfVjDIwQclfi b43LPkYZ95rc72c385Ca+tyxBUKsYJYUB7ZNyTI0s6a3ACHn0U67fTFxuOC7qgTGUQRq7GfS7C1A MiWh6QudrQmZRB6YnJ0fHMUIBpyicwXQ4OaaXAZL6Hztcx5tlSmciu0N8lSc5hdqCGN0TL6I2uD8 Jq4Ods0QIGYZ1S8ZmMmp1Cvi6tqAOZMpLFOpAFNEMr+FbGLC5Fb1jP6KrwVyk5y08h+TvWj3mt93 yKDnP0CeRWfWfBFk1ZCuyZtzVTneJvfS47Cnu0M2M0hONwaZSi05Jr+pIwFWc+RCoUqsyHstoXJ2 1drQqjvJLOsW5CkAv9tsHckLebncXykD20id0CPN5KodNXnr1cvl/upeOI5FySON3IqmP5BDeyAP xx1pZcak5J4VZhNmRcaAppqc2KVgdlGOgybQTCHln32pyAvt/DbPN3b+URaHhM/7E4TgSdPkqrNT Um/cGrOcVmOZzE4Spj6gtAcyH75tRT6csCNnUjerhROsPsxHcqpe7Laz3ApGrvnJnCa9ccy97kcy nqz6QtJ/Bo1GKSNXoG7IukXTTYbsr8n59jXuB3nG7etb/t/y74/nyz9I/vnX8+XvLyZ/yx+V7fSO wmUmvfqaW4kg6pvC2hw1KxlG/Flcd19l/gw56WfWDZXuM+Q9PlDxnyI/UuF4voD81hryJZ7uky9y LK3xiryWN6WohpHv7/jA5yr8jgH78fBMrXSCZj5eZyBDyTkg6LfOiqetE/LUCjm5JmJ2EDL00I/T MHVweL/ky1AGuKkhn8EQyKdsBtfDPJQHeyVvXsj4+uw5G0uHXpBStsreb6CBN4Pcs+IwuQmdG+hN Hu2VvIeeyCuVCTwCGXO9UAJyA1+m3MILDpi19GbATISsR8UQCjl5Nqa5hKQCldwIHT1RbgCA9pDc mcl0eezFs6WQ98Bhjdizib/gRjg3NCRvEDnjGMUwhldO4tAhr6P0SQJ3OxXhJdLlmMnZjc2BI9Dg BZUDosy/8cemRJOxHNZmyNCGPxHJcQu/2dnsvBQ4dKBZWWAUKpAp5t2QwSkgRxYm53+kL0P9BjnP oasWOIEcBDh1WiiWbLLxiJzjdTSRFTnA+kxNNjrmecRarcjTMc+Q0lIb51nI06sHJs6MqQ0yK1Un ZKnairy5Us+yUmB1wI/DWEw9s5muFCFzPU8VGR9XXmbq8+o60KDvprIG2SzJ6hYyLebN1WQs7yBb g0wYfSY2FJXuG2ImO5KSE3wqhbYm7wE94u1MyAP62XFUsteJme6iQqbvofmKnPBBvmaYfEZvex6b P+OUDE7X5Pw95Mer44nlEsdb3STx0csi23sn3a8L1/O7Z+ivS+Kq655O3nGO1gcXst8WKJ831z4f TLXl568g59p6cB17jvw3ABIWPpoKZW5kc3RyZWFtDWVuZG9iag0xMDAgMCBvYmoNPDwgDS9UeXBl IC9FeHRHU3RhdGUgDS9TQSBmYWxzZSANL1NNIDAuMDIgDS9UUiAvSWRlbnRpdHkgDT4+IA1lbmRv YmoNMTAxIDAgb2JqDTw8IC9MZW5ndGggMTI0IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJl YW0NCkiJYmBgEBUVVVNTMzExcXZ2DgoKSkxMzM/PLygoqKioqK+vb2ho6Orq6u/vnzxp8uzZs+fN m7dkyZK1a9euW7du27Zt+/bt279//4kTJ86dO3fx4sVbt27du3fvyeMn7969e//+/bdv3/7//88w CkbBYAUAAQYAyVY69AplbmRzdHJlYW0NZW5kb2JqDTEwMiAwIG9iag08PCAvTGVuZ3RoIDQzIC9G aWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJYmBg0NLScnJyqq+vnzRp0r59+44fP/7/ /3+GUTAKRgAACDAAT94MBAplbmRzdHJlYW0NZW5kb2JqDTEwMyAwIG9iag08PCAvTGVuZ3RoIDYz IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJ7MEpDgAxCADA/f/3OAQhaXAIBAaB gf5j0xkAQEQiYmYRUdVzjpm5e0RkZlV198zs7vc8/3IFGACeCihmCmVuZHN0cmVhbQ1lbmRvYmoN MTA0IDAgb2JqDTw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggOTU5IC9I ZWlnaHQgNTUwIC9CaXRzUGVyQ29tcG9uZW50IDQgDS9Db2xvclNwYWNlIDkyIDAgUiAvTGVuZ3Ro IDc1NzEgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSInslz2S4zgMRpEg6MRnQ6LA ic6kxIETna0TBUx8hiUByu6pnWrZGpk0P+Jtj2fHFmRUP+GHt5sDzHftBJy34n6xcb/YuF9s3C82 7hcb94uN+8XG/WLjfrFxv9i4X2zcLzbuFxv3i437xcb9YuN+sXG/2LhfbNwvNu4XG/eLjfvFxv1i 436xcb/YuF9s3C827hcb94uN+8XG/WLjfrFxv9i4X2zcLzbuFxv3i437xcb9YuN+sXG/2LhfbNwv Nu4XG/eLjfvFxv1i436xcb/YuF9s3C827hcb94uN+8XG/WLjfrFxv9i4X2zcLzbuFxv3i437xcb9 YuN+sXG/2LhfbNwvNu4XG/eLjfvFxv1i436xcb/YuF9s3C827hcb94uN+8XG/WLjfrFxv9i4X2zc LzbuFxv3i437xcb9YuN+sXG/2LhfbNwvNu4XG/eLjfvFxv1i436xcb/YdOZ3Gc61UyhLX34DCfUl uC+/FyLh2kkUpS+/EuuX5tpZlKQrv0GiYBlrp1GSvvxSqt+xdhol6csvJ8Fj7TRK0pVfnb/en3GR WMC+X+GyEPNX7SSK0pffMHRWvp35vYXrtXYKZenMb3e4X2zcLzbuFxv3i437xcb9YuN+sXG/2Lhf bNwvNu4XG/eLjfvFxv1i436xcb/YuF9s3C827hcb94uN+8XG/WLjfrFxv9h05vfCw1w7h6L05Xci YZ5rZ1GSrvwulDjXTqMkXfmNxctMNNfOoyA9+Q2s9Sun2okUpCe/38QkIvRVO5GC9OR3EpJYv0y1 EylIV36jXokl3NMA7smvFW8cwHPtTMrRld9YvNTZAt2VX1a7Xr+gTKk7p/lbO5GCdOU36+XaiRSk J79Lbs9ftRMpSE9+Q1qgRWisnUhBevJ7m9IC3VV77stviOXLXZVvX35vS+zQ59pJFKUvv7dwudZO oSyd+e0O94uN+8XG/WLjfrFxv9i4X2wa8BvkPNfO4W9c6Fw7hW0a8EtMXDuHv3Ah5rF2Ept8vt+F iXiuncX/ESIaaiexyef7nWKd0Kl2Fv8jkMhH9pU/+Xy/qU4+cNItMS+msXYaWzTgV4Tpq3YW/2Oh mBaPlbPYpAG/sUw+0m/US3PtNLZowG8UzJ/nN6T27H7/ne/0exxrZ/F/Ynv2/fkAUqF84p46tbBe NeD3diH+vPU5PncDDXPtJDZpwO/teq2dwV8Jn5nWn7Tg19mP+8XG/WLTmd/rdZ5r51CUvvxeBpEG lt4D6cqvUERkrp1HQXrye2GS+B9x7UQK0pHfkMo3ChYZa6dSjo78pvLV+u2pgDvyS2T1GzWPtXMp Rj9+l1S+Wr9CX7WTKUZHfonz/I0lXDuZYvTjV7LeNH/d78cShuG6K3CIWnUCp9dxzx2uw3De9dUV ac1vSDN0129ZVK5Evcw87rjBEou/OcGt+Z04VeD8emAg0col2lu/6cmiYcdX16Qxv4vq3bP/hrxc xSa974C02PGqsQJuzO9Fq49pfjky2Mko/iTB4+tfLel8Fb/89ciaNObXpifx+HLkQtqgRR+QXX7T 08F7Hq2aNOaXcv2OL0cGLb5Uu7yrPy+pu+9evavRlt+Fbf7uGcBJD1v1yg5Ji9jTIafXv7oibfkN Wr+pSb8ea+NXq5dkfjl8SdM3Cd7xaFWkLb+L6o2WhtdjJ7H5qWN0z1fn2eB+38eiZ9h4EN3xS9YF y+pf9vjV3WzfaKhIW36DTc9dv+RAj/o9vR6+5NV7x2pWk7b83jgvWOOOWLnX754zTqBcv3u+uh6N +U21m1r0vCN2obV+d7VYSes37VnNatKY3+9cg/OO2CB5w+Jxz1dP9njsGN01acxvsF/yeVfwYvW7 Mzpo42isPbfm93bRLWneGay1v+NspUxJ8N7gWrTm9zZERePe4Gts0fuqN/EPT1Y1mvN7u17nf4n+ t6/+l+gqtOfXeQX3i437xcb9YuN+sXG/2LhfbNwvNu4Xm+b9hmEY5vfc+SLD+S13Lkjrfi/MxPwO DYGJSIY33LkkjfsNA1H0wOPxd5Z0Y+Kvw+9clMb9CgnHH+bD73yh1Bii4vnwW5ekbb8LRcGUfo4u s0C5fun4R6ckbftNZlXv4XNyUr063OeDb12Upv2GKDb25/hyeBuVdGfW4d70BG7a77c151TEcjr0 zsE6Q2rP1PQK3bRfydM31fCxY/KbdGtjlTwfeuuytO03uTXHIofeedJnJhVwZDz01mVp2W/Iu5XO 32OrbNDeLFbA45F3LkzbfmWdv9HFfOSthfNyler4fOSdC9O2X7L2LGkNGg++sx2O0vPT8gLdtl9t zFa/T/m9DsNwnp+4s+TWrO3B67cOVr+6XT1Vv+GSngMZ5u1bJ6+6P6cHyOu3Dov9+kXrl8ety8Og emPEdfNSfWIoW/b6rUMQsmOM2pi3LhfdibWbP3FtuqXY4zAekGstmvYbxbJp4O3z0YXY1qW0N23d ejK9olePh2Rbh5b95opk3Z+3lAWtdLZ2LuPG1ZN1fq/fquiWy+Z4y++F9cQj5m3r6mXd3FLRzwel W4Om/X7rCcYEnzauVbu6MmmTHn+/OtjBS7T3H5RtFZr2G1jPMCpu/v3SRS9izno3zzzCeb0SOR2T bB2a9qsDWOt3c/xOZOvVvUvPv1//Tff9eePKz6Ztv4vVI2/uQIHsPMv5b+KtgPXgRedjUq1E235T WYqW5cZ1wc6zdjjSkC1tl1y/TW9Xzfu1PYg2JUx5nvL6I5tb02R62y7f1v3ermnF5evWZRPlfdjm bxK8FRKGNH0b19u831u4DOd58yqxpYr4fkp6Ym26DsP2RR9O836fQ8dvNCy2bKfWO9bOqQh9+LX1 Oc9qrV6SU+2kitCJX7GZy2z7c+rVrU/W5+jD76JDV4+0duyh7QMSBn34DTp01a225/T6VTupInTi V+XaXsV51/L6bZZ4sBmuP98IuSun0k1npCT4/GeEDOe5YIqlAPQbBuvF8+OtxVbmXL82jE8/IuLB Kb11vcGB5zcMuRfz+HhPtHIl71g6jE8/IvIHjNez4fxGlZJ3qR8VbH3ZFiu2+h3vH0ou7D8eCRDg /F6Su7xEPcoxv0lml5Puef1sWd9Jr/NfbtkyaH4Diax6icb17SkvzespKb7O9wjKEzmdntA6NJrf SU1ZucZ+u759WU9FNntTo14/Wig/EBr2cytDAMxvIDvk2rh9tNvFijT36NSL74WaG7PuXzF4rJL3 2wDz+y3rLJUki7/WD9YFK32qf4/5g8V0a/HqllUn8XcB5lfydmy6fjTo1TvnOr5XtjV0eyJSBT/2 Lgiw/AY7//C6Sj3a7ZJPReo/evxZ2LL2bL3gVCHv94Hl1446eRlWlaf8Scj7MefaHtcIsTOT2Ofx r6+/3bhZsPxOujSrXPUcK3n96HLv2bpKzfnttD0zPfbnH4s1BFh+TVWau2x/HgM42JhVvY/yvU2S i133Z23Sc43M3wWYX7K2fK/fH7YueQAnwY8anWzZZitufQTGCom/DSi/Qav2Pn8l/Wu+fyiPLnx/ 82aLs0XYGH7UNgJYfq07q6cs+IdKE5w+OT9C1qlr4nV0n0qn/U6g/C75eGsz1oSNj4/DoKX9U2+g deqK5NWboRZoLL+yDl+zlhr06cfn4TrIcJ5/vpOfhLxg6Urmfj+Vha1+JS9Lydrp14hUv9qUmXL9 irjfT2Whe4MWyUek068RWr92otL61aeiTLJlAPNr+/N9lj5Zv5RPVPpA+Pz9WBYR3Z/XXTj9c/w9 JPVlPVLdG7TP348lrENXnZmv+feQHPDzSDWWSLUUWH7zELWl2ATPv4dIDlgvj69jiVRLAeX3lvuy 1aMdlubfIyYbvrz26fhojCUyLQWW38GORrKWYnzdiLgw23bF6+q9VfFtgeV3sqMR2SjV+t2IWOgP vWl/LpJpKbD8LjZKRfusdujTRoQdkHg9HqGtz2B+zZaaFWvP41aIWOmq4vR/WOMXzO9NsmAr4+32 HFu6qFwbvow2ftH8XvKOlAynGbzdbBeTqw9DMow1ftH8BmbzpXs0P1OMcj9TaZMe351iWcD8xg36 Xr9PbM+JxfSuHXp+d4ZlQfMb1JbkETw+EyF55db9+ZmIlkDzexvyepWsPXfUWeK1a/2ilS+e32CV qw16fi5kyvWL150B/ZpgPdVen40YUr3HuufzWzOrAZ7fpCvxSi2+HtEKgH5vt8sQeSkixIDz/I5c KgPp17njfrFxv9i4X2zcLzbuFxv3i81/7Jc7mts6DIXRoLiN14aGhRutSY0KNVzbNCzYeA2XBEBZ lJ3II+tFRudLnBk9KMa/DnBw8a1bF9+6dfGtW/Xw9Z0xxq6zTP/9dk6iavg6Q1H3L5fxYRlE/HaZ 06gWvg6iEOC/75ehsA6utK+jVQlfH6gE20U231jPJ7zfvienUSV8GQn7l6BZvgxBKPERcHhVvljm RKqDbxttR+LfL0prB0//Atr19necCuPb93f75nBMVgqGcHmFjvZFWYfeGzg8f+nix6govmF2gXfZ duS7yMcsXN7pEhFvXOn1+XzYLlz+EBXFt2OKrzOQAa3NyD14KYGhBHC1hxcDe4O8gYXLH6KS+Cpe IJsf9xysxHccj5bVUB9vZrxc7V/rfKfcS5qdSuKrJqXp7PITirJmK3kDlgFwMPZvXDI/7/kh8WxB PfisfDtj+smhVujFD5ufkLyLPNbQ4ujbAkkJkHgVFs2XkUdzBpvc6Y0xix65vU7Kt41OavJjhulx VM4NDAwWtPvSH6LvrCiVAB2QgG7j056YPUev6c5iZbFLnrm5zsnXyTybHePwqgCyhOwFLGp6jr8t KaAeIL0+qJ08W8ZJbeBd3LI7O97ApGmcRGfg63s7OWLYj7kNnXy10UV5AXYAWpZBcjQu+q69rK89 gDtttkzL7xxxhc7fPJL3Yfqf6KcN5gidgK8304bpSQpt/v3KVyvlc3z9jzBN5TX+lvn7QzlprnGh WAv4lRmf58erf7MTXlPXLV+vBWwWbGNlnYBv92I4J8TyQcTAEHByZ7egwRZlfInldcE2nGKS3sur ZK+RSY+ITxk//0fDe/5QF7diH0frBHy5aWZHOtCvzI4vE08L49voRCt0Kf3FZQPSD9+fagE7eFxW vCzNLRqwme4/FpXX/0PzOFrH8/X8pTbjQ536KLOJwWeHzZJPq/1Xqyf/uGAfrYZjJqsOtqNtDi0+ /sk2xo+fupXLzZKct66O5+u4EjfjQy1JJcyOqqclIY/rOTtWHMT5d2r8D9WKC/UtEgePlvH6aknI G29Mcz3mIYLflOMz9Qn4ctlrxodaycn5UUIYBiHEydUgwVb8Swv9i6n/Dp3YPk8//UsT/w65y46X 4xft4itz56RTtfr1Tm2CWp1z/7aUUi2Jh2Gaxz9Sm/qvhueIe7SMk3ilL9FkY+L37KG83as+P/T9 t+MjnRTiPL8KWiGYFb52mJv0dPh1wTZ+Uv+V6Mw7GD3fkzzlNT9LvpukfWnntwX7WFcn4BuNkw+s Ln3R44PiX/3b5PdLqiX27+tX/Zl+ZCZKeCOezJHDyzV9HVvpGZPQ7mDakQ/RCfj6SaOV2ZFr5fhg y45WD4+v7zS+UgpItGg+clLeQTsATJchzXAwzW8O4aVnPDgvHN9+z8D30ZtpnyLx7218zMn3Lv61 4xNMVsylCXfJF+sghefUgfMy0Cp/rhLZnfwqTN/Rhzd3+zhcZ+D7qg6m8SaaGtUok+/X83dOai4m sISvJzEvphJN+XjTwdO/+fotx4UlNWN7nZPvw8QvcuJqniilembfr3/6F3TGaZY8E4fZSPrAa/0g adA0Wd/zu3V8Vn6nk/L15vUL4wFVskyTnTA8uIwB2yXPJFSsXIfjs+zkvEy6+DJ++dPiPSvf0JTt 9IjXoQUgT9sh+eLgX/lh0RMdJHtqE55gdOntejPWvu72JDot3zfiQfNNknEy1gzpZ1H7lQAn7ZUk RE+X4SoBk3R3cpXE1+sQ9IIPpP9yxoKX9vixpEAr3ZcuIEH9tfueWyXxDYAjwdfq2KH03/iBL+X7 Y/0k/2qBttMLHJ87aad9r6L4PnxH2NvXwzDGu7A8x2VSged/33D0HZp+4erHqCy+f1L7LM84jb2/ UJf8S6/puVDVwddz44W36es3yxBohgorFVWG/6g6+AbnUdTy4UgkDZaX+maZE6kSvgGwTjbGfrkM sX9fw1WZqoXvw8hw9C2XTvNz8/2OTqFq+D56A2Tu9vtlEMzSCet8qofvpXe6+Nati2/duvjWrYtv 3br41q2Lb926+Nati2/duvjWrTr59n3/yzv8r+8oQzXy7QwBmPsv7ugNEuHdbrOfI1UhXwNBgfB/ H9/RUbyeAO12uzpI9fHtAFBwmQ/vcHo9kLFb7uwIVcfXAWIEHHBB89EdPpRmkD/4m6JehGrj6yNX 5L8Bm/3klg70lvjRbLu93VUb3y5ijRZmYp/Y0cerQ+/lexA33+G+qoxvtK9YkYv0JwZuOY2h3APY bLvBvVUZXxdjElvx03LrY+dlwPxq0OepuwhVxjcUZ4Ebfwig52k5ULzAGQvAbr/LHVUZ31SbZd6B D2gRX0hcnONd2Gy9x11VF1/HTVdAQbQyNjN3eL6W2PTRxQh1Fei6+HbcdCNgMSXM0nIk16p/Y03f Zad7qS6+0nSFLHFmMjN3OI1Vyb/hF7vHTvdSXXxNys1EpHacuaONdTzdwV2Ymj12upeq4usBJD9z xZV2av9+SwsgeJN/6YOZqiDVxVdAiRslGM9VW0lWCBLL+KPZY6t7qSq+TvFS8i/O+hdTtHq6vqoA XRdfSmk44BL/zlRbz+8APD0c3499NruP6uILMs1KuJJ62/z1jljRuQNjwot4+fescjr08nAkzXSO L6EMUuJhuPx7ZrnEldKYNKnPvu/6PjsgbTrFK77X7LrnjVUZXynQjEqn2mZ03phwzBg7OvRM3CgG xitfnVbsX5CMhRqhm+dpE5FTNjP5mMKG7CyQL75nlZfOq22Uy+5oPjICPRr7eZAvl+Is1yPc99/4 dqqMr+Ai6amMy6aTjnMy4x0hbOWN0IzFn80BO99MVfF9SOcF5RXD1hCGPedpQKHcpMPtMEtB8m/z buVSVRlfhitNVpLWwLdjtIoXhoz8o659+ndUvCtQXXx/uI9KFZYcPYQlrdmoJm70sGPX0si/+H7p QlUXXycOBDVxYN2kMwQ65UaOiKkD+3H/5c+q4nNlfB9aYqUOR1k90UECLLPT4FKT0rYUaKqr/dbG tyXtpGxSxMGMiheGuanREz+opRm1b9tDNr6VKuPrhrgkatJxLtYpXeOoMXsYLM19u672WxtfHwGK h8NPOJjRCXHJ1ezfIUF3ye3StZtD9r2ZKuMbaSXA4adbOtzG1oswODX8ZvWUY09LtB7NU5WoNr6e /Svp6snwQZjmW9Q5CJ7nJFfLdNTsvuNtVRtf6cA67DbDUe6uqQizU2k46UlPjrpyNaqObxh4ZNyB 0STrQbsrx2fMoleo6ZjiVWXh+VEj39CCmSPcn4c8yliEKUUFh9+ep13Kz3bnrW6vCvk++thtsR8d 8Wpd0PwcHT4uxb2JF9ztvvvcQzXyffi+7+34gMM05Uqwjj/nrXZ6Ry2qku+LfMrOjDfGL6L7/G0V 6F/hS+xZ+KN/a9W/wddF03KsklEo+BdvR29qF/0bfD2RWpckKYcfbkdvahf9I3wj0Bif+YNbMDRH b2oXVcDXfnBNjFTi21ie+d9mpaXPreL5dkQ4H4VbNnDswOpfRDt3jzdgzOxVJ1fpfInDsJm7rOXi DNp8GfDcLTGTAZUOuHC+HeemeQdrgEb9E+6ZG498GqPsKhs9SmXz9YwtutHOXCg+T/4Nf+feiFDB a8hhZfNtOTQFCDSPC9i/w3hk/369Q/Xv7KtzbhXN1we0qFPtzKVdfA8gVej59tvF64gBNyvt9hAV zdcRqM1wrox6lNFI/Etwm1ma+K2hWBvmOvWpVTRfkprL/8xRaHFUnWGu6DoJ2eUX6LL5klCIMOZG JMdYpUDjbLpyOk/xh11pu0eoZL6ejRtDbuysc1ezgdW/s5ZsQReOFbpZZbfHqGi+8PQvzrrMD+MR 4Zx9H0Qj/95W2OtRKptv8m/0pJ273Gl+htnqzJ09LUyzrf3MKpmvS/5lCs389cSO/ADvg52bqsPF 9xg5VIcxBzt/gzehoZr+gwulJGh1uPgeI47Ev/BvkO/tR0vHpA2Xf4+Vj/ZF5bDqFONpABwecPE9 Rl6GHR1l7JpLE8ocxZ+3NVfeWUXz5QQkMyrAqku3PHOhuPi26tL7qmS+j9gdSWZUwlVXbiNcKQ8E zapL76ui+bYMeIsQ5Aik/4bF0a669L4qmm8HJM03cGjWXTq+NpzbEMy6K++rovn62H11iFl5aS78 EsyblZfeVUXzfZCGZ6C1TeZA8a4czPdW2Xy99kgCu/bSHMtjeVi3se+tsvnGhBVLKNxXX9lpOC86 XRXP1xvOz1tA6ARws/7Ke6pwvg8fMODdbrF0F93bbLHyjiqdbyDcb7Z0bzdbei+Vz/fS33TxrVsX 37p18a1bF9+6dfGtW/+zX+5IbsMwGEaDwo3PhoaFG53JjQs1PJsbFWx0hvDhJJN4ZzzCzhLC48tk O5j/6BNAKPzaJvzaRp/fIh1AFer8rglu7OKyplvmH03IP1oIbX4LAcKFW5yA+I7q0foEK/PbnnH9 t3CLG4ld3QTzisVQ5nfreimxiu9Q25eq5Mw7upYDe3YIocxvgi4YM6O2APUGJt4FnhDb+8E6Wg5l fsd4RtaAbnIJ+5DN3HIA5Bwthy6/G77693q8ttTe64bqDyyMo6GXc+9+KZT57YJ4G/QG4+Vo7we/ HEjXBazMb7NT/yMer322uT7uX2KUb/QSHH5/ju/0b/ODiOMHMuNo7L0f/fuDbASjfzl+sfcv9h9Y jh+NoPEDSZff1ycOwvV4addLY0Tjcrh+669GnQDHj5ZEmd/Wv+07dGGV4qhm9W9pY6P17/FSSXT5 3al3IOJyuLIAjv2oNSFHEr3Gez5eKogyv8/+iIGx/5ZR2eY7rwkfYz1jHC2JMr+lzVdW+xUc/dtn LHF+YHxeMSolUeZ3f7Qu5PRQv39xCObcv/t+b6XK2led3z0RpIVTOO7P8YfXhalWZ06hIOr87mvO rLq2/44vnNrAvJ9YV1aZJPr8crn/vn/bH+kw0/DjdwP4I/giHWYafvyWv3q1LcHfwI/fvv/WqxcV bkl8HPkttXGpLc90k44yD0d+awNXwW3LytJJ5uHJb6mtW+06un19+d1LG8/oaDo787uXNd1W6RBT 8eXXH+HXNuHXNuHXNgr8lpSkI3zJmhSsagr81o+ai3SGL9jqt1aWDvGR8/t9AuEZH2SCc753/3J+ v1T7FxbpFO8AEWGWTvEJBX7rg8TzNUobzwoGtAK/rX9v0ine2LA18CId4xPn9wvtOZ6wf9trd8Z7 41/O77f371U4xDsFsHZwlo7xifP7vbf9eZFO8U596xCkQ3zk/H63+hjP+BzvtX/Pd238z/n91gbG 861XbUArWJ81+C3rKh3hS8otS0f4jAK/wTcIv7YJv7YJv7YJv7YJv7YJv7bx5bckxCwdYiq+/CYC gCydYiau/D4AEAGlY8zElV8grIZhkc4xEU9+C3S9eJUOMhFPfp/Yx7OrAe3J7330L5B0kIl48kvw Epylk8zDlV/sAxooSyeZhyu/0b+maf3b1ivM0knm4cnvncb+DNJBJuLJ7/M1nuP7yCZbb17Ei3SQ iXjyuwP0/XmRzjERV34f2A1Lx5iJK7+FgAgX4RRTceW3CYabdIip+PK772uWTjAXb369EX5tE35t E35tE35tE35tE35tE35tE35tE35tE35tE35tE35tE35tE35tE35tE35tE35tE35tE35tE35tE35t E35tE35t481vlg4wGWd+H4RZOsNUfPktAHSTDjEVX37vUJEOMRVffoEqi3SKmbjyW6j17yIdYya+ /AIhRP+apa5XQNG/dqn3b8xnw9T9GWN/tstWx/NFOsRUfPndCdIinWEqzvzua5ZOMBdvfr0Rfm0T fm0Tfm0Tfm0Tfm0Tfm0Tfm0Tfm3z/DUApEZTMwplbmRzdHJlYW0NZW5kb2JqDTEwNSAwIG9iag08 PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvSW1hZ2UgL1dpZHRoIDU1MCAvSGVpZ2h0IDEzMiAv Qml0c1BlckNvbXBvbmVudCA4IA0vQ29sb3JTcGFjZSA5MyAwIFIgL0xlbmd0aCAzMjQzIC9GaWx0 ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJ7FeLquwqDJ1OW0EQhCIIgjD//5XXR6xR205f Z8++s7POufd0rI0xWa7E14tAIBAIBAKBQCAQCAQCgUAgEAgEAoFAIBAIhFVYoT7tAuHXY+LMftoH wm+H5vLTLhAIBAKBQCAQCATCbTBKfdoFwq8H42z6tA+EX47JsYR/2gnCL4fhnMlPO0H47TCCCg6B QHgHe4NSGEG3oO+G4sxctSE4u8MVwq+F4eKyDc2oc/nrsOay2BC+HZpxdl1uCF8Bu/bCkYQzefgz wvdBszW9cN0t52vd6fpnhC+EJ4JefOMKjufJ0c8IXwjPBLX4xvqCw1fuMJ4/xJI/A6cYa6XDilWS vDQ1tn8LJ9tQ6l4JBEILK5k0eyZOfKWNIXw9QqfKdtBEMjdxrVkhfDcmd2XhXL6dZ5m/E7Ef8Ijw +yCDllQsUf6OUzapxk1zPKHO9U9CheSXDYfXF1dfSkbwQJOfdI3weyAcI0QxokNxcX+LUeOHdrW5 hC+EVrocCEXI86QUE6voikN4WROlQnIQEwOj9zQj/JHwfq6KEyMt+71f7cNjEUdOwB0OwQ5PlPCb w3EMxuuH8MQQsVGNLJH+x3THAt2ckffmdJwYRW74CZbo9x/OuMMh2KF4P/NfrH4Wlgdi+EtM7Eti qyr40h3o1AI5I+8PUKElP8KSI1pyh0PnteSTLAma4Sjh+SBiY2JCyxpG+fW2VeSMPN9OvlNLbOU8 acl5MGhZmf8xuYdQfFQavd64DiglbyffpyWGdZVOkJacR+pYwW1oWFUavc6SDqXkfQFT2v+JXlwI ixpbnRgintFqBz+PiOWNWnKCJSbG5tLqZwFVhpdua+hj+WWnND6446FPzydFbugEXLmGE2Zv1JIT FeeTSKpRRZQzrDAXwCI9YmzeNyYY55MCTFikuPgsS85ryUchAx1qbpvQu97QvPax1PQbeVvD+aQI 0pK7oaWQbfbsJORkLxu3wI6xbUysFAMX9mVRL+Kf3d84oUmKcuVx5AK5FT81se9OnFYqraZUu4NW S6yzO7q2vSKVmTjvBZ/ySZkd0oKNYiqd0MGJ6lbobQyco/g2WgJTpvI76cbcPk0wHR3TITTISSXr cPjNTFwMTMjr5/sHMUFgpyY3EtpartHB37rjcOg7HVgKDJgw8Kq3eXBN2RstYXN/3aHZKt/NhpSZ 5FCqnwo70ZdOeJhxtvFMnKq0RC8sM/vYGY5C0JfhEAvhcDe7bO4zfe4pjJAS8yj3mGjjoxr/CZvC jClZYnM4fdDhqEBA5jz3eTCiZUmlJeaJp/dplnwsWAEX8hcGOfEsnXjlc1DYKLWkXCYxaSZXN6IQ lIdmxB92esFcN4vdRSjfkWxwbvLvL5WdGFGRHnKrUCRnfgMnLbpUhIWVs6EPro2EkoZ/t/W/0pK+ /B6GdTkKlXKoRoEPjRORArYrR6ONQkvqZabF0SWW8HLC0yLbMy5UHSNnXkzxIrNGEytYuOoATcwk DrNTz7sfcQBn1ndszn6rJVhiUwAYe+IwP5KZdLSG1zEtyX50yI0X/HgOsFoXYpBYMs45Kp1Ie4m2 YXZX2ii0pK+XwR8++cxgPBxzMXvSoXDE525SsEd2NF0mtaLaJz6G2PDIgrVbmYpvGUez4xtdd3pr gCTY+Qk0IEmDzvnf1pI8H1QoBvRR248JEthMhVJLMhNNjPEY9vqYn2FhiRzyzx1yGpwws/tBYmze MLaBtaRdZsr2vDwk7jfh4Hn5fg6Hycv0RbDfwXIJ/zKgRtSHoEVToID7uyImIrx0/wv8kuExhMW4 xygrhsvlbwHZW403jPf2wod4tS955hRm5s19SfWIg9juCk80KGdjdonnJWAGQw7Z/KjXnZDZB4ts YC2BZUxehucZog4aDkfXrGOKI4b7mfdQLBLC+ITHAIbUh/hJkAq2IgyJJNGEDI/xxLtH4BzfVrUm gFCd4deIdr+tJSibENCpGsbJ36ElsaXArJxyzuLKUa6e2S5ySKAPkcEhT8bXkyHPwFqCc9nPU3Rj GhtpwmHnuOocgmkI2MxNhpWQlTmvUT7CQZwYlJw3WhKJILMImZlZZrv0TIgZsElgFdomTumalijE DPiWLz4e1BKO1kYM7dEUlGHkEP4QGRzqxzAZ0wFrCZoNz151ZWP6sRIOXoYjNSvD6a7VTtB5hvZC wmPUkrW+RAclSbOtf4Qrg5n29SU4W5CdeEBxevGkNS2BcamUVvg0No8H+5IkCc6qQiUCTzHupftT OdRqyTJLerwM2klwGckUso2dx1RDz0C1UZfhmO9gqxeS3TCTTCl+c8fx9WRuSzzR5D5uZHRok6lL jURHsdqjJe3l8FYtwRCVLYQTWlIsg3bSuJxtt6Wqea6vvI9IR5N/Dpd5kqEdD8SGPimvJKf1K5XM TnDBuUBZOKslGKjZuawlGPy1hyV7taRYxrxKLcHLNFpSr/jm0AQrcE8L6M/n7Wch281Ax2pRrE5q SWhw4iNpSbJiWR7pjir/hzC2m4HGBF38jmiJa8HnPyhBzcHEOayxpCUC2a37EoSrWoJ2sqElO1ii 23BA21nw5Maq89JSyNaeux2JyV6zvESS4vwFlhzQksZNZGYh+cfuOI3ZhiV9dugfasmOioOp1sDO ZfS5+P4UQovK6gU1D53rJTYulIkqQGHVA1rShAUNx8d4p9itJSsTOzSFDUPv/lYO7dWSR7SxvBNM pGUteS6zBFNtCUlPbqs5E1x3K4OBI+7FFdOwXRfjiCdKJNrmiDK1piW4Qin03Dwe1JIJTURsxbdY 5MWhvmRsEtyhHfJqGH2JfXosswTCIcpwGH9J4OE2wtCEOyACHWo+aCBJTZ5DAIGee+3Uc/k69sh5 eKKorGkJzA+dbxvF830JrCf9M8vu4gx32dOWJRtagu41sMUerYhVtNog8ildbVfCMaBwyOKIwe3g 0iHH8FTwWgIGTWxFFIsVh19gCXja1QMP32j1cx4Tdza1BOIcbGHtQcHAyd+tJfDVMy/xQHO8oyqn b6HibGiJyjawEmItkdlNTNcct9T+N+FAzzkcKpuzyNwdYFBb/MXSTqkXUYE77JKW4PhGADX8UhDl jvHugVK6qiUwf7Tp7MTOLD6vacmgX23/XWhJirGcpcQ7l4L8tC/b52Qf0xIQoc7Mi0xoh8FlMy9T 6C5EpOPzHbEJh6zD0WWve1uepDsgo5gwvwURq4yzbVLFuXDLwbGJ4DnBSUsztrWkmR/NxudGS9Q8 rWV5qSXqUSFuuK9GO1s59F5L0m5nRF4XjSerpsQPZe3TowmHrWeEcCR6pQ6wuaSdhoXSYryA5JY1 NrXsimSBqyaPAO/DUI7hrr7kNV/vigDEH42WpHgtnaZSS2ZRf+DsvUxXDk+1Qzu0xFY2oivFba2a 0sUpduZot3LHaZjUF/Etzd0C47sS4c+KjA0K84x5TRfrTTo0TzSUTgBuFh8Ct5qrWlKlk8Fg/NVo SZaed1pSycZ8KdBF/kAPj2nJSz+xDVmEhS8s0yVnLSzUmR6FoAxHIUOpqhcx6tD5vAEGWlaZyoyB 4UtWZZnPgB5nyMhx6N29Dbea61riIjqk/Q9zFUO5LZNvGbaEUWmJOw9zMkfEKctSAjuWAnFMS5wN MZNgTKW70BKHhWX8LDEMXshXOtYyHM9c1FXmvLjQLCBoVYVQpopT2rdK3bPgCraurRWU5ILrnd4Y pXc6bpXgQjR2tXTD8ppsmymY3pqytUzFjAqL4VCTG5zuKjaCM1byTQeKuP9e9Si7scKBVcdRpYPZ rWvrn4QKsQnKElnSf84V1jYe05KUhBpUEucGIGkGLfmnevW/QoxHCHlXVcefhgy6Ud1htGtNpjJd Jpahu3PIMjX6TVX9g4jU8B0ptPvs7Sf/Cv6qy2qWLMAEKbldS6C3HU3iy+ci8eswpgbXPD9djK3I t5lNSD/vPZuOorglUsHBUFVonh/0xUghd911Xc/8D8hsykjcT8P/MXgRmm5Xlr4UZkCBuP0O9f+G 6HJsPtW62g1y2nXl33h1DoaPg8e/kKr/OxQPoRnlpyqxZkysvdpoQfxn1Dz8GQh3Z1lWeBtuvdPR zwhfCHe5YcvpVjzQ5OhnhC+EZnylqmh/6eUrLHEv1woV4Rux2l54JWErFWfjM8Kfwlb3SiAkGBKM vw59Q2ehSWy+HIozc9WG5OwOVwi/Flasdqa7YYS6wRMCgfDdUEJerkmEL4dhnJpTwhu49pbzTztB +OWwnLPrDS7hy/Ef+3VsAgAIA0CwE6wEEQQh+6/pGAnhboJvP46DAQAA+rhzZSdQ3h4jshuo7q2T nQAAAAAAAAAAAABAd1+AAQClUBnXCmVuZHN0cmVhbQ1lbmRvYmoNMSAwIG9iag08PCANL1R5cGUg L1BhZ2UgDS9QYXJlbnQgODEgMCBSIA0vUmVzb3VyY2VzIDIgMCBSIA0vQ29udGVudHMgMyAwIFIg DS9Sb3RhdGUgOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYx MiA3OTIgXSANPj4gDWVuZG9iag0yIDAgb2JqDTw8IA0vUHJvY1NldCBbIC9QREYgL1RleHQgL0lt YWdlQyAvSW1hZ2VJIF0gDS9Gb250IDw8IC9GMSA5NSAwIFIgL0YyIDY0IDAgUiAvVDEgNjUgMCBS ID4+IA0vWE9iamVjdCA8PCAvSW0yIDk5IDAgUiAvSW00IDY2IDAgUiA+PiANL0V4dEdTdGF0ZSA8 PCAvR1MxIDEwMCAwIFIgPj4gDS9Db2xvclNwYWNlIDw8IC9DczUgOTEgMCBSIC9DczEwIDkwIDAg UiAvQ3MxMiA2NyAwIFIgPj4gDT4+IA1lbmRvYmoNMyAwIG9iag08PCAvTGVuZ3RoIDYwNyAvRmls dGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIiYxTTW/UMBAViTfemFUTKAhFpLum3Rb7EK/H sePkigoIOCEicej2VNEKpAoBB/4+42Tb0hVFvcRyMm/ex0yAX7DV20/AL34x4F85qxtlHfeuUXXN Xd2ptuWVh+H8+YWdsx9Mc2jxc8Mra1VjuebOO2U7XvtaWeBnl2z17tLy4+/sI3vVs9Ubw4H35wjs VIew4Ykop0O5N4Gxv8TPF0wrrbXh/Rne+t9MPIhk/42terjq0CAWeDUe2MP60KMF1cLYA6Hi84cB 9X/eFj0NvCciJrICUJ2YJDSepnStweAbo4x4KE/796wC5a3neBhtW94fj0IhsJ2I2Swg3E6WP6Kz KJJgxOMB1mAHZEK6zgaQmO7sPqGzIO51PyZZOzSjlTcoDIxX4LnrbEhkjNH8FeN1CCH7OlgBrVXj biKsBln1EMKLebZfFmQrCaQIUYxHmKRVpuMen+4mv6cj6IoPh9yGsLdsPyNFnEUHhJIsKzO6JIms cA8EkbUCEW9uUTmhcZ5maRzuTiQkiwpZecwWKwAro7igsgqY8lBWxnZi+nyKJQaXEEQ6WdIhTB3i 18Zdx2/8qGOaE+mUF7EE5Nsr5oviaEBc2642FrBBY5pmaHAfo3bL6GAJxMt9UtDycPRdUJLGOV2Q JC+XNCroASkSWsyjTTGulsbN+peJcVqDC3FYZke3RnVLs6nvrdlsaV6LPE1ogppvJA/Zimg2nYW9 Dbcwh/BatSIv5xL/LyPWUmrUvZ+VCYmTO/XDqD8rF3fqr3V3X/1jt910j9CMxOViQSbLNMEl2vw0 fwQYAOeK0BMKZW5kc3RyZWFtDWVuZG9iag00IDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVu dCA4MSAwIFIgDS9SZXNvdXJjZXMgNSAwIFIgDS9Db250ZW50cyA2IDAgUiANL1JvdGF0ZSA5MCAN L01lZGlhQm94IFsgMCAwIDYxMiA3OTIgXSANL0Nyb3BCb3ggWyAwIDAgNjEyIDc5MiBdIA0+PiAN ZW5kb2JqDTUgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvVGV4dCAvSW1hZ2VDIC9JbWFnZUkg XSANL0ZvbnQgPDwgL0YxIDk1IDAgUiAvRjIgNjQgMCBSIC9UMSA2NSAwIFIgPj4gDS9YT2JqZWN0 IDw8IC9JbTIgOTkgMCBSIC9JbTQgNjYgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgMTAwIDAg UiA+PiANL0NvbG9yU3BhY2UgPDwgL0NzNSA5MSAwIFIgL0NzMTAgOTAgMCBSIC9DczEyIDY3IDAg UiA+PiANPj4gDWVuZG9iag02IDAgb2JqDTw8IC9MZW5ndGggOTM4IC9GaWx0ZXIgL0ZsYXRlRGVj b2RlID4+IA1zdHJlYW0NCkiJpFVLj+Q0EBZpJ+6E0bp5DBprM92mHzs2EE/s2Hlc0QICTohIHCac VuwIpBECDvx9qux0b88urFbiEndS9VV99VWV24j74vabH424/6sw4ldRNK12XnS+1U0jfDPovhdV Z8L55y/Fy+KPohamB3MrKud060QtfOe1G0TTNdoZ8eKhuP32wYnnvxc/FF+Oxe3XVhgxvgTgoAeA hSegfI3uncWM4wOY74ta13VtxfgC3sa/C/leosbfitvRHCO0gDWiigfEcB3G6I3uTYwBUPnT9wH1 9rw91BTy3skFUZUxepBpRhfLnE61sfDFaivfVz+P3xWV0Z3rBBy2dr0Yn0eiBrPdyYsLRPgnbPUB vUgSZaz8MMBaiACZIN3gECSXTz76mH6C5L4ao5KNh2Jq3VkgZmynTSf84FCRKKM9k/EkAmrfYCmm rnXrX0lYIa2Z1aeMbBJOlNNGUqhGHkpG+XpLWFKyJLuh5YEGlnWo61gUEEAJl0+Xr2kI5FDEeOAM OG0H0cHTv1L+MoKOTGE8emzTUbAuUluWm02ZTuqGrsik9nQHz4QckNFrBJdPVeW1l0soAA66yznZ 0zI9Y15b408JfEywjbXTbIXwQeYZnJJuSq5gdHqZUUh4jTYniWogMpIhjIFE2+gcHtHGUk4XZcoT ns14pHYgkzyn0SCJO4lBUlKuOWhudSd3ScafBbeTltWsC6BaawPundSz87itFyua012Sc4osdSNL BTGhCJmzPad8oyoHX/NJxR+TCrXhSy8Zp4zwNRZiwaJqOQfhSBhqS1I6SZJfpwS+wESeDOea+7AF 1fkaAG5PQqxWkkWWpJjCyytVNTiCZbYi7JHG4JcRNru9VaS2fReR4vTPKn22wrwN9IPQnOFvYEVR AicX+ToaOb5bSeYzrETwiHKhQuxoi4YMtXByDp7jxzArLbSaYheMZMdwcxLsggfzDJ0DwTUByDS6 zMggSxtd8SDgAw1LH5GdfSeZZKsj+o1leNSYSeXsitB0HeailTvoA1Qaa/TImMMKJukVjfFmt5IR eqwguSa4IfEVGkjwiiTpG/Mf8/ahUeTzdbJl5S7fTYo8e3Sb/M8NwNv2QGG858E+DVQTBopDwwmF 9MHjhp5IwMVBcLwvv6gu/2WaQwo3p5AJglEJmN18uyU5i9tmZAlqhf8IiJ/s6T7P8pMJmwMd2aFS g4SZhytsSfN0/x83lm3nK3FFFC7EAlreyGu+3sR9gP+JfwQYAN5rQfgKZW5kc3RyZWFtDWVuZG9i ag03IDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA4MSAwIFIgDS9SZXNvdXJjZXMgOCAw IFIgDS9Db250ZW50cyA5IDAgUiANL1JvdGF0ZSA5MCANL01lZGlhQm94IFsgMCAwIDYxMiA3OTIg XSANL0Nyb3BCb3ggWyAwIDAgNjEyIDc5MiBdIA0+PiANZW5kb2JqDTggMCBvYmoNPDwgDS9Qcm9j U2V0IFsgL1BERiAvVGV4dCAvSW1hZ2VDIC9JbWFnZUkgXSANL0ZvbnQgPDwgL0YxIDk1IDAgUiAv RjIgNjQgMCBSIC9UMSA2NSAwIFIgPj4gDS9YT2JqZWN0IDw8IC9JbTIgOTkgMCBSIC9JbTQgNjYg MCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgMTAwIDAgUiA+PiANL0NvbG9yU3BhY2UgPDwgL0Nz NSA5MSAwIFIgL0NzMTAgOTAgMCBSIC9DczEyIDY3IDAgUiA+PiANPj4gDWVuZG9iag05IDAgb2Jq DTw8IC9MZW5ndGggNjE4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJjJLdb9Mw FMWF6+QuoVoKDNGIkJp1BQcpru3Y+XhFAwQ8ISLxsPA0sQmkCQEP/PtcO23XoQnxEiuJzzn3/GzF LuP1m4+KXf6KFfvK4qoWxrLG1qKqmK060basbJRff36JL+IfsWSqxd81K40RtWGS2cYK07GqqYRR 7PwqXr+9Muz0e/whftnH69eaKdZfoLATHcr8E1VWuu2Ndon9Ff6+jKWQUmrWn+Nb/zvmd0jRf4vX vdo61KhVrBwX9DCN82iVaNXogVL+6b1X/Tu3xU4+94xPaFEqJToehDA5iGCQSuMXLTS/W3zu38Wl Eo1pGC5ampb1p+OgyqWd8enUKexhMrsHU0IKpfl9L6vRAZMwrjNOxA8OHxzBIzfcq34kWVksI0Wj cTClG6EaZjvjiIwY9R7GHQTHvnJVlJSittcISz+W9RCeJtkyWpKhIOFzyFa+VJSQLE/hOApDmgTp X5Qq6TGNy3jKXcUaI7S9ZvtwFG1nEba1FW7eImlHJGJGA5iuH4NMyTyIhgLCDIKMpLRAUw5IVyie RCHJFJDCIukUwllhRMMpTKI8pZAVhq8KTMDtjqZ0/KXaoKRJkiXpEUQpkAAimGSLRR6QibMMUsDa wZOjXVNwU48GGi/2dlp/0/jASTiDFR04LEno/bw7RAmFRYaPY89yKOizG8jKTX20rbUf678YmT1G MzKfQbQ4dseUn9AdI1dai4qfkDTMk2UUTCgQh4jeAmPv1DfoVrDlsyMwthyKKJljTL5Y0sTtux1M 5c1GDzjJExLMffb+OJMogOUGkHPpX2zu30j1JjO8738EGABnmrSdCmVuZHN0cmVhbQ1lbmRvYmoN MTAgMCBvYmoNPDwgDS9UeXBlIC9QYWdlIA0vUGFyZW50IDgxIDAgUiANL1Jlc291cmNlcyAxMSAw IFIgDS9Db250ZW50cyAxMiAwIFIgDS9Sb3RhdGUgOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzky IF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSANPj4gDWVuZG9iag0xMSAwIG9iag08PCANL1By b2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUMgL0ltYWdlSSBdIA0vRm9udCA8PCAvRjEgOTUgMCBS IC9GMiA2NCAwIFIgL1QxIDY1IDAgUiA+PiANL1hPYmplY3QgPDwgL0ltMiA5OSAwIFIgL0ltNCA2 NiAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSAxMDAgMCBSID4+IA0vQ29sb3JTcGFjZSA8PCAv Q3M1IDkxIDAgUiAvQ3MxMCA5MCAwIFIgL0NzMTIgNjcgMCBSID4+IA0+PiANZW5kb2JqDTEyIDAg b2JqDTw8IC9MZW5ndGggNjM0IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJrFTL b9MwHNZcJ17CtGQwRiOqzmpDsUFx/czjigYIOCEi7bBwmtgE0oSAA/8+Pztb90ACCXFo3Cb+ft+r jqLn6fr1B0XPf6SKfqapqYV1tHG1MIY604m2pVWjwvr9U3qWfkslVS08rmllragtldQ1TtiOmsYI q+jpRbp+c2Hp0df0ffqiT9evNFW0PwNgJzqAhSugnPTbG+0Z+wt4fJ5KIaXUtD+FX/3PlG0h3n9J 1726mlADVtFqXGCGbfyMVolWjTMAyo7fBdSfeVvwFHhP2ATzSinRsSgmk+2EDFJpuKOFZvf4x/5t WinR2IbCoqVtaX80ClWe7YTt7HiE283yPbKDEFea3Q+wGiYAE9B11oPY9u6DfbLlxb3sxySNAzNS NBqEKd0I1VDXWZ/IGKO+EeMmBJ+98VaUlKJ21xFW17LYI1TE82yZRBNMJkkEn3mBycCTbIq9Ahns eFkjqg2ogc8mycCJNo/vZGhkCHFcgLoDFFBbod118g9H0JVS4VpnYPNVYC5sEjmOSHlDXrFIoghF U3I4H3iMFgPHZInigkxA7uGiIBu9yhq7mabH+As0jbzkmFdGGDYjGF1+zbmECnEGrrHvFA98FYrZ 2KouJYaCbUjjb0bGtEzYtRuVo9AkwwRFJV5guETxrej+geMOBUxGXj8q597ohnGPzDJyQAaG4pzg 5+Hhb1GdsEU2WyZLNHAUP135Zkk5z3zccY5v9QCDBu4JMIfXACMhq/7ZaFmN5R2s7H9xV4/l+YPz hJRJnBDEHSvg2ImWgcWkwOU+0dxCj6DY/4OXHF4zDJMcTXMoNc7h3sGKGzi4NkiFQ/VLgAEAgLvR HQplbmRzdHJlYW0NZW5kb2JqDTEzIDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA4MSAw IFIgDS9SZXNvdXJjZXMgMTQgMCBSIA0vQ29udGVudHMgMTUgMCBSIA0vUm90YXRlIDkwIA0vTWVk aWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1lbmRv YmoNMTQgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvVGV4dCAvSW1hZ2VDIC9JbWFnZUkgXSAN L0ZvbnQgPDwgL0YxIDk1IDAgUiAvRjIgNjQgMCBSIC9GNCA2OSAwIFIgL0Y1IDcwIDAgUiAvVDEg NjUgMCBSID4+IA0vWE9iamVjdCA8PCAvSW0yIDk5IDAgUiAvSW00IDY2IDAgUiA+PiANL0V4dEdT dGF0ZSA8PCAvR1MxIDEwMCAwIFIgPj4gDS9Db2xvclNwYWNlIDw8IC9DczUgOTEgMCBSIC9DczEw IDkwIDAgUiAvQ3MxMiA2NyAwIFIgPj4gDT4+IA1lbmRvYmoNMTUgMCBvYmoNPDwgL0xlbmd0aCAz NDE5IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJjFfLblzHEUVokWNNBNF5LBJA i2sIgYcBeNmv6sfWcGzEQRaGBvBC8sqJBAtiAiUL/X5O9bN6Hgw3vHPYt+p2dVedOqWXd9u7717p 5d1/t3r5ZdlavzpaAvnV2oVsWmNcboPOz//8c/t2+3GrFh2x7Jdb51bvFrVQoNWlxQa7Or38fL+9 ++u9W7759/aH7df77d23ZtHL/i0M05pglv/CihS/Hgx/cX+P5XdbtSqlzLL/GWj/abv71cXN/v32 bq+bBw9bvdyWB3y4wD6iXqMuPmC6+/Fv2erh70bElL/7evfZk5tbrde0u7zafPb5080bpQ3+Y1az +/XNT/vvt7d6DS4seBjl4rL/pmxU89de7549Ywt6fv3FbzbPLi5utNn9Npt5eMCX8Lnk2Gj3+fPf /X7zB97cX/blJC0hGLUGg41pE1YdFkqOT6QcoxHH2A+Bz95yKFqp1dM4wtu8rZgP4cvrFy+fvrx4 c3Nx9dXmxZ82zy+/2tDlk42/4PCeXG54GyrHNAe0uwwvL642Xz69unpyfbl5s7u4+mITN0+ur19c /7HvnXOAM0VFfugVX7312BSS5Mc/L/9CNr3DOzi3sARtV41s4mS59Y7W8tpbxDX8mJXCYk1c4Uet 2px0lOO0Nq6as2/1sflRy3dbs3yPI3mP4042LZ9wOMvfl9c/qeUfW9wa57OOcU1hud8a7Minhj9s nUmcQH3dIfFxE3292htkjqJhX3G37+vVvq9Xe8tJJr5fcbfv69W+rzf74Fbc0bAveNi39Wbf1qu9 Q3XL/Vfc7ft6te/rzT4FvshhX/Cwb+vNvq1Xe6I0fb/ibt/Xq31fr/beIM/SsK+42/f1at/Xm320 0/4rHvZtvdn39SkfRn6kyG77ugcBejphb41isun2DTf7irt9w83+VSEA1whA416TBg+2H2pxZFfT U7hR4eudurm10aJWdoRf3qrV7jT/0ymzxp3m/1owjt0Z/q/NrxrqL9jxrh3vOpUJDkxwVFrTUR8f /ZmjPVwfpXlw9PP3+GgU4tao/8wan5bXhnvVI/5kZpgz/f6gUvB5HVcS4VUsbuZjr07KSxGkD3pW ncAaHxUq0skzlVmbGxc+GAJbWZf4wQ5/OLCY30ByWJ2LG90l5OSZMNa9b2SQT2jGWE+gfJBHDJww Hw4wQlT4AZpNqSb3hPkIAnO1AzlRvqEJY924fKTG8mc/HGBej5zcDidQyUtizoDcCxzi7UfMxkw4 AemLZIDciHzGsk/MZuzGsShxXrNJOVh2hDYEpsIt4QEaIH/sSBqyI8/n47g4tXCULxCdPVcy+tMJ R9KQHeWDdJHYZDiK/B1Snq9HqzVO2TMbDSeg1hSOnECoMQlUJ/NmpB37ifwqaVWyoPlJRfgZDg6k iI5tc3MffqTd8GOy9jv0wwqRmp+D/Ui74ccSWxz6oVw0zc+Rk2I0nEAVhnDkxCdOsIc2U+yGHygz dRwUxIh5OKhiN/xAuJjjoJLlPHvIT7EbfmLOzAM/XuWUPns4xWg4AQH5o6A8eC0+fOPFrvvxKpX6 nv2w7YOHU+2GH2MK1c5+nOXaechPsRt+ULbm+HCIRCYfHU41Gk6cP5HG3of/k8bVbvhBLzhOYx/g 4uGgit3wA8KLx0GV7lP8QOOc8FPshh9YdLqwoBqQaVKFzyvUpbNVvoWsZ4GL+QI9BQTPY2AhudHW VEDXreyvMU4l0R00mkoY/Wx6NVFtFA2HwtAVG6UrtTRsa6No2FcqbLiedcMY3ErCN1wTo+NYq7Ph vPWxbpqKa5imRmdwCUb6t7pQaMduOgpjfeH7hp2a4jfOlKru2M/xu1ilUsWk5/ih/ILcH4U5fqho J/fnm9RqmJrWqThWFqwYueTl/iAknNxf8FUoVBzbFNRwm3IaJvYn1kMhpIaROlJoGPCkkftLvmmx imOpkYqtMpWoGnYlsTuOlVUrRivTYn8WqiAmuU5VyDRchVjDqEAn9meRLzrI9VCnlIpR+VHuz1Yh 17Fnf2I9TvVhkS9K7s+5qT4sdzG5vyYtG0a+WLk/ojl+SlN9WK/m+L2b6oP15RQ/8kXWhw1mjj/Q VB82hDn+2IRkw3aOvwvVhtMcfzJTffDEIvnBIl9kfTilJn5wyBdZH07RxA8ODUzWx7EQpil+fkh+ cEZN8TtjJn5wxk/xs7aT/MBCV8bPelXyAys9GT+POJIfnLNz/I4mfhgKumLSEz84cnP85Cd+cMgX yQ8Og4nkB+f9xA/Oh4kfXNATP3Th3XGYBxF0VckPrO0lPzgIIMkPDrOc5AeHRijrgyW95AcHvpH1 4VKa+IHQr2R9EPJF8gNhfpL1QVpP/EDgG1kfpP3ED4R+JeuDkC+SHwjzlawPQr5IfiCrpvogayd+ 6JK74zTxAzk91QchXyQ/EPqVrA8iNfEDkSnxvxr/8VOFEMX5BDB6yQohb+cT8GGqEFb20wkgY2SF UKD5BEKcKoSimk+gS+aG/XwCyBhZIZT0fAJQOLJCKIWJITwyRlaIV3ZiCI+MkRXiVZwYwkPhyArx 2k0MwWJexu+NmhjCQ+HI+L2hiSE8FI6Mn0W9ZAiPjJHxexsmhvBQODJ+78zEEEMtNxwnhvBk5vjJ TgzhoXCm+NGxJEN45ItkCA+FIxnCQ+FIhugqu2M3MYSHwjmUxlWD61PSuJkhrSSReBCT7iLcZpbq IrxCUy6qQSr7bLCOtRVqpFHeZse2XFPHoXy/41SuqWGkUSayjl25pldj1DAQR0aPUcM4MQAEnVjW lxFjTAAFVxl/MLqwGO8jEIiM/Dl/YwxomE6MQsbk5lL8MVGas/76WNCwLUk3+0MzxVWdGtGO/LUk q7iNCQf+MvnXeNU8QM3+TBuLGo6NNqU/l5vDY/z1MaThUNrU7I9yTjwm3mI//JEq2XTkr9/vw/dB bUwY+MT9htwsH5Mvfeyp2ItqG/6iEfcBoXPeX7Ef/tBUTtwHxIHv9wEhdbY+xljUcB17Jn9W5VZV /aHtxPP+QiWpirETdeyPxWY/v/BA/Y6xqWJdx7YDf2HUB4SHO7u/ai/8xRP1YVl8Puo+xlhVsfUn 7sNSbnWPyJcxZlUMmaOP+MCWVtPzOZ4/vz52NRwbm0p/IbfGXm/p/P6K/fAHURSOzy8YsT88zror 5sKdnbeXh0yetZBEIP2Y6f7U7nJTGUNew27OvuIu5cZb3J1jF2k//LHkkrcbuLU5lWu8dNtbqBg6 77BPgTaLtzr1DYfE2c6dGaQGKid2iJM54U/as78stlgGTHSaj9NBKeGrmABdPM8HltXEGBPRfeUY 2Bzm++LhDYFjmw/wi7Rnf1nN8sO1DfZ3TVaSfEbltFHCmke+WEecCfO64cN0LMiKwMrK0HFVFGUT cyxeNYEoMa/n2cP5VAUW6A1Z60IdkWzKs49D2mUBbWNuly5W/5zejDmPsrLxuf25FKowojx7Ebir KCOXX+wjGCsbfJ+KivlwgHk9/xjrxpWRyxSBa1WeFQl3le+GSTLbh1JMTMKMcVk5uZjknR4jmgk5 a8hUZTZjrHvHL5Kp8XETi4x9OX8DUor8flVuxuakqSMd1o0qWLdk/Lg1yvEZk3N8hrgQ6w+lAmsX z2amhKFT5q4+CWaBik6Oq+8CtUCt6ijUcNeQDYfG8l/vt2qB3FNJa3BT/aEgbHKp65Ia+/vtTqub /Xt+GcmmDRNZ/YGXtWd/uuR4flnzy3ff4lvL/u3/iq923raBGIy26eJfkVEeZOh01D3Wol06tai3 plOBdgmCjP75+T4eT+dzpEwFuhigSfF1/Pg4jAwsUPT8+TC8KzwpvE3zIjohXGkT1KjWv9BbAAQJ R0E6v9hwqHSyC+2Wb8tvoRtfLC0rXS+sjm7fV/6PA/s+sOAWvQlFG7G93afdnPosLJBAXIWcS6IY FtLj8pIhC6XeaZLm3WyL08MwxMxHppIny43PWa+e6EqTqXRal/Ld50ZtcUNXREHlz+HbcQSAhstx FOwsMgAyx1/nr+1NMXdCdvak77snRTw51+eG3wEN8fz7MDwMD0eLN8SZF9kpoZQhNJG/ZXT5H0bD PzMqyr8fE23Oq9FBajmzyUYer3UvNrpcpY8bfJ39Rjd+dqWXrrRYr+/o9n3l75eEYNhz6obIJeEa gqbThQqvSqdyq73ml30lVPgZv/zdvo8Vfh3dvo8r/HbL2OtI5FhOV9CY+OgSMO1XZCGMWbyOKS7f Xtu40esOovXAQZLYeA3lKb2NcsubzKE2w4sZdPd/Dy7pQht1bfTctcZg+/4fjgVRtp8V4hMvs9FN cL5NBfFFIlACE4IC8WQavjcd86zduCwl3VZnKuYy/zjlsPfV7zWsTLX8QUqxJSQKe04dhDMhCjSD UuhAz+WIQIfnW8gsAfU7tYp/jRsfTQDhxbwUdQqcEVddVQln4AodWUQd4ZZijlDvpTzwhsdpMo/R zOkMpd1zN5eQ4M7JfjKhaFwNAo1xkqXD9q1nfoo1Raci+XS5sSYhxDfswe9VgHgm7q4MNkVb4UZZ H4gffdCeB7dTDXG4+3hViFdVUpagSvuyOvT8R9TGC4YZuVMKZW5kc3RyZWFtDWVuZG9iag0xNiAw IG9iag08PCANL1R5cGUgL1BhZ2UgDS9QYXJlbnQgODEgMCBSIA0vUmVzb3VyY2VzIDE3IDAgUiAN L0NvbnRlbnRzIDE4IDAgUiANL1JvdGF0ZSA5MCANL01lZGlhQm94IFsgMCAwIDYxMiA3OTIgXSAN L0Nyb3BCb3ggWyAwIDAgNjEyIDc5MiBdIA0+PiANZW5kb2JqDTE3IDAgb2JqDTw8IA0vUHJvY1Nl dCBbIC9QREYgL1RleHQgL0ltYWdlQyAvSW1hZ2VJIF0gDS9Gb250IDw8IC9GMSA5NSAwIFIgL0Yy IDY0IDAgUiAvRjQgNjkgMCBSIC9GNSA3MCAwIFIgL1QxIDY1IDAgUiA+PiANL1hPYmplY3QgPDwg L0ltMiA5OSAwIFIgL0ltNCA2NiAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSAxMDAgMCBSID4+ IA0vQ29sb3JTcGFjZSA8PCAvQ3M1IDkxIDAgUiAvQ3MxMCA5MCAwIFIgL0NzMTIgNjcgMCBSID4+ IA0+PiANZW5kb2JqDTE4IDAgb2JqDTw8IC9MZW5ndGggMTU0OCAvRmlsdGVyIC9GbGF0ZURlY29k ZSA+PiANc3RyZWFtDQpIieRXS48bRRBWZj3ueFjFCwSBhbXbZDebmSD39rt7OEaBiCAkoljKIc4p kBVRNgQ4hJ9PVde87HX2wQEOXDwed9fX9fjqq7bip8XJo6eKn/5ZKP4rL4wX1vHgvDCGO1OLGPki qPT845fiVfF7IbmKsOz5wlrhLZfcBSdszU0wwir+8qw4+f7M8oe/FU+KB8vi5DvNFV++AsNa1GCW PsHKSdweNJ64PIPl00IKKaXmy5fwtnxflDeyavm6OFmqFsGDreILegCGDYgRlYiKMMC0fPZDsrr4 3AgxpXOflzujaqGUqMt8zHZuTthKKg2/aKHLj6oXy8fFQolgA4eHljby5UNyVOFpz8vdXbRwt6Z7 H7PdLKuULj9JZh4Q4CQ4rrZoVN689eltNkPnvl1SJo2DYKQIGhxTOggVuKstZoTSqAdp7JKAuTcY ipJSeNencJHccikJX03nh5PDbFVl43vs82w2nmaH++NsnrP5XVbfZuiFTCGha2Tpk+V4b8Ty/YPD 0RS3jqbT+XTGspzl7DAbz443cgtOY3LpgdywQtc8wKfrK/IZGbURAG0ilq9NpKZEdi5XCytMCY7T F1Yp+ARf4jlvWDZjKdedR4sGHSLz2vt0xiU+UPCRnBB71UIDF0bTEbvDwIEym1WLKFTJxu0SPuD9 DvqLLqqyTTTunwN3hCsxddXCAInAZQLKyZ6NJjsNFKskhDZuNmYN2qpiMjvG73gY8vCYokwFk9r1 HAxdybT5Uq0X55+mQjXEPtkntxIVqBQptljerRZOePS+XJWTeylBtmQYIBo0m1flXhPXzl67IMtx 833U7MKSVtDRqpzOKHy2MwGgADlkh6MPBe7Ix6OW2aMjNjmY51+kKjlMHFEEXsoZJhvQLiaLNlfJ UDrcbCGsbgmrE2EtErb+JuKhWJ7ZPgV5gE2FP9zpGzQbzfDXyapqoT4UdU0Hz0aVgRC/3qdnjlXR qGP4NmM783w8yfIMsgEtk36kvpnkRywPkeXzPAx/h35KBy7vF70QwDHzA9wFhZ5Pj0bTY3Z/r3mH AEOps3x+kIHXqC8YXSg7lWkIm3TOqCRQXidhS2q3sBrnybP7/C0MntPCyJq2BHzAEbCjmThPegQX oYwBJIdD4dRWBNyCCAYksrVP6mn7YZBmAYqV0VoAkTzIqB2o1VsqvmuLb01wVPxyTEsNGLBKetA5 7Pe0/NdVLRsyKTpv08zH1owcurF+qqJTCUHUoSbFX4FurAHpc0DJCRSe3q+dbhwZGPMg3U2ZIOPR rtVJ8keF5o/B7jUU0fP3MH/4j/z5C8l/Lox2OIpcTMJ/1r57bXDMvimeQh1SmfpKYkdhFaGWMM1S rUyIuGolWNXNKrSSbe8eNgZc8NIieBQBSpkmTudkt8MJE7ixwqUtDm8qHZ2sJ0ISjEH7TRic3WoA 4+N5GLwHYSy1S3NbWHTWDGjZrwPLlBSR5m0PEOy5xvCytw9qsym8P2ftvMWmUAHPX28LsscN8IBK 2Ks0hYoRS+Z8KmBzMSvf0X2hhl1pj057Qts4xPytOHA5o6tAQ1vkoXq3wfhL+K5ausPclXad7ptO eRnbu4dw2/vLeh8uOA8K1W2AzjYmbO+vreHCXXh482nbC7k/KJjFHhm81wZ6ZH39DTHeEFVr01NE ubonqibhrH3HEQVl66lO9s7YrcrZmOPylVXT2pha3QwJAkmFNsWIfyIqiE6iYGlTojCSpE1dCf6X mnuholpLihoMsaV598qgmqCi9tXFfwA1/JtA9YHLj7RdgR/gKfDnBrUJP7F8BhnqfGh7lzXuJL5Y QzSGAR94UkY71LRteDXpRYd3cw0PLj3mmh66kCLvESdDxKYjYFqhqhJiEuhLY/bKimASYvqPd87N 64GSmwPQ3Q7xsubxOjbNs6Gs1uq0bsyHlbXFsPZfUtbWqaD+C2VtwzVxq7IiI2wSfu2hMXnAMXsp Z9NIhjt/WwXgg1vjA0VMkBFNr9YGA8hba6QlF0nGr+NirJGYiDfd4h/hXce/Hm9vyNbtJJVNxinZ cMPvxa0c5YPR1o0Zt+3i9vcAFwZ1LgplbmRzdHJlYW0NZW5kb2JqDTE5IDAgb2JqDTw8IA0vVHlw ZSAvUGFnZSANL1BhcmVudCA4MSAwIFIgDS9SZXNvdXJjZXMgMjAgMCBSIA0vQ29udGVudHMgMjEg MCBSIA0vUm90YXRlIDkwIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJveCBbIDAg MCA2MTIgNzkyIF0gDT4+IA1lbmRvYmoNMjAgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvVGV4 dCAvSW1hZ2VDIC9JbWFnZUkgXSANL0ZvbnQgPDwgL0YxIDk1IDAgUiAvRjIgNjQgMCBSIC9GNCA2 OSAwIFIgL0Y1IDcwIDAgUiAvVDEgNjUgMCBSID4+IA0vWE9iamVjdCA8PCAvSW0yIDk5IDAgUiAv SW00IDY2IDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDEwMCAwIFIgPj4gDS9Db2xvclNwYWNl IDw8IC9DczUgOTEgMCBSIC9DczEwIDkwIDAgUiAvQ3MxMiA2NyAwIFIgPj4gDT4+IA1lbmRvYmoN MjEgMCBvYmoNPDwgL0xlbmd0aCA1MjQ1IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0N CkiJjFdLjxy3EUbW0o49EbxOYiRQIDgdCEFmDWyLzyKJ3AInQhzkIHgAH2Sf7FiIYdlQcvDfTxVZ fHWze+ays9Xs7+siWY+v5PTm+OLlF3J687+jnP4zHTXMxk7Owqz1ZHWYvZ8enIy///338bvju6OY pMdlmB6MmcFMYrLOziZM2unZyOmbt8cX/3hrps9+Or46/vV8fPF3Ncnp/B0CwxwQFv8iygp63Sn6 4vktLr85ilkIoabzN2idfz6efnFzf/7++OIsMwMgVk4P6Qc5jCMOL2cvEwdCT1/+M6L2v+txT/G7 r0/vPbp/kHIOp8e3h/fe/+DwlZAKn6hZnX55//X58+ODnJ1xE/4oYfx0/iw5Kulrr09PnhDCfnj3 0a8OT25u7qU6/TrCABnwS/i5YAh0ev/D33x8+D0597dzOkltcTNidgodk8rN0k02GDqRdIyqOcZy CHT2mrYihZjB1iN8iG7ZeAh/vHv2/IPnN1/d39z++fC7m6e3dzfPP7m9efb48OxPh9uPHpEXIm4p uvb4kz88f3RHa4/u7p7dPf34EA43jw9/+e3h+c3t0+KyBB8DRKsZwiRngeEBjvZJ4fHlp9OPGEdv 8DUdHdOOAsPODj0GuiWTouhVpcLdWzdpfBOpxCzViAhfwlc13TaG3ewzi5heHtX0OR7F93jMAdT0 Mx7K9K/p9ddi+vZIgYpuKoXnYqe3RxmArp7tH44G41zJum7AUiCV9Yz38ZgrPtkVn9czPq8zXhs8 5QbPdsGXdcaXdcYbISlWC57tgi/rjC/rGQ8xMSs+2RWf1zM+rzPeKjOLBs92wZd1xpf1jPdYKFp8 sis+r2d8Xmc8GEuvFTzbBV/WGV/WGe8EzM5VPNsFX9YZX9e7eGjiI3TrEGLiDvAiELDisx269YpP dsFrqykQCr7YoVsveLYLXuGCafDFDt16wbNd8RiI0OKzHbr1ik92wa/yL9uhWy/4/r5+OH6RCp/J hU9iZdQWa0n+R0wGc62mcG4Br0/iHksk8oaTxf+MpvIu6aGMDyU9VYA0J1WfKlv+1fWpzu/akxGp H4jZY5BhybGBmgjWT0ElEp87LbFQzQGx6fmsUqXVCLGGnxl6dv6U/oX6r49vZgoXfKKQXH03TnN5 2peiebleq+Ei2vvv0W28w4yxVFroopEj9VUlqX/lqk3XIbEsGzwwehNXuTYkmzpHYnvV8FlDgZv4 NB7KJl2CVzpMdalXdLiT6l4glbLFl/CVDyvCwD3nr3QvwSsdhJF7gS75GroEr3TYQmNh7Oi0UNW7 /d0mfOULeuCelnGxuLdzuQlf+LSwtLElH+oafc12GV7ppE91vKfTvrq3u13GVz6qO2v3rLzSvQSv dEaN3MPmiVl0BV2CVzrMAUy3SodPSVRbKskS1Y2K23XbhImACHWEQlTxlVDNAZWWx5qmUWySGNty sMUTn4xIzALf7hfrAomTKKelpTjZ4mvxyGcDIXVAmZM3XN7t1t7GU8XgNhjjPrX81sZ1E0hiGvx+ 5OptWldUCgzGYCqCnY3rKDyxPhkcdZIk6mxaV3GXGCQ+8nc2rQvqXXQFOvJ3dsXjNXb4ZNfve5/x rV38t3jKSXJ1dtm/xfs0zf7ZLudHkiy4en5sl7uwNPiFUuTjzVrr6Wd0sz2qRIg1Pn2lizjrosQf RVyPK5Fr8bGxy1SwPibMMBV6YMkpiyOpCMsUpQkLE2WQoj2s0nhLO1jQgIiyZYcmwSpN8LmTtjRY pYo3fT3rcYUHUCYO3EGNs+sOwyqNUiN3jK69s6/+Pa7yaDPomWBjBu5si3GVxwBtYMkDrvNnva0E qzQYtgN3SCcOW3gPqzRODnojBLmlLHpc5fEjwePEluDpYZUmjIQOazl2R4zcCb3AcRi0pQdmve9T 32BT4iusMqpk0fHO04dkKwiE2xF7GClhLX907EDM5WbvN+mAi2S2Fw2Itahu8gcZzBbdUjti/K3T SFEBtVfReW4RWUqKvt2ystU1DTDTt8/OmC6M2V7RxdH2Cu9WdDCohAoEFd5r6ALdWaOTJf2s6EzV Prubxfehvdlkr+ig6uRd71Z0bqCTlYs98Qq6RRdR2Fz0+iqca24W++FmGGMFa4WeAj+62XbG2Kfz /c2OZwxvKBWv2KxTufeybZJSWdCFrtDs0PlOJCusu4OBCquSvepmcRumvVmcSc1gApJNr9mjC7EL NAOQGlyFbhv7Ph3089mwwWupar3bo6NttPVOCz2od7rTC7t0rqt3eikb0oCBSrAZMMZ8aR6QOhc8 tm1f8BKfFq1K2+ULOZCTTd2jjTxNx6a1pfiUkvrEFl88YNoJ1yi2oa9Raf4xkgKU1eguH/XlRt1q 6h+y48MM08ZRhOLZbLsXu6ImsWyZLtq+VpXyqqAAakZDtnUKU4MiBQOOvpn6rI2Rqkk/67gOBNRW 8XST1L7Go3S6qn+2m3UePekOMZE00ialiC/iFWsbeLrAso8hQDNu2gpOCoZmXsPTEzqCea3Bc5qZ SKSdYP+NpZDUpK/Tevywdnk6SdNNnRTxsvAf7QJPbziWUaR4TTdB01PcuPaxnKTpKc64uTWnaUoH bs0GgylOl7k5ogSg88ziyuh40DrkeYSnNZy2eHqL4s6gekvnx9MeyrBY5LKN0Z+qFkpEvA9D01g8 n2wDn0+2Y4eoeMqEuM7fU4rPh/2hyG6nUxJG6ftxP/Q4RW7aLz2GxA9ETMdi2umTpIxtps9kN+uB /U/nbTAek//pPugak//pvigM+H7jfdJQyvcbA49Sxzf3T2GX7hdVCjpKYZ7uN8UPpUG63zTdUsFI 95ttrkQcnwZyZcEPYY4Zij9d49tg/KX7zbbJ+dPaJT+Ms3z/Kb8MNTlX8894kfMn5qfBrmUTPsTz 95Dn2XfRRzwci8UefZaGxthF4eC0N9StQq0YJqh0zF0BsiIGz6gAFVyZkGIhM9SmwrQojFbGn2Fh ZCD53FZYSw2qrbCxYlsZJzau2L2UKUCXSyvbi5ktthKrNB3jsJVwzyCnWR+wbXp9wESBDnKfyHfK wCoxUAaW6oMeCtyCM90IxPaKx9ZxaunPBo8djFIWZw4/VnkFFzhU2caKslbalvLQ7u5LWw7pzl7x AOXrnj8rHpdTo+WhvB1LpowzMucG27rPDeZxzX0tIpFxjktVtv3ovmzMmF0e39+X8ank9TwQE2Zv X1QaWx5rUute8AQqkLs8nls42yBSC+95sLBtjEQFZ7pZyOJMtp6FrJd1eBnzONENQRbHjvXUYr3b mloKznbjisUpbD2u2KC2xpWM87KbUyxJinUcBr81pxSc6wYU68NgQAFhtgaUjAu6rz8kSVbnDFJc qhshdHEIKF3WdQNQuu/XDfK4rRtAEmd1zoBCZT/fyeM234EK9uqcQbkL+Q4kmULL4wf5Dlifxe45 A0qpNt9B6UGego7ZW3gkrHlcl6eAddav92VUPedhXSWPod1Xslc8sQrs7WvFY/thJfFYcSHfgaRe ez5YZ9f5DliV9G4fBFT6bb6zveKBC3VjzeMGdQNAXagbtPO2bgDV2fU5AzT3Narz5HFbN8C60X2B q/Vng8f192X9oP6A0xfqD+28rT8AZlR/XLhQfwB6/QNupH/AX9It4HrdAm6kWyBc0i3get0CfqRb IFzSG+B7vQF+pDecuKQ3IPR6A8JIbziskvXe8UbXNC5LcbZDL8UHNFGQL3jI4dBsywlTy49WMTtx SIkRxibNKORwNrmvZ5PbOpuBu3U2FQVHXQVyuq6yZGJT4pTgG2qJU4JzZfqJO1QL4erqSaEhw/Qy 70wp2QWUGgpY1QlPTZPHFl/CN3yjgFA4ypSE+T/1Vc9ryW1DgZT3V7zybfHsEfU5rZHYQOrXxWmS wEECPwfu/PNz+KGRNJLuLoJ1kWZ3uRqdS1Lk4eFIcHc8UHP/ombf8fjwihfiLXwxHgAW8Q4CkHiv 2eHp/Ya3FIIEGUVrgXvHS3kgZAJxLOItZ6tgf3yzf1693uC4TOfnQAFe+wl+fQemlzswv1hS/OF2 ZHjH4zbo8eDGHKt3skFeseb9W+j9C8+ryJrwtuLshudvIs0vRZon6e2aPP5rh4dIQve23mXt+xHP 5xbv01q2+w2PyipeiK0Sv6T27H6Hdy6WI49R3+J91hvep0Hs+OBW8Sb3pfHq/YYXaRUvRmQTYXuw eA6T3ae4UGJ+mMhg/H3y9H7DW05mf4r+Mzz81xau0NBrHr3iJ7hwlE6Qpyc8b/c7vDIK88CPEMDt cA9LBV71OWA4LuKT/gwujA56Dj9gHEBPOscjYxdwf5/xZPYGIh17Fc/xQwToUwwifJs54rxmA8eh BSo6J0OSrS6gD65J1L7tz/AtdCL8DZgSIj84zyi5EA/TA4O9OA9SVRyTyMTR7s6jJitAf+OZQ65r E34YWQ2oYcUfbJzz5gT/UJYyZQL2EBRAYAUh+J7UPhnv55vN56I742GyOqDhcDG6YP7pe0UKmvzR 5nMv9+nUzgtocQBH7nQpBrQoyDUG0yyjzeeeKSUiX7IeBAgLDNoI2a/Vjg8xUmKdoqP9IXWK4RKz 00kRogi+mFPVa/KeEfLL1rOrdBKL0FY6Q22P1z5qCUVWofFe0wm0m1tN34CkeCPybeuQFHdCnw7r onRbotB3WxiFaH+xAbm4oIEUaJA/eRbGVAts2GcnnLyTPeO9Dqcs5M5X21/7fXGQOTuY9bqYj17e 9BN6ty7i/Wdl89XWvGE9GybeeO+z65k8xiTbxksdSGDunJahToovQKbFbKW/h40KGV4sZqPstgu2 UKFB+4VKzTMy8/xczbb2VDtVJfvrw5XCvU3Mbv5Fxswb2mqQOoSU//Bw2BD5y7r20CED3uwaFh2i LHiZAXvqU7+h4QetqICE/84KKB1PTh7F7AsQX4FgK6A6uMbDr/GXFx7JiL/jkYhY8tLW1UGa0bws RkS2zxKIGtxl9oXmpTIIzY0kmHc0KlnDA3/zlxdeFElG5JXPDS+IuKh41bslIDqPP1UABjz5H3fA KMTEgPyxOuhH6Wl4SQZKw8syMIik7C68LOKi4pmDa8AilWEAACwS+h0Qq5EUjMxpczDwgJjxTq2X ioeCDJxBedCqTvRbXkBcrmcofdS81IJ96x3p69ZzkjFJmEmSPE+yTJrN56f0x3UO9SDJr+d43cS+ ZI3NpL7Z3HqkybzOT81Nsd9PSWu/mH/Zay3X8yzTq50XLV9VEzhHbspl4/xMWm31/CwaX1Ufh9A/ t32W6c/5OKuNc6w7qT8nofV2TkLX3JVKlKpmzGb1cdGAnqsa464T6mFlEVy1P0QgcHztvLc/qloh pN2mCzMoaU07USfEIuC2ppkaoXBRsdm0WKcohW643DSF1Jx158XFFONiuBAk6kXpd3/6ex3Oxcg9 ThZp+WRoEqRNvznxL9PsTyldXP0iUa9pTzcYTOtFWNpuz9wpybRwtctCcnknb/DEHZ5f/T6IkVKm 7HDHpqfZ8cfJPdBweJL0YYlE4k5G8OfkizCDhwYzmaQ2fzRjaPefsyN6iau4B+Gh4ScQpYxz1n16 KbgqkHp7Ainch7toJozSD7OKoVy0jSZG68lq51EcKQgXr4DcO0Avpbo2VTty8HcQTBDSt7m1Y3+p gWRZ2SYQocZtONkottrFKG4EwVoSFMRNCEUaoyFApPoJIWAikSLc1oP+UgdiIm4E4Y1nXyOhKr3R nkCE63cJmUF8E5wNBDKhJmRcC2J1VNfOaodVTnjKnk9AwpCTwEHPnvAo9vtwKOiIvuzU5EIDCVLE u9cJ3g81H3h5mxOLDt4Vid1oCMGtEsKy4MnThFJ3UbPPXtpWELRB9Nuu4a9Sn5AkUmMC8c/61y51 IGHRvwEqxT15miyavoFk368SFYSljN9RWsCsyn2NcDNOGPGItdCw58yFxjtWD4KfnAsturPy4sKR eNBQZxEdMNNiJBmJu4xEl6tQifUnaY6GxdU2I5GibhjV5tqdMVQ1bR0JIx3Fu2JSEFM6G0eaWlL7 rnIUI4f6vEtHkh+IhH9y8b7FGbkueTFmGt7X7AmkVCJZRTNhnAseiefFI8toitRyAykrHklYVmzs LUGaPor1N+exl0xmAcT329R46QJJd5GlICayNp6kprBi/c1ZYSUsDRbORIvppq8SuVUsTaOt2Nku dSArkZaCrHw7RkueBiJJ3DYLEG8VvwHxA7cmKL255FMMlQRWYy8FN8iaFOKCBFKUYbgHiWNiMSqO 2ZOULCfr941R18fLzqucoBOMXFeVlm5SL+VjQa6pSb2lJ3qpA1lJvfRc6qWb1EtLqZehdyyc1fhM N7WXeHZM4eRL7a3rRC91ICu1l8EDVmx+JqR8uIFe2e+r6lnVIp34U8rRTLCPDGkzT68lUk3zwUxX deBlX/T93fvjeIE4g/TiFdf+cbzQcXDUjjcE9/L+8XiFXE3u0/u/H3/iKz886JBndtH65LKTDsv7 uTtlXprdzkvUqXDZRSMb7Xa/nu99D0mS7VRPiO/md0iZ/8vlXKtHbVSP+DSfy7ppdjuvPl+2+TTa 7f7nfSbUIr4sh6X7L6//+Nu/fvnPxydwAjL/+tunt8AD6fWXT399//Pj2++B+vL+0wOFAg4CwPsf H69/4DC//T7oEajnxCvrERSMHtq9N4iC81zee0MxntHpGeoKXf/y/vfH64+vP37iD99AviAKeI+/ 8UL46uAPvr7L4fdwGf38O7qc/t+y/M1xuP/R5+zdOfgMpUkBvr5hioBcL59fQ21AEOIp60V2siBA ZWDM9Goigvv4HN1/cAteth+EwCnyJCMFIDq0UU53wTgDyR5rdgNSd0h2jb07Lurku+zCk+XujqE8 c0cvNnd6oB058EICXMJKgVHTE9oVAOGnma93ARACoNgCIPbO3wMgErBnAXRAEoDZt3wS1BIE+N4d Vn++c4fiIMHNHUN55o5ebO70QJ/Jp22+nM/fLJ/u5Z8Pl0TJg8qhstCZ4eUN45J/+KfHfwcAeAlb cAplbmRzdHJlYW0NZW5kb2JqDTIyIDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA4MSAw IFIgDS9SZXNvdXJjZXMgMjMgMCBSIA0vQ29udGVudHMgMjQgMCBSIA0vUm90YXRlIDkwIA0vTWVk aWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1lbmRv YmoNMjMgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvVGV4dCAvSW1hZ2VDIC9JbWFnZUkgXSAN L0ZvbnQgPDwgL0YxIDk1IDAgUiAvRjIgNjQgMCBSIC9UMSA2NSAwIFIgPj4gDS9YT2JqZWN0IDw8 IC9JbTIgOTkgMCBSIC9JbTQgNjYgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgMTAwIDAgUiA+ PiANL0NvbG9yU3BhY2UgPDwgL0NzNSA5MSAwIFIgL0NzMTAgOTAgMCBSIC9DczEyIDY3IDAgUiA+ PiANPj4gDWVuZG9iag0yNCAwIG9iag08PCAvTGVuZ3RoIDc1NiAvRmlsdGVyIC9GbGF0ZURlY29k ZSA+PiANc3RyZWFtDQpIicyUy28TMRDGxWZ33SxRN+VRKWrUWnm0ttA6ttfeRykcUAEBJ9SVODSc KlqBVCHgwL/PjL3bhKiqeuDAxc7Gnpnv+3lsRa+SxdszRa9+JYp+pUleCGNpaQuR59TmtagqmpXK zT+/JJfJj0RSVcFyQTNjRGGopLa0wtQ0L3NhFL24Thbvrg09/Z58TF41yeKNpoo2lxBYixrC3AhR VuL2UmPF5hqWrxIppJSaNhfw1fxO2IOAN9+SRaO6DAXEKpr5CXKYEnNUSlTK54BQ9umDi7q7bgWe XN1z1gt5ppSoWRST3lafLKXS8I8Wmj3kn5v3SaZEaUoKk5amos2pF6qw2jkbDDDCbqfDHTIIAq40 e+TCCsgAlaBcbTCIbW0/fkL2UNzrxpPMLZiRotQgTOlSqJLa2iARj1GvYbyBgOxztKKkFIVdIcyc LOtliWHQG5JJP47DNBqRfhqSgzEMk3Q87U+XPCSzcBJyOEMWcYi0bDYiTrd0TlGxT1g4quOI6Hxv gyxIRrR+ws4wQte0hNGuzuOpD+r0Q9NUeHgdRuP1IsM5liAnZPeQZ7komeF4CARd8MwKxZwXjufP CFegecmCeEjAzWhE4mE/IsP+ks9Ju33I8ehYSKZB3IUF0ZpLqW21Mlp5JWGajjG+Yinp88xAohSK KcgDX+ygW1xjeehWBlxiNb9tAh+aLfk4xsI56JUwzscpePRJ947RpwFRz52iG65ZywgEFrooHKn7 kLSbJF+0JAtHEjRgO4B6FvIc4GFfeIAHqBi7Y4OhJ1isEaz+T4Iv7yCo83sTVJsEj4GgIcBjybsu NKydEKJi3eUKn+27bUfrngZgZeIdYMQt3FZPXusMakLJw78u2j9riJMTtDML2qOM02C6HwdwtXu+ I6K47QxoF9N2yYzj4wAex/vRzD2O7Z4jMve/lgybrLrNXrbyt2oLzdLRio3/J8ZGA0LzbgfHm0sw sWEaHyjF8u68c7br2ml3x1WE1/SPAAMAUFolgQplbmRzdHJlYW0NZW5kb2JqDTI1IDAgb2JqDTw8 IA0vVHlwZSAvUGFnZSANL1BhcmVudCA4MSAwIFIgDS9SZXNvdXJjZXMgMjYgMCBSIA0vQ29udGVu dHMgMjcgMCBSIA0vUm90YXRlIDkwIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJv eCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1lbmRvYmoNMjYgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BE RiAvVGV4dCAvSW1hZ2VDIC9JbWFnZUkgXSANL0ZvbnQgPDwgL0YxIDk1IDAgUiAvRjIgNjQgMCBS IC9UMSA2NSAwIFIgPj4gDS9YT2JqZWN0IDw8IC9JbTIgOTkgMCBSIC9JbTQgNjYgMCBSID4+IA0v RXh0R1N0YXRlIDw8IC9HUzEgMTAwIDAgUiA+PiANL0NvbG9yU3BhY2UgPDwgL0NzNSA5MSAwIFIg L0NzMTAgOTAgMCBSIC9DczEyIDY3IDAgUiA+PiANPj4gDWVuZG9iag0yNyAwIG9iag08PCAvTGVu Z3RoIDUzNCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIiaRSTY/TMBAVaVI3ptqU XZA2KEstkQr7ENffTq5oAQEnRCQODafVbgVShYADf59x3HYBCbQSF0/imTfvzRtLssXrV+8l2X7H knwiWDtuLPHWca2J1R1vW9J4OcZv1/gGf8WCyBbSjjTGcGeIINZbbjqiveZGkqsdXr/eGXL5Bb/D z3u8fqmIJP0NADveAWw8AWVFKPcqMPY7SG+x4EIIRfor+Ot/YHovYf1nvO7loYMDrCRNDNDD+NCj lbyVsQdA6Ye3I+rfvC3MNPJu6CRljZS8o9kUTWY5GoRUcKO4ovfZx/4NbiT3xhMISpiW9JdRqAxs GzqfB4Q9KRYP0DxJmFT0dIQ5gLUadII7JqA2dHZy9hCdhpJkLHnRR0e1haEE9woESuW59MR2JjgT 7VS/2Hk0I+xAh5GkENzZWyubUZ6N8s7SySLPknJZoiRDyXw2D2pZo6yiqzzMWhVJdo5W6UDRKEqM kwbBsZEZXU2LoirKP5zVikO6iQGEdJKDLd5wZW/38SiCDrq5ddZC8bG/PwhdVsUz9DhI09zSIs/q aon2vEHYkbjZNwmL0T5u5E5UOlK5Iq+f5ANDdXoOn/k0YZZrWjEBZ4aqFVrkRT3QvEj/Tmykuwtx 4B1fCp0VVVlOgfbiNxf/d5h5WiRhloHFKTImuaMXJRpoMl3s79AyqeuBpcxBDj2t2rQ8PMCfAgwA sli56gplbmRzdHJlYW0NZW5kb2JqDTI4IDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA4 MyAwIFIgDS9SZXNvdXJjZXMgMjkgMCBSIA0vQ29udGVudHMgMzAgMCBSIA0vUm90YXRlIDkwIA0v TWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1l bmRvYmoNMjkgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvVGV4dCAvSW1hZ2VDIC9JbWFnZUkg XSANL0ZvbnQgPDwgL0YxIDk1IDAgUiAvRjIgNjQgMCBSIC9UMSA2NSAwIFIgPj4gDS9YT2JqZWN0 IDw8IC9JbTIgOTkgMCBSIC9JbTQgNjYgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgMTAwIDAg UiA+PiANL0NvbG9yU3BhY2UgPDwgL0NzNSA5MSAwIFIgL0NzMTAgOTAgMCBSIC9DczEyIDY3IDAg UiA+PiANPj4gDWVuZG9iag0zMCAwIG9iag08PCAvTGVuZ3RoIDQ4MCAvRmlsdGVyIC9GbGF0ZURl Y29kZSA+PiANc3RyZWFtDQpIiaxSS2/TQBAWjh3HW6sOLUiNFJERImJ98GZnH177RIUKCDghLHFI OFVtBFKEWg78fWZ3aVMhBD1w8Xi98z3mGyNs2erNR4Ttd4bwBZhuhbHgbCu0Bqt70XXQOAz1+oJd sismATu6bqExRrQGJFhnhelBOy0MwvmOrd7uDJx9Yx/Yy4GtXitAGC4J2IueYOFJKCt9u1NecdjR 9ZZJIaVUMJzTafjB+IOkHr6y1YA3DC1hEZpYiMM4z9Gh6DByEJR/eh9Qf9ftaKagu+ajtG4QRc+z cT6aFPlGoqIvSih+UH8e3rEGhTMOqChpOhjOolH0amtelh5hD6vpw7xMkhoVPwqwlmCdJp+UjvGo NZ8cHj/Kj/Ytr4aYqLY0lBROkUFUTqAD2xufTIxT3YnzNgy/A+1HQilFa/dRNsGejfaO09G0yJLZ YpYnWZ6Uk9K7rRtlFV8WftZ5lWQn+aLIngdPMgzq/UYeE0JNq2pezX4LVitB100s5KNHQak4I5Td r+NxBN3YFra1lppv+V3oKufL8YYXVfpimVSL8Jb/QbP5hfcr0S7u4l4qcYpTP/upzUfzbFzNN/W/ JAy295bAEDhXT2b/1XhkPSiqkzR/VoyLp8V4ekeA/qCfAgwAtbOvdwplbmRzdHJlYW0NZW5kb2Jq DTMxIDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA4MyAwIFIgDS9SZXNvdXJjZXMgMzIg MCBSIA0vQ29udGVudHMgMzMgMCBSIA0vUm90YXRlIDkwIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5 MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1lbmRvYmoNMzIgMCBvYmoNPDwgDS9Q cm9jU2V0IFsgL1BERiAvVGV4dCAvSW1hZ2VDIC9JbWFnZUkgXSANL0ZvbnQgPDwgL0YxIDk1IDAg UiAvRjIgNjQgMCBSIC9UMSA2NSAwIFIgPj4gDS9YT2JqZWN0IDw8IC9JbTIgOTkgMCBSIC9JbTQg NjYgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgMTAwIDAgUiA+PiANL0NvbG9yU3BhY2UgPDwg L0NzNSA5MSAwIFIgL0NzMTAgOTAgMCBSIC9DczEyIDY3IDAgUiA+PiANPj4gDWVuZG9iag0zMyAw IG9iag08PCAvTGVuZ3RoIDIwMDkgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSInc V0uP48YRxs6O3Tv0gJRGxMA2VlIPJTLUxuKwm81X7JNhx8gaPhgWkMPIJz82MbIIYh/277uqu5pN ajmUF0EuOexqRHXX46uq7ysK/sq7/+o7wV/95gn+T+4VVaZKXpdVVhS8LNqsafi+Fvrz15+8n73/ eDkXDfxc8b1SWaV4zsu6zFTLi7rIlOA/vPbu//Za8S/+7X3rfX7w7v8queCHn+Fim7VwTf8Pt8oc j9cSPR5ew8+vvDzL81zyww/w7fDGS59c7A6/ePcHYS1UcFfwvfkAG6pGG43IGmFswNX071/rW9N+ G8hJ+31In17u9kJkbfre++zpsyt2zIWEJzKT6Qe77w8vvb3IalVz+JC5avjhCxOoQG8P6fU13ij9 YDZn1xcXOyHTG32tgmtNAXECOgpvPaTP/EXIbvDItT7y5cEgWpSQVJ7VEgIUss5EzctWITIGTtmD swMDa1BgSiLPs6p0UO51eI0G4+4yiJ8H66v3nl4y+IPFl8eUXQbB8+AjRAkCyPlX8O8l2PkFQhWS vwGD/Bv+8H3Of/REJbOq5aIW+FE0mYSOAMDhHDTEd1MVFhAQhCWwPVx5HtLny9U6DJiGWKT8jl1F m8U2ZOsoZmEQ8CCx2JBzBQ3nnBei7FyPe1RtTh5HnP1pGeCXNNkBTCKNT53VAKbsnElRZBW6G/76 2pPwR1vb7//y/mGfTCF1Eq4UcKPoA9T1f3rDqDxkVkqVQceR2VJmxTmzEvqoLtBsen1irJBFz1jb nrFUyMpauj2xpOBE1VlqagPWhC0lFHJL16V6iOAvIYUe+XV4TPnmuDtxM+iAusry5pyfrgdyxyc3 Q6sV9ATEQlaFLJHPps1WAnyXBtShsWHfABLt41CoNoN0YcDhwrDsL/784rYzDPwrS9tQIi+1jemO MpbfGrlPtts9i+4YMs+z7D5nuunzPqU9pHM/YbHg3awE5tThRY/uuqK9gQsrHhBzhmaY3AWHeRwG q0RGhU0KKgnsJiEbmNRcCwlgJY28UMrU7Dbl6W43KbtuH8kWM/N3OlJ258/i86mJKJ6zWwYH304S MqInq2Qk44fUh/5l8TxkQUg6MuMBPNrar5sd9Eq6ABf2+kMaJfB7AhXQ7DT3N2DBOmYqDCIOFuxx JDLMTEYO2LbRwFYFirEBFnp2CCwNvgV2YvINqm7yNapMBbyMjrsQQnmWIby2lTAmjbLF976a1Td7 zRY6vSBc+ICBThGCP0HG2uhxQroNN4DQcBaIbrpZmOIbk4HjG7McYBaAJPjckDKYCBHRROtQgzrU xa0FyQhGP/sz7eO6jt0BTJlKT7pOj8YQIxPfCU49/GY6CmokhAaKD9Un9SqkxPKrusnq2pYfHNPe 9jY+hRQaH7jQOBYy0LfaL6xZqhHtYO/p2NrkiYsPT2LGsReoc23TnmQ7gNJMUmuDt9U1LG+rO03z VN6e1GNpl4rFnN1e69mHQiYs2kBd/xK6Nh30sC3TBhr2wqGNpNXqtc6ADilu/WABOZkecdbWyyAB LzIqTxiulC1G1lVCDCeRtMfmekZ8TLJOfIjj3nEcXX6aoZLlqrcX6UR6fETjZ4qZsOOO25FN2IYe 8/6F8H5mT2imlCn4QK/amTs30sloXT8Lk97BDZo38fVtwy1Lhzggbw2iGT7Z1aMoFNajUgU2uqkH LP9qUA5S7x6xPC7fRVmiwU6+009wXoNOtwtVGhHWug3HynOyTRadbGN5h9ZIEsnapCKSNaeIHfNp VQT8+9DDfHRtsT5C58PIJE6qHlOks6vBcbfc0uklyF2RJnv24ZMn1/r9qB2TzU9BMg0/iCgJbemF 4YoIhnsVJdjSrkOSlQ+Pk+U6sc0RM63cny1t49KW78ZC63K0mMcsUf4MpnmIMykk4fy4QBLIfYG8 4qsuapqAnngswnhM5nqrbzwP+HIxH8ZDime7aELwKKITwTNKcxWtIiAw5EbYMDvFGa8dDpuKacib lJ/sOJb9QbpXahvGYU/A8PrMjHIZBviye8fdy5eW3d582yJpHUYKIr7uamUUY4iH0QjCY1IiLCBD idiTRtx86DRiHc78ZRL2Yjvu/NhujDgPcySxLXLPx2y90+ngRT/WRvxee0WrcBgxMT1FPE30FPIJ 0VPEHxNAeqMo0jOUGuknZtPsHT034u4kDgpVrptmqWdZ4iyP6SmJoR90PWH8MX8bLTZwoh1CQ6zr mvtx0lUggXVx8s4kzNvN+9CFszm1qyZi2isUkHtd95i4as4wMbkZZWKy5pgYrU0yMVl7ZyZ+7KUE Tycsu2aminmvjALLOMaq8ZC4DS+xlZvJ5Zbve2UHjjZ9f0ufLdPEDJw8VHdgWmhNd5HYeYyT3SFi Z9g0TrZF4uSuq4ic2yH2jp0R+8fZmYD/37EzxePYWXfWBDtTRP+v7Gzx6NgZ8ZhkZwvIODv3N3jK uN9FXdtsoOOAyfUrG5xXfgyr+wbDxS686K+TMxiEhdkyTfkTdITHbuBVK2BlNOv/2nR5AlaJL8vI LpSUZAm9OLbgExSO9hGKadonLBztj7xiaVwwR8oXheD0DesP68Bob43zS//tlNzol9P/eivkeh/E jG6fjDDXO+kKrPtRV59SSqxP1ejZpPqoTA3K46SHJtdKj+Av4Q687r7hnqhkpszLAVS6qIHOFN9X edud/n0AhgJ+VAplbmRzdHJlYW0NZW5kb2JqDTM0IDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1Bh cmVudCA4MyAwIFIgDS9SZXNvdXJjZXMgMzUgMCBSIA0vQ29udGVudHMgMzYgMCBSIA0vUm90YXRl IDkwIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0g DT4+IA1lbmRvYmoNMzUgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvVGV4dCAvSW1hZ2VDIC9J bWFnZUkgXSANL0ZvbnQgPDwgL0YxIDk1IDAgUiAvRjIgNjQgMCBSIC9UMSA2NSAwIFIgPj4gDS9Y T2JqZWN0IDw8IC9JbTIgOTkgMCBSIC9JbTQgNjYgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEg MTAwIDAgUiA+PiANL0NvbG9yU3BhY2UgPDwgL0NzNSA5MSAwIFIgL0NzMTAgOTAgMCBSIC9DczEy IDY3IDAgUiA+PiANPj4gDWVuZG9iag0zNiAwIG9iag08PCAvTGVuZ3RoIDE0NzQgL0ZpbHRlciAv RmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSImMVl2P20QUVbfbztas7HyMtqponFnHtpwFTzz2+KtS QaoKFUU8ICL1IelToRWVCgIe+vc5d8Yf2e6m8BI7ju+dc88999wo8c7ZvPhFiXf/OEr8LpyikroU dVnJohBl0cqmEWmtzPXv35y3zl9OJlSDnyuRai0rLTJR1qXUrSjqQmol3nxwNj980OL5n87PzrOt s/k+F0ps3yKwlS3CzCeiyoxer3M6cfsBP79zMpllWS62b/Bt+9FJ7pyst++dzVb1GSrEKpHaC3Lo mnI0SjbK5kBo8upHE/X5cxvUZM7dJXdP16lSsk3u3Wd3zx6wfaZyPMllnnyxfr196aRK1roWuOSZ bsT2uQWq6LRdcn5OEaXrTabs/ORkrfJkZsIqhDUFcIIdTVG75MydczajVy7MK99tLaNFiaIyWecA qPJaqlqUrSZmLJ35AZ0DGdSDgkpSWSarcqQyNfAaQ8blqRc99pYP7t09Zbhh0ek+Yaee99h7RCwZ AKrMJfiktiOBaqTKqev4Rk1/dSX+wGkvoJCXOOy9UFJr8RGnip/E7nUmfnUUSgUdddWYBJLC84yC n119XgWKQCuhm0EFBDoPig6cgqgKkFHRZ2ZkV8s2t2JE3tvTlS3BMc09k5uM+GZBzKIpZ1dfmdaq 5GoNmvBci3CfdAJYdVfhB27I3LV9uF9TsLhk+zWfe5yzLsOqF0dWQbSplllbmT6nozxQylBLrrRp bEPNsrUoQDhWS64KmSsBavN2oGaX3EMRmnsBgUgEsAEYFWkK3Cc6mLhhuDYAUZcp8bxaNTMmPNxI 3GXNrEeeN61F3ujiFuQj9FLTuECibdFDB9PNUexlIZtWVIUs60+wb6pJPWOX7iQCIN99ymPA7xhF FR7frxeTYBnyls3qjW0B1eayifB9YcpeheYCJS/iblIR1ieNLTeriblwFkQMlAhFAugazLvme9x3 g1X3/rxLkDI1tY8m6yyZ0uEdvoXhsz/SfkNq4NBoCHGqyVZKckWwmfhBFJQ8ZcRiWipomAZ9tJFd Egm25NTCaTA/AGfyxwRSJRE6B+V5THUopoGtbGq763eP3U3gB0+4lblna7cvUjwL+Woe2TgwaW7K 62wQ9a2pIjuu5rKXRJHVpGNIoqxHSaijci6ySlZGEvno1UiNCFg9aaNHw8SKKL24YEu0TgUoni0X XhyhMjeidiwwk7bAqCeLWhVxO6o32qwFWcDBnJsBX6c5jK9Ngijm4WU/ErpR7ZGRsEspHymAbdGe bGhCLQVovz7KALYVXMwO0TgUX1sLuu49vgsGUPhYH6r3Ah1yyHs1x4W44Z4nPIwGWdR+bZu6WIYd XTHrg8v+ZsWCS5oHvMzj0ceIJBH136fu8H43rS6N3yWZSpHYGVsuokOdWZ47tZlPKzk7Ew3qL1o7 E4dqgiCn6G8YdgOCHAoSOdyzliKMSOxj0LjXsvs4nW9cn6ggBKedqLmFYovhEdMWzA1mqfR+wNjw U8Sm7kDWtarHeg1S05OJcRRKNbJtjjNrIg6emLTNoL8vifaBC/yXyrrpJwZHBQ+zarXYvXvNL8wU G0sKhyogeEuBu2KzixP6C2JMEwqJD5cYQYaG7G7Q1yUUrFCrxyfCsojxyhIcQblN2vPZw4eMhsgk DcVq/l9OUQ7LQ2ORqeL/O4WmX+tPnGKXfCMUC4OnnLwhMDYA84RWB9sATlPN4IgDsyjBGnrsD1xF vQuuhgxe1Fs7LOgTC1GmMZcdP74h8wn3bAret0CwYfneSBFDQNGBcXfjijnq7rrXzdIZVpIYzczr B1TZxUCI4m9v+NbtzRi7gdUM/6lyWuhdN7Q8alqaLBLNMH/xx2Y8ED6/xXK6IUCh0x5sT6l2JzAu Bpn5izAyTkazIg6aZYjGw0OWuJnAbjQGy6KXxn3hi24d2I0c9kzGTF6kd1Lzj/yRuZzfsePwKGOu h8AgjoelB27+HQCAOy1uCmVuZHN0cmVhbQ1lbmRvYmoNMzcgMCBvYmoNPDwgDS9UeXBlIC9QYWdl IA0vUGFyZW50IDgzIDAgUiANL1Jlc291cmNlcyAzOCAwIFIgDS9Db250ZW50cyAzOSAwIFIgDS9S b3RhdGUgOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3 OTIgXSANPj4gDWVuZG9iag0zOCAwIG9iag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFn ZUMgL0ltYWdlSSBdIA0vRm9udCA8PCAvRjEgOTUgMCBSIC9GMiA2NCAwIFIgL1QxIDY1IDAgUiA+ PiANL1hPYmplY3QgPDwgL0ltMiA5OSAwIFIgL0ltNCA2NiAwIFIgPj4gDS9FeHRHU3RhdGUgPDwg L0dTMSAxMDAgMCBSID4+IA0vQ29sb3JTcGFjZSA8PCAvQ3M1IDkxIDAgUiAvQ3MxMCA5MCAwIFIg L0NzMTIgNjcgMCBSID4+IA0+PiANZW5kb2JqDTM5IDAgb2JqDTw8IC9MZW5ndGggNDk1IC9GaWx0 ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJfJFNb9QwEIZF1llvzKopBQQrItaULrUP8Xr8 ESdXVEDACRGJw4ZTRVcgrSrgwN9n7HS7FQcuGU887zszj4Fv2frdZ+Db3wz4d85so5znwTfKWu5t p9qW1wFS/PWNXbGfTHNo8brhtXOqcVxzH7xyHbfBKgf8csfW73eOX1yzT+x1z9ZvDQfeX6GwUx3K 0hdVXsfyYGLHfofXW6aV1trw/hKz/g8T9zLZ/2DrHvYODWqB12NADxeiRwuqhdEDpeLLx6T6f98W d0p9N2JCZA2gOpFP6WRW0EGDwT9GGXFffu0/sBpUcIFjMNq1vL8YB4XYbSPm86jwR+XxAzrPMglG nCRZg7LW4pxIx0XVRsyOHj6iJ7HkSSp5049ErceltAoGBwQTFATuOxfJjDjNHZy3MOIb2LgSaK0a f0BZp/HaBOMFKVdVuSzyCaF4oCsyCErKsioX/1CyRmmHjimgaadV2/HglPEHto9H0X4G5RvvsRiX q+8yiUBWdLEsBjlImj9fviSyUa0oJUoErVY3I9CzapnlxZTQ6THJ6ezZjCYuOsIGZ+ytM5jReVLk srYKBJW1wUCKRcy9yNC6EYM8p6dlNT6hxRfMzghdxmMQVUnopLpJTgdJpjE/lBJpRFk8jYlLyav9 G/0VYADGnY7PCmVuZHN0cmVhbQ1lbmRvYmoNNDAgMCBvYmoNPDwgDS9UeXBlIC9QYWdlIA0vUGFy ZW50IDgzIDAgUiANL1Jlc291cmNlcyA0MSAwIFIgDS9Db250ZW50cyA0MiAwIFIgDS9Sb3RhdGUg OTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSAN Pj4gDWVuZG9iag00MSAwIG9iag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUMgL0lt YWdlSSBdIA0vRm9udCA8PCAvRjEgOTUgMCBSIC9GMiA2NCAwIFIgL1QxIDY1IDAgUiA+PiANL1hP YmplY3QgPDwgL0ltMiA5OSAwIFIgL0ltNCA2NiAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSAx MDAgMCBSID4+IA0vQ29sb3JTcGFjZSA8PCAvQ3M1IDkxIDAgUiAvQ3MxMCA5MCAwIFIgL0NzMTIg NjcgMCBSID4+IA0+PiANZW5kb2JqDTQyIDAgb2JqDTw8IC9MZW5ndGggNTQ3IC9GaWx0ZXIgL0Zs YXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJhJLNb9MwGMa1xI0XUy1lgKBSWE1Lh3OI628nN0ADBJwQ kXZoOE2sAqlCwIF/n9dOu3UXuMSO7ed5n/dnS7ohq3efJd38JpJ+o0Q7biz11nGtqdUtbxpaexnH X1/JNflJBJUNbDtaG8OdoYJab7lpqfaaG0mvtmT1fmvoxQ/yibzuyOqtopJ21yBseQuy+AWVFeG4 V6Fit4XtDRFcCKFodwV/3R/CjpKq+05Wndw7ONBKWg8DeBgfPBrJGzl4gJRdfoyqf9dtoKdYd81S VNVS8paNMpwe57gXUsGK4ordq750H0gtuTeewqCEaWh3MQSVodqajcdBYU+KyX08TpJKKnYaZQ5k jYacQMcE1Zodnzx4iE/DkaN45E03ENUWmhLcKwgolefSU9uaQGbAqQ5w3sAId6BDS1II7uwtyjrG c0O8Z6gy3LJiWRazfJQiDBMMsxcYFUVZTGOOA1hacWHAOA7g3QretNQbruwt4kcD4X0Ubp21cDj2 +BL8nySV5Y4FkmiER2ezBarg0lhRQVKGyyVOJyh9ihdJNsULlGW7bRzTiIBaGqWDYWwmomZzdLeL JerZrguczOZ9laApnpdlgcAIAg4+unE3PkIPUA6dejYJ1R0b7b3yAuFRmYF/XO8ZvAZAeH4XVb1r O7wO7Ydn8R84QwY7ZHicLJdJetZXGc5K4JEXeZqhInmFcKAnIc9zlOVJX53v38pfAQYAqn+hIwpl bmRzdHJlYW0NZW5kb2JqDTQzIDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA4MyAwIFIg DS9SZXNvdXJjZXMgNDQgMCBSIA0vQ29udGVudHMgNDUgMCBSIA0vUm90YXRlIDkwIA0vTWVkaWFC b3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1lbmRvYmoN NDQgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvVGV4dCAvSW1hZ2VDIC9JbWFnZUkgXSANL0Zv bnQgPDwgL0YxIDk1IDAgUiAvRjIgNjQgMCBSIC9UMSA2NSAwIFIgPj4gDS9YT2JqZWN0IDw8IC9J bTIgOTkgMCBSIC9JbTQgNjYgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgMTAwIDAgUiA+PiAN L0NvbG9yU3BhY2UgPDwgL0NzNSA5MSAwIFIgL0NzMTAgOTAgMCBSIC9DczEyIDY3IDAgUiA+PiAN Pj4gDWVuZG9iag00NSAwIG9iag08PCAvTGVuZ3RoIDcxOSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+ PiANc3RyZWFtDQpIiaRUUW/TMBAWqRu3YSLdoKIR0Wa6DOyKuLYTx6n2NgYIeEJE4qHlaWITSBMC Hvj73MXJuoE0TeKhdc+577v7vrtUs4to+eajZhe/Is2+sqioZGmZs5UsCmaLlaxrljvdnj+/ROfR j0gxXcPjiuVlKauSKWadleWKFa6QpWZnl9Hy7WXJTr9HH6KTJlq+Nkyz5hyAK7kCWPsNKKsw3Rms 2FzC44tISaWUYc0ZRM3viN8LRPMtWja6Z6gAq1nuD+AoHXLUWtbacwCUf3rfom6vW4Omtu6aD4jI tZYrPgzpYDSmG6UN3Bhp+H3xuXkX5Vq60jE4jCpr1pz6RjVWW/OdHUTYB/Fkl+4EgdCG77WwCmB1 AX2COyWi1nz04OEjuocpszblVeMdLSyIUtIZaFAbJ7VjdlWiM95Oc83OKzNwBgVK0krJym6tzK+3 9zRO50RYHoMkWfEwIAlNj+hohE0vFm70ZJRPaduNaiVip57BtnYepTFNguFsIwg9DEJK4jiNk78s hp7RY3/gipTSrJiDb7sdzNSDegGwPTVOsfezaJMWE0Ln6UY8P6G7L+sXx/DJ8Uc+vVEy7+AoylRV S3LnImv+mIRkEIrcwpATOt6IjaBpltHh/sEhidGgXnInN3ecxmQ2jjO4THwstCw5pG7NU8bWW/+M rxWTQRpnFHBDAatb8Vk4edZi/kfNjSFv5VS9HJHD64k98izdP+yG5zM6TXQep3ihud94XPggIxkN YQRd0/9Kw7K69GWBVcCW8iEookFC+zhJgGM8pFMYndu9Tasp7qy1k7qYiLyAnklMYDMJRZ2Gn44P UIrl3QGvoXR86FNnkMNpFiR4huPudoCKC94z4NMaYjrGuOAhRQ+RiwjDu0vPgFO0YK0CluMOvu9J 50JxGqY0xrBsoeCjDzJ4A/l1R4urrVy1BuCGoJOwUr1z4Aj8R/wRYACtYPkQCmVuZHN0cmVhbQ1l bmRvYmoNNDYgMCBvYmoNPDwgDS9UeXBlIC9QYWdlIA0vUGFyZW50IDgzIDAgUiANL1Jlc291cmNl cyA0NyAwIFIgDS9Db250ZW50cyA0OCAwIFIgDS9Sb3RhdGUgOTAgDS9NZWRpYUJveCBbIDAgMCA2 MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSANPj4gDWVuZG9iag00NyAwIG9iag08 PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUMgL0ltYWdlSSBdIA0vRm9udCA8PCAvRjEg OTUgMCBSIC9GMiA2NCAwIFIgL1QxIDY1IDAgUiA+PiANL1hPYmplY3QgPDwgL0ltMiA5OSAwIFIg L0ltNCA2NiAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSAxMDAgMCBSID4+IA0vQ29sb3JTcGFj ZSA8PCAvQ3M1IDkxIDAgUiAvQ3MxMCA5MCAwIFIgL0NzMTIgNjcgMCBSID4+IA0+PiANZW5kb2Jq DTQ4IDAgb2JqDTw8IC9MZW5ndGggNjU4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0N CkiJtFPfa9QwHMde21zrWOfpcMUy4s2NZJJckqZN+yIo20R9Egs+XH0abigMUR/89/1+k7vdFBk+ 6EEv5Mfn+/nxTTS9zBcv32l6+T3X9BPN61bahrqmlXVNm7qXXUeF03789jG/yL/miuoOtlsqrJWt pYo2rpG2p7WrpdX0/CpfvLqy9ORL/jZ/MeSLM0M1HS4A2MseYP4fUI3C484g43AF25e5kkopQ4dz mA0/cnYn4sPnfDHodYUWsJqKMEAN67BGp2WnQw2AsvdvPOp23g48ed4lm8RcaC17lqRkMs3IqLSB FSMNu8s/DK9zoaWzjsJglO3ocBKEamRbsq0tRDTbxc49shVFXBs287AWYF0NOiEdi6glm27ff0Bm eKT0R06HkGjdgCklnQGB2jipHW16i8mEOM2NOK/DwB7UaEkrJdtmE6W4Ke9xUc1j3rACLMmWpVFc kuqQTKco+vjYTR9NxS7xapS3+Lu/w6ogUADSKDh8MDvYL8o0GLiRMsjGmMOAt8RJ01NnpWk2vdkN rVl7gAvUYSNXlNoGyodxGnOwzyZpSbKRj5yUESxoluyNPGytVKCXMnoeE7F79Ksisaruu4caTv6/ BkHCRseB3eDNwtk8JenOCoFpRgDZBK4bdZ15HbirkfGaJcTTzbllGXwp6GlZkfwbm2sqtDlJucD+ lmQ2O5qF3ykXSP7s9AzdBf8jz4q9uACvyR4Xjo2cKzgT+xhwAXxFhwg07A8GxeZ5L5mviEcdHFXw +OZZ6mU4FhcJhyfewhsRNZbKCnihxrbI9XQ/mhfVQXYAPbgtibb76yT6VRLVPhYNPSVAAnnXIGyS RlVCkiolYe5769h85FURVp54A0frF/1TgAEAhXrsGAplbmRzdHJlYW0NZW5kb2JqDTQ5IDAgb2Jq DTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA4MyAwIFIgDS9SZXNvdXJjZXMgNTAgMCBSIA0vQ29u dGVudHMgNTEgMCBSIA0vUm90YXRlIDkwIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3Jv cEJveCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1lbmRvYmoNNTAgMCBvYmoNPDwgDS9Qcm9jU2V0IFsg L1BERiAvVGV4dCAvSW1hZ2VDIC9JbWFnZUkgXSANL0ZvbnQgPDwgL0YxIDk1IDAgUiAvRjIgNjQg MCBSIC9UMSA2NSAwIFIgPj4gDS9YT2JqZWN0IDw8IC9JbTIgOTkgMCBSIC9JbTQgNjYgMCBSID4+ IA0vRXh0R1N0YXRlIDw8IC9HUzEgMTAwIDAgUiA+PiANL0NvbG9yU3BhY2UgPDwgL0NzNSA5MSAw IFIgL0NzMTAgOTAgMCBSIC9DczEyIDY3IDAgUiA+PiANPj4gDWVuZG9iag01MSAwIG9iag08PCAv TGVuZ3RoIDc0OCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIiZRUXU8UMRSN3dl2 ZyTsAm7i6shWQWgh02077XQmJD4YFD+ejJP4wPpEhGhCjPrg3/e23S9AibxMM+09995zzm0Vvcgm Jx8VvfiVKfqVZmUljKXOVqIsqS0bUde0cCqsP79k59mPTFJVw3FFC2NEZaik1llhGlq6UhhFzy6z ydtLQ4+/Zx+yl202ea2pou05ABvRACx8AWWlD3faV2wv4fgik0JKqWl7Bn/t74zdQ7z9lk1aNc9Q AVbRIi6Qwzifo1aiVjEHQNmn9wF1e90aOIW6p6yT8EIp0bAuJp1eSqZSadjRQrP7/HP7LiuUcMZR WLQ0NW2PY6PKVztla2seYdf7gw2yhhBXmm0GWAWwuoQ+QR3jUaest771gGz6kEch5FUbFS0tkJLC aWhQaSeUo7YxXpkop16RcyGG96D0lJSUorJLKYuljqyXjzDJn5On+ZTvk04+fjblSY3w/jWNoKgX KS7eYyN0Qx187VLZYQTNOwD7a2/DTJCZHl6MUCjtJ7sEhYLjpItTjPLuiOBBisku8iKj3YQbRnb2 YTMhpPe4R4Iq0ksttV1K3cTUgUQ3H6N0yglOn6DuQ1IMyQ7CIw5DAbmAqc+Fus+2MYFCeTchGI2T vZB4wbeY9Q51Kl1Voc7/MLSxj17Pkzw4cNAy1B+kI3JEMJARlvXHoz0ySWpMoCcraoa5FCVLcUJi wCEvNGxvz/T4O2NVhnY6EIGCgdftMuCVv3ylbgQEKwW9qn875axd0pg7ZUhyOG9jyhAeENB17AcG SOEEmJyQ4ZUYboGAD3wxP19tf5E/jt7wRnoTUcGifjJOUdc7OCxuBB4R3397ELUI7bIA3bt9bI2W 4QG7y9hKNzMVvORFKRwbvgmWbkQR/F7NoqneUc2wC+NWgbWFgbOUw02f/yTcp4iM4k7gFZ1cwCTo 6F29MpUrtpoqXOa72RpnZmt2473IxaoHg9RfuoR0pjwfwbXIFzYHVeEt+iPAAGhX/foKZW5kc3Ry ZWFtDWVuZG9iag01MiAwIG9iag08PCANL1R5cGUgL1BhZ2UgDS9QYXJlbnQgODMgMCBSIA0vUmVz b3VyY2VzIDUzIDAgUiANL0NvbnRlbnRzIDU0IDAgUiANL1JvdGF0ZSA5MCANL01lZGlhQm94IFsg MCAwIDYxMiA3OTIgXSANL0Nyb3BCb3ggWyAwIDAgNjEyIDc5MiBdIA0+PiANZW5kb2JqDTUzIDAg b2JqDTw8IA0vUHJvY1NldCBbIC9QREYgL1RleHQgL0ltYWdlQyAvSW1hZ2VJIF0gDS9Gb250IDw8 IC9GMSA5NSAwIFIgL0YyIDY0IDAgUiAvVDEgNjUgMCBSID4+IA0vWE9iamVjdCA8PCAvSW0yIDk5 IDAgUiAvSW00IDY2IDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDEwMCAwIFIgPj4gDS9Db2xv clNwYWNlIDw8IC9DczUgOTEgMCBSIC9DczEwIDkwIDAgUiAvQ3MxMiA2NyAwIFIgPj4gDT4+IA1l bmRvYmoNNTQgMCBvYmoNPDwgL0xlbmd0aCA1NzQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0 cmVhbQ0KSImEkktv1DAUhUXixDOmaoZCpUYNE6sPsCvF42ecbFEBAStEJBZNVxUdgVQhYMHf59rp dAoSsInl2Ofce75rRddk9fqDousfRNHPlJhWWEe9a4Ux1JledB1tvIrr90/kmnwjkqoOjlvaWCta SyV13gnbU+ONsIpe3ZDVmxtLz7+S9+TFQFavNFV0uAZhL3qQxS+onAzXvQ4Vhxs4XhMppJSaDlew G34S9iDhwxeyGtTGoQWtos20gIf1waNTolOTB0jZx3dR9e+6HWSKdS9YinijlOhZluN0NsejVBr+ aKHZQ345vCWNEt56CouWtqPD+dSoCtUu2M5OULjdYvEI7yQJV5rtRVkLss5An0DHBtUFm+0+foL3 wpXDeOXlMBE1DkJJ4TU0qLQXylPX20Bmwqnv4byDEWZgQiQlpWjdFmUT23MRxiFKq7pG2ck8T6rs DzBGC2nBJC7g00vR9dRbod0W5/4k2pQVrnUOLkOeZjsviHY4wydLhPMKJ3mJj1GeowKnVaCJivkB wqdVwcGa4bKej3zkOCKQgauy2mwdTaybLetjcKhOMSqKqihxvphneLZA6dNyWZfPfsvS3PYVRmX8 NKP/dH9/hrNZGOHZmYcUzX7IwT3bRCnCZuQAMGYZ+XM88gqiWDYyHrpn96OYrr2zN5N9Uh+NHNXg A/GzIGlZHgcCgMocjyzJF8GxZcvbUziojwIvzyou4XHWSZlkB39l1se09TJd4CSbiOETlKN0M3Z4 ar8EGADm+KzWCmVuZHN0cmVhbQ1lbmRvYmoNNTUgMCBvYmoNPDwgDS9UeXBlIC9QYWdlIA0vUGFy ZW50IDgzIDAgUiANL1Jlc291cmNlcyA1NiAwIFIgDS9Db250ZW50cyA1NyAwIFIgDS9Sb3RhdGUg OTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSAN Pj4gDWVuZG9iag01NiAwIG9iag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUMgL0lt YWdlSSBdIA0vRm9udCA8PCAvRjEgOTUgMCBSIC9GMiA2NCAwIFIgL1QxIDY1IDAgUiA+PiANL1hP YmplY3QgPDwgL0ltMiA5OSAwIFIgL0ltNCA2NiAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSAx MDAgMCBSID4+IA0vQ29sb3JTcGFjZSA8PCAvQ3M1IDkxIDAgUiAvQ3MxMCA5MCAwIFIgL0NzMTIg NjcgMCBSID4+IA0+PiANZW5kb2JqDTU3IDAgb2JqDTw8IC9MZW5ndGggNTIwIC9GaWx0ZXIgL0Zs YXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJfJHNb9QwEMWF48SNWTXl61Ap2rW62WIf4vVnnFxRARVO iEgcdjlVdAXSCgEH/n3GcbdbOHDJyPG8N29+1mxH128/arb7RTX7yqjtpPMs+E5ay7wdZN+zNuip /vxCb+kPqpju4bpjrXOyc0wxH7x0A7PBSqfZzZ6ur/eOXX2nH+irka7fGKbZeAvCQQ4gm76g8iq2 BxMnjnu43lEllVKGjTdwGn9T/giJ8Rtdj/rg0IFWszYV8HAhevRa9jp5gJR/ej+p/j+3h52muRue YdFqLQeeFyQ7KclWaQN/jDT8sfg8vqOtlsEFBsUo17PxKgXVcdqGz2ZR4U+rsydkhpDQhj+dZB3I egs5gY6Lqg0/OX32nMxiC5paXo+JqPWwlJLBQEBtgtSB+cFFMgmneYDzHkZ8AxtX0krJzh9RtlM8 N8GI0VakQAtMLnC1QMU5ub78h481ErrbVMBuULIfWHDS+CPVF0l0mC595z00w1rTOG0SjTiPnInW Ss1LoCg7nnhawImF4aTBBc4K3JCszlPfSoAdn5/Hk+eozkm5rOfpriDFnRsmDVqtcIXvVBkmE0MV H0Y7Y++zqJCyLHGx5RiDXU5QVeFmnqHlVmDRQRaSwtUVuagmOvVWvMzrBSq34qGx7bujsU3GZd6Q MuqEl5YvRGtkz1GB4qZ/O10envmPAAMAkkyZJQplbmRzdHJlYW0NZW5kb2JqDTU4IDAgb2JqDTw8 IA0vVHlwZSAvUGFnZSANL1BhcmVudCA4NCAwIFIgDS9SZXNvdXJjZXMgNTkgMCBSIA0vQ29udGVu dHMgNjAgMCBSIA0vUm90YXRlIDkwIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJv eCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1lbmRvYmoNNTkgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BE RiAvVGV4dCAvSW1hZ2VDIC9JbWFnZUkgXSANL0ZvbnQgPDwgL0YxIDk1IDAgUiAvRjIgNjQgMCBS IC9UMSA2NSAwIFIgPj4gDS9YT2JqZWN0IDw8IC9JbTIgOTkgMCBSIC9JbTQgNjYgMCBSID4+IA0v RXh0R1N0YXRlIDw8IC9HUzEgMTAwIDAgUiA+PiANL0NvbG9yU3BhY2UgPDwgL0NzNSA5MSAwIFIg L0NzMTAgOTAgMCBSIC9DczEyIDY3IDAgUiA+PiANPj4gDWVuZG9iag02MCAwIG9iag08PCAvTGVu Z3RoIDY0MiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIiayTwW/TMBTGRZrkLaZa OmBokaLOa8tmC+Lajh0nR9AAASdEJA4tp4lNIE0IOPDv8xy367hMO3CxFT9/fr/3vRdFr8jy7SdF r34TRb9RUjfCWOpsI+qa2roTbUsrp4b911dySX4SSVWL4YZWxojGUEmts8J0tHa1MIpeXJPlu2tD z3+Qj+RVT5ZvNFW0v0RhJzqUDSuqrPTXnfYZ+2sMXxEppJSa9hf41f8h7EHE++9k2avtCw1qFa3C hm8Y599olWhVeAOl7POHQXV33hZrGvKu2CjmlVKiY0kKo70M1lJpPNFCs4f8S/+eVEo44yhuWpqW 9ucBVPlsKzYee4XdzycHMI4irjR7NMgalLU1cqI7xqtWbG//8RMY76687oOjtcWipHAaAZV2Qjlq O+OdCXbqW3bemOF7UPuSlJSisTsrqwGvDXieLc/jxXQU8UYoNl/zGMocTvLoOIaTcs3PkvI4ytZ8 ILplG3J438Lm226E7qjD1e7MPgxeb6FwIlrfma1HdYBYlrwywrEUFpG3NyqiOZ4wTCqRKYb5GWTJ Gcw8DpTPIINFfJTHMeCSpbyy2IyY16JmOaSTLIGnMJ+mMCBL3xmp7U1n1LZ0HnMjWlYUQZOcwr81 Vhte1De6aQb9PapSemdtWsDMZ9Esj5CvY+UCogKSMoXDF4k7gGkSMLTHmKHpURouhjELhfnwKVfC sP9DKE0gPM8KOJ6ueRrNfNsL7n8dFj+fxskoLrzReJwcDQYrNuF+0hlgHMcB0hJti7G8PCoXd1lX y+7eYHoHViYDkfMDgI5oFrB2VBO84on1Jg6OV9p6rzzXSz9SzSaUl1Bsyjrd/ll/BRgA15fWAgpl bmRzdHJlYW0NZW5kb2JqDTYxIDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA4NCAwIFIg DS9SZXNvdXJjZXMgNjIgMCBSIA0vQ29udGVudHMgNjMgMCBSIA0vUm90YXRlIDkwIA0vTWVkaWFC b3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1lbmRvYmoN NjIgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvVGV4dCAvSW1hZ2VDIC9JbWFnZUkgXSANL0Zv bnQgPDwgL0YxIDk1IDAgUiAvRjIgNjQgMCBSIC9UMSA2NSAwIFIgPj4gDS9YT2JqZWN0IDw8IC9J bTIgOTkgMCBSIC9JbTQgNjYgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgMTAwIDAgUiA+PiAN L0NvbG9yU3BhY2UgPDwgL0NzNSA5MSAwIFIgL0NzMTAgOTAgMCBSIC9DczEyIDY3IDAgUiA+PiAN Pj4gDWVuZG9iag02MyAwIG9iag08PCAvTGVuZ3RoIDc2NyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+ PiANc3RyZWFtDQpIiaRUS2/UMBAWXiduwqppoZWIFHXdbbp1BPHajhMnVygg4ISIxKHLqaIVSBUC Dvx9ZuKkD1QhJA5ra2LPfA/PrOaX8fr1B80vf8aaf+Fx1Uhbc1c3sqp4XXWybXnp9LD/+BxfxN9j xXULxw0vrZWN5YrXrpa245WrpNX8/Cpev7my/PRb/D5+3sfrV4Zr3l9AYic7SBtWyKoVXncGEfsr OL6MlVRKGd6fQ9T/isUDUvRf43WvpwoN5Gpe+g1qWIc1Wi1b7WtAqvj4bsj6O24LmgbcMzGjRam1 7EQQstlWxDZKG/hipBEPi0/927jU0lnHYTPKtrw/9UQ1op2J+Rwz6u1kZ5fNCSm0EY+GtAbS2gp4 gjsWs87E1vbjPTbHK/PhysveO1rVIEpJZ4CgNk5qx+vOojPeTnPLzmsz8A0qlKSVkk19Y2WJ9HTl 6R0mRWlkK8iChEVZCYLyKDvMNsVJkC1ItCkGKrf8AgJomN/wva00HXew1jcu73uTJzbQCi0+yWSO 9egRWyYeuZZ6wl7ewmYkZVHASJLQ/GBGjjbFH+cDOYXmK1O33saN2MlSuEcTkuVQoSidYIWWVuw/ CwpoDy0cIhqxu8fIiq7YImrJ4sAfLQAgHXI8HPyCJ+HO6q4N5SgJgBvTNIOwfxCuJ9dR9T16U+RQ CRYllAVZyAI6o2kaJQSusEWWglNW1uKObF1f2zoMh8hpShIaIfvV0Hwhy+mSBnnKsoCBO1HIaJJk CXwO6Swk8BVZ37Fx6BLlPN9lkoGbIZIzghRGUCDaAo2ygsGICgWcgbEPFwXkdiJLMNSCMrJYTsFY Igp814Ur1NOOev7HXN/UxtOVlO0gXiUiP61ONsADOYN5lAb+kCG4RU4wGmLK8A6H42E07sfRLATD fJDB/IPCY7aLsREvIBbt0GkYsqcHvhbBys6LdoJCC4LUnI7XyOAJEIuOsBtrAaNWYpfSe57XP8c4 tFHK/MygcwpWeB8USqA2y/KcLaFjxgsnq+mv5LcAAwDg2PjTCmVuZHN0cmVhbQ1lbmRvYmoNNjQg MCBvYmoNPDwgDS9UeXBlIC9Gb250IA0vU3VidHlwZSAvVHlwZTEgDS9GaXJzdENoYXIgMSANL0xh c3RDaGFyIDYzIA0vV2lkdGhzIFsgNDcwIDQ3MCAyNTMgNDk0IDQ5NCA0NDYgMjUzIDU0MiA1MzAg ODc5IDU5MCA0NzAgODE5IDQ3MCAzMDEgNDQ2IDQ5NCANNDcwIDQ4MiA0NzAgMzM5IDQ3MCA0NzAg NDcwIDQ3MCA0NzAgNDcwIDY2MyA0OTQgNzM1IDQ4MiA0ODIgMjc3IA0yMjkgNDk0IDIyOSAzMDEg NDM0IDYwMyA0MjIgNDgyIDU0MiA1NTQgNjE0IDIxNyAyODkgNDM0IDI4OSA2OTkgDTI4OSAyODkg NDgyIDQyMiA0NDYgNTY2IDI4OSAyMTcgMzg2IDYwMyAyMjkgNDQ2IDYxNCA0MTAgXSANL0VuY29k aW5nIDc4IDAgUiANL0Jhc2VGb250IC9GRU1MQlArQWdpbGVudC5UVC5Db25kMDgzIA0vRm9udERl c2NyaXB0b3IgNzEgMCBSIA0+PiANZW5kb2JqDTY1IDAgb2JqDTw8IA0vTmFtZSAvVDEgDS9UeXBl IC9Gb250IA0vU3VidHlwZSAvVHlwZTMgDS9SZXNvdXJjZXMgNzMgMCBSIA0vRm9udEJCb3ggWyAx IDAgMjUgNDEgXSANL0ZvbnRNYXRyaXggWyAwLjAxNzI0IDAgMCAwLjAxNzI0IDAgMCBdIA0vRmly c3RDaGFyIDc1IA0vTGFzdENoYXIgODcgDS9FbmNvZGluZyA3NCAwIFIgDS9DaGFyUHJvY3MgNzUg MCBSIA0vV2lkdGhzIFsgMjkgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDE3IF0gDT4+IA1lbmRvYmoN NjYgMCBvYmoNPDwgL1R5cGUgL1hPYmplY3QgL1N1YnR5cGUgL0ltYWdlIC9XaWR0aCAyMjkyIC9I ZWlnaHQgNTUwIC9CaXRzUGVyQ29tcG9uZW50IDggDS9Db2xvclNwYWNlIDY3IDAgUiAvTGVuZ3Ro IDE5MDA4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJ7NYtm+JIGAXQf1yRHcnK Rm7kYGNZRyQrwUYSuVjkxPamvxhINwreVJ6Zcypg762Y3JcXAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADu0TZV 0+YuAQAQqq9SSkWq+txFAADi9M9FKoYnLXM3AQCI06Ti4/ybuwoAQJjqvHmq3FUAAMKcJ0+RclcB AAhTnjfPIncVAIAwq8/Jk+rcVQAAwnTnzdPlrgIAEGf/MXn2uYsAAERql8PiWba5awAABOu7PncF AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADgd3TIXQAA INihfkoplbXdAwD8xtYpFUWRipTWuasAAER5ft87r3/Fc+4yAAAx1p975+2sc9cBAIhwuhg8r6fL XQgAIEA92jx17kIAAAHK0eYpcxcCAAgwmjxFyl0IACDAl83T5W4EAPB4XzZP7kIAAAHK0eQpcxcC AAhQjzZPnbsQAECA4/XkKbrchQAAIvxztXnWuesAAMRYXkyeZe4yAABR1ufNs85dBQAgTleXw+Ip 6y53EQCAYPYOAAAAAAAAAAAAAAAAwG/svz53AwDgQdrnlMr6lLvGDB1/pJQW29w1AIBHqFMxnPR0 zF1kdo5Pw5spilTnLgIA3K9Nb5unSIs+d5WZ6Z/exuDwa3JXAQDutnifPMOzy11lZnafLyYtclcB AO72PnnS8KxyV5mZ+uPVDOeUuwsAcKfj+bueqtxdZqb69WoOubsAAPc6f9eLOneVmVn/ejW5qwAA d6vOX/Y2d5WZ6c5vpspdBQC429GH/ZbV5+g55m4CANxvX7592Jd97iKz07+PnrLNXQQAeITTuqpW +9wtZqlbV6utMQgAAAAAAAAAwB/r1G6aTXvKXQMAIE7fLNKgSGmx6XOXAQCIsXka5k5RpLfz1OSu AwAQoK9ScXX+7nNXAgB4tJ/PxWjzFM9GDwDwu6nGi2c4Ve5SAACPtf26eIqUtrlrAQA8Ul9+M3mK VPa5iwEAPNDm28lTpG3uYgDAn+DQbJrNYYKgv25snuUE2e2mqjbtzwmSAIBZ2i3SoEiLNjrpdGPy FOkUHb19GlKGp2yikwCAeapfR0fxOjxSHRzV3dw8XXByfU5K1c/gLABgjt7XwMcgCB49u5ubZxsb XF+GVbFZAMAcbd93wOcg2IWGNTc3TxOau7/KSpvQMABgjsrr8bEIDdvf3Dz70NxFugwryj40DQCY nzaNxkcbmdbd3DyHyNh22DkXWSn2kgDADDXj9dFEpvXjhXU+fWTsprjKLVLoJQGAGfoxXh/r0Ljl jcnzHJrajPOq0DgAYH6q8fqoQ+N230+etA9N3YzzbB4A+NM04/WxCY3rF99unkVo6NSXBADmpx3P j3bivLcTHHqY+JIAwOz05fUcKKMD628mTx0dupj4kgDA7Gyu98c2PPDr6AmfPC/t1JcEAGZneTkH qgkC69HmiZ88o8zVBIEAwNz0zxeTp58isV1cLJBFO0XkS/0rcTXJJQGA2WnK9zVQNlMl7lcfA2S1 nyqy/etjY+2mSgQA5qbfNVXVtP2UmV136LopA1+OzWDaSAAAAAAAAAAAAAAAAAD+Z7eOkdtGsjAA 7w3kG2husL6BcAPrBtYJGsqISIRDMbAChGSNEiOlMiqEQzFSLUMi4FaZKcVNhNQLUbKGANEPDTSI p+7+P2mqpkbAw2v0G/QPAAAAAAAAAAAAAAAAZls/zO5nyw13G7V+zWcvjXK3AQAAACbaxAPhi5ff UZJxN0N4vg/FW6MxYg8AAAA0k03FS4x4/RGDBXc/UkmQ9/ne6OSZux8AAAAwyfo6DxJ/kkT+LyLm 7qhaNn7NOu+hJ1hztwQAAADmWA8KSeLlZ8zdU6XRn1D2ns4GCD0AAACgKAsLcef1J+buqsKPt972 Gx1k3F0BAACAIeKKyOOLBXdbBxbvze33OeZuCwAAAMywrIw8fsjd14HwvblCpyl3XwAAAGCEuDrz iAfuxkrmkj5j7sYAAADACJIo4U+4GysZSxoV3I0BAACACVJJ5BED7s5KAlk4W3J3BgAAAAaYyaKE v+FurWAj61P85G4NAAAADHAvixJi2UH1zTJdLpfrDiql0swz66A6AAAA2G4myxJ+qlk5exiLF3l8 EpOHTLPaUpp57jUrAwAAgAvuZVFCLLXqbmKxS07+WzURb7Tqpcg8AAAAoCGRZolMp+xM/IlO7xlK L5xk0j7nOmUBAADAEaksS4QaRTcj4R8U9MX1RqNmKMs8a42iAAAA4IxAEiXi9iXXg/z+g4L5b6CR T2JJn9/alwQAAACHTCVZIm1dcb2LUZVVB7/aV61u05+1rggAAAAu2VRniah1wefBrkB1Qgk2retG lRWDrHVBAAAAcMp9ZZZYt64X+bsK1VFKI0s9BVUVk9b1AAAAwDGjiizxs3W1RFSGnQ5CykNFtUnr agAAAOCa7DD0xO2LBXTkEYOsde35QbXr9sUAAADAOdmklCV+tq81q4k8vj9rX/yhFKhiRB4AAABo Yr4fJqK1RqVvdZlHhBrVn6K9SuFCoxIAAAA4KZtHr0EiiFOdOovayOMLraiynoavfU7mOmUAAADA XekyTTeaNaaiPvRMO2g0060BAAAAoCGqTTzCj7ibBAAAANCkEHmE4G4SAAAAQJNC5PGReTqx+peS LXefZYm81zPpTVfym4a9dW6BM7WhaeWMe3EtYbYo9u03jRhw7tbgI9ooZZ41d5tWuFU7ie64+ywj Mo8nvWkov2nYX+vm89SGphX57n1smC2KfftNIwacuzX4iFKhEnpS7jat8EXtJLrg7rMMmYeRpzY0 rZh6BmK2KPbtN40YcO7WgME6+RFFIoyi2X8kV6hEHvFceevzYhpdCyGi8Wx5vCXY45PaSXTK3WcZ Mg8jT21oWjH1DMRsUezbbxox4NytQd+eZ6HIA8vLb/5PEFcGE6XMU3Xjcrwr+/YzmG6OuxjzPaoe RSvuTkuQeRh5qlPTgqlnIGaLYt9+04gB524N+pXNgjyLvEUSfxd8ol+Hl4UKmSc8vG0d5UWLl02f j78okxFf6qIb7k5LkHkYeapT04KpZyBmi2LfftOIAeduDXr1K9wFkf1Ukv/ODq6b1EceMTm4aypK tV9+gmUfCzPWmepRdM7daQkyDyNPdWpaMPUMxGxR7NtvGjHg3K1Bnx6qskv+Gz2XLkzqI4+flO7J Rgd557V++UL4x/+Uj6IT5k7LkHkYecpj05ypZyBmi2LfftOIAeduDXr0IA0wo6x45aY+8oin4i3P 18KvvNAXcX9LNM1U/SxKuHstQuZh5KmPTWOmnoGYLYp9+00jBpy7NejPnAgw16VrJ7WRJyrekI3y /1Z5Yf6b9LVE41yon0VD7l6LkHkYeepj05ipZyBmi2LfftOIAeduDXrzi4wwcfHitDbzpMUbfrz8 R+nVy/6WaZZTY88iZB5GnvrYNPbB5kwZZoti337TiAHnbg36koV0hlkUL49qIk9UvHyxyzvSqwdZ fws1yX+bHEZb7m4LkHkYeU3mpiFTz0DMFsW+/aYRA87dGvRlVhNiwuLlTwF9/aZw9XO4yzvSq/1p jys1yPcmh9Edd7cFyDyMvCZz05CpZyBmi2LfftOIAeduDXqSBXTk8UVSvCEhry5dfP+ad6TXC1HM SPDqvMlhdMndbQEyDyOvydw0ZOoZiNmi2LffNGLAuVuDnszkgeTtJyzdERMXx6Vrw9e8I73BF9O+ FmqURofRKXe3Bcg8mv7+d/t7vUaD04ypZyBmi2LfftOIAeduDXoS1iSe/GdRukUeesqRZ0EGnt1P 0NdCTULkhior7n73IfPo2A7/0vn4es0GpxFTz0DMFsW+/aYRA87dGvRjXZt4DpOMNPQoX7j/k/az UKNcNjuNbrn73YfM097KP9H7+HrNBqcRU89AzBbFvv2mEQPO3Rr0I6kPJX54cNc8qLgumB9cFypk nlkfyzTM52an0Tl3v/uQedpKvmp/fL1mg9OIqWcgZoti337TiAHnbg36MVbIPOLwtiw+uCh+OrxM obYfHX+Rplk1PI1OmPstQOZp52+vg4+vJ32N+kw9AzFbFPv2m0YMOHdr0I9IIZaItOLGp2mwd0lQ lXh+pyqBCpnnwG3T4+iRu+M9yDwtbG9OO/n4ek0npwFTz0DXZ4tm337TiAHnbg36ESikEn9ZfW96 P46+iW/R5L4qFOWWKsXF8RZnqoumx9GQu+M9yDyNrfyTjj6+XtPJacDUM9Dt2apj337TiAHnbg36 oZRKJJGmVorM08oJ8T/mh/88bROpR+lNLp9LjxfdfXy9ppPTwEcasiZcnq169u03Tf51Srhbg34c NfMskXnaeGx+Hm25e9bk7rl0d3a45PbVvOajo8zUM9Dd2VJh334DkEYqqWTZsniqknnCTtdjA+Ij LXPH3bMmR8+l7e1p1ZKP90ByiI73WFaOzpYi+ctB5gErRSqxZNOy+EaleNTpemzgkSdTpUvunjU5 eS6trk6ql3y8R5JDdLzHsnJytpTJXw4yD1hpqpBKROvqgULxaYerscKWPJiqnXI3rcnBc+nxq3TJ x3soOUTHeywrB2erAfnLQeYBKy0UUknUuvpEofq8w9VY4U72EfpEnFcr7q71OHguEZvJ8lBkHifJ Xw4yD1gpU0glP1tXnytUzzpcjRUuJd+gT7I/vLjl7lqPg+cSsZksD0XmcZL85SDzgJ0m9bFk07q4 QqKadLgWO5xKvkHnd8R5dcHdtR4HzyViM1keiszjJPnLQeYBO6W1qWSsUT2uDT3LzlZiiZX0A70l zqsT5q41OXguEZvJ8lBkHifJXw4yD1gqqoslqUbxTV3kiTpbhy1uZN+g5PdfxIH1yN23FgfPJWIv WR6KzOMk+ctB5gFLpTWpJNaqPq0JPeuOVmGPc/mRdEEcWDfcfWtx8Fwi9pLlocg8TpK/HGQesNWU TCVBplU8C8nq047WYJETySfo8+/ft8SBZfYXysFzidhLloci8zjJ1i8KAGFEpZK1ZvF1QFSPOunf KonsE3T5+/ejtSeWg+cSy1ZSA2T2BMk5OFsNyF8OMg9YK7uWx5K5dvVUSIuPsg66t8yl7BN0m//x E3Fi3TE3rsXBc4nYSpaHIvM4Sf5ykHnAXtnoeJEnDz0BIo+6z7JP0GP+x3PixLrk7lyHg+cSsZUs D0XmcZL85SDzgMWycWUqCdJOqq/DyuoxIs+hrewL9OnlrzfEifWZu3UdDp5LxFayPBSZx0nyl4PM A1ZLgsNUMu4qlGR3h4knXHRU3C635BcooY6sFXPrOhw8l4idZHkoMo+T5C8HmQfsls2CYiiJ0g6r b+JS4rnvKk9Z5oL+PFNH1i1r43ocPJeInWR5KDKPk+QvB5kHbJfNJ8GfUDK633RdPXmvHsaLjovb 40T2BUp2f/aII+ui9UOT26HvneU/X4bfk203C2mm03PpMbm5Gl6cea8/V8Ph9JiLery7Gr4+Kn/Q Sv0+YieP1usxMk+SfB8Ov7y97PwdXCVJhw13sJcNZmtvL6+SVScL2Nfx/2edzLn85TTPPNskf3/D Pw19HearfGxc5MD7pnzJN2WrX687x1nvziqflbdh2VWeJh0VhrJNurhP/s9+1SO3jTRR3oC4AXkD 4gacGwg3IE5AIrMiQakcCIFCu4RklZKZHE7qjfCtQynAVn1OVeNESvnxFyKA6Tc9AAiB+vi0drlW r9909/TMPDw/H0396fn56UjinwIJeQOpze/B/d3rV1vxShR0RmHaUDV8NOV5klt/pG+OmMVp42mn t17BpPa9WPFiwU42nidn0QrLyitvQHQ7XKi6ucpbr5G9ZM5WeisKvx343L1kIJkVixkFSXU10Bu7 ztPNsfI86X0w1suM/CixUcrLljalTtcahJoHY/336ciLZC1pSdxhA69GI884o6OIqAvI3f5e0ldU r5fYrxfrb87xImMk+8+Y0g+x3CZg/V/xz1jMyDya8Dwq9vW3UIbRjMi5hJisWtxnJHmhX2ZCrzLL ZATIc9Ov/c+MWz8HsD1WSiomqn/v9m1aOU8VF81kEQP2XrJmS+pf7P7EWMOMnJT3E6IivTcc3CqL puzFjL3hz3kznicJ9OW9t9Gf29e5PoX68t6PYMI4pnnQt9OYX2+ovzbf6/XuVZWCl2mET9VgIivp nnFGVyGoYfe3v1foQES2q90PSa2x3HFolyX1osCW0bdofc+TTAwPwRaDSHHUGPkQr+QGFymhKzg5 5sF/ehiAK1noUHavgHFcKUs5YakPwpSjVnMvTa5H0KFyR7mlR9MJgbQOzDkfxYqnR0swB0/dGgzP Fn0/tax0QesO5Y5DXzahdb3M+Vf3BsOzg707Ubcc5UGobIXPOKO7ICc93hFccBosX8dUwLM127JO w/OgR6uA/pUy65nzCfAqV3pdwU4zQwc9D3iNihgSjQCw2EujIVnDuJcqgGs4uARBR8oNIXGh/EAe pzesOa/teVTosDPKvqNYwthVT7bFte55jlbvMuUZ/RWYO3vGGSeABTnnyY4RoMNgt5bpe9FVa9op eJ4U349FOJFR0ZSPMn6RbbtXhLBKdIPOeZ7EwpOsMFhYZZjaqe8fPwDTXibGvRRoDUHHyfXvY9M5 47n6Cr1hzHldz3NlLC4HjkXdIjHeThuptj3PvV29Y3a9fMezBmtnzzjjBBCQQ75nxOgoLCyWgkJb jNTyJDxPZHcRrStLDZKGfIw38gr9RKMrbDPtnucBrSFwofgJ2u9lf1E94XDJ28tRQssLOkwuWees NzlWbxhvLh1sHrzEtU2oH/NKjR2zVLJs2/OkwrreiFWusj5UfDd1xhldxoAa8ez+SdFBCPgrca7i nqtOwPOoC04pBZjuXpwP55nsaU2PsE+1W55Hjewr6DmaTujVxxXUe56Cooa9dDhL9FNSXtBRknnO ej6nN5XmfGGSpWONgxdbe7B1qYpRK6trjmrZ8yyq1GsYzg0S8t6nYd7ZM87oPmhDE2acITgHI/ZK C97B8rrveZTLK6WIK6gK81FD3hIDVdIV9ol2yvPw3F4ZMSu71K2mPkqRKtzL1GEuoSh5QQdJpuXp 9SbG3lSdc1Pn6UjT4AXVEqI7mYE5ZaN2PQ93L4tZpqZyKwrHxj6ecUbXQQ+/zDg+OgZ/mAul3Kcr 6rrnUSNmJSX4SBbmI7hLlOtlhyKRGoArmcOrWh7eBV1dvZ8AWbiXLncJj5IXdIzkV7Q4Wm/gnNfw PJOqCbnKUKsaMJWCNj1PRWfSw8NZp5GxoY9nnNF5eOR4q4wToVNwz1xIcI9VP+2251Eut5IyZkAX 5QN+V0RUufHv6JDnSRz79PeIjeqqhjp6VxraywUhL+gQ6bLV+wr3hq9Uwi1UpuPw4PnVExrhWsE1 WETSnuepbHl6hr31qwvHuI9nFPDfx7sv0zWuv8mXD8rh7efDzSaH6d381wfl0CWQ33LuOydBh8Dw TbeHxfn1u+15RvxKyohpXZCPZ7FC/09BV9hn2R3Powb22b8jManX2suhIoUb2ktqBUGHgF+VAI9u vd7ESBpkj8JqWADTSIMLpCTUmudZ8JMqw1W08KyOcAIbecYh3uT1dHbwc/cRjuPpYbpdfv3XdHr5 +PoBSXQJ9AE+vBAdcAYGrHUU6a00oC83aVtF854nsChEg6RKPla4KugKe4nueJ4KyR8Af+3W+t5d Y0wKN7WX8TG68o70aL3pJ0CaDkODB7+8zJgB6aWNs6b3NrSuF81/anNlljEhhWt5x95AoUaecYB/ 8o5n7Tjufrecw8u3XAqrHGaXjy3n0DHQBzg+YHnoEPyn3joaOORvpF69Rc9j8UWoxUg10iIAp6Ar 7CU643miai3I4EH1RU11ek6a2suBXl40JB8crzcu6DsdBQZP2fgSHSStbeUCHPI3oXW9aP5HNklp sCB0a3opNDRnHOD1e8HxbC1Hu4bj52XBdW1cz03bzqtTcMnRTg5Y8Aq/5azj1Dtne0i9enuep/bN e/x3snjZCXuFrnief2tez/TFv4aqrZ47JIdobC/1C4iG1AdH7E1IN54OAoNXu6V0scthXe0tqJJB CJ1TXbvf6yu9sKgrnNBJn5Hh903Z8mx+vr+1l8RcY7rWfy7/j02PIgfbOaQBT9HreYx1rD6kAKRe vj3P08Bjlh5Pegs/ryvsFbrieS6qNeAA1MW/RlBbnWxUY3sZaOVFU/IJ1Zv6BYDO00H04KW1E6JP NbzeGlgAhJD1NmDIfa0wLnboh2EYCMjhXPifFG9P8lE+vTCYvy8JyzOb3rwePc8dHkprZ67nH0b4 qtofj6xqTwm0F8nfPvAMMNZxoQAfUi/fmudRTv0aJvb52GGU1xX2Ch3xPE28RSGp3sArSg5kY3up 3wnRlHx0xN7on9w16Bh68Pz6CZEurAHtDULreun5DxrIJ9UJCxDgJTuSCh1b3c+PV3kzXduG6fT6 8c3EvSYtz2z2vZV0l8sfmrX3rufytyn658261BX5ev6pbI9PznWY46GDQt37B8BXqOMv0hUpiYaQ BpZqzfPgt0xE6/yUDF1IS6tor+DHmzbFnomY1xUmehkd8TxjGOrFyYqTymCIWI6i1Ce4B44IwtAT mFT0l3uY93I78rZ7uYMpq2Eg/6xo0niiKF/C6o1rEE+pztMh5OCZTJjwwtAXDiaFem2Fo/zNnCWx MKTQqOfBOTnBbnx8B2auEUbfEfEBLwHKAZX1p4a83PuG1Z8vEpPvSMezdhGPrST8S7d09o/rVxx8 vfNILSbcDobkXMscL0BHy3wEIhTuKR6vnFWGtjzPnz7IzT1ITg4BkXhoDO+km2bMxMXUfJcEJuvQ Dc/zL0zxvR3L2AHEiFDHr4qf7HkLAYlSK27Yy4PkTXupdPI4pcPxjRzIHBO9gVFZb1Q8ZKaRBx1C Dp6PFhrG+y5JyOvrtWMUc3DsDL1s1PPAAQpVxlMBIqZlYZ9mRzliQhMHVNafGK8P0+mhb5jevQG2 nBJ2Z/fzu4WM375oFj745xwFz3fV7qO+ompPCmCuVY64QCfLNa4zBtE5BwCvn96He557bh1L5QOq ss1nBS+n7kIu0aVKl299wFRR4ATEhTkm6seAUEf+ephwmfl9yYD3MjcqVfZSwJD4kJo4iDqs25sA MPtE5yt4HmhQw0OmRPXGWnEfROS2F1yX5URY9ZLzDz6unCTHRAVrEqKFi733ad2krHvSeHl6MTDe vk4zw7D76+aVZl/OtFYn+7lrNn8t5jiF2fSJjn2YZmXunM8XY4eMPewG6Lut4GPgnUO84bzowkIB XOijPY9Lh5TePkFzI9t8Vi+NynfUQWSiS6fmeUCNQYGKjMNCrz6kI1yVp4LxIoYf7mXxbIE6q3ie MM/FnxHWvSkMIrRHsV69gudBRRRWQSZPLw/8RaFa6H6b9DzoGzMpcMF0DkrCgLzg5xARaZ8kXv66 nK7wDViA5fKv9fM/zRzA+q8ZbVwep1OD33huvIwiXkwpgPzneeLmr5s3tNhfXzY9PH5ZtXFBTrVf YLrgEJJ32x7oBMs8VQ3RQh/seVI6wlFFMnjKXI224Z2MC2T4khFdOjHPA6am3MCEJhdH2RhQ/oy1 ftjhXso65A0ECBjakPX9t+qNR3NL3wE70BHUaR3RIUX7Cx1qalltXOC6gNuk5/HpiKhEBvOz4HMd i7SpjT1FzDNv8O2VJP29dTkHFmD9PyRFvzTZjem3Y5SSww9TCrPpCxH6rGXP6bXm2+asWnJH97Aj 4J8scAipV+Ud4FCWLjn8KSX1C7TkeUBucZkNbElqmU/5IVs6gE106cQ8j29VYECS+1p1mq/b/SHN vtCpo70sGTbgjit4nqhIRl8c+v4HNL/kMGD2Sitv73msPjbsDAM8p6VjBz81GvQ8Dj+lJZrO0s3s kdSyj3FJ7ohI+wTxcPCqf30jSK9bE7P3Onv+JcH/ZXQbs+n/2C975ESWLAqzA9iB2EGzA9UOmiWw Ahrv4RVtCqciBlOKwMIVXmPWmE8WMTKFgYNLVzuNy0gCJKi851TeqhQUkEcT84w+N+/JrPz5QK2c qZcdAlHMnWiHiPS2hh8muIYlEbkNpykrPew3GX1ucekw7U1YoxMzD56HdBORqyhS5hGemg5xg1U6 M+apwiLpm5JnMZZG/wbtwitKt780OvuWkeFuEbcYPiAFc8NdU4YnayNRTAe7x+LweuYhyx+abrIV zIddiUh1bHbIPORiHgp2/FvMuJfwFWaGD7SxS6e/k7t2+/5/xDE5eNbvgetp8+8/Dv7z+vdLto9E Zjj8+7fo3DK0yI7w404uRcg2Aq0mexz4o/0fFmr0T7v3+Lfo3IqoY7+pyT1SMQkppRquTAxzkzWK 5QbHYZ4/uCCS/B1o/67MI8ybXIuXwTxTXDSW/E1oN4GR7ueWNHoN+2PBzr7l1HAzopJGZ6+RQN9N Mrw0OvnZIa0NWUpp5dc5mIfMINHZdYs5N8wdbHbIPLhLTbKTDza1jmKGJ5sYTLRkWt1tHmL0WK/X y3b74Fmfyba7zb+2D/7z+teT7T0L3sCZ3Ci24K4fK7EUIpvcabm/hK/Ug2nu342zu3Awv7zCv+fM u6eODwB48j9EjqRwx+EfLZUTMw/pMpf8+NUWby72TppuBqFglc6LefA+EFePgEND5ZaJqoX9kWDX fUuCd2rmEahEF4bitLg2DWiXVn6dg3mqsKAp2cnHjTVpBH5kPzXcMQ++mEUgJ5AXWUcZG6OSfZOA mZZLg91DHCPHY+pZvxddq+0/mxiwFP1tw2f+DVxMkGhkgTztF7G0q7MfrOHr/+5QpNluXf7562aO OUSezdAw42NVATfPhwgrmH3oA3Ba5sEFwuX4phosmOrySONj92UwTxPWyLuN7GbB3VFGIs+oFId8 S2lDYreaeULTzd5pafQOticu/GrmIZ82UvrN5SFmATDI7zeHzIP9Q9GPfx98tx7ZDE82cQxmWiot Pp7iHrL00s+66HqBEDMT7QIcmX+OJok0sIjw41mqXMLZTsRO/T3L+/+tQKT7DzCKHU1SL9VvXXys XlWlfUip2Ye/kbHc4TjMcwv98q8v8jRFujzSJPDgl8E8N7AmlAtqmvXA31LeMITEG4Jd+S1xdDXz DE03OR3i+uO1kaZKmUpMr2YedYM69JuAqvtFtm5guzPmIZHmyoL0ryU8srky5H2IwUxLpcnn470A FuNZf5FcM0gNv3T2YzKPTQZlfNG+NmziGu4b2wNXs1Srifd0YpjJpZ91BsgDMBXsgb7PcZinpvST DhIkeebZ1x/19PCCRJpQHfUsBDf5ltL4OLqaeQT/XJddvTYJLgiVaymfVrKaop/cbDeGl/wii1VD u2MeHKkmNyBfILGOkqRHVaNXyfTr8xmeAYvxXovGX+bzv/2bSPaXUjCPTQhlfNFuuYYHxtMxTxVu 6fSvgzeR04JP+7uauE6yt0ifWO5wFOYhFwuIhe8uKdUXv5NvwiWlYx79hduBBSZhEoKP5NHruCIx 3cq9FWB7LIVR+rFbWn+CSGBtarBAYns98+DpNuRAGkgi3rluaGfM04J2dJvhBrF1lGZ60GkABWKU S5PPZ3gBLOnnWjY+Qwr4JdlnpWAemwzK+JbM8wIifTpOxjzk3peuqoAcF3T3ZFVKbEVvlVjucBTm IU2m6ljF8wTYDlbprJgH8yKqwet3a3jH6sULVBXnzDxk16rX5lb04wbyaa0r/arP28Je5dDOmCeA dpkhWUVku5SVyhAMfq5atHePbA9Z7qxIBFPATLK/lIJ5bEI8S5VLJfP0074ViHT/4YgdTVItci9H OntF/K37IXwgxSuL3Sqx3OEozKN/hUksYb2UeTrYDlbprJgHr4YIymvd9yFrPZdHD3BFrElffuYh 2afy2jRV4+uZB/tDORA5eMO09xZaa8qhnTFPVd0hgBUdW2Pl8qDnYffIim/7m8ap9/pBdK0gBSxF vw1uDNxMEWpkw10isq27yP4i2icpF5zZYmfpISr6cjXw5p8KdnbYK5UxaYSrxCuONYrlDkdhHuIH HaaqiSjzEDtYpbNingCWoC9KNoHh7agTkeUeq9yharLlYh792oh+bBe/LTlGoRxorqgIdGHI0M6Y R98BT+L20NghWQxAOnet7jaP7Bg6llYQsO4DCPgp23sWvDEqPj2q2AJ52nLpCNi7sn3ZPfRBwFw/ bZw/Fw7ml0uJu5P4qhbppDzCZWWeJvbrZz4unOfCmecWluRgnjjtDdSJdF/nnJmHDK5fGzE+tmt/ BA1BIlxhXFR1XRi2mKE6jbiihPEi0KEDKxqHxjHJ8uaeggZnqqdBrz8CIPOuycF7/YCGARQQy/aR BW48FZxZlhYWzNOXS2dKTJscuO5JqOVo0B1MVkXnlltk98uHvcEOyw3phKtCpf+kzBPoe6g6eOax q/pa5kGje+apVOpgbYbK+Ngurr52eNrh1t4q/47DflfMo9nIW5H9dmhkv3Tf1UlAiwvVaJ8B4Gvc EymgC/zPFrzx5Q9/LzvDf0HpQPYvUSu7NSyBWtqz26FnZQ4bkWMmNyJdYrnFUZjnVt9D1cEzj11V DuaJio9+PcxThWYXK8/ziB1y7PSafQc8eKgc2hXz5GA8a+ZZN0mYd9XCBDS5TE0+nutH/Fw/ixAQ I3s3EzcevmIqB5pkMw+CmJnofiS9PudVZuRZ3+BtPxYLxvSoRLARuRGHYgHpEqtbuGOeHD1wSbNw nstmHgLKOV7e0D4THD2Eik33OTNPjrXRzfYIzBPAipp9GDE7W3u5QH3scsyXlMwPnQSodqpeF/X8 fuy9vta90YyZRipumYjQsP9HmznRqp0VQpe/z2hms4ZdvoYn15Rs+kSsIM9QRXzEtyI3YiwWkC5y wRkyj1DimWdP5Iuinfa1zKOTZx4yW22HFvZPQaLAfsbK7GfHPOmSOkmzU+26qMdGAw0ErLoZvDE4 QuJJFnctcK2JeBR5zkQR3vF1UEJPSxV2UjNPQ1twhsxzWzjP1TJPqC9ppawE+K+eeeb6tSEr35H8 yg5ktiCQooT8kgu1Q8sFJWMecvHvqzUHra5VaQx4YBAQZ/AGwQ1nWvV4hkdWPLlA5Fk3yXYHJS16 SGLUSc08gbrJMZiHvAW1AEgVyzPPnsgXrYPFbuCS9GprH2m1zph5cpwlbYnSTmYLAilKXN5OIUiD K8QJNLG9AfZ+XTEL4j1QIM//avX8c48Buk/cPKC8MTlK4BnN0OMUMzuY7XECf7XIZo9AyZAekRB1 uhDmIT3yqHCeq2WeHFIwT+hkzpfJPOjHEPk5UD7mSS1QCZmHdMihYXp4+6N1Y9Ret552JNN/4sCw Xq/6BDdGR0n7+qVJhu4iq/rpfuvtx1mzPQ+xjT8FNVN6QBo5WsViQaAt8MxzKLBK58Q8EV89pTzz IJnmPGuDSzzz0PkegXnMUGRzplUPE7TE16nZ7GU2s0GARRfiRv9oCDEqgDxvenmb7mUAz6s6ZKPD oho9Hwmo8swjqXCey2Yexb1sIc88SKa5fMxT18TXLpD6dmriArQ+uOIkzMOmYKjqqSefln2AG6Mj QsSkEPJcmL7hXY4pgZ+VMagit0oiFgS4IFa38MyzJ1UeV1Iuw0aeebaKpeGVfrIyprl8zKOKr10g NfMoP22OCZDwOSSEShqaAWqeenJp9SDixuSoIZ67Uob+8qghyqG57pBsFdGz0QJV5FaRCwJcEKtb eObZkyqPKymXYSPPPFvF0vBKP1kZ0+yZBzi3ugDmUUJPpeqpJ5eeewZuDBZHzrAaGcTTPS52lUVD ssPHsIo/+3V9lVwQ4IJY3cIzz55UeVxJuQwbfS3z5Hi3dPLMs5VnHv0ESPgcEkMlTd0gtSFaaS+m p58HtDGYnSDD71GKeFYnCFECtcj+7oRQ/GDM5V6eeSQVzuOZx16eeZBMc5RjbXDJjc7umYeHzyEQ SnvAvsmr4ZWhxWTQfWeNnw/x7xNlWD2N+hviGTw+nyjD6VV1eqy2Gsq9PPNIKpzHM4+9PPMgFY6u b+CZJ2MCJHwOoVDjmnKgToJW28ur9Jo6PVU7fZebeeaRVDiPZx57eeZBKhxd38AzT8YESPgcQqHW SVM50s0UDeXlVXa5fVF2qsnNPPNIKpzHM4+9PPMgFY6ub+CZJ2MCJHwOoVBvc68rx4rwWF5epZbb U/WhqdjMM4+kwnk889jLMw9S4ej6Bp55MiZAwucQCrWZTE03WIsN5uVVWiVOD9WnQrGbZx5JhfN4 5rGXZx4k0xzlWBtc0tDZPfPw8DmEQm2UKKknSOhwXl7l1NjpofrUrdjNM4+kwnk889jLMw+SaSb7 XIqe0UAJMZ55ePgcQqF2UlLPtyRjPC+vEqrl9FDt6Y/UzTOPpMJ5PPPYK70J8rzrKnnm2cozj34C JHwOoVCfSqK6YrxG5nheXqXTjdNDtaex1O3ymace6lU4z9UyT6Bf7GFqdM88O5lmsjYdMNs5LvHM o58ACd/S7315FikNGyRjOoPNgF5eZRK5ogpKvBQvn3lwD4088+yJzM7BnZvjXdfpMpkH7XNtidJO fqKBQBfLPHIkF4pbJOWhoi8L4eX1NYqsd7dWN1I7zzx28syzJzI7B6tdtm8ZYHssDa/0/5/9asdu G1mi2AGxA2EHxg4GOzCWgBWAzMToQRNawfCcYeh3jGSYihNpwn7hOIJHoRQgsFIdODFTPlKjD8iu ut0FNCGCwpU9x/bcul1V6Oq+zbN74XlAtUxCbjxPJZXOmGz4iOPxPOt1mfkg0TqKA2bxTnC7xcHU Hzbiq4Op9xGx5dZugJJYbvA8dhg8Tw2H7faxfcuIpytKXsjn2X33PBWTEQjZa5DL0yljsuEjjsnz rNdVHoBUX+FmRt4tbhYXaZqONz8Xi79/Olf/4/JRPU3nV/euxXuLkdXGboScWO5EPE/RZA0JBs9T w+B5nqAoeSGfZ/fd85DdwSGVde5i6YzJho84Ls+zwRIs/or80GmcLlbXG8MzHo/Tp5/p1cqh+s/r afoivbE9F18divcYYMpbI5atRycY8QFKXJIzzyM9qcUYPE8Ng+d5gqLkhXyeTfS/gbcH7aQOhLf1 PPa5V1LpjMmGjyD3/0Rer0MoUOAzzg6fxoni5vzJjrz6kvNrZ+pqWlPe/Gz+dnHrTL3HACPVGiNi PbHnAfqKjnhjz+PGMAyep4YDu5LDqvfa8zToDfhYVLUdeB7fumKQOy0OsiGLhfWS+x9snpxZwSlU CBLuMI/Tw2rx6kae/7D5/WnlRP3nfMfwPDqrze8rJ+L9xgfjjm6BQl9v8Dx2GDxPDQUfMniet/I8 IVPtUlattPsNdrp9xWCj0eKRsFhRNqZ6uRUcY+aDlLdwMyXvDavLNH1xI/U/TL87UL+f1qxUzVm5 slQ9xg/Ddm6HTF/wVDyPL61DiMHz2EUd1vOMuJCMxReCzC+QEdoRT1dULkI+z5Z5Hu5ruXs9HN7z aL5NKu4Li4VLHKnnWZcRyHmLsqNETgmrT5opef6Z3rdW/z6tW6maoUoH0/PFsJvbgTi0gCEpyAyB vKJr6sTzRDzf0HI7DJ7HLipwkFMgz4iPILZYnz1PKGIbql22bmUjDyBYgacqobQrzwNOM24F9wBN 3yLvLJHTwaVuSl7MyfnPluKrKSO9tVmfnOTfXyR4M7dFpS1Y8mRFJQhG/lg9T2nouRUGz1OHf9BU I7F6wUecmOcB4kxvpFuRp5PTuuT5GZ0QOHL67XkmzAoHQA6y9ryP3SVyKrjiDM+jL5m3VP+dFd/+ 58pJBb3FCO7l1ljqK/JkRSV4rJ4nEaeV8Shb53PinieSxpSSbgP1ipaXbbE+e56YZxd0b6QRPJ2c VtD6jE5IEiEUB97XleeROLanevmtnzM5mQFNT9BY9r3ijnc8jz+qlboaY/VbN0X0E9/QTnYA4iXC k2dUhmjWFF1UJ55HbjJkhQyep46IjynIAJFVljfv/XgeeW9AOiSfp5PTCjxATCck8TyBPdUg7crz SPsDP9kvTE4WmIG8D3henCguDJ5numohvjo3qF86q6OH+A3s4zCygo9mIdSXHLHkjMoQDPCbeh7g xUj+4HmMi4JlxeWhy6i9+uB5tljSvQlF+vI7XcpHt7VWQsRSSUOFjEBGZyMeO/7A9OkFwCfb6xBP VLpsBBIvmFoH0PhqMCXj9LqF+l9G9RtnlfQPDvZxDCQ8rxQsSR5ZSF7RGXXiecAiE3oFWSGD56kD 3C0zMkDkeQA5o/MB7Sa+fp89j7w3fIAbDxOw/IBOSDIbEUslnm/4Rca0Rzx2fEpeJa23jedBI0XQ BwDMja5k2kL93CQ+njurpHeowDZmHhEa0FPH83KNH8lWDOWT1onnAa2jFylAIe3zOXHPI3aYIEDf ZuDTZHQ+E1FEnz0P6A3deTAYCRnA8+lBAuWSfPRs0rgTmTjIxZnnSXi+kjZoNyfwbSnlUJrIAAYP RsszTm8aq9+YxdMHh9X0C0swf/QBpQPd5ZTKhCcXujpyZW/qedY+yx9Jsxo8j2nZko+hP6n9Wxcn xWyYiI/IRclkMnElTIbi82yy/zyb7g3Y51Sx8gVANwsyIGT5I4m4IqR9nu7M84CUZmRAZJsT+FRU seB1S9EHsFBmVzJeNFZfmMXHXx1W0y9MwPzltiI+EPFGGl02wsiVva3nifiAbxQfnBhUVoPn2cFI GDTh+bHODllyIK5C6exeex5e3SdbI74ZeT49rTkfsGy9ADhvJroyfO9lZDLysQPHGf0uBbOibJUV oSukD2DxX7MrST81Vr9MjeJpc0fVd5yB+SttRWIgQjy9wOSEuvgEaSs6oW48D8gsF/Kpo2vwPDuI hPUBfqazE55dUeroriMCeu15Ep5euuCLPQ9oftaaD84Owv4Cf+fO81SilNbrH2CB0lY5lyWimFoH kPhktjzjcWN1C0M1nrsrpl8owXQwD1wCcPC9mWTRUhMPkLaiE+rG84DnZkzxQ56v9Uiez6l7HlDf RLiM0tlgCy+F2fgyekbQI1Huh/Y8YJ+TvQlZOnOk8PrMtAoDZB+XJxPN5Gv13HkedAYWBH3J0/c3 pzB5nq63ZgCAhSlJ04eG4vdW6k7r6RGQW0msVdCjlzqGJIuCA5eftG48D/JulU4HjySyjrf1PEQB jgAWRZ4HHOTUVQroVG1gC5OOKuD51A7rtecBvaHOCDAW5FOggecB9VYEPRbRQ0E64KDxHHqehA+g dieg738BmTA4wohGDuBhYUrG6V1D8Vsrdaf19AgxGL/cXsYHMsQcR4Cs9rghlN5nP6Ebz4Muvlxn I/fmIB/HnoeNaQ2wKPI8QsuY8OyQkh+xdMpRobsuI/i99jzC3oB9PiPl5Z4HtHNJ0H1R/gkvrqlH gOvQ84CWUhXw30v7AjyTaj2/7X2m1AE0rFzJbUPxu8HzAIDp8Ep7mRjIEMcQOLK8oNqhzqDyG3ue hI8409kBz95/fTXJx7Hnydk+tQVYFHke5H4TjVz5PJt6waItrHR2BEooCPV+e56E5y9FyZSkvNzz gPkmRgk4BmoroCPH361gCaieQ89TgohcUq+2OSNRKvw25o/VARSsXMkdHfv9+vN8Pv99/vn6nibc Dp6HB3quUg8IDtiZaAcLWtYLqxoTje8WSlyXS8+Djry8FblBPo49j24jXAEsCj0P2mRqn4xsdUGp g52mtwJtYPLF22/PA3qjjxPoDXekSOT/BXirlZLuLAltZDC8sC5f+Ijq0PMgv683NRCQY55L9IZP gyt1AA0rV7IiAh+uzrf/64lxfvVAcWzExweu8FiBbgbJpVcAHc/7oPEhPVAvPOylvLf2PGufD/Gr XWoVgDIqSvxtPc+oZBvVEmBR6HnQJgv3uT7PpW/eH0Bd7XPPAJmcm357nh8gQNMHuWiPH2M+3LQm fEi8zwWPDdKfIs+wCclfeDMfET2XnmciWAUdmtoXABtTZPUVk/gAGvNmruTnIt0jpQvCGdmIzw9e 43HiAxiPXCLkAyHi6RVDuhfO1IakZgGm8aPWledJUBGVdcnaMd0knyaeJwRJ8Y1qCbAm9Dww290D ukLUjFb/yEcE1a46GhtvSYn32/MIegNv3IJWb+B5gI/ZP7hKn6fS77oJEN9WPFPV9nQKMc1z6XlK tMxuweDsI05i1MhijwuebbR5HMBiaWFLdFdyMyVo5/9ovEsL9UUXZR4ffqDxqCRKMVLS7VMO6QIo Op2uPA86MbwPtQ5WsbyKDjxPhLIK2bB2QGtCz4Ou053rqwwRs6TV0Z6sf0qDekCK99zz/Akidr09 6mJIizfxPOsArJPXifBrkc1cFyBChExcL7v/URU7BecjQNTbiczUWbVDRQ8J2jwOYPHVwpX8tR90 lVI8wr5cmcXHX7sp9NjwBWx49oAiAa8jfSIqH/Ltoeh0uvI88Oj1/PyZBg8i5prswvNMUFqbq36S LZVa/mdSsE1rALgkCqxg5ItFqzLYbfZ49kHQ66c0qDNbpeeeZ30GQvy8eqYVMeDpTx9zPuy0goZu PnH5TKsyH/CYwcNTLUAmrpfd//iVGD8XXMaQp3ThEND9Ol8hZsHlPYDGysLz3O/FLEjLs/3HP/aY 92bxdNVVqceFBOziiUgJP41GopUlUHQ6nXkefBR5oyTbIMa3JHcXdOB5DOmbBZoArgQjE5zl2WO3 I0MtJacOr1HP/7gVz34xqPuVVDsj6KAGRckL+SD/Zr2JHnuThJhFtwbmw04r9r9eOMks9kLOiOMX nD0ycb38/vfxUh+25U5CTKK6OYER4az4t93LSCo8AGJhdCW/6hEkcftrsce9MKrvR7wXoNfbUibl w9Ep9uklpNtD0dl05nkqv30NXEYdeB7bz8AKNAFcCUa62DUJq+7iW3I7pe+ex0VvmNbAfPhpTdon FHDaTnaC59bzYNdpB0Xo4tfqFqMoMjza3J4P7wN3JlMy/ns3QKUpy01TtUv+n9Hz3HVW6VHhG9rG lUwrhjMx0/iJcdSsoOhsOvM867x9DYWbfJp4nnVolyIv0ABwJRya2KUL4Fe8uoNrhZPvu+dx0BvQ eT6In1YHviRnxV0YDM+t56mC1unEpHDYWhd8pQEsPhtcycUu/WGajlnuxvXc77BXFwb1eYeVHhN+ A9s4FGrNhNNW+vYjBaiKzqY7z9P+yJg4yqeR58GfzUKgAeBKOLT9RZcj9aCtOivfe8+zDnDhZoDO 80HgNm3tS/7PfhkjN8psUVg7GHbwswR28FgCS2AFWNkowxOOE6qewnEVyVNqsnHYE44iqhxaAcHv VEM0SvWQPPZY0Pd20y3TGJ3PLhLdPpxumstppsX12WcB/VPae77M/rc+XRGh0/7U5lW0a0CxXfCx ZHNavuRqm/9WiHngI0+yHW6ioyJk9jH5JSYo+deiO0C/ZcWMTyF3M2DmYW6lBd15h8g8ms2dFjCA vZNirGZGI5EfdV8oLNXpzfXxM4/tPmfSi1nmsT5tlIy2/k4IDdoGI8d4CrU9ySkIXd9Sd5YxpgHJ OuFSyd1p8SNbLMlIKzby/BhumuPC5AUh8djXQnQHhJpvlFczlRLdAwNmnv1ccx5yvPJcfowyj2b0 ZAT6w95JNTjU8ksR1Lx6bKXebFVK+ONnHtt9XjHLTg/jMo/imKUiZaT1d4JXDZZ5Kk9/chLIU6xt 0ufPEYCEiyU3rdpbNsIcLv89HbC7YepXg81xZLB7ve6rFvV94WpP75USvb8Aw2Yeu/NmSesOknn0 ngIj0B/2TqrBta/jl4AJmH/UAwt17mFOIPPY7fOcETbMPPvcxhCrrL8TcpO2wehxnqzCSUzrRja6 ynMEIFmRqeRmd1q5ZSPP1fGyPR3ChJ7b4aY4MubcRu6tlvEvhmRE6em8Unn/L8DAmcfmM5wzuoNk Hr1Gygn0hr2TcrTerpGijDz7fWWuzj7MKWQe3UOKDNkkdfzwySQ2N6T8UuvNdm7UNhhB1hPfY1m4 +VqdIxB5LFgRoWTZijx7wUae5HgRrTFk6FkNNLsR8g+zk+e91Ur+1ahlQzz1K5XvR595zD/DXs7J DpN5tD4crEBf2Duph5uvdqlhziJS5YzsFDKPxdrEnKx55jEPPV7FCx9mG+jNa8DMYz7fqGan6pnq IvLYsV7IUsldp+4bG3mujpdv7UG7O2n1j0FmNkoqbiuL/nr8e5NLLQSqV+o4LOztc9jMsy991Tyk KD7CA2WefaR2ygv0hL2TxnjDDh1UWu5M+79XcKqTyDzGa6OIPOaZxzQEaO2FOlTJZIeyITPPft57 qkdUD8A49CDyWLK97WSS5aZb9oWNPMnx/7o7bHPdqV1u339SoyXn9rKBXmTw1tVzdpBfHqtCukLI vQycefZ1wM5DjqrxDpV56khplRfoCXsnHQGjiMmfdN+gDuIygpIVnUbmMdvn8vOOnh9V5uEWllGt VbI64p5QFaW956vc/2zXpsiUMzUMPXO9dQQMT6uTUHIrSTz7fcJGnqvnf9nA9VJD/WKImL2sbDUS Mvbt8IlRZUgO8dL6uYYuGUvmMTmAKTWHyjz7fazyqhLoBXsnLQWNlNbCy/T9KYK4lHnNa04k85js 8z8HFw56sLoRCa+3o1Qp+kIV0yovj3zYzGMQ+BVx/Jk67CvbPFmhtYhAwe6hWC6aQLJYrtY7eQkf eZ4TUSIf+0vc3hx/p9Uvhk/Mbk4N9Er+BSmpcSKW1r8mno+RefYi4OfftkGuh6kfi8yzLzzerVKg D+ydNDUKnzfcIq56ORT91GeBUClOJvP0XhtVGuT9aBy++kZUrQjwShV7MhHv744aOPPsa+aGMqeU iTaZdKIMOk8WnIWNIvM8/21c2xw7bEQRJor8O5PRA+s89lsvapy/+T3sbdRB5tnvc58WaJsgfNv4 sck8qg+Hjl1t2DvpitSpx+q8RWu1T+nxLGd+rtabTubptzZ6YZMer5F5mlbGTL+NzsNqUcTBqYYX 5fXfn4fOPIccpj3dNydHJbW+7Kz3MQLYsNWJPMmTa5tjhz0tGClG7DsS8YNrkadpeCBNi+r0t5BW FXI1J5mn+RoE7BK8LgXh2s6PVeZpGikXIvQMa8Kujb5Mnfms0h88w+as+SxnUaGjNqXMo7022ktP S2hlnib1xFqGZmGup9fh0J2iQ3eap0V5+tPwmeeQejyd6fpZ3WuWderrraM3r3oJA0s0Is9V4trk 6AmZLa3ZaVpk/HvyLlaFfISjzNM037nPr8Is1O5Dw2ae/eFAS5nXFNCDXZ5eSurVjvLa2KdavXmY lZ7WtDKP1tr0WHpaRLsT1XmkMhS8z4faRebRma8fC4PZFOoH2zxZA2FgAzLPhRHSr59w7a1LlccB 4TacF7VrezxVkcZh6D3b9ZpzbZymQtSubdGUWUQ0aS9Mha068ywb+bE/zPeFXxsxvKG6SMlOEcR5 9U63PcdRyQyRvrypbfwoK41lq3xOruOhhZ3NP9BmqRF5lq5NgvNBv4FjzDxHyiJNoyYy+Mc+0dBE h9K1qckimtU+LLI3ew5qUbPa1buoz4JjDszOKP+hGd3aVCI7pPYwOH76n/dCUb7nHd1lngMv8w1f 5hun+TnOKJXInzvYcws7KM/feSGnyu7x8V9bjTt15Lm6O4dZMA4+YOYBAFwEbjMPGDc/v31OmkCS fBVbG5mNOvIkDzY3+C2WyYGvq0cbGXAmkHkAAOMEmQdQ/Lx+SSRN7Fn9tlD6oow8XyzUd3eNvz86 yTVSj3uQeQAA4wSZB8j5/e0klCQLizDxXRV5ru7Nxf/9nBxS2avRO3MpcB6QeQAA44TJPJlrb8Ah u5urdjD5aS62UESezztj7XVyavQqufltLAbOAjIPAGCcMJlHuPYGHPI1STrRZGOsJvjIk/wwVn7s 2kxujdXAWUDmAQCMkzm6E5DwvQkPnWjy+bex3pKNPEtj3d2iK3eVfDfWA1L+Q3aJVFbOZJ5qWOMA gIlDd6dQVo4TGZCwTaTZZGUsKMsmr3+LnbHu/yRyydViaywIZNBtIpWV+3RXGdg4AGDi0N0JmQfo siLSiXmWeKJDz+LJWHUrE2xSj3k4AzLoNhHLyummgswDADgr836Zx0d3Ah12VD4R5ppk6LGIPHsh EzxomksCCVGvrlLSTUXahAAAwJSUbDeBrJzuTsg8l8tDQuSTpYXo9kYqebO10JRKHi6PFqKgQ7+u ktNNJR7cOgBg0mS9Qoygq3Eiu1y+yxNP82cley9RvLdSlJo8XOxkQYt+XSWmq7OhnQMApg2TYqpu NX1+m80Htw7GwoqKPImd7q+28OqXld5WavJwubMzCk5hukrRrfboajG0cwDAtKnofpN3q4Ne1eBC WFKR5+rRUnknbl/FbsXOUu1RavJwWVoqg1PoPhF3agu62HNgHQAwaT6RDSfs1DIBaVYNbx2MhFsq 8iTbM6g/PW7Wj5unMyhtpCYPl9UZ1MFf6MPRp7pdG9JNJRreOQBg2kQ9YkxM1wYOrIOR8J2KPFeu nZ2ySyQej5d719Ymxlw/xxR06Sx34R0AMGUyuuOErdKS6U6ZC+9gHKypyHPt2lmLhcTj8fLTtbOJ IbSDTOXRlZ9qF94BAFOmYrpTfFJZB0xp5cY9GANbKvOsXDtrsZKYPF5+uXY2NXymV6Rv6iquqcSU OgAAmMI2nfpvHRt5Imf2wQi4IULPo2tjLR6IbPbVtbHJkTLNYvZPXj9X1dknrq5yOQMAwDTJtbrT Pme7k3A4AeCctTxKLF376nAtN7p27WtyVFy3aAjjNE0jj62JXU8CADBBar7xzIJjd2ITzyx0PQng lhtplti4ttXhQepzfNns4xMrQo8ar3I9BwDAFEmtu9OsdD0H4JanhSRK3Ll2JWEl8bl4cu1qgqiO Umoy11MAAEyS2rftTnPXUwCuWXejxMq1Jxm7m67RtWtTkySzbCqh6wkAACaKsOxOQe16BsA5D4uP EHma0LNs+Vw8uLY0USKrpuLVrv0DAKbK3K47la79gxGwXZ4kiR+u/ZDcL94aXT659jNV6gBNBQAw Smy606xw7R6Mg83yJUlc3+9cm2HYrhYvRpdr12YmTOUh8gAAxojNkSx3bR6Mht3DfYN4cu1Dyeb+ frm6X29d+5g2pWlbQeQBALwrxqHHy11bBwCMkzo0aipB5do4AGDi1JFRd/JL18YBAKMlNWgqc9em AQAXQOb1705R7do1AGDElGHPnhII15YBABdB1bc7+cK1ZQDAyBF9+oqfu7YLALgY0J0AAOdGxJ5e T4kK11YBABdFqdudwty1VQDAB6Eu1I0lyirXNgEAl0cR+6ruFKA7AQB6UebzkAg+XpgK1/YAABdL xXanonbtDwDwMREiS9M4fCFNU/F/duuQMXEsjALoP34Z17jBNjaylRObOpBUglxkkItlVjV2F6aU lrSo8vGy7Tkv/t48kdx57k4Ah6/Tj+PX6cfPnw/zf3KXIrNuOp3ON9eN7NZdd9XEv+dXf0kAYEzm N6nYnXR7tQ2yuN/n7Z77xbUiu/r5JSdXSwQARuXpeQz8GQTtVRJXk/QaOVldJbN9jWz6qyQCAONS HcdAUVxl9DSv8+NPZnOFzPZNYvp1hUAAYGSmb8bA7unCA2/3O+ck8y48szuZWWkWHggAjExfFidz oI4ObFLx7jTRofVpXvkUHQgAjMwiDfbHNjZv9cHkKdIqNnQzfMllbB4AMDrNcH7MQ+P68sPNM+lD U+eDuNSExgEA41MP98csNG7x4eQp0jI0dTbMq0PjAIDxqYfrYxoaV53ZPLehqbNhnM0DAN/N3XB+ PEam9WcmT5H6yNjpMO4uMg0AGKH5cHxsItPWZzdPFxm7GabNI9MAgBHaDtbAJDRtkc6NnmVo7mSQ 9js0DQAYofZ0DqxCw2bFmcmTpqG5y9OwNjQMABijvno7B5rYsEU6M3mKZWxw8zar6mPDAIAx6qvX ORA8ef5dF2cmT+pig/v716xbkwcAvqW+PayBchkd9ZTOTJ4ifIfMy0PUg8kDAN9Vv2zrera6QlJ1 ZvLcxkf3y6au7+fb+CQA4Nubfzx50mPuYgAAF9SXH26emz53MQCAS5p/NHnSY+5aAACXVX8weerc pQAALqyv3k2eqs9dCgDg0vp6MHlqkwcA+Ipm5ZvFUz7mrgMAEKOfTQ6TZzLrc5cBAIizXc1ms9U2 dw0AAAAAAAAAAMjlaX5XTze5W4zSdtq0iz53CwDgElY3Rdppc/cYoXZ/Memmy90DAPi8LhWp2D1G zzvtn4vZLcJN7iYAwKfV+72zP2mbu8rIbNPL1dS5qwAAn7VNxz/7PHeXkZkWyRwEgK+iK46bZ5a7 y8jMUnG4mqLL3QUA+KTuOHkKm+fUtEiHq0nr3F0AgM8qXzZPWuWuMjLL4xpMuasAAJ82e/mxT3I3 GZt+8nI1be4qAMCn9dXzj73c5G4yOpvy+WqqPncTAOACpuXuv36/zV1jhDb1fgy2Jg8AfBGbde4G Y9WvXQ0AAAAAAAAAAAAAAADwv9d3fe4KAACxuuYmpXTTdLmLAACE6duUilTsntT2ucsAAAS5fd47 +5Oqp9xtAABCtKl4c9rcdQAAInQnk6dI69yFAAACNKeTp/iVuxAAQIDydPKkMnchAIAAg8lTpNyF AAACvNs869yNAAAu793myV0IACBAOZg8Ze5CAAABmsHmaXIXAgAIsBlsnnXuQgAAER5OJk+buw4A QIzqzeSpcpcBAIjycJw8D7mrAADEWTflbvCUzTp3EQCAYL9zFwAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAL6ibbfNXQEAINiqSilN/spdAwAg0iKlYn8W uYsAAMRZPy+e3elyVwEACHN33DxN7ioAAGHKl8lTTHJXAQAIk9Jx9OSuAgAQpjpunjp3FQCAMPPj 5nnMXQUAIExfHSZPlbsJAECgw+ip+txFAABCrdq6XeUuAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA /MceHBAAAAAACPn/uiEBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC2 EkCAAQDgbLAuCmVuZHN0cmVhbQ1lbmRvYmoNNjcgMCBvYmoNWyANL0luZGV4ZWQgOTEgMCBSIDI1 NSA2OCAwIFIgDV0NZW5kb2JqDTY4IDAgb2JqDTw8IC9MZW5ndGggNzgzIC9GaWx0ZXIgL0ZsYXRl RGVjb2RlID4+IA1zdHJlYW0NCkiJAAAD//wAAAABAQECAgIEBAQFBQUGBgYHBwcJCQkKCgoLCwsM DAwODg4PDw8QEBASEhITExMVFRUXFxcZGRkaGhobGxscHBwdHR0eHh4fHx8gICAhISEhISEiIiIj IyMkJCQlJSUmJiYnJycoKCgpKSkpKSkqKiorKyssLCwtLS0uLi4vLy8wMDAxMTEyMjIyMjIzMzM0 NDQ1NTU2NjY3Nzc4ODg5OTk6Ojo7Ozs8PDw9PT09PT0+Pj4/Pz9AQEBBQUFCQkJDQ0NERERFRUVG RkZHR0dISEhJSUlKSkpKSkpLS0tMTExNTU1OTk5PT09QUFBRUVFSUlJTU1NUVFRVVVVWVlZXV1dY WFhZWVlaWlpaWlpbW1tcXFxdXV1eXl5fX19gYGBhYWFiYmJjY2NkZGRlZWVmZmZnZ2doaGhpaWlq ampra2tsbGxtbW1ubm5vb29vb29wcHBxcXFycnJzc3N0dHR1dXV2dnZ3d3d4eHh5eXl6enp7e3t8 fHx9fX1+fn5/f3+AgICBgYGCgoKDg4OEhISFhYWGhoaHh4eIiIiJiYmKioqLi4uMjIyNjY2Ojo6P j4+QkJCRkZGSkpKTkpOUk5SUlJSVlZWWlpaXl5eYmJiZmZmampqbm5ucnJydnZ2enp6fn5+goKCh oaGioqKjo6OkpKSlpaWmpqanp6eoqKipqamqqqqrq6usrKytra2urq6vr6+wsLCxsbGysrKzs7O0 tLS1tbW2tra3t7e4uLi5ubm6urq7u7u8vLy9vb2+vr6/v7/AwMDBwcHCwsLDw8PExMTFxcXGxsbH x8fIyMjJycnKysrLy8vMzMzNzc3Ozs7Pz8/Q0NDR0dHS0tLT09PU1NTV1dXW1tbX19fY2NjZ2dna 2trb29vc3Nzd3d3e3t7f39/g4ODh4eHi4uLk4+Tl5OXm5ebn5+fo6Ojp6enq6urr6+vs7Ozt7e3u 7u7v7+/w8PDx8fHy8vLz8/P09PT19fX29vb39/f4+Pj5+fn6+vr7+/v8/Pz9/f3+/v7///8CDAAY ooHBCmVuZHN0cmVhbQ1lbmRvYmoNNjkgMCBvYmoNPDwgDS9UeXBlIC9Gb250IA0vU3VidHlwZSAv VHlwZTEgDS9FbmNvZGluZyAvV2luQW5zaUVuY29kaW5nIA0vQmFzZUZvbnQgL1RpbWVzLVJvbWFu IA0+PiANZW5kb2JqDTcwIDAgb2JqDTw8IA0vVHlwZSAvRm9udCANL1N1YnR5cGUgL1R5cGUxIA0v RW5jb2RpbmcgNzkgMCBSIA0vQmFzZUZvbnQgL1N5bWJvbCANL1RvVW5pY29kZSA4MCAwIFIgDT4+ IA1lbmRvYmoNNzEgMCBvYmoNPDwgDS9UeXBlIC9Gb250RGVzY3JpcHRvciANL0FzY2VudCAwIA0v Q2FwSGVpZ2h0IDAgDS9EZXNjZW50IDAgDS9GbGFncyA0IA0vRm9udEJCb3ggWyAtMjE3IC0xOTMg MTAxMyA5MTYgXSANL0ZvbnROYW1lIC9GRU1MQlArQWdpbGVudC5UVC5Db25kMDgzIA0vSXRhbGlj QW5nbGUgMCANL1N0ZW1WIDAgDS9DaGFyU2V0ICgvRzkxL0c1NC9HNzkvRzQyL0c5Mi9HNTUvRzgw L0cxNC9HNDMvRzY4L0c5My9HODEvRzQ0L0cxNS9HNjkvRzMvRzgyL0cxNi9cDUc3MC9HNTgvRzgz L0cxNy9HNzEvRzM0L0c3Mi9HMjMvRzg1L0cxOS9HNDgvRzczL0cxMzUvRzM2L0cyNC9HODYvRzIw L0c0OVwNL0c3NC9HMzcvRzYyL0cyNS9HODcvRzIxL0c1MC9HNzUvRzM4L0cyNi9HODgvRzUxL0cy Mi9HNzYvRzY0L0cyNy9HODkvRzc3XA0vRzExL0c0MC9HNjUvRzI4L0c5MC9HNTMvRzc4L0cxMi9H NDEpDS9Gb250RmlsZTMgNzIgMCBSIA0+PiANZW5kb2JqDTcyIDAgb2JqDTw8IC9GaWx0ZXIgL0Zs YXRlRGVjb2RlIC9MZW5ndGggNTg1OSAvU3VidHlwZSAvVHlwZTFDID4+IA1zdHJlYW0NCkiJbFNr VFNXFs6B5OZCkYcaLJf0UinaLKOZvEgujkOJgxxBECggAiJoeBiNBDW+UOqIy9EAcbT1MVLeqDwk mhFIVV6CIIJWEHEtOyxbdVh16tiOzrh6bjixM5eurln9MX/2Oufsffb+9vftDXh8Nx4AgIpaERe7 PEGqyzcYcwvMsuRk2e9NBTlyRjXjXc6mu7EZ7uw8Pt7g5f7ai8+u8/J3JbjeF5OUWJgkoGt5K0sO /qes7H8HLyG7zJdd58NO+p0QB6ydzVsOAF/oOct3jn+AOCh4gUQqk6s0Ycs+Wr5i5arVicmp6es3 5ORv3lq4Y9fe/QdKDh8tO3bi5JmKqtqGC82tl690XO3sefTcCcMUMFQNtWFQrYRhShgaChk5VKih WgU1DAxTQUYB1WqoCIWaMMjdlFChgVo5DGUgo4IKLdQqoIpLoIRKzhsKFVwmBmo5lyoUqjRQqYYM Z+VQHQa1aqjSQg0XylXRQiVXWg61XBgDlRrIMDBUAZVKqNVAjRoqtZDhvmihggMgh5pQqOTwcIVV UMtAhRKqFXGmApN5b2Fu1pKsX1jOSk7OmmH5/19/rQEP+Ljz3gHe7nM8gnkhbhK+3D3UbSkvnBcB In1SyCNcwGnQDQbAMXAS/BlUgnrQDOzgC1ALGsEl0A6ugxugGlwAraAN9IBBUAr+BD4DDaAF/AU4 QD84CspBBbCBa6AXHAenwFlQBWrAOXARXAFXQRe4CSzACj4FZ8DnoA40gcugA3SCPnAElIET4Dwv gpsiHp8n5Ol5PbxveD8CIfAHQ+A7t3Xus9zD3e/yD/HrBUeJcKKLcAo/JaVkDLnJw9ND6rHMo9Sj 3bPQ8+47qV5+Xou96r16vL7ycs267L3U+wufFT73fOf5fuun8EvwO+3X5zc+O2v2lTmhcwbmes/t FQlFGaKDoiGRy/83/h/7W3+K8nZFeOMf2RIXIcK+BETiFCTRIy2pR5rdSIYkFKKqUcglJKPtSNqH NBMonEQ+RBKen40XmrCULMSyYqzCDIUjKjDTiKV0E5Y6cPAgnkeWCbnc6JhYYBc9xOF9WGPnPtix rBKHYIrCkt1YpscaWo+1KVgCsZjkINxC8xwouBFJySYkPYsYFEEhphipTByEQiTNRgsT0XwS+RLe aPP0ODslcjEuLZawWpcWcZZlkMTFsAyWEN6uUpZAaSIscREC7EM8ZAkBSiNm2jWwCSIXgXxYgnvn 4lCXP9ahNKTDnJ81zwCeskyYBzNvZdgjK6Vk1eLP8BK8kcJ5h7B0/xJ6nzQvMjGDzEiIMMsDdTGV 9oSgxMv629ufkeann3CYj1Bo5ymkrJqiq/9mHx3oGrj+oOGZlePjmHj6jqi9tPVsfUt9U8XVU4Pk yYHjaD7aTqE1f0QBJQ/og+NFXdsad5w3ns23Gixbi03bTKY9qSU6suSjwzgQp1O49ixmGhLpC3Ht BTfLZzhmp/xxhJg/SsCkMP2i7SSSCJGH7fnA6GMSBYunDYT3tJf/22YkIxzDYy9/QAE/oHfHXznu OUabhqr7yKobp/pvUmPFfUY7bby8vmq1Nc6SuWtzFmnMLooP53QlUEDLP2/fpYdH/nERzbUiD8u3 W27HkcMxLXLsScUdiC2MobdF58I10CV3Od5l386Qzz5wlole73uYd3U1eW1VnQrzKey2eXHMb+mw Ve/nYEkg9pHZJmKD4sa3/AsFUZwwskrYkkW3ruvadbd8sLS73t5F2q9XjXxF/XvvA72DzumIrAsp J5FO+MgyXOPoIB1tdUOT1P19/cYO2tiWXZNsXWPZaDauJwsy98VHUpyUyL/l5e1RevjOcxvytSIf y99/wb0E+1C67dHpCXR6/Kqc8CLyiJDjiOV2IMJFYB1LuAxIQbDyabEAFf20AEVwHeHvkUTkPCDE L94eEKAX3Px96ewR1dltzY5zjrprZzqtyN0ymdv9O7Ir/BwW4sXUB39QmXX0Tl1uQpKe3JC8bHdw IPaQtY5FB0WPGl6iuRSKIHS18W3r6bbMXvOX5ROlPedbbpItvRVPkJBCbge+LrxJb+tf2xhhTbNk FemNGw1m/eF0EuuIc85+gVPGTfZB4hOX4fgRgTfqc9aIGm2tF9qr26qun7pR/l3po/yhKHIoshkL sJTCUiP2iA2jmVWqHMwPxOJFPW/igxLeGBFEJyl0vxltGnhND776vgcFWt9YHhcMJZHDiU0LsITi 6k0WDxe206b27NqPreGWNRtNWrKQKcYUjqOOESjpbYhobAZ8L9l6o+Ip8qVeFj009NKGnozzq60r LenbDOmkYW2RTkIFVy+1xdGXYjuNQz/PrnNSLCBEriDx9LSAU4zVigXuAtcdJHHuwQsIb6fAeVo0 uXMwuz2WbIup0WAehQMMH66IpCOjZAbsMUOobSIqCI5veoXE1JuaJ20jdNvwrbbJGvK4MLIq5Up+ P7mpb8/EFPXG9nT0Hj1654ntNdffN1tGY8jRaJsmhFIXReal0JuSU/SRZvKoEH84JrpmaTvX0kW2 dH8+8ldqpLjr583Iqcq0ZlhydxfkkVv1+1NjKF1lfEsG3ZxmN3fM7Paht9miW6WdDZe6SXtn5dgU 9WL/fUM3behMb4ixRlvSdmzJIDenFa1YTCmroy5l0Pa0zh2/kCBxvicy5Rm26Yv0ezMPplpllnBb /DgZf9+I+EhGIclF5DnyNX3n8bN2xLeiQMvL1NvB5ND8JgzxCQo/NOH8xIX0woRFafi9wA+WXhhY /V+iyz2o6SuL485ifrldHd2xhsUf6S/K2BGLA4tPkF1KFA0iCoJAEsozGAlEIgjyEhHqqolk3bZU EBEC8lA0CgQDQZCHIC8VdYqypVBZ7bpbH1XXnZPkJs7eH+1u/0rmd09OTs4938/5RhDcp3wHbjQI Ka8zQQ0JTEOCMauveFLT01r3PWr4rhScYSd9gsLiCt6YpvO8vgvpOysevabf5E4ljTB7hz+7tEMr UotVykikDM9dh+fRrhV+ejGjj/hf1UvHrSYeK7FbY8zA6A96+K32tXpSNbQbDYc1Cj3o9QXbVBIm TSL29/2DZGdKQnY6yk0riguhMR+7QhJ35vLU0ENmbMI0VGtCde1l3cP00OGOA5eY9MbECmlxmCYu MzUBqWIPBbixI2gWvDfxfqU03ObP7XbECRR2gJcc2wiF5+CXHBxPmVP4nFIO5FPmI4R9HNtccyEP F1niOMcpnGWP45gJD6HVMmjJ483k35abdqGO4JqNK+iteVsU/kzKZrF0azpK2+J3xMfZlktSmQ9Q INC8Sh7YgW5tv+CO59NrcrfKxYw8Ijxpcx4ZGFv4Ax40Un2aDt0VI2q6VjH4LT2e3SdvYeRNoZXC YoTVNgVld58LFdQz9V1dcytqMZy7+4yezBlNNDHy9m01biSo3FZOCvOynjB7kpWlwA4if0YoWq7A jDP2xNxrTzcJ/J/IIRV+ouHONYj47hHz/fh/msFRCzz1i9SxEDQWrF+PHRQnD+QR+LVQD/P6Fa2M ollaFagNVktzZIrEvWnigm3osOhzTOENNPb4Ei8s8WBKVtUFGBKuJfRnfcOCV2foQIa2qtFpuu5k tbqcAamSWnDH+rGjPZu0wxJLqFNuU5iLyGuRrdxsYGmfzOrYgWdT2ArNIeQgxO6IJWy0O3sKZ62m aV60ImD/2myUs3Zd4XpnzPGoNm4SbDZGPcqAOSgT5hTABkijIaAUBOefMzU/dj4YN423v6qBhVpY rH6nmoxEU+EGV+xC52BJHXcsszfGwMQ2hVRt1m5Rh+ZEK6L3qsIPC1Gh8Ch2YKnl/yV2Kl3DlHo2 BJkUHUl9uSPFHZqOU11lN05fKas8i8yBOJ21DHZfLpkm6wNHLLWIbOV20WzlILE9NZebn4J0doPa /MyU4y8Hsw8s+/mcmdknfMr2Ed/6lIN9KdtyPmcxh299QfYxiRzlQpo9t2KXzrveZUndsgqfkvC/ NjtVvTWAxyDEg8qS7ATPZrOZPfhzdY74ImW7xedwSAb8cwYT12zk46Uc/tzH4EciSaWt5resPwFK /fzgQ9mj2BsBDatQ/aoy7I6VNI47gldmrmQOrowR+kuQdKtLOl7m7LXtq0qpIOpszMV4Q4JBeSf7 Ocp5dgT4IKLh5FnYeOEJc+HvnaP32+4Znuhgzv+NilFzvaSruqvqaqduAlVPlMDvIZgG+VFwyb3P 5N1P7Yy/KKsPPeOvlaqjCqIzYtKV0hx/lCssJP6O1FJfi726IpnO8JG0iVliEGckJL+KZ46h4CPN m+Q7QWgs8JIrdiJISFohDGQ2BWKHFOyqRTjzlwEiBs7O40EWNUmWvsGIDIaq0Rn62+xBeSsjN4Tq fpZXIZmxXVbSQAlV0PQ5LDoGihMPjvYXGotaMy8qKpPPRXzNBkbZ3WeTmnXUO83UvqFQdsN5L6dd D/rGhDBxwSLZ6ix0jPuvgqHkxsjG3ZX4N19g71PY7avlpSGkfru7pY8HRdS4pr/GYEKG9qqRKXrs UDe7HfRRlUHkG/Q2li5LLV9Ydbx67nj2oKyNkbXt1m3SBqojc2RyJJNniXfQkadjdclMtaJxX3OG IX1432MZOIv+GXBjO+oW1X66nsaL5G6+2xiRL56jxMQ1uGCHqw99Bb7jstewiO4HCRfPr/YyfsYY pcMZk8UTmsFzxhbU1lTV+4C+nm2Ss1T55Db2nMbOY+vuyEZQ4nDW+GP6XevMxD1mYuytHj7UAqP+ d8q9HehekN6VuHEVlnCJylWzMqb4VjvbKQcbF1ytvqyGLc/51Bv28mzzWKxisjo8LdlK+EQBAUkg X3KMUuG1ciyKxmL8R3u+E86l8PtZEpPLPvA+uwH/rnJtScSSv1C14PUP8wZYbfFGUEbBx3a3aZv3 eeyPTlGl/Xpw64MTEGrpdILgWa+eTBTGw1F8y0uQ4oX4FZ/7AfzE2nZyFsL+bXkJC4kmqQ+xmLx5 wTcDwZF1Prkq7EnhcnMUIWsUB6+mzHPMdg5wKHhvs3MWWPoc8Qbqz8MnjLX0MSo9Zo9cvA+lisPy dzr7bK/tjhXE3Mj45kca5hmeTNxkJnrBpR5kWgQCbmv+ZVUNo9LFlYUV79LEHEnNRKkZeSnxdC4W cvu/vn6uhalsunKhp7a35n7pD1oi1Jm0/gg0EF7/Jxfa1szdeVIST5/kYj8AHjBc7KFZkRWdiKJl B8ID6eDTsdUqRqfU57drB9W9dV29qLOnduAu+cCCVvibxcqrN13SG2oNNabSHtZxTsf3bETdG+rw YuxH49D9mB/mw3jtXp+IKWdJ3OmaPYI9NSlN6V3oOJ6HWfWB0DZwioIEeylvSG263HgXNd49y5on cMx/obzNpA6JL/sXf6qJTFYFIVXgoZV4CY0Xl7s2iJgLW28qplg6EN73wTgPf5C2NjyU2RXus3+Z M2ZWtEwRPz+ZAothEw2BjcAfmGEGH78ygoDtwnRajxh1R9Z5sybOT0f15F1LucQoG+UVsmIfTViy MggpA3NXYD6NnStcG4OYy4H9ykmtSd1U0dCEGq6eabtJ68GP9MEy6YjXUEXtx/Q6+jiVHpeSkpCF smTEOZG94tUyHSGInEwjVfjT8N/SqzWmjSuNqgoznlZamko1CddorK1Km6ZKG/WxhaZqoGlDmtaE RwyU4uLEZGBgYMAuL1NCHkqLX20SpVAXQh0etuPgMNTEwZDwDIYkrLpNN9FuKKGrPlbqI7tUqa7H 10a94yRStVqpK638817P/e4595zvO1lOCKaX6Okvl85A0nbeNNwi1Ar6U+W9b1EfhKawHWH3jZA3 5ddMEx7XPOWcs/8bKgBc3fxj1RWa/8vOgVSrylzEctsobltTsgQF6FjvVtGe18e4KzEoxFuwR37w sdL0TOy3qmf1aJUCKTEU25XbMRRroQrATR744Mw3dPDrn4dhsu0fps+Nk+wU61P3bqH60j9C96IU EPmWRA90PSXk0YJ6suaK7ZJp1O51e13Hhw/P4MyDQwk8J4fKhq91Ar1H0HSprenmXJ7Jp5jchhce AqEU3KLiQ74k4hk5b+YtldYSM3+gsYkyNu6vxLnVGNNOYfQWShNv4enrfrgO7pVdPx4UBmlBCH6y YJswBSw+bLyrY/i2yR8/mNaophvUZcXqcorN29T0R8X6F7t82cocX9V1eB8eLKvJXe1sL0/zPUN7 xxX+T492eZXe45/0HRugjp1+f2wewJT3ZHwey+j0lEFXtA+Tc09qjJzFGggkXJ5xwvjpG/T0AiZn lW3SNNIi6Af1HuaEBnvBhMXnxqeslGB+YJz8mhkTNEc5Z+3LMBnAPxj/yV6k2dlCT4b1ZXMBW51B 8VubHkFrcCvp3ODNp73q8zWXYl3nz+KtBJQu5pKzzSNVJ+lKT3FHFhZuUX0ZQ5XqGtUZILrxrjJG ZK6h/v6z3X7HWNuM7SfTF3xATY3s7N78JIiqZfGRGng1Ke57aQzFPzyUJqGEOJQe5nGdPPlflsPh uKdNGYc09CHNO6W8oUbPG7lD1EFO25qrgKtlZ/uOfNyubLcf9oyCpLgG/BVJW/Bqwp16pNcZupyA NpORNaHL8F+hFAK7IhIjL4UeiEwSsQN/uzc+NJ1EqKTbNpP+d05zTprrKfuoxKo2axoYhmL21DMs iDJ4nZGNtfh4D13jZDo01jxzccNultpTVq/JBtHX8Pp3soAlMCzFB8ZSshu4AoIXx9zu8fY525xp rNln8FW7mM5C17nTwogz0HPBPm+7YZrX+zXU8JuOjGdBtPZ/BR4f5bWcDUpHFVpK68BdAs61BW03 /5OAodCBcCt2TdPyrosp1NyfPGgNehmgrAoEXn2B3rR9HYPWYvkl+5ZUyswbHB5YtgGY6YVrLy/Q 89dv+mCipNmV5LgvTRfc3jnKO9OxDB8B8KGmZXaWZoM5nuetz5lzuIosqnyHMRkPm2hd58OeHLo/ a6ZqKaZ37MLhVXKn/1S/r9vnCLT9vgsXt3WXKJnbLtwaaZb4UuHb6i2MBsSc2PF/ODFaDH0jLsiN 1jpbtS3HVLyPq+fq6ssPvEXt17yXiu4FkX0Y4zHZKctnyzhIksdKj2rfL6SOkD3+fkFwD7qC9gUs +3RZfDb+kFY+1eLnT9K8U9eRb9Way/caDJRB31LFAr6t5kQt7TBMlC/tnmfHqn31vjov5y6hTurs hVlgQ+WWzAK6QDJAQvFwavcoJmCUu7QAgjBNJ9N0lniraC93pvkcdpohu8dNeVz2AT8QWrzVp2iD 57Uzj45nu4p6yrrYTq5X308Z+vcOnQeLJy9PjdIXAou9v+Bw+beayUJqMq9vywYpEEpXn9fLX1FZ dirS1V2DFUpOMOJYJa6+Y3BvyGDaVQL13PG+dTLmw+pOI328se/ggM1jGrSM3Pa72++UDFgGegCv YSu0tVSdtmB/lmL9lt6pIqVm4u1rPwCxAKZH82SqjzWuStpdOdwUtM2YRrtOD1IDQufkVSBuxOvf ycosRTulXp0qhsO8HH22whOII9FfwzwBt5HwyApJHCZhe5gk4BNYro1YutI+h7SvlkQ6cYVoJfUI bEVn0aqImkJ6ErmlP79Cwl3RRVggLhIQP5r86FewVPyKgJvJ+Gyp/jQSHQ0loncjiQS2C/j3kANN hnQEfJGEDQhCHkJCaiMXIk1wMdJK3C1wMfrL86giFz3HIzIRPUFi4KbQ9bCPgHtIyETbpGr7xTYC Porx3ihqE6JPwkQy1CLyRISB94sJKJ6MmKNGImSOzWSHQ7Nii1xVm6/dRWu1+XU7FJkFDr9WqfXX jl8EsyfG/X7af2bcMWudNY/X+bWUX3sif4eEWDjvwfBW+a+/kwhyCmVuZHN0cmVhbQ1lbmRvYmoN NzMgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvSW1hZ2VCIF0gDT4+IA1lbmRvYmoNNzQgMCBv YmoNPDwgDS9UeXBlIC9FbmNvZGluZyANL0RpZmZlcmVuY2VzIFsgNzUgL2M3NSA4NyAvYzg3IF0g DT4+IA1lbmRvYmoNNzUgMCBvYmoNPDwgDS9jNzUgNzYgMCBSIA0vYzg3IDc3IDAgUiANPj4gDWVu ZG9iag03NiAwIG9iag08PCAvTGVuZ3RoIDEwNiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3Ry ZWFtDQpIiTKyVDBQMAFiI1MFE0OFFEOuQi4jQyDfAMQFSSTncjl5cumHKxgZcul7AEW59J0CnBWA lKevQklRaSqXpwsX+///RKGG/2wM9iwM8gwM/Ax/+Bn/sTP/Z2enDHG5enIFcgEEGAATfUSFDWVu ZHN0cmVhbQ1lbmRvYmoNNzcgMCBvYmoNPDwgL0xlbmd0aCAxMDAgL0ZpbHRlciAvRmxhdGVEZWNv ZGUgPj4gDXN0cmVhbQ0KSIkyNFcwUDAEYTMFY1OFFEOuQi5DUyDfAMQFSSTncjl5cumHKxiacul7 AEW59J0CnBUMufQ9fRVKikpTuTxduP7X/6j/AIcMjBD4oZ4oaP+B8QcQ/mPkcvXkCuQCCDAA3pw5 lA1lbmRzdHJlYW0NZW5kb2JqDTc4IDAgb2JqDTw8IA0vVHlwZSAvRW5jb2RpbmcgDS9EaWZmZXJl bmNlcyBbIDEgL0cyNCAvRzE5IC9HNDQgL0c0MCAvRzU1IC9HNDEgL0czIC9HNTEgL0c1NCAvRzU4 IC9HNDIgL0cyMSAvRzQ4IA0vRzY4IC9HODUgL0c3MCAvRzc1IC9HMjAgL0c3NCAvRzcyIC9HMTM1 IC9HMjIgL0cyMyAvRzI1IC9HMjYgL0cyNyANL0cyOCAvRzQ5IC9HODggL0c4MCAvRzY5IC9HODIg L0c3MyAvRzc2IC9HODEgL0c3OSAvRzg3IC9HODYgL0czNyANL0c5MiAvRzcxIC9HMzggL0c1MyAv RzM2IC9HMTUgL0cxMSAvRzkxIC9HMTIgL0c5MCAvRzYyIC9HNjQgL0c4MyANL0c4OSAvRzY1IC9H MTQgL0cxNiAvRzE3IC9HOTMgL0c1MCAvRzc3IC9HNzggL0c0MyAvRzM0IF0gDT4+IA1lbmRvYmoN NzkgMCBvYmoNPDwgDS9UeXBlIC9FbmNvZGluZyANL0RpZmZlcmVuY2VzIFsgMSAvbWludXMgL2Nv bW1hIC9kb3RtYXRoIC9jb2xvbiAvZXF1YWwgL2V4Y2xhbSAvcGFyZW5sZWZ0dHAgL3BhcmVubGVm dGV4IA0vcGFyZW5sZWZ0YnQgL3BhcmVucmlnaHR0cCAvcGFyZW5yaWdodGV4IC9wYXJlbnJpZ2h0 YnQgL2JyYWNrZXRsZWZ0dHAgDS9icmFja2V0bGVmdGJ0IC9icmFja2V0cmlnaHR0cCAvYnJhY2tl dHJpZ2h0YnQgXSANPj4gDWVuZG9iag04MCAwIG9iag08PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAv TGVuZ3RoIDI5OSA+PiANc3RyZWFtDQpIiVSRT2+DMAzF73wKHzvtQAJrt0oIaaOt1MP+aGy7Q2IQ UglRoId+++UlXaUdeHJ+thP8nFbH3dEMC6UfblI1L9QNRjuep7NTTC33gyGZkR7Ucj0FVWNjKfXN 9WVeeDyabqKiSNJPn5wXd6FVfRnb6XQv7ih9d5rdYHpafcnvHw/qs7UnHtksJKgsSXOXpNVrY9+a kX362hq4vD44aZ5to9g1pmcqhCypkKIkNvp/LvG/G1raLp5jbZAsk1npQQYQRGQVQI44DxXVGuAB IIjInwHWiNcR7AA2iDfxDgnwiHjr5fC0fwFoAFQAhw2ABtCxYg/AABwrBEAH0EWw9QATRvHAX+qn /RsLg2MLN9/U2TlvaVhVcBP+DYZv27SThV34kl8BBgB2+pCTCmVuZHN0cmVhbQ1lbmRvYmoNODEg MCBvYmoNPDwgDS9UeXBlIC9QYWdlcyANL0tpZHMgWyA4OCAwIFIgMSAwIFIgNCAwIFIgNyAwIFIg MTAgMCBSIDEzIDAgUiAxNiAwIFIgMTkgMCBSIDIyIDAgUiAyNSAwIFIgDV0gDS9Db3VudCAxMCAN L1BhcmVudCA4MiAwIFIgDT4+IA1lbmRvYmoNODIgMCBvYmoNPDwgDS9UeXBlIC9QYWdlcyANL0tp ZHMgWyA4MSAwIFIgODMgMCBSIDg0IDAgUiBdIA0vQ291bnQgMjIgDT4+IA1lbmRvYmoNODMgMCBv YmoNPDwgDS9UeXBlIC9QYWdlcyANL0tpZHMgWyAyOCAwIFIgMzEgMCBSIDM0IDAgUiAzNyAwIFIg NDAgMCBSIDQzIDAgUiA0NiAwIFIgNDkgMCBSIDUyIDAgUiA1NSAwIFIgDV0gDS9Db3VudCAxMCAN L1BhcmVudCA4MiAwIFIgDT4+IA1lbmRvYmoNODQgMCBvYmoNPDwgDS9UeXBlIC9QYWdlcyANL0tp ZHMgWyA1OCAwIFIgNjEgMCBSIF0gDS9Db3VudCAyIA0vUGFyZW50IDgyIDAgUiANPj4gDWVuZG9i ag04NSAwIG9iag08PCANL0NyZWF0aW9uRGF0ZSAoRDoyMDAxMDMyMjA5MjAyOCkNL1Byb2R1Y2Vy IChBY3JvYmF0IERpc3RpbGxlciA0LjA1IGZvciBXaW5kb3dzKQ0vTW9kRGF0ZSAoRDoyMDAxMDMy MjA5MjAyOC0wOCcwMCcpDT4+IA1lbmRvYmoNeHJlZg0wIDg2IA0wMDAwMDAwMDAwIDY1NTM1IGYN CjAwMDAwMjM1ODAgMDAwMDAgbg0KMDAwMDAyMzczMiAwMDAwMCBuDQowMDAwMDIzOTcwIDAwMDAw IG4NCjAwMDAwMjQ2NTAgMDAwMDAgbg0KMDAwMDAyNDgwMiAwMDAwMCBuDQowMDAwMDI1MDQwIDAw MDAwIG4NCjAwMDAwMjYwNTEgMDAwMDAgbg0KMDAwMDAyNjIwMyAwMDAwMCBuDQowMDAwMDI2NDQx IDAwMDAwIG4NCjAwMDAwMjcxMzIgMDAwMDAgbg0KMDAwMDAyNzI4NyAwMDAwMCBuDQowMDAwMDI3 NTI2IDAwMDAwIG4NCjAwMDAwMjgyMzQgMDAwMDAgbg0KMDAwMDAyODM4OSAwMDAwMCBuDQowMDAw MDI4NjUwIDAwMDAwIG4NCjAwMDAwMzIxNDQgMDAwMDAgbg0KMDAwMDAzMjI5OSAwMDAwMCBuDQow MDAwMDMyNTYwIDAwMDAwIG4NCjAwMDAwMzQxODMgMDAwMDAgbg0KMDAwMDAzNDMzOCAwMDAwMCBu DQowMDAwMDM0NTk5IDAwMDAwIG4NCjAwMDAwMzk5MTkgMDAwMDAgbg0KMDAwMDA0MDA3NCAwMDAw MCBuDQowMDAwMDQwMzEzIDAwMDAwIG4NCjAwMDAwNDExNDMgMDAwMDAgbg0KMDAwMDA0MTI5OCAw MDAwMCBuDQowMDAwMDQxNTM3IDAwMDAwIG4NCjAwMDAwNDIxNDUgMDAwMDAgbg0KMDAwMDA0MjMw MCAwMDAwMCBuDQowMDAwMDQyNTM5IDAwMDAwIG4NCjAwMDAwNDMwOTMgMDAwMDAgbg0KMDAwMDA0 MzI0OCAwMDAwMCBuDQowMDAwMDQzNDg3IDAwMDAwIG4NCjAwMDAwNDU1NzEgMDAwMDAgbg0KMDAw MDA0NTcyNiAwMDAwMCBuDQowMDAwMDQ1OTY1IDAwMDAwIG4NCjAwMDAwNDc1MTQgMDAwMDAgbg0K MDAwMDA0NzY2OSAwMDAwMCBuDQowMDAwMDQ3OTA4IDAwMDAwIG4NCjAwMDAwNDg0NzcgMDAwMDAg bg0KMDAwMDA0ODYzMiAwMDAwMCBuDQowMDAwMDQ4ODcxIDAwMDAwIG4NCjAwMDAwNDk0OTIgMDAw MDAgbg0KMDAwMDA0OTY0NyAwMDAwMCBuDQowMDAwMDQ5ODg2IDAwMDAwIG4NCjAwMDAwNTA2Nzkg MDAwMDAgbg0KMDAwMDA1MDgzNCAwMDAwMCBuDQowMDAwMDUxMDczIDAwMDAwIG4NCjAwMDAwNTE4 MDUgMDAwMDAgbg0KMDAwMDA1MTk2MCAwMDAwMCBuDQowMDAwMDUyMTk5IDAwMDAwIG4NCjAwMDAw NTMwMjEgMDAwMDAgbg0KMDAwMDA1MzE3NiAwMDAwMCBuDQowMDAwMDUzNDE1IDAwMDAwIG4NCjAw MDAwNTQwNjMgMDAwMDAgbg0KMDAwMDA1NDIxOCAwMDAwMCBuDQowMDAwMDU0NDU3IDAwMDAwIG4N CjAwMDAwNTUwNTEgMDAwMDAgbg0KMDAwMDA1NTIwNiAwMDAwMCBuDQowMDAwMDU1NDQ1IDAwMDAw IG4NCjAwMDAwNTYxNjEgMDAwMDAgbg0KMDAwMDA1NjMxNiAwMDAwMCBuDQowMDAwMDU2NTU1IDAw MDAwIG4NCjAwMDAwNTczOTYgMDAwMDAgbg0KMDAwMDA1NzgyNiAwMDAwMCBuDQowMDAwMDU4MDgz IDAwMDAwIG4NCjAwMDAwNzcyNjIgMDAwMDAgbg0KMDAwMDA3NzMxMSAwMDAwMCBuDQowMDAwMDc4 MTY4IDAwMDAwIG4NCjAwMDAwNzgyNzQgMDAwMDAgbg0KMDAwMDA3ODM4NCAwMDAwMCBuDQowMDAw MDc4ODY1IDAwMDAwIG4NCjAwMDAwODQ4MTYgMDAwMDAgbg0KMDAwMDA4NDg2NyAwMDAwMCBuDQow MDAwMDg0OTQyIDAwMDAwIG4NCjAwMDAwODQ5OTIgMDAwMDAgbg0KMDAwMDA4NTE3MyAwMDAwMCBu DQowMDAwMDg1MzQ4IDAwMDAwIG4NCjAwMDAwODU3MjggMDAwMDAgbg0KMDAwMDA4NTk3OSAwMDAw MCBuDQowMDAwMDg2MzUyIDAwMDAwIG4NCjAwMDAwODY0OTYgMDAwMDAgbg0KMDAwMDA4NjU3NyAw MDAwMCBuDQowMDAwMDg2NzI0IDAwMDAwIG4NCjAwMDAwODY4MTMgMDAwMDAgbg0KdHJhaWxlcg08 PA0vU2l6ZSA4Ng0vSURbPGNiYWRiZjAyZjI4NmQxNjRkNzg3YjkwY2I0NzFjNTlmPjxjYmFkYmYw MmYyODZkMTY0ZDc4N2I5MGNiNDcxYzU5Zj5dDT4+DXN0YXJ0eHJlZg0xNzMNJSVFT0YN ------_=_NextPart_000_01C0B2F8.CAFB3C30-- From owner-ips@ece.cmu.edu Thu Mar 22 17:26:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA23586 for ; Thu, 22 Mar 2001 17:26:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2MJahk27266 for ips-outgoing; Thu, 22 Mar 2001 14:36:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from chmls20.mediaone.net (chmls20.mediaone.net [24.147.1.156]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2MJaMr27238 for ; Thu, 22 Mar 2001 14:36:22 -0500 (EST) Received: from breinhold (p-39.ts-6.qnc.ma.ttlc.net [140.186.41.130]) by chmls20.mediaone.net (8.11.1/8.11.1) with SMTP id f2MJa4a10001; Thu, 22 Mar 2001 14:36:04 -0500 (EST) From: "Barry Reinhold" To: "Ayman Ghanem" , Subject: RE: foils Date: Thu, 22 Mar 2001 14:34:40 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Ayman, The foil was drawn showing how TCP reads might take place. Since AHSes are very rare the first TCP read would be 48 bytes + digest size (assuming digests were being used). That is the "first read" is done anticipating a AHS_length = 0. Since the foil is no longer part of a presentation, but a description of a format, the "first read" should now be removed. - Barry >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >Ayman Ghanem >Sent: Thursday, March 22, 2001 11:07 AM >To: ips@ece.cmu.edu >Subject: RE: foils > > >Bullet 3 of slide 6 (new PDU format) says that first read is 48 bytes + >header digest size. I assume this is a typo as the header digest >comes after >the AHS. > >-Ayman > From owner-ips@ece.cmu.edu Thu Mar 22 18:26:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA25551 for ; Thu, 22 Mar 2001 18:26:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2ML1kZ03208 for ips-outgoing; Thu, 22 Mar 2001 16:01:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2ML0rr03136 for ; Thu, 22 Mar 2001 16:00:53 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id QAA17539; Thu, 22 Mar 2001 16:00:46 -0500 (EST) Message-ID: <3ABA67EB.77CFF4AF@cisco.com> Date: Thu, 22 Mar 2001 15:00:27 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: "CAVANNA,VICENTE V (A-Roseville,ex1)" CC: "'ips@ece.cmu.edu'" , "'ipsan@rtl.rose.agilent.com'" Subject: Re: CRC vs CHKSUM presentation slides References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Vicente- I just took another look through your slides after seeing the presentation on Monday. They were very well-done. I have one question, though. If the CCITT-CRC32 is considered "good enough", then would the Ethernet CRC32 also be good enough? The reason I ask is that every hardware vendor involved in building iSCSI stuff already has implementations of the Ethernet CRC, which is used for both Ethernet and Fibre Channel. The Ethernet poly has more terms than CCITT, and perhaps is not as good as CRC-32C (any thoughts?), but everyone has hardware and software for this, with proven interoperability (bit and byte order, etc). Performance-wise, it will be there for 10Gb Ethernet, so it should be fast enough. So if the Ethernet poly is deemed good enough (even if it's not the best), and fast enough (even if it's not the fastest), why not use it? I think we would stand a much better chance of achieving interoperability in a short time. Please let me know what you think of this; I realize that a few of my questions were speculative. Regards, Mark "CAVANNA,VICENTE V (A-Roseville,ex1)" wrote: > > Here are the slides I presented at the IETF-50 in Minneapolis. > > Vicente Cavanna > Agilent Technologies > > <> > > ---------------------------------------------------------------------------------------------------- > Name: CRCorChksm.pdf > CRCorChksm.pdf Type: Portable Document Format (application/pdf) > Encoding: base64 -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Thu Mar 22 19:05:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA26867 for ; Thu, 22 Mar 2001 19:05:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2MMNhZ09053 for ips-outgoing; Thu, 22 Mar 2001 17:23:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2MMNKr09039 for ; Thu, 22 Mar 2001 17:23:21 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id XAA111156; Thu, 22 Mar 2001 23:23:13 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id XAA134980; Thu, 22 Mar 2001 23:20:46 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A17.007AC1D9 ; Thu, 22 Mar 2001 23:20:50 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Mallikarjun C cc: ips@ece.cmu.edu Message-ID: Date: Fri, 23 Mar 2001 00:23:36 +0200 Subject: Re: Retry question Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Yes - but it is not mandatory to follow. It is meant as a hint. Julo Mallikarjun C on 22/03/2001 07:24:44 Please respond to Mallikarjun C To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Retry question Julian, A quick question. I interpret the current draft to mean the following: Retry bit doesn't mean retransferring the entire data always, it may mean partial data transfers. The EndDataSN in the command PDU is meant to help achieve this. Can you please confirm? Thanks! Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Hewlett-Packard, Roseville. __________________________________________________ Do You Yahoo!? Get email at your own domain with Yahoo! Mail. http://personal.mail.yahoo.com/ From owner-ips@ece.cmu.edu Thu Mar 22 19:06:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA26919 for ; Thu, 22 Mar 2001 19:06:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2MLhwY06176 for ips-outgoing; Thu, 22 Mar 2001 16:43:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.giganet.com ([216.41.107.5]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2MLXRr05434 for ; Thu, 22 Mar 2001 16:33:27 -0500 (EST) Received: from sachmo (gn14-113.giganet.com [172.16.14.113]) by mail.giganet.com (8.8.7/8.8.7) with SMTP id QAA20995 for ; Thu, 22 Mar 2001 16:31:35 -0500 Message-ID: <008f01c0b317$33a58880$710e10ac@giganet.com> From: "Jim Williams" To: References: <3ABA67EB.77CFF4AF@cisco.com> Subject: Re: CRC vs CHKSUM presentation slides Date: Thu, 22 Mar 2001 16:29:39 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6600 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit From: "Mark Bakke" To: "CAVANNA,VICENTE V (A-Roseville,ex1)" Cc: ; Sent: Thursday, March 22, 2001 4:00 PM Subject: Re: CRC vs CHKSUM presentation slides > Vicente- > > I just took another look through your slides after seeing the > presentation on Monday. They were very well-done. I have > one question, though. If the CCITT-CRC32 is considered "good > enough", then would the Ethernet CRC32 also be good enough? The > reason I ask is that every hardware vendor involved in building > iSCSI stuff already has implementations of the Ethernet CRC, > which is used for both Ethernet and Fibre Channel. > > The Ethernet poly has more terms than CCITT, and perhaps is > not as good as CRC-32C (any thoughts?), but everyone has hardware > and software for this, with proven interoperability (bit and > byte order, etc). Performance-wise, it will be there for > 10Gb Ethernet, so it should be fast enough. > > So if the Ethernet poly is deemed good enough (even if it's not > the best), and fast enough (even if it's not the fastest), why > not use it? I think we would stand a much better chance of > achieving interoperability in a short time. > > Please let me know what you think of this; I realize that a few > of my questions were speculative. Since the iSCSI messages will often be encapsulated in ethernet packets, there is some value to using a different CRC. Link errors are double protected with two different CRCs. If ethernet and iSCSI use the same polynomial, there is little additional coverage against link errors. This point may not be decisive, but all other things being equal or almost equal, it is worth considering. From owner-ips@ece.cmu.edu Thu Mar 22 19:44:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA27936 for ; Thu, 22 Mar 2001 19:43:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2MMmjj10784 for ips-outgoing; Thu, 22 Mar 2001 17:48:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2MMmEr10756 for ; Thu, 22 Mar 2001 17:48:14 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 42521E2C for ; Thu, 22 Mar 2001 14:48:09 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id OAA14436 for ; Thu, 22 Mar 2001 14:48:02 -0800 (PST) Message-ID: <3ABA8235.ED090F7@cup.hp.com> Date: Thu, 22 Mar 2001 14:52:37 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: Retry question References: Content-Type: multipart/mixed; boundary="------------003644D6431E81196172C9BE" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------003644D6431E81196172C9BE Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit It looks like Rev 05 does support partial data recovery and the target can send back either all the data or data starting from EndDataSN on a "retry" command. I thought the Orlando consensus was to perform error recovery at command granularity and not do partial data re-transmission (?). Quoting from David Black's mail (http://ips.pdl.cs.cmu.edu/mail/msg03221.html) : "- DataSN (formerly DataRN) functionality will be removed. Error recovery is now at the granularity of commands, not within a command." Regards, Santosh julian_satran@il.ibm.com wrote: > > Yes - but it is not mandatory to follow. It is meant as a hint. > > Julo > > Mallikarjun C on 22/03/2001 07:24:44 > > Please respond to Mallikarjun C > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: Retry question > > Julian, > > A quick question. > > I interpret the current draft to mean the following: > > Retry bit doesn't mean retransferring the entire data > always, it may mean partial data transfers. The > EndDataSN in the command PDU is meant to help achieve > this. > > Can you please confirm? > > Thanks! > > Mallikarjun > > Mallikarjun Chadalapaka > Networked Storage Architecture > Hewlett-Packard, Roseville. > > __________________________________________________ > Do You Yahoo!? > Get email at your own domain with Yahoo! Mail. > http://personal.mail.yahoo.com/ --------------003644D6431E81196172C9BE Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------003644D6431E81196172C9BE-- From owner-ips@ece.cmu.edu Thu Mar 22 19:44:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA27975 for ; Thu, 22 Mar 2001 19:44:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2MMhhN10417 for ips-outgoing; Thu, 22 Mar 2001 17:43:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2MMh0r10388 for ; Thu, 22 Mar 2001 17:43:01 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GAM00BCRF3G44@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Thu, 22 Mar 2001 14:42:52 -0800 (PST) Date: Thu, 22 Mar 2001 14:41:32 -0800 From: Douglas Otis Subject: iSCSI: Out Of Sequence due to null sequence with multiple connections. To: Ips Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit All, Out of Sequence caused by multiple connections for iSCSI: I attempted to discuss this issue at the IETF meetings but was tabled by David Black for the Wednesday meeting. At the beginning of the second meeting, I was then told I should have attempted to discuss this issue at the first meeting, and upon reminding him, I was then advised to take this topic to the reflector. So here it is and hopefully this will reach the reflector. If this becomes repeated, my apologies. Perhaps the inability of the iSCSI protocol to maintain sequence for immediate commands is not considered important as was already mentioned, it is seldom that it matters. The point in time it is likely to matter however is during these seldom events. Should a command appear to hang and the initiator attempts to bring the target into a known state, the likely method per iSCSI would be to issue an immediate command that carries nulled sequential information. The management command could arrive before other commands that it was attempting to cancel and as such fail to do so. Rather than bringing the target to a known state, the state becomes less known. Solutions- A) As suggested by David Black, deliver all messages to the target in sequence and depend on the task attributes to sort out any problem. This would remove the use of the immediate command at the iSCSI level and thus remove a need for the null CmdSN. B) If the iSCSI sequencer is hung as a result of an unresolved hole in the sequence or due to a resource limitation delivering to the target, use a flag within transport to indicate the need for immediate processing but assign CmdSN to the next sequential value. Commands normally before the immediate command pending a send to the target would then be rejected back to the initiator to allow the initiator to resolve their status. C) Place the immediate command on all connections and filter duplicates based on the initiator tag. From the duplicate packets, the relative position of this management command could be deduced. I favor B myself as it assures the acknowledgement of these immediate commands as well as eliminates the need for sending duplicates. David's solutions assumes there will not be a need to deal with the iSCSI sequencer in the event of a problem. Sending commands back to the initiator by means of rejection should allow a freeing of the iSCSI sequencer lock and ensures the initiator can track the state of the target. The present scheme does not allow management nor provide for condition created by multiple connections. Doug From owner-ips@ece.cmu.edu Thu Mar 22 19:50:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA28201 for ; Thu, 22 Mar 2001 19:50:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2MMjrU10578 for ips-outgoing; Thu, 22 Mar 2001 17:45:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2MMjRr10529 for ; Thu, 22 Mar 2001 17:45:27 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id E6B0A2D8 for ; Thu, 22 Mar 2001 15:45:26 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 2B6C4199 for ; Thu, 22 Mar 2001 17:45:26 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id OAA09708 for ; Thu, 22 Mar 2001 14:45:24 -0800 (PST) Message-ID: <3ABA80A2.CF1CE1C1@agilent.com> Date: Thu, 22 Mar 2001 14:45:54 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI: Target Resets are Management Functions References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit John, Are you saying that iSCSI allows the function, but does not specify how to use it? Or that target reset is to be removed from the list of management functions? John Hufferd wrote: > > Julian, > Target Resets are management functions, that is where they belong, not as > an iSCSI or SCSI action/command. It is not like this is going to be part > of some automatic error recovery function. Target Resets need and deserve > this function as part of administration management, it is up the vendors' > products to perform, or not, this function, with information they get (or > not) from an Admin interaction). I do not think we should support it in > the normal iSCSI Protocols. And we should not have to specify the problem > avoidance approaches that the implementer SHOULD/MUST take to support this > function. You will probably say that it s an implementation decision as to > how they avoid the problems, and that is what I am saying, and it belongs > to the vendors' Admin function to implement or not. > > If it is a big enough problem for FC to take it out, with their limited > network domain, we certainly should do that also. The reset function is still a management function in FCP-2 (table 3), and it's actions are described in FCP-2, tables 4 & 5 (and elsewhere). -Matt From owner-ips@ece.cmu.edu Thu Mar 22 20:28:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA29200 for ; Thu, 22 Mar 2001 20:28:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2N0AkW16114 for ips-outgoing; Thu, 22 Mar 2001 19:10:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2N0AFr16083 for ; Thu, 22 Mar 2001 19:10:15 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 3742A11F0; Thu, 22 Mar 2001 17:10:15 -0700 (MST) Received: from axcsbh2.cos.agilent.com (axcsbh2.cos.agilent.com [130.29.152.144]) by msgrel1.cos.agilent.com (Postfix) with SMTP id 1AF20146; Thu, 22 Mar 2001 17:10:15 -0700 (MST) Received: from 130.29.152.144 by axcsbh2.cos.agilent.com (InterScan E-Mail VirusWall NT); Thu, 22 Mar 2001 17:10:14 -0700 (Mountain Standard Time) Received: by axcsbh2.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Thu, 22 Mar 2001 17:10:14 -0700 Message-ID: From: vince_cavanna@agilent.com To: vince_cavanna@agilent.com, mbakke@cisco.com, jim.williams@emulex.com, ips@ece.cmu.edu Subject: RE: CRC vs CHKSUM presentation slides Date: Thu, 22 Mar 2001 17:10:12 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk To save Julian from having to write a memo I would like to add that the ethernet polynomial has a certain weakness for 3 bit errors that the others don't. The ethernet polynomial is only guaranteed to catch all combination sof 3 bit errors when the message length is up to 12144 bits. For larger message lengths some 3 bit errors may get through. Vince |-----Original Message----- |From: CAVANNA,VICENTE V (A-Roseville,ex1) |Sent: Thursday, March 22, 2001 3:42 PM |To: 'Mark Bakke'; 'Jim Williams'; ips@ece.cmu.edu |Cc: CAVANNA,VICENTE V (A-Roseville,ex1) |Subject: RE: CRC vs CHKSUM presentation slides | | |Mark and Jim, |I think any of the 32 bit CRC polynomials that have been |proposed are more than good enough. My main reason for |recommending hte CCITT-CRC32 is that it is half the cost of |the others in terms of gate count - and unless you are doing a |serial divider, which would be too slow, the gate count is |very significant. Also, the leverage may not be as high as you |think unless we are willing to use the datapath width of |existing implementations. For 10 gig we will probably need to |have larger than normal datapath widths. If you change the |datapath width to handle, say, 64 bits at a time you change |the implementation. I would otherwise have no problem with |using the ethernet polynomial. I will try to explain later why |I am not concerned about using the same polynomial and |potentially giving up some cross-checking (I am not quite sure yet). |Vince | ||-----Original Message----- ||From: Jim Williams [mailto:jim.williams@emulex.com] ||Sent: Thursday, March 22, 2001 1:30 PM ||To: ips@ece.cmu.edu ||Subject: Re: CRC vs CHKSUM presentation slides || || ||From: "Mark Bakke" ||To: "CAVANNA,VICENTE V (A-Roseville,ex1)" ||Cc: ; ||Sent: Thursday, March 22, 2001 4:00 PM ||Subject: Re: CRC vs CHKSUM presentation slides || || ||> Vicente- ||> ||> I just took another look through your slides after seeing the ||> presentation on Monday. They were very well-done. I have ||> one question, though. If the CCITT-CRC32 is considered "good ||> enough", then would the Ethernet CRC32 also be good enough? The ||> reason I ask is that every hardware vendor involved in building ||> iSCSI stuff already has implementations of the Ethernet CRC, ||> which is used for both Ethernet and Fibre Channel. ||> ||> The Ethernet poly has more terms than CCITT, and perhaps is ||> not as good as CRC-32C (any thoughts?), but everyone has hardware ||> and software for this, with proven interoperability (bit and ||> byte order, etc). Performance-wise, it will be there for ||> 10Gb Ethernet, so it should be fast enough. ||> ||> So if the Ethernet poly is deemed good enough (even if it's not ||> the best), and fast enough (even if it's not the fastest), why ||> not use it? I think we would stand a much better chance of ||> achieving interoperability in a short time. ||> ||> Please let me know what you think of this; I realize that a few ||> of my questions were speculative. || ||Since the iSCSI messages will often be encapsulated in ethernet ||packets, there is some value to using a different CRC. Link ||errors are double protected with two different CRCs. If ||ethernet and iSCSI use the same polynomial, there is little ||additional coverage against link errors. This point may not ||be decisive, but all other things being equal or almost ||equal, it is worth considering. || | From owner-ips@ece.cmu.edu Thu Mar 22 20:29:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA29246 for ; Thu, 22 Mar 2001 20:29:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2MNHjF12651 for ips-outgoing; Thu, 22 Mar 2001 18:17:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2MNHQr12634 for ; Thu, 22 Mar 2001 18:17:27 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GAM006AEGO475@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Thu, 22 Mar 2001 15:16:53 -0800 (PST) Date: Thu, 22 Mar 2001 15:15:32 -0800 From: Douglas Otis Subject: RE: CRC vs CHKSUM presentation slides In-reply-to: <3ABA67EB.77CFF4AF@cisco.com> To: Mark Bakke , "CAVANNA,VICENTE V (A-Roseville,ex1)" Cc: ips@ece.cmu.edu, ipsan@rtl.rose.agilent.com Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mark, One small note on these slides. Although the number of instructions required to implement CRC-32 with 128K table lookups compares closely to that of Adler-32, Adler-32 is not accessing a 128K byte lookup table. These slides made a comparison of instructions without considering additional memory access required or the size of the table required. If you were to use the same CRC-32, you will find the same levels of protection as this CRC is to protect against various types of equipment failures NOT protected by the Ethernet/FDDI/Fibre-Channel CRC, then a replication of this polynomial should not be a problem in that sense. For those that wish to include a different polynomial tend to justify this effort by indicating this would improve the present physical layer checks. The problem that I see with that conclusion is errors seen outside of any potential Ethernet CRC failure are of such orders of magnitude as to out-weight any replicate benefit derived. In that light, even Adler-32 may provide benefit in that case. The next area being looked at is the number of bits detected in a burst error. The unprotected equipment is not likely using a serial method of delivery, so you are again left errors that may provide strange error patterns. The concept supporting a different polynomial was its greater ability to detect burst errors. The large piece of information missing from this presentation was a characterization of the types of errors seen by intermediate equipment. There was a bullet that suggested this equipment will produce burst errors. If so, what is the nature of this equipment and what are the prevalent errors? From that perspective a comparisons could be made. But as we are familiar with noise distributions of single bit errors, this became our focus. It is a bit like looking for your keys under the light post even though you dropped them in the dark. You wish to look where the light is better. Doug > Vicente- > > I just took another look through your slides after seeing the > presentation on Monday. They were very well-done. I have > one question, though. If the CCITT-CRC32 is considered "good > enough", then would the Ethernet CRC32 also be good enough? The > reason I ask is that every hardware vendor involved in building > iSCSI stuff already has implementations of the Ethernet CRC, > which is used for both Ethernet and Fibre Channel. > > The Ethernet poly has more terms than CCITT, and perhaps is > not as good as CRC-32C (any thoughts?), but everyone has hardware > and software for this, with proven interoperability (bit and > byte order, etc). Performance-wise, it will be there for > 10Gb Ethernet, so it should be fast enough. > > So if the Ethernet poly is deemed good enough (even if it's not > the best), and fast enough (even if it's not the fastest), why > not use it? I think we would stand a much better chance of > achieving interoperability in a short time. > > Please let me know what you think of this; I realize that a few > of my questions were speculative. > > Regards, > > Mark > > "CAVANNA,VICENTE V (A-Roseville,ex1)" wrote: > > > > Here are the slides I presented at the IETF-50 in Minneapolis. > > > > Vicente Cavanna > > Agilent Technologies > > > > <> > > > > > ------------------------------------------------------------------ > ---------------------------------- > > Name: CRCorChksm.pdf > > CRCorChksm.pdf Type: Portable Document Format (application/pdf) > > Encoding: base64 > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 > From owner-ips@ece.cmu.edu Thu Mar 22 20:30:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA29287 for ; Thu, 22 Mar 2001 20:30:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2MNXml13827 for ips-outgoing; Thu, 22 Mar 2001 18:33:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2MNXXr13810 for ; Thu, 22 Mar 2001 18:33:33 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 459BA965 for ; Thu, 22 Mar 2001 16:33:32 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id F334918F for ; Thu, 22 Mar 2001 18:28:41 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id PAA14175 for ; Thu, 22 Mar 2001 15:28:40 -0800 (PST) Message-ID: <3ABA8AC6.E59A591A@agilent.com> Date: Thu, 22 Mar 2001 15:29:10 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI: Out Of Sequence due to null sequence with multiple connections. References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit In simple terms, what Doug is asking for in (B) is an "immediate" command that will go and abort all pending commands in the iSCSI queue that have not yet been delivered to the SCSI layer due to missing commands sequence numbers. I'm wondering how useful this is. The initiator could simply look at the ExpCmdSN from the target and resend the missing command(s) if the target appears to be "stuck". Given the way TCP works, the only time the target could get "stuck" is if a TCP connection fails. In which case the connection would be re-established and the missing commands resent. -Matt From owner-ips@ece.cmu.edu Thu Mar 22 20:30:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA29288 for ; Thu, 22 Mar 2001 20:30:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2MNgkq14399 for ips-outgoing; Thu, 22 Mar 2001 18:42:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2MNgGr14380 for ; Thu, 22 Mar 2001 18:42:16 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id BF52C98F; Thu, 22 Mar 2001 16:42:15 -0700 (MST) Received: from axcsbh4.cos.agilent.com (axcsbh4.cos.agilent.com [130.29.152.145]) by msgrel1.cos.agilent.com (Postfix) with SMTP id 95164113; Thu, 22 Mar 2001 16:42:15 -0700 (MST) Received: from 130.29.152.145 by axcsbh4.cos.agilent.com (InterScan E-Mail VirusWall NT); Thu, 22 Mar 2001 16:42:15 -0700 (Mountain Standard Time) Received: by axcsbh4.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Thu, 22 Mar 2001 16:42:15 -0700 Message-ID: From: vince_cavanna@agilent.com To: mbakke@cisco.com, jim.williams@emulex.com, ips@ece.cmu.edu Cc: vince_cavanna@agilent.com Subject: RE: CRC vs CHKSUM presentation slides Date: Thu, 22 Mar 2001 16:42:13 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Mark and Jim, I think any of the 32 bit CRC polynomials that have been proposed are more than good enough. My main reason for recommending hte CCITT-CRC32 is that it is half the cost of the others in terms of gate count - and unless you are doing a serial divider, which would be too slow, the gate count is very significant. Also, the leverage may not be as high as you think unless we are willing to use the datapath width of existing implementations. For 10 gig we will probably need to have larger than normal datapath widths. If you change the datapath width to handle, say, 64 bits at a time you change the implementation. I would otherwise have no problem with using the ethernet polynomial. I will try to explain later why I am not concerned about using the same polynomial and potentially giving up some cross-checking (I am not quite sure yet). Vince |-----Original Message----- |From: Jim Williams [mailto:jim.williams@emulex.com] |Sent: Thursday, March 22, 2001 1:30 PM |To: ips@ece.cmu.edu |Subject: Re: CRC vs CHKSUM presentation slides | | |From: "Mark Bakke" |To: "CAVANNA,VICENTE V (A-Roseville,ex1)" |Cc: ; |Sent: Thursday, March 22, 2001 4:00 PM |Subject: Re: CRC vs CHKSUM presentation slides | | |> Vicente- |> |> I just took another look through your slides after seeing the |> presentation on Monday. They were very well-done. I have |> one question, though. If the CCITT-CRC32 is considered "good |> enough", then would the Ethernet CRC32 also be good enough? The |> reason I ask is that every hardware vendor involved in building |> iSCSI stuff already has implementations of the Ethernet CRC, |> which is used for both Ethernet and Fibre Channel. |> |> The Ethernet poly has more terms than CCITT, and perhaps is |> not as good as CRC-32C (any thoughts?), but everyone has hardware |> and software for this, with proven interoperability (bit and |> byte order, etc). Performance-wise, it will be there for |> 10Gb Ethernet, so it should be fast enough. |> |> So if the Ethernet poly is deemed good enough (even if it's not |> the best), and fast enough (even if it's not the fastest), why |> not use it? I think we would stand a much better chance of |> achieving interoperability in a short time. |> |> Please let me know what you think of this; I realize that a few |> of my questions were speculative. | |Since the iSCSI messages will often be encapsulated in ethernet |packets, there is some value to using a different CRC. Link |errors are double protected with two different CRCs. If |ethernet and iSCSI use the same polynomial, there is little |additional coverage against link errors. This point may not |be decisive, but all other things being equal or almost |equal, it is worth considering. | From owner-ips@ece.cmu.edu Thu Mar 22 22:10:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA03634 for ; Thu, 22 Mar 2001 22:10:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2N0ZlY17651 for ips-outgoing; Thu, 22 Mar 2001 19:35:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2N0Z2r17611 for ; Thu, 22 Mar 2001 19:35:02 -0500 (EST) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by palrel3.hp.com (Postfix) with ESMTP id DC433124F; Thu, 22 Mar 2001 16:35:01 -0800 (PST) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA29740; Thu, 22 Mar 2001 17:35:00 -0700 (MST) Received: by xboibrg2.boi.hp.com with Internet Mail Service (5.5.2653.19) id ; Thu, 22 Mar 2001 17:35:00 -0700 Message-ID: <6BD67FFB937FD411A04F00D0B74FE87802A08F1F@xrose06.rose.hp.com> From: "KRUEGER,MARJORIE (HP-Roseville,ex1)" To: "'Black_David@emc.com'" , ips@ece.cmu.edu Subject: RE: SACK (was RE:Async Message PDU) Date: Thu, 22 Mar 2001 17:34:56 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Sorry for the delayed reaction, I'm just catching up - You stated that a decision was made at the Orlando interim meeting to "change some of the terminology with the goal of each term and acronym being unambiguously owned by a single layer in the protocol stack". I think that's the right thing to do, so can we agree to rename SACK to something that doesn't cause confusion with the TCP construct? Marjorie Krueger Networked Storage Architecture Networked Storage Solutions Org. Hewlett-Packard tel: +1 916 785 2656 fax: +1 916 785 0391 email: marjorie_krueger@hp.com > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Thursday, March 01, 2001 11:37 AM > To: cbm@rose.hp.com; ips@ece.cmu.edu > Subject: RE: Async Message PDU > > > Julian will doubtless pick up the rest of these, > but I thought I'd take this issue, since it was > part of what we did in Orlando. > > > 4. Somehow, "Asynchronous Message" seems at odds with the > rest of the > > usage in the draft in regards to PDUs (since a message is > introduced as > > PDU in section 1.2). Should we may be just call it an AEN > PDU? Section > > 2.14.3 calls this so. (I realize that it may not always > result in a > > SCSI-level AER.) > > One of the things we did in Orlando was to change > some of the terminology with the goal of each term and > acronym being unambiguously owned by a single layer > in the protocol stack, in particular making a sharp > distinction between SCSI concepts and iSCSI mechanisms. > > All the *RN entities changed to *SN for this reason > (SCSI has a CmdRN, so all *RN entities are SCSI, and > all *SN entities are iSCSI). Similarly, all AE* > entities are SCSI, and we went with "Asynchronous > Message <*>" to name the iSCSI entities. This matters > here because there are circumstances in which iSCSI > will send Asynchronous Message PDUs for its own use > even when SCSI has disabled AER. The usage of "AEN > PDU" in 2.14.3 is probably an editing oversight. > Suggestions for words to use instead of "Message" > are welcome, but they must not start with the letter > "E" ;-). > > Thanks, > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ece.cmu.edu Fri Mar 23 01:19:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA12703 for ; Fri, 23 Mar 2001 01:19:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2N4qvi03880 for ips-outgoing; Thu, 22 Mar 2001 23:52:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2N4qSr03832 for ; Thu, 22 Mar 2001 23:52:28 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 22 Mar 2001 23:52:22 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0708015314@corpmx9.isus.emc.com> From: Black_David@emc.com To: marjorie_krueger@hp.com, ips@ece.cmu.edu Subject: RE: SACK (was RE:Async Message PDU) Date: Thu, 22 Mar 2001 23:52:20 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk I think picking a new name for SACK to avoid confusion with TCP is an excellent idea. Any suggestions for a new name? --David > -----Original Message----- > From: KRUEGER,MARJORIE (HP-Roseville,ex1) [SMTP:marjorie_krueger@hp.com] > Sent: Thursday, March 22, 2001 7:35 PM > To: 'Black_David@emc.com'; ips@ece.cmu.edu > Subject: RE: SACK (was RE:Async Message PDU) > > David, > Sorry for the delayed reaction, I'm just catching up - You stated that a > decision was made at the Orlando interim meeting to "change some of the > terminology with the goal of each term and acronym being unambiguously > owned > by a single layer in the protocol stack". I think that's the right thing > to > do, so can we agree to rename SACK to something that doesn't cause > confusion > with the TCP construct? > > Marjorie Krueger > Networked Storage Architecture > Networked Storage Solutions Org. > Hewlett-Packard > tel: +1 916 785 2656 > fax: +1 916 785 0391 > email: marjorie_krueger@hp.com > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Thursday, March 01, 2001 11:37 AM > > To: cbm@rose.hp.com; ips@ece.cmu.edu > > Subject: RE: Async Message PDU > > > > > > Julian will doubtless pick up the rest of these, > > but I thought I'd take this issue, since it was > > part of what we did in Orlando. > > > > > 4. Somehow, "Asynchronous Message" seems at odds with the > > rest of the > > > usage in the draft in regards to PDUs (since a message is > > introduced as > > > PDU in section 1.2). Should we may be just call it an AEN > > PDU? Section > > > 2.14.3 calls this so. (I realize that it may not always > > result in a > > > SCSI-level AER.) > > > > One of the things we did in Orlando was to change > > some of the terminology with the goal of each term and > > acronym being unambiguously owned by a single layer > > in the protocol stack, in particular making a sharp > > distinction between SCSI concepts and iSCSI mechanisms. > > > > All the *RN entities changed to *SN for this reason > > (SCSI has a CmdRN, so all *RN entities are SCSI, and > > all *SN entities are iSCSI). Similarly, all AE* > > entities are SCSI, and we went with "Asynchronous > > Message <*>" to name the iSCSI entities. This matters > > here because there are circumstances in which iSCSI > > will send Asynchronous Message PDUs for its own use > > even when SCSI has disabled AER. The usage of "AEN > > PDU" in 2.14.3 is probably an editing oversight. > > Suggestions for words to use instead of "Message" > > are welcome, but they must not start with the letter > > "E" ;-). > > > > Thanks, > > --David > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Fri Mar 23 01:21:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA13138 for ; Fri, 23 Mar 2001 01:21:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2N42wd00680 for ips-outgoing; Thu, 22 Mar 2001 23:02:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2N42Or00621 for ; Thu, 22 Mar 2001 23:02:24 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GAM00HRYTVIOY@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Thu, 22 Mar 2001 20:02:07 -0800 (PST) Date: Thu, 22 Mar 2001 20:00:47 -0800 From: Douglas Otis Subject: RE: iSCSI: Out Of Sequence due to null sequence with multiple connections. In-reply-to: <3ABA8AC6.E59A591A@agilent.com> To: Matt Wakeley , IPS Reflector Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt, If the sequencer gets stuck due to some target resource limitation, you are now sitting behind a queue that you have no control over. If you use the immediate command as a bypass to this problem, its placement with respect to pending commands is unknown. Tearing down the entire session does not resolve the state of the target with an ambiguity of the delivery from the sequencer to the target still unresolved as well as commands in progress. If the goal is to be sure of the state of the target, severing the session does not accomplish this. Assigning a sequence to an immediate command with those pending at the sequencer to be rejected is a simple control to have in place that will likely provide the needed actions to resolve a problem. Disconnecting simply leaves the target state unknown permanently while hoping the target is not sensitive to replaying commands. Doug > In simple terms, what Doug is asking for in (B) is an "immediate" > command that > will go and abort all pending commands in the iSCSI queue that > have not yet > been delivered to the SCSI layer due to missing commands sequence numbers. > > I'm wondering how useful this is. The initiator could simply look at the > ExpCmdSN from the target and resend the missing command(s) if the target > appears to be "stuck". Given the way TCP works, the only time the target > could get "stuck" is if a TCP connection fails. In which case > the connection > would be re-established and the missing commands resent. > > -Matt > From owner-ips@ece.cmu.edu Fri Mar 23 01:25:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA13810 for ; Fri, 23 Mar 2001 01:25:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2N4Evj01526 for ips-outgoing; Thu, 22 Mar 2001 23:14:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2N4Epr01518 for ; Thu, 22 Mar 2001 23:14:51 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id C04A010EC; Thu, 22 Mar 2001 21:14:50 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 00F0D176; Thu, 22 Mar 2001 23:14:49 -0500 (EST) Received: from agilent.com (cos1nai132175.cos.agilent.com [141.184.132.175]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id UAA09428; Thu, 22 Mar 2001 20:14:47 -0800 (PST) Message-ID: <3ABACDB6.4461738E@agilent.com> Date: Thu, 22 Mar 2001 20:14:46 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Douglas Otis Cc: IPS Reflector Subject: Re: iSCSI: Out Of Sequence due to null sequence with multipleconnections. References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Doug, re-read my message. I did not say anything about disconnecting as a means of recovery. -Matt Douglas Otis wrote: > > Matt, > > If the sequencer gets stuck due to some target resource limitation, you are > now sitting behind a queue that you have no control over. If you use the > immediate command as a bypass to this problem, its placement with respect to > pending commands is unknown. Tearing down the entire session does not > resolve the state of the target with an ambiguity of the delivery from the > sequencer to the target still unresolved as well as commands in progress. > If the goal is to be sure of the state of the target, severing the session > does not accomplish this. Assigning a sequence to an immediate command with > those pending at the sequencer to be rejected is a simple control to have in > place that will likely provide the needed actions to resolve a problem. > Disconnecting simply leaves the target state unknown permanently while > hoping the target is not sensitive to replaying commands. > > Doug > > > In simple terms, what Doug is asking for in (B) is an "immediate" > > command that > > will go and abort all pending commands in the iSCSI queue that > > have not yet > > been delivered to the SCSI layer due to missing commands sequence numbers. > > > > I'm wondering how useful this is. The initiator could simply look at the > > ExpCmdSN from the target and resend the missing command(s) if the target > > appears to be "stuck". Given the way TCP works, the only time the target > > could get "stuck" is if a TCP connection fails. In which case > > the connection > > would be re-established and the missing commands resent. > > > > -Matt > > From owner-ips@ece.cmu.edu Fri Mar 23 02:04:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA25201 for ; Fri, 23 Mar 2001 02:04:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2N4ivA03378 for ips-outgoing; Thu, 22 Mar 2001 23:44:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2N4iHr03357 for ; Thu, 22 Mar 2001 23:44:17 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GAM00HSZVT7VI@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Thu, 22 Mar 2001 20:43:56 -0800 (PST) Date: Thu, 22 Mar 2001 20:42:35 -0800 From: Douglas Otis Subject: RE: iSCSI: Out Of Sequence due to null sequence with multipleconnections. In-reply-to: <3ABACDB6.4461738E@agilent.com> To: Matt Wakeley Cc: IPS Reflector Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt, "There will never be a time where the immediate command will be used to reach around the sequencer and the only reason a sequencer could be stuck is due to a hole that can be discovered and eventually filled" if I understand your statement. There are also resource limitations between the sequencer and the target that may lead to a stall or a long term delay that needs to be halted. As the sequencer has exhausted these resources, it waits and waits. If there is a tag left open for the 'get me out of this' command, the sequencer needs some means to understand when to employ this trump card. An immediate command seems like a good means of creating such a signal. The problem is what is the final state of this action and is the reason for maintaining sequence on all commands sent to the target. Sorry, but I looked to the next possible action and tried to answer that as well. I tend to be long winded I guess as well as long in the tooth. Doug > Doug, re-read my message. I did not say anything about disconnecting as a > means of recovery. > > -Matt > > Douglas Otis wrote: > > > > Matt, > > > > If the sequencer gets stuck due to some target resource > limitation, you are > > now sitting behind a queue that you have no control over. If > you use the > > immediate command as a bypass to this problem, its placement > with respect to > > pending commands is unknown. Tearing down the entire session does not > > resolve the state of the target with an ambiguity of the > delivery from the > > sequencer to the target still unresolved as well as commands in > progress. > > If the goal is to be sure of the state of the target, severing > the session > > does not accomplish this. Assigning a sequence to an immediate > command with > > those pending at the sequencer to be rejected is a simple > control to have in > > place that will likely provide the needed actions to resolve a problem. > > Disconnecting simply leaves the target state unknown permanently while > > hoping the target is not sensitive to replaying commands. > > > > Doug > > > > > In simple terms, what Doug is asking for in (B) is an "immediate" > > > command that > > > will go and abort all pending commands in the iSCSI queue that > > > have not yet > > > been delivered to the SCSI layer due to missing commands > sequence numbers. > > > > > > I'm wondering how useful this is. The initiator could simply > look at the > > > ExpCmdSN from the target and resend the missing command(s) if > the target > > > appears to be "stuck". Given the way TCP works, the only > time the target > > > could get "stuck" is if a TCP connection fails. In which case > > > the connection > > > would be re-established and the missing commands resent. > > > > > > -Matt > > > > From owner-ips@ece.cmu.edu Fri Mar 23 03:10:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA28074 for ; Fri, 23 Mar 2001 03:10:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2N5lxp07145 for ips-outgoing; Fri, 23 Mar 2001 00:47:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2N5lir07102 for ; Fri, 23 Mar 2001 00:47:44 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 23 Mar 2001 00:47:38 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0708015316@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: iSCSI Naming: WWUIs, URNs, and namespaces Date: Fri, 23 Mar 2001 00:47:37 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I don't like naming issues. :-) :-) After suitable consulting with some members of the IESG and IAB, I have some news to convey about the current approach to iSCSI naming. The IESG will not approve another global namespace for iSCSI's use - this means that WWUIs as currently designed will need to be revised out of the naming and discovery draft, and that it will not be possible to proceed with the WWUI URN draft as an official IPS WG work item. The best course of action would probably be to allow the WWUI URN draft to expire without further revision. To a first approximation, WWUIs are/were a "grand unified theory" of naming, in that any namespace could be glued into the WWUI world (as several were). The WG is being directed to instead focus on the individual namespaces and make sure that the ones that are used are in fact necessary. iSCSI can use text keys to identify which sort of name is being used (one key for each sort of format, for each instance in which a name is used), and it may be possible to encode the name format in the parse rules for the values of iSCSI keys to avoid proliferation of keys. Taking a look at the namespaces in the current iSCSI naming and discovery draft, here's some initial guidance from this WG co-chair: iscsi - canonical target -- This should be fine. eui - WWNs -- The use of these for storage makes eminent sense. I don't see a problem here. dns - hostnames -- Use of these should be abandoned as not only are they not really URNs (as indicated in the draft), but their intended usage is straying into the tarpit known as "URN resolution". Faster progress will made by staying out. A DNS hostname can be put into an Alias or something new can be invented to carry it as a Location Hint, BUT the relevant URN WG RFCs and drafts on URN resolution should be reviewed before proceeding too far in this direction. iscsi - Reverse DNS and oui - Org. Unique Identifier -- The rationale for these is not clear to me. Assuming that WWNs are going to be available for use in naming iSCSI Initiators and Targets, what are the problems that these sorts of names solve that WWNs don't? It appears that one of the problems may be who can get/create them. Discussion of this on the list would be appropriate. In any case, the fewer name formats we have to deal with, the better. I want to try to anticipate an objection to this, which would note that from a functional viewpoint the basic impact of this is to move some characters from one text string to another (e.g., from a WWUI type designator to part of an iSCSI text key), and wonder if this is a distinction without a difference. One of the reasons for the that started this post is that a functional view is not sufficient for naming - how things are named, the intended usage of names and their scope matter a lot. This is particularly true when considering the structure of a namespace and how that structure may be extended. The upshot is that avoiding introduction of something claiming to be yet another global namespace is important (i.e., use existing namespaces with global scope in preference to inventing new ones). The resulting need to define the name spaces/formats in the main iSCSI spec. is probably a "feature" as it forces us to pay more attention to the sorts of names we use and raises the bar for adding additional sorts of names in the future. I will be working with the naming and discovery team in my "copious spare time" to make sure that we don't lose the valuable work and progress they've made to date as a consequence of this change. Discussion on the list about what sort of names are important (e.g., the Reverse DNS and OUI namespaces) and why would be useful. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Mar 23 07:12:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA06863 for ; Fri, 23 Mar 2001 07:12:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2N9i3u20280 for ips-outgoing; Fri, 23 Mar 2001 04:44:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2N9hZr20227 for ; Fri, 23 Mar 2001 04:43:35 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA40556 for ; Fri, 23 Mar 2001 10:43:07 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id KAA12072 for ; Fri, 23 Mar 2001 10:40:37 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A18.003529D2 ; Fri, 23 Mar 2001 10:40:41 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 23 Mar 2001 11:43:02 +0200 Subject: New Format foils Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk The new format foils on my site have been updated. Julo From owner-ips@ece.cmu.edu Fri Mar 23 09:17:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id JAA13835 for ; Fri, 23 Mar 2001 09:17:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NC46U09268 for ips-outgoing; Fri, 23 Mar 2001 07:04:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NC2ir09199 for ; Fri, 23 Mar 2001 07:02:44 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id NAA192618 for ; Fri, 23 Mar 2001 13:01:01 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id MAA81066 for ; Fri, 23 Mar 2001 12:59:19 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A18.0041C21B ; Fri, 23 Mar 2001 12:58:15 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 23 Mar 2001 14:01:05 +0200 Subject: Re: CRC vs CHKSUM presentation slides Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, I think that bot Vince and myself have stated tht polynoms of the for (x+1)*irreducible are better than irreducible. We will soon agree on one -:) Regards, Julo "Jim Williams" on 22/03/2001 23:29:39 Please respond to "Jim Williams" To: ips@ece.cmu.edu cc: Subject: Re: CRC vs CHKSUM presentation slides From: "Mark Bakke" To: "CAVANNA,VICENTE V (A-Roseville,ex1)" Cc: ; Sent: Thursday, March 22, 2001 4:00 PM Subject: Re: CRC vs CHKSUM presentation slides > Vicente- > > I just took another look through your slides after seeing the > presentation on Monday. They were very well-done. I have > one question, though. If the CCITT-CRC32 is considered "good > enough", then would the Ethernet CRC32 also be good enough? The > reason I ask is that every hardware vendor involved in building > iSCSI stuff already has implementations of the Ethernet CRC, > which is used for both Ethernet and Fibre Channel. > > The Ethernet poly has more terms than CCITT, and perhaps is > not as good as CRC-32C (any thoughts?), but everyone has hardware > and software for this, with proven interoperability (bit and > byte order, etc). Performance-wise, it will be there for > 10Gb Ethernet, so it should be fast enough. > > So if the Ethernet poly is deemed good enough (even if it's not > the best), and fast enough (even if it's not the fastest), why > not use it? I think we would stand a much better chance of > achieving interoperability in a short time. > > Please let me know what you think of this; I realize that a few > of my questions were speculative. Since the iSCSI messages will often be encapsulated in ethernet packets, there is some value to using a different CRC. Link errors are double protected with two different CRCs. If ethernet and iSCSI use the same polynomial, there is little additional coverage against link errors. This point may not be decisive, but all other things being equal or almost equal, it is worth considering. From owner-ips@ece.cmu.edu Fri Mar 23 11:04:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA21051 for ; Fri, 23 Mar 2001 11:04:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NDO8s13781 for ips-outgoing; Fri, 23 Mar 2001 08:24:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NDNar13758 for ; Fri, 23 Mar 2001 08:23:39 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id OAA43010 for ; Fri, 23 Mar 2001 14:23:23 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id OAA46546 for ; Fri, 23 Mar 2001 14:20:52 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A18.00495315 ; Fri, 23 Mar 2001 14:20:53 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 23 Mar 2001 14:43:02 +0200 Subject: Re: foils Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, I am not going to answer to this -:) Julo Matt Wakeley on 22/03/2001 20:25:35 Please respond to Matt Wakeley To: IPS Reflector cc: Subject: Re: foils Why don't you submit the new format in an I-D format instead of slides. julian_satran@il.ibm.com wrote: > > Correct - I there are several typos (all hopefully obvious). > Barry (who has the source) might correct them and send me an update. > > Julo > > "Ayman Ghanem" on 22/03/2001 18:07:15 > > Please respond to "Ayman Ghanem" > > To: ips@ece.cmu.edu > cc: > Subject: RE: foils > > Bullet 3 of slide 6 (new PDU format) says that first read is 48 bytes + > header digest size. I assume this is a typo as the header digest comes > after > the AHS. > > -Ayman From owner-ips@ece.cmu.edu Fri Mar 23 11:06:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA21217 for ; Fri, 23 Mar 2001 11:06:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NDr9b15569 for ips-outgoing; Fri, 23 Mar 2001 08:53:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NDr4r15562 for ; Fri, 23 Mar 2001 08:53:04 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id IAA04566 for ; Fri, 23 Mar 2001 08:52:58 -0500 (EST) Message-ID: <3ABB5526.38875E76@cisco.com> Date: Fri, 23 Mar 2001 07:52:38 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: IPS Subject: iSCSI: Slides from IETF-50 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian has placed copies of the MIB and Naming & Discovery slides on his IPS archive, at http://www.haifa.il.ibm.com/satran/ips. They are: IETF50-iscsi-mib-presentation.pdf IETF50-iscsi-ndt-presentation.pdf IETF50-iscsi-urn-presentation.pdf IETF50-iscsi-slp-presentation.pdf -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Fri Mar 23 12:24:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA26571 for ; Fri, 23 Mar 2001 12:24:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NEbDa18362 for ips-outgoing; Fri, 23 Mar 2001 09:37:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NEagr18308 for ; Fri, 23 Mar 2001 09:36:42 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id JAA23339; Fri, 23 Mar 2001 09:36:30 -0500 (EST) Message-ID: <3ABB5F59.718AC63D@cisco.com> Date: Fri, 23 Mar 2001 08:36:09 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Black_David@emc.com CC: ips@ece.cmu.edu Subject: Re: iSCSI Naming: WWUIs, URNs, and namespaces References: <0F31E5C394DAD311B60C00E029101A0708015316@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David- This is starting to seem like work. In case they help, please see my comments below. We can take this up in more detail in the N&D team. -- Mark Black_David@emc.com wrote: > > I don't like naming issues. :-) :-) > > After suitable consulting with some members of > the IESG and IAB, I have some news to convey about > the current approach to iSCSI naming. > > The IESG will not approve another global namespace > for iSCSI's use - this means that WWUIs as currently So I take it that WWUIs are fine, as long as they use existing global name spaces. > designed will need to be revised out of the > naming and discovery draft, and that it will not be > possible to proceed with the WWUI URN draft > as an official IPS WG work item. The best course of > action would probably be to allow the WWUI URN draft > to expire without further revision. > > To a first approximation, WWUIs are/were a "grand > unified theory" of naming, in that any namespace could > be glued into the WWUI world (as several were). The reason we glued these in was to avoid creating a new global name space. We allowed the use of a few currently available global name spaces. How can this be a problem? > The WG is being directed to instead focus on the > individual namespaces and make sure that the ones that > are used are in fact necessary. iSCSI can use text > keys to identify which sort of name is being used > (one key for each sort of format, for each instance > in which a name is used), and it may be possible > to encode the name format in the parse rules for the > values of iSCSI keys to avoid proliferation of keys. > > Taking a look at the namespaces in the current iSCSI > naming and discovery draft, here's some initial > guidance from this WG co-chair: > iscsi - canonical target -- This should be fine. > eui - WWNs -- The use of these for storage makes eminent > sense. I don't see a problem here. So far, so good. Does that mean that we can keep the above two, in the current WWUI, as we have it formatted? EUI, although it's not that flexible (see later), is required for interoperating with and adding iSCSI to existing devices. > dns - hostnames -- Use of these should be abandoned as > not only are they not really URNs (as indicated > in the draft), but their intended usage is straying > into the tarpit known as "URN resolution". Faster > progress will made by staying out. A DNS hostname > can be put into an Alias or something new can be > invented to carry it as a Location Hint, BUT the > relevant URN WG RFCs and drafts on URN resolution > should be reviewed before proceeding too far in this > direction. I don't like this one that much, either, for the same reasons. > iscsi - Reverse DNS and oui - Org. Unique Identifier -- > The rationale for these is not clear to me. > Assuming that WWNs are going to be available for > use in naming iSCSI Initiators and Targets, what > are the problems that these sorts of names solve > that WWNs don't? It appears that one of the problems > may be who can get/create them. Discussion of this > on the list would be appropriate. These two formats use existing global name spaces, and allow the implementor or end user to take a more flexible approach to naming than is offered by the EUI-64. When dealing with a large number of logical targets, burning WWNs (essentially, MAC addresses), and attempting to keep them tied to a logical entity without tying them to hardware is not practical. Both of these formats accomplish the same thing, and we could make do with just one of them. The only real differences are: - OUI is a fixed length naming authority; reverse DNS is variable and could result in longer names. - A human looking at a reverse DNS-based name can easily determine who the naming authority is; a human looking at an OUI-based name will usually have to go look it up at IEEE. - Only a hardware manufacturer (and a few software manufacters) will already have an OUI. Others, such as storage service providers, large end customers, or software driver providers, would either be out-of-luck for naming their devices, or would have to register for an OUI, which would otherwise be a waste of IEEE resources. OTOH, a storage service provider could provide its own names by using the reverse-DNS authority. Everyone has a DNS name. This would enable the SSP to provide its customers with a WWUI that would not change if the back-end storage device was replaced. After all, it's not the device that's important to this customer, it's the DATA. - If we had to pick one or the other, I would pick the reverse DNS, since it offers the ability to generate unique names to a wider variety of users. > In any case, the fewer name formats we have to deal with, > the better. Although I probably should have discussed this with the rest of the NDT first, I would personally be happy with: - iscsi - The canonical name. - eui - For compatibility with existing device naming schemes. - reverse-dns - For better naming flexibility moving forward. Perhaps the IESG would accept the WWUI with just these three. Or perhaps we could just make the WWUI into a URN? I don't know if that would help, but perhaps... The main point is that we are NOT creating any new name spaces; we really are just using what's already there. Maybe by specifying fewer of the options that are already there, we would be in better shape. > I want to try to anticipate an objection to this, which > would note that from a functional viewpoint the basic > impact of this is to move some characters from one text > string to another (e.g., from a WWUI type designator > to part of an iSCSI text key), and wonder if this is > a distinction without a difference. One of the reasons > for the that started this post is that a functional > view is not sufficient for naming - how things are named, > the intended usage of names and their scope matter a lot. So that leads to the question: Has any other WG come up with a world-wide-unique-identifier for something else that the IESG folks deems acceptable? If we can see a few examples, we would be better able to anticipate what they want. > This is particularly true when considering the structure > of a namespace and how that structure may be extended. > The upshot is that avoiding introduction of something > claiming to be yet another global namespace is important > (i.e., use existing namespaces with global scope in preference > to inventing new ones). The resulting need to define > the name spaces/formats in the main iSCSI spec. is > probably a "feature" as it forces us to pay more > attention to the sorts of names we use and raises the > bar for adding additional sorts of names in the future. > I will be working with > the naming and discovery team in my "copious spare time" > to make sure that we don't lose the valuable work and > progress they've made to date as a consequence of this > change. Discussion on the list about what sort > of names are important (e.g., the Reverse DNS and OUI > namespaces) and why would be useful. From a requirements point of view, please respond if any of the types of names are important to you (plural). Hopefully, we can get this out of the way quickly. > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Fri Mar 23 13:21:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA29983 for ; Fri, 23 Mar 2001 13:21:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NGQMQ26010 for ips-outgoing; Fri, 23 Mar 2001 11:26:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com (mxic1.isus.emc.com [168.159.211.82]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NGQ1r25976 for ; Fri, 23 Mar 2001 11:26:01 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 23 Mar 2001 11:27:19 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A070801531D@corpmx9.isus.emc.com> From: Black_David@emc.com To: mbakke@cisco.com, Black_David@emc.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI Naming: WWUIs, URNs, and namespaces Date: Fri, 23 Mar 2001 11:25:53 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Mark, I need to read your note in more detail before I can respond completely, but we may be close to a solution here. At a high level, the notion of the WWUI as yet another globally unique namespace (which is what it is, even if it's assembled by gluing in existing namespaces) needs to go away. The set of {canonical iSCSI, eui[WWN], reverse-dns} as usable namespaces may be the right way forward, especially as it looks like you've made a good start on the rationale required to justify reverse DNS. We need to describe how to uniquely identify Initiators and Targets without introducing the notion of a WWUI as a new first-class global unique identifier abstraction (e.g., that some other set of people may try to use and/or extend). In other words, we need to scrap the current WWUI structure without losing the important functionality that iSCSI needs in this area. Some amount of this may just be word-smithing to avoid unnecessarily waving red flags ... More to come ... --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Mark Bakke [SMTP:mbakke@cisco.com] > Sent: Friday, March 23, 2001 9:36 AM > To: Black_David@emc.com > Cc: ips@ece.cmu.edu > Subject: Re: iSCSI Naming: WWUIs, URNs, and namespaces > > > David- > > This is starting to seem like work. In case they help, > please see my comments below. We can take this up in > more detail in the N&D team. > > -- > Mark > > Black_David@emc.com wrote: > > > > I don't like naming issues. :-) :-) > > > > After suitable consulting with some members of > > the IESG and IAB, I have some news to convey about > > the current approach to iSCSI naming. > > > > The IESG will not approve another global namespace > > for iSCSI's use - this means that WWUIs as currently > > So I take it that WWUIs are fine, as long as they use > existing global name spaces. > > > designed will need to be revised out of the > > naming and discovery draft, and that it will not be > > possible to proceed with the WWUI URN draft > > as an official IPS WG work item. The best course of > > action would probably be to allow the WWUI URN draft > > to expire without further revision. > > > > To a first approximation, WWUIs are/were a "grand > > unified theory" of naming, in that any namespace could > > be glued into the WWUI world (as several were). > > The reason we glued these in was to avoid creating a new > global name space. We allowed the use of a few currently > available global name spaces. How can this be a problem? > > > The WG is being directed to instead focus on the > > individual namespaces and make sure that the ones that > > are used are in fact necessary. iSCSI can use text > > keys to identify which sort of name is being used > > (one key for each sort of format, for each instance > > in which a name is used), and it may be possible > > to encode the name format in the parse rules for the > > values of iSCSI keys to avoid proliferation of keys. > > > > Taking a look at the namespaces in the current iSCSI > > naming and discovery draft, here's some initial > > guidance from this WG co-chair: > > iscsi - canonical target -- This should be fine. > > eui - WWNs -- The use of these for storage makes eminent > > sense. I don't see a problem here. > > So far, so good. Does that mean that we can keep the above > two, in the current WWUI, as we have it formatted? > > EUI, although it's not that flexible (see later), is required > for interoperating with and adding iSCSI to existing devices. > > > dns - hostnames -- Use of these should be abandoned as > > not only are they not really URNs (as indicated > > in the draft), but their intended usage is straying > > into the tarpit known as "URN resolution". Faster > > progress will made by staying out. A DNS hostname > > can be put into an Alias or something new can be > > invented to carry it as a Location Hint, BUT the > > relevant URN WG RFCs and drafts on URN resolution > > should be reviewed before proceeding too far in this > > direction. > > I don't like this one that much, either, for the same reasons. > > > iscsi - Reverse DNS and oui - Org. Unique Identifier -- > > The rationale for these is not clear to me. > > Assuming that WWNs are going to be available for > > use in naming iSCSI Initiators and Targets, what > > are the problems that these sorts of names solve > > that WWNs don't? It appears that one of the problems > > may be who can get/create them. Discussion of this > > on the list would be appropriate. > > These two formats use existing global name spaces, and allow > the implementor or end user to take a more flexible approach > to naming than is offered by the EUI-64. When dealing with > a large number of logical targets, burning WWNs (essentially, > MAC addresses), and attempting to keep them tied to a logical > entity without tying them to hardware is not practical. > > Both of these formats accomplish the same thing, and we could > make do with just one of them. The only real differences are: > > - OUI is a fixed length naming authority; reverse DNS is variable > and could result in longer names. > > - A human looking at a reverse DNS-based name can easily determine > who the naming authority is; a human looking at an OUI-based > name will usually have to go look it up at IEEE. > > - Only a hardware manufacturer (and a few software manufacters) > will already have an OUI. Others, such as storage service > providers, large end customers, or software driver providers, > would either be out-of-luck for naming their devices, or would > have to register for an OUI, which would otherwise be a waste > of IEEE resources. OTOH, a storage service provider could > provide its own names by using the reverse-DNS authority. > Everyone has a DNS name. This would enable the SSP to provide > its customers with a WWUI that would not change if the back-end > storage device was replaced. After all, it's not the device > that's important to this customer, it's the DATA. > > - If we had to pick one or the other, I would pick the reverse > DNS, since it offers the ability to generate unique names to > a wider variety of users. > > > In any case, the fewer name formats we have to deal with, > > the better. > > Although I probably should have discussed this with the rest > of the NDT first, I would personally be happy with: > > - iscsi - The canonical name. > - eui - For compatibility with existing device naming schemes. > - reverse-dns - For better naming flexibility moving forward. > > Perhaps the IESG would accept the WWUI with just these three. > Or perhaps we could just make the WWUI into a URN? I don't > know if that would help, but perhaps... > > The main point is that we are NOT creating any new name > spaces; we really are just using what's already there. Maybe > by specifying fewer of the options that are already there, we > would be in better shape. > > > I want to try to anticipate an objection to this, which > > would note that from a functional viewpoint the basic > > impact of this is to move some characters from one text > > string to another (e.g., from a WWUI type designator > > to part of an iSCSI text key), and wonder if this is > > a distinction without a difference. One of the reasons > > for the that started this post is that a functional > > view is not sufficient for naming - how things are named, > > the intended usage of names and their scope matter a lot. > > So that leads to the question: Has any other WG come up with > a world-wide-unique-identifier for something else that the > IESG folks deems acceptable? If we can see a few examples, > we would be better able to anticipate what they want. > > > This is particularly true when considering the structure > > of a namespace and how that structure may be extended. > > The upshot is that avoiding introduction of something > > claiming to be yet another global namespace is important > > (i.e., use existing namespaces with global scope in preference > > to inventing new ones). The resulting need to define > > the name spaces/formats in the main iSCSI spec. is > > probably a "feature" as it forces us to pay more > > attention to the sorts of names we use and raises the > > bar for adding additional sorts of names in the future. > > > I will be working with > > the naming and discovery team in my "copious spare time" > > to make sure that we don't lose the valuable work and > > progress they've made to date as a consequence of this > > change. Discussion on the list about what sort > > of names are important (e.g., the Reverse DNS and OUI > > namespaces) and why would be useful. > > From a requirements point of view, please respond if any > of the types of names are important to you (plural). > > Hopefully, we can get this out of the way quickly. > > > > > Thanks, > > --David > > > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 From owner-ips@ece.cmu.edu Fri Mar 23 13:21:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA29993 for ; Fri, 23 Mar 2001 13:21:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NFiEw22736 for ips-outgoing; Fri, 23 Mar 2001 10:44:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NFhBr22673 for ; Fri, 23 Mar 2001 10:43:12 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA215352 for ; Fri, 23 Mar 2001 16:43:01 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA212598 for ; Fri, 23 Mar 2001 16:41:08 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A18.005613DE ; Fri, 23 Mar 2001 16:40:11 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 23 Mar 2001 17:43:06 +0200 Subject: RE: SACK (was RE:Async Message PDU) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk how about SNACK? Julo Black_David@emc.com on 23/03/2001 06:52:20 Please respond to Black_David@emc.com To: marjorie_krueger@hp.com, ips@ece.cmu.edu cc: Subject: RE: SACK (was RE:Async Message PDU) I think picking a new name for SACK to avoid confusion with TCP is an excellent idea. Any suggestions for a new name? --David > -----Original Message----- > From: KRUEGER,MARJORIE (HP-Roseville,ex1) [SMTP:marjorie_krueger@hp.com] > Sent: Thursday, March 22, 2001 7:35 PM > To: 'Black_David@emc.com'; ips@ece.cmu.edu > Subject: RE: SACK (was RE:Async Message PDU) > > David, > Sorry for the delayed reaction, I'm just catching up - You stated that a > decision was made at the Orlando interim meeting to "change some of the > terminology with the goal of each term and acronym being unambiguously > owned > by a single layer in the protocol stack". I think that's the right thing > to > do, so can we agree to rename SACK to something that doesn't cause > confusion > with the TCP construct? > > Marjorie Krueger > Networked Storage Architecture > Networked Storage Solutions Org. > Hewlett-Packard > tel: +1 916 785 2656 > fax: +1 916 785 0391 > email: marjorie_krueger@hp.com > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Thursday, March 01, 2001 11:37 AM > > To: cbm@rose.hp.com; ips@ece.cmu.edu > > Subject: RE: Async Message PDU > > > > > > Julian will doubtless pick up the rest of these, > > but I thought I'd take this issue, since it was > > part of what we did in Orlando. > > > > > 4. Somehow, "Asynchronous Message" seems at odds with the > > rest of the > > > usage in the draft in regards to PDUs (since a message is > > introduced as > > > PDU in section 1.2). Should we may be just call it an AEN > > PDU? Section > > > 2.14.3 calls this so. (I realize that it may not always > > result in a > > > SCSI-level AER.) > > > > One of the things we did in Orlando was to change > > some of the terminology with the goal of each term and > > acronym being unambiguously owned by a single layer > > in the protocol stack, in particular making a sharp > > distinction between SCSI concepts and iSCSI mechanisms. > > > > All the *RN entities changed to *SN for this reason > > (SCSI has a CmdRN, so all *RN entities are SCSI, and > > all *SN entities are iSCSI). Similarly, all AE* > > entities are SCSI, and we went with "Asynchronous > > Message <*>" to name the iSCSI entities. This matters > > here because there are circumstances in which iSCSI > > will send Asynchronous Message PDUs for its own use > > even when SCSI has disabled AER. The usage of "AEN > > PDU" in 2.14.3 is probably an editing oversight. > > Suggestions for words to use instead of "Message" > > are welcome, but they must not start with the letter > > "E" ;-). > > > > Thanks, > > --David > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Fri Mar 23 14:57:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA02572 for ; Fri, 23 Mar 2001 14:57:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NHdJM01079 for ips-outgoing; Fri, 23 Mar 2001 12:39:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NHcdr01033 for ; Fri, 23 Mar 2001 12:38:39 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel2.hp.com (Postfix) with ESMTP id 44FEED95 for ; Fri, 23 Mar 2001 12:38:38 -0500 (EST) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id JAA27940 for ips@ece.cmu.edu; Fri, 23 Mar 2001 09:39:37 -0800 (PST) Message-Id: <200103231739.JAA27940@core.rose.hp.com> Subject: RE: SACK (was RE:Async Message PDU) To: ips@ece.cmu.edu Date: Fri, 23 Mar 2001 9:39:36 PST Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Thanks to Marjorie for raising this question again. How about SRR (Selective Run Request)? -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com >I think picking a new name for SACK to avoid confusion >with TCP is an excellent idea. Any suggestions for a >new name? > >--David > >> -----Original Message----- >> From: KRUEGER,MARJORIE (HP-Roseville,ex1) [SMTP:marjorie_krueger@hp.com] >> Sent: Thursday, March 22, 2001 7:35 PM >> To: 'Black_David@emc.com'; ips@ece.cmu.edu >> Subject: RE: SACK (was RE:Async Message PDU) >> >> David, >> Sorry for the delayed reaction, I'm just catching up - You stated that a >> decision was made at the Orlando interim meeting to "change some of the >> terminology with the goal of each term and acronym being unambiguously >> owned >> by a single layer in the protocol stack". I think that's the right thing >> to >> do, so can we agree to rename SACK to something that doesn't cause >> confusion >> with the TCP construct? >> >> Marjorie Krueger >> Networked Storage Architecture >> Networked Storage Solutions Org. >> Hewlett-Packard >> tel: +1 916 785 2656 >> fax: +1 916 785 0391 >> email: marjorie_krueger@hp.com >> >> > -----Original Message----- >> > From: Black_David@emc.com [mailto:Black_David@emc.com] >> > Sent: Thursday, March 01, 2001 11:37 AM >> > To: cbm@rose.hp.com; ips@ece.cmu.edu >> > Subject: RE: Async Message PDU >> > >> > >> > Julian will doubtless pick up the rest of these, >> > but I thought I'd take this issue, since it was >> > part of what we did in Orlando. >> > >> > > 4. Somehow, "Asynchronous Message" seems at odds with the >> > rest of the >> > > usage in the draft in regards to PDUs (since a message is >> > introduced as >> > > PDU in section 1.2). Should we may be just call it an AEN >> > PDU? Section >> > > 2.14.3 calls this so. (I realize that it may not always >> > result in a >> > > SCSI-level AER.) >> > >> > One of the things we did in Orlando was to change >> > some of the terminology with the goal of each term and >> > acronym being unambiguously owned by a single layer >> > in the protocol stack, in particular making a sharp >> > distinction between SCSI concepts and iSCSI mechanisms. >> > >> > All the *RN entities changed to *SN for this reason >> > (SCSI has a CmdRN, so all *RN entities are SCSI, and >> > all *SN entities are iSCSI). Similarly, all AE* >> > entities are SCSI, and we went with "Asynchronous >> > Message <*>" to name the iSCSI entities. This matters >> > here because there are circumstances in which iSCSI >> > will send Asynchronous Message PDUs for its own use >> > even when SCSI has disabled AER. The usage of "AEN >> > PDU" in 2.14.3 is probably an editing oversight. >> > Suggestions for words to use instead of "Message" >> > are welcome, but they must not start with the letter >> > "E" ;-). >> > >> > Thanks, >> > --David >> > --------------------------------------------------- >> > David L. Black, Senior Technologist >> > EMC Corporation, 42 South St., Hopkinton, MA 01748 >> > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 >> > black_david@emc.com Mobile: +1 (978) 394-7754 >> > --------------------------------------------------- >> > > From owner-ips@ece.cmu.edu Fri Mar 23 14:57:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA02598 for ; Fri, 23 Mar 2001 14:57:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NI0HC02651 for ips-outgoing; Fri, 23 Mar 2001 13:00:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NHxYr02622 for ; Fri, 23 Mar 2001 12:59:34 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 7E4043DE; Fri, 23 Mar 2001 10:59:30 -0700 (MST) Received: from axcsbh1.cos.agilent.com (axcsbh1.cos.agilent.com [130.29.152.143]) by msgrel1.cos.agilent.com (Postfix) with SMTP id 67C52103; Fri, 23 Mar 2001 10:59:30 -0700 (MST) Received: from 130.29.152.143 by axcsbh1.cos.agilent.com (InterScan E-Mail VirusWall NT); Fri, 23 Mar 2001 10:59:30 -0700 (Mountain Standard Time) Received: by axcsbh1.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Fri, 23 Mar 2001 10:59:29 -0700 Message-ID: <1BEBA5E8600DD4119A50009027AF54A00568461B@axcs04.cos.agilent.com> From: pat_thaler@agilent.com To: vince_cavanna@agilent.com, mbakke@cisco.com, jim.williams@emulex.com, ips@ece.cmu.edu Subject: RE: CRC vs CHKSUM presentation slides Date: Fri, 23 Mar 2001 10:59:27 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Vince, Your statement is correct except for the number 12144. 12144 bits is the length of the maximum IEEE 802.3 frame (before we added 4 bytes for VLAN tagging). The polynomial was actually checked for coverage of 3-bit errors out to the Token Ring and FDDI maximum packet sizes which are about 3 times as long. I don't know the length at which the first undetectable 3-bit error occurs, but it is something beyond 32000 bits. I also have trouble with an argument based on leverage. Hardware design of CRC generators and checkers is just not that hard. It's a register plus a bunch of XOR's. Regards, Pat -----Original Message----- From: vince_cavanna@agilent.com [mailto:vince_cavanna@agilent.com] Sent: Thursday, March 22, 2001 4:10 PM To: vince_cavanna@agilent.com; mbakke@cisco.com; jim.williams@emulex.com; ips@ece.cmu.edu Subject: RE: CRC vs CHKSUM presentation slides To save Julian from having to write a memo I would like to add that the ethernet polynomial has a certain weakness for 3 bit errors that the others don't. The ethernet polynomial is only guaranteed to catch all combination sof 3 bit errors when the message length is up to 12144 bits. For larger message lengths some 3 bit errors may get through. Vince |-----Original Message----- |From: CAVANNA,VICENTE V (A-Roseville,ex1) |Sent: Thursday, March 22, 2001 3:42 PM |To: 'Mark Bakke'; 'Jim Williams'; ips@ece.cmu.edu |Cc: CAVANNA,VICENTE V (A-Roseville,ex1) |Subject: RE: CRC vs CHKSUM presentation slides | | |Mark and Jim, |I think any of the 32 bit CRC polynomials that have been |proposed are more than good enough. My main reason for |recommending hte CCITT-CRC32 is that it is half the cost of |the others in terms of gate count - and unless you are doing a |serial divider, which would be too slow, the gate count is |very significant. Also, the leverage may not be as high as you |think unless we are willing to use the datapath width of |existing implementations. For 10 gig we will probably need to |have larger than normal datapath widths. If you change the |datapath width to handle, say, 64 bits at a time you change |the implementation. I would otherwise have no problem with |using the ethernet polynomial. I will try to explain later why |I am not concerned about using the same polynomial and |potentially giving up some cross-checking (I am not quite sure yet). |Vince | ||-----Original Message----- ||From: Jim Williams [mailto:jim.williams@emulex.com] ||Sent: Thursday, March 22, 2001 1:30 PM ||To: ips@ece.cmu.edu ||Subject: Re: CRC vs CHKSUM presentation slides || || ||From: "Mark Bakke" ||To: "CAVANNA,VICENTE V (A-Roseville,ex1)" ||Cc: ; ||Sent: Thursday, March 22, 2001 4:00 PM ||Subject: Re: CRC vs CHKSUM presentation slides || || ||> Vicente- ||> ||> I just took another look through your slides after seeing the ||> presentation on Monday. They were very well-done. I have ||> one question, though. If the CCITT-CRC32 is considered "good ||> enough", then would the Ethernet CRC32 also be good enough? The ||> reason I ask is that every hardware vendor involved in building ||> iSCSI stuff already has implementations of the Ethernet CRC, ||> which is used for both Ethernet and Fibre Channel. ||> ||> The Ethernet poly has more terms than CCITT, and perhaps is ||> not as good as CRC-32C (any thoughts?), but everyone has hardware ||> and software for this, with proven interoperability (bit and ||> byte order, etc). Performance-wise, it will be there for ||> 10Gb Ethernet, so it should be fast enough. ||> ||> So if the Ethernet poly is deemed good enough (even if it's not ||> the best), and fast enough (even if it's not the fastest), why ||> not use it? I think we would stand a much better chance of ||> achieving interoperability in a short time. ||> ||> Please let me know what you think of this; I realize that a few ||> of my questions were speculative. || ||Since the iSCSI messages will often be encapsulated in ethernet ||packets, there is some value to using a different CRC. Link ||errors are double protected with two different CRCs. If ||ethernet and iSCSI use the same polynomial, there is little ||additional coverage against link errors. This point may not ||be decisive, but all other things being equal or almost ||equal, it is worth considering. || | From owner-ips@ece.cmu.edu Fri Mar 23 15:01:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA02705 for ; Fri, 23 Mar 2001 15:01:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NHdMc01083 for ips-outgoing; Fri, 23 Mar 2001 12:39:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hub1.san-jose.webnexus.net (hub1-20.san-jose.webnexus.net [209.140.224.20]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NHcCr01016 for ; Fri, 23 Mar 2001 12:38:13 -0500 (EST) Received: from rmaranon (h-209-140-231-182.webnexus.com [209.140.231.182] (may be forged)) by hub1.san-jose.webnexus.net (8.9.1a/8.9.1/WN-1.4) with SMTP id JAA04315; Fri, 23 Mar 2001 09:37:40 -0800 (PST) From: "Renato E. Maranon" To: Cc: Subject: RE: iSCSI Naming: WWUIs, URNs, and namespaces Date: Fri, 23 Mar 2001 09:36:17 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-reply-to: <3ABB5F59.718AC63D@cisco.com> Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I'm with Mark Bakke on the minimum: - iscsi - The canonical name. - eui - For compatibility with existing device naming schemes. - reverse-dns - For better naming flexibility moving forward. Renato Maranon Maranti Networks, Inc 920 Hillview Court Milpitas, Ca 95035 Phone: 408-719-9600 x309 Fax: 408-719-9631 email: rmaranon@marantinetworks.com home: www.marantinetworks.com -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Mark Bakke Sent: Friday, March 23, 2001 6:36 AM To: Black_David@emc.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI Naming: WWUIs, URNs, and namespaces David- This is starting to seem like work. In case they help, please see my comments below. We can take this up in more detail in the N&D team. -- Mark Black_David@emc.com wrote: > > I don't like naming issues. :-) :-) > > After suitable consulting with some members of > the IESG and IAB, I have some news to convey about > the current approach to iSCSI naming. > > The IESG will not approve another global namespace > for iSCSI's use - this means that WWUIs as currently So I take it that WWUIs are fine, as long as they use existing global name spaces. > designed will need to be revised out of the > naming and discovery draft, and that it will not be > possible to proceed with the WWUI URN draft > as an official IPS WG work item. The best course of > action would probably be to allow the WWUI URN draft > to expire without further revision. > > To a first approximation, WWUIs are/were a "grand > unified theory" of naming, in that any namespace could > be glued into the WWUI world (as several were). The reason we glued these in was to avoid creating a new global name space. We allowed the use of a few currently available global name spaces. How can this be a problem? > The WG is being directed to instead focus on the > individual namespaces and make sure that the ones that > are used are in fact necessary. iSCSI can use text > keys to identify which sort of name is being used > (one key for each sort of format, for each instance > in which a name is used), and it may be possible > to encode the name format in the parse rules for the > values of iSCSI keys to avoid proliferation of keys. > > Taking a look at the namespaces in the current iSCSI > naming and discovery draft, here's some initial > guidance from this WG co-chair: > iscsi - canonical target -- This should be fine. > eui - WWNs -- The use of these for storage makes eminent > sense. I don't see a problem here. So far, so good. Does that mean that we can keep the above two, in the current WWUI, as we have it formatted? EUI, although it's not that flexible (see later), is required for interoperating with and adding iSCSI to existing devices. > dns - hostnames -- Use of these should be abandoned as > not only are they not really URNs (as indicated > in the draft), but their intended usage is straying > into the tarpit known as "URN resolution". Faster > progress will made by staying out. A DNS hostname > can be put into an Alias or something new can be > invented to carry it as a Location Hint, BUT the > relevant URN WG RFCs and drafts on URN resolution > should be reviewed before proceeding too far in this > direction. I don't like this one that much, either, for the same reasons. > iscsi - Reverse DNS and oui - Org. Unique Identifier -- > The rationale for these is not clear to me. > Assuming that WWNs are going to be available for > use in naming iSCSI Initiators and Targets, what > are the problems that these sorts of names solve > that WWNs don't? It appears that one of the problems > may be who can get/create them. Discussion of this > on the list would be appropriate. These two formats use existing global name spaces, and allow the implementor or end user to take a more flexible approach to naming than is offered by the EUI-64. When dealing with a large number of logical targets, burning WWNs (essentially, MAC addresses), and attempting to keep them tied to a logical entity without tying them to hardware is not practical. Both of these formats accomplish the same thing, and we could make do with just one of them. The only real differences are: - OUI is a fixed length naming authority; reverse DNS is variable and could result in longer names. - A human looking at a reverse DNS-based name can easily determine who the naming authority is; a human looking at an OUI-based name will usually have to go look it up at IEEE. - Only a hardware manufacturer (and a few software manufacters) will already have an OUI. Others, such as storage service providers, large end customers, or software driver providers, would either be out-of-luck for naming their devices, or would have to register for an OUI, which would otherwise be a waste of IEEE resources. OTOH, a storage service provider could provide its own names by using the reverse-DNS authority. Everyone has a DNS name. This would enable the SSP to provide its customers with a WWUI that would not change if the back-end storage device was replaced. After all, it's not the device that's important to this customer, it's the DATA. - If we had to pick one or the other, I would pick the reverse DNS, since it offers the ability to generate unique names to a wider variety of users. > In any case, the fewer name formats we have to deal with, > the better. Although I probably should have discussed this with the rest of the NDT first, I would personally be happy with: - iscsi - The canonical name. - eui - For compatibility with existing device naming schemes. - reverse-dns - For better naming flexibility moving forward. Perhaps the IESG would accept the WWUI with just these three. Or perhaps we could just make the WWUI into a URN? I don't know if that would help, but perhaps... The main point is that we are NOT creating any new name spaces; we really are just using what's already there. Maybe by specifying fewer of the options that are already there, we would be in better shape. > I want to try to anticipate an objection to this, which > would note that from a functional viewpoint the basic > impact of this is to move some characters from one text > string to another (e.g., from a WWUI type designator > to part of an iSCSI text key), and wonder if this is > a distinction without a difference. One of the reasons > for the that started this post is that a functional > view is not sufficient for naming - how things are named, > the intended usage of names and their scope matter a lot. So that leads to the question: Has any other WG come up with a world-wide-unique-identifier for something else that the IESG folks deems acceptable? If we can see a few examples, we would be better able to anticipate what they want. > This is particularly true when considering the structure > of a namespace and how that structure may be extended. > The upshot is that avoiding introduction of something > claiming to be yet another global namespace is important > (i.e., use existing namespaces with global scope in preference > to inventing new ones). The resulting need to define > the name spaces/formats in the main iSCSI spec. is > probably a "feature" as it forces us to pay more > attention to the sorts of names we use and raises the > bar for adding additional sorts of names in the future. > I will be working with > the naming and discovery team in my "copious spare time" > to make sure that we don't lose the valuable work and > progress they've made to date as a consequence of this > change. Discussion on the list about what sort > of names are important (e.g., the Reverse DNS and OUI > namespaces) and why would be useful. >From a requirements point of view, please respond if any of the types of names are important to you (plural). Hopefully, we can get this out of the way quickly. > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Fri Mar 23 15:09:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA02573 for ; Fri, 23 Mar 2001 14:57:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NHXGV00699 for ips-outgoing; Fri, 23 Mar 2001 12:33:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcamail03.zca.compaq.com (zcamail03.zca.compaq.com [161.114.32.103]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NHVgr00576 for ; Fri, 23 Mar 2001 12:31:42 -0500 (EST) Received: by zcamail03.zca.compaq.com (Postfix, from userid 12345) id DFBBFD4E; Fri, 23 Mar 2001 09:31:34 -0800 (PST) Received: from exchou-gh02.cca.cpqcorp.net (exchou-gh02.cca.cpqcorp.net [16.110.248.202]) by zcamail03.zca.compaq.com (Postfix) with ESMTP id CD604F46; Fri, 23 Mar 2001 09:31:33 -0800 (PST) Received: by exchou-gh02.cca.cpqcorp.net with Internet Mail Service (5.5.2652.78) id ; Fri, 23 Mar 2001 11:31:34 -0600 Message-ID: <78AF3C342AEAEF4BA33B35A8A15668C659C11F@cceexc17.americas.cpqcorp.net> From: "Elliott, Robert" To: "'ips@ece.cmu.edu'" , "'Mark Bakke'" , "'Black_David@emc.com'" Subject: RE: iSCSI Naming: WWUIs, URNs, and namespaces Date: Fri, 23 Mar 2001 11:31:32 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2652.78) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk The SCSI over RDMA protocol (for InfiniBand) chose: 1. initiator port identifier = 64-bit EUI-64 plus 64-bit identifier extension 2. target port identifier = 64-bit EUI-64 The target port identifier is the same as the InfiniBand IO controller (IOC) globally unique identifier (GUID). (InfiniBand GUIDs are EUI-64s) Targets will often be hardware devices that already have IOC GUIDs. In the cases where they are not, we assume the software provider can provide GUIDs just like it provides license keys. Software could allow customers to specify their own EUI-64s, too. Initiators are more likely to be software entities. We assumed the software could take one of the GUIDs of the hardware on which it is located (e.g. the InfiniBand port GUID) and append some sort of unique identifier to it. It is unspecified how to guarantee software generates unique IDs. If there is a name server on the fabric, it may check if the ID it chooses is already in use and pick another. Alternatively, the software provider could generate its own EUI-64 or the customer could provide one (as with targets implemented in software). I proposed using the iSCSI reverse DNS WWUI format so SRP and iSCSI could share a namespace, but the group preferred the EUI-64 based approach (among other concerns, the 255 byte string is too big for some of the packets). If iSCSI drops the long WWUI format and goes with EUI-64, we may achieve the common namespace goal. --- Rob Elliott, Compaq Server Storage Robert.Elliott@compaq.com > -----Original Message----- > From: Mark Bakke [mailto:mbakke@cisco.com] > Sent: Friday, March 23, 2001 8:36 AM > To: Black_David@emc.com > Cc: ips@ece.cmu.edu > Subject: Re: iSCSI Naming: WWUIs, URNs, and namespaces > > > > David- > > This is starting to seem like work. In case they help, > please see my comments below. We can take this up in > more detail in the N&D team. > > -- > Mark > > Black_David@emc.com wrote: > > > > I don't like naming issues. :-) :-) > > > > After suitable consulting with some members of > > the IESG and IAB, I have some news to convey about > > the current approach to iSCSI naming. > > > > The IESG will not approve another global namespace > > for iSCSI's use - this means that WWUIs as currently > > So I take it that WWUIs are fine, as long as they use > existing global name spaces. > > > designed will need to be revised out of the > > naming and discovery draft, and that it will not be > > possible to proceed with the WWUI URN draft > > as an official IPS WG work item. The best course of > > action would probably be to allow the WWUI URN draft > > to expire without further revision. > > > > To a first approximation, WWUIs are/were a "grand > > unified theory" of naming, in that any namespace could > > be glued into the WWUI world (as several were). > > The reason we glued these in was to avoid creating a new > global name space. We allowed the use of a few currently > available global name spaces. How can this be a problem? > > > The WG is being directed to instead focus on the > > individual namespaces and make sure that the ones that > > are used are in fact necessary. iSCSI can use text > > keys to identify which sort of name is being used > > (one key for each sort of format, for each instance > > in which a name is used), and it may be possible > > to encode the name format in the parse rules for the > > values of iSCSI keys to avoid proliferation of keys. > > > > Taking a look at the namespaces in the current iSCSI > > naming and discovery draft, here's some initial > > guidance from this WG co-chair: > > iscsi - canonical target -- This should be fine. > > eui - WWNs -- The use of these for storage makes eminent > > sense. I don't see a problem here. > > So far, so good. Does that mean that we can keep the above > two, in the current WWUI, as we have it formatted? > > EUI, although it's not that flexible (see later), is required > for interoperating with and adding iSCSI to existing devices. > > > dns - hostnames -- Use of these should be abandoned as > > not only are they not really URNs (as indicated > > in the draft), but their intended usage is straying > > into the tarpit known as "URN resolution". Faster > > progress will made by staying out. A DNS hostname > > can be put into an Alias or something new can be > > invented to carry it as a Location Hint, BUT the > > relevant URN WG RFCs and drafts on URN resolution > > should be reviewed before proceeding too far in this > > direction. > > I don't like this one that much, either, for the same reasons. > > > iscsi - Reverse DNS and oui - Org. Unique Identifier -- > > The rationale for these is not clear to me. > > Assuming that WWNs are going to be available for > > use in naming iSCSI Initiators and Targets, what > > are the problems that these sorts of names solve > > that WWNs don't? It appears that one of the problems > > may be who can get/create them. Discussion of this > > on the list would be appropriate. > > These two formats use existing global name spaces, and allow > the implementor or end user to take a more flexible approach > to naming than is offered by the EUI-64. When dealing with > a large number of logical targets, burning WWNs (essentially, > MAC addresses), and attempting to keep them tied to a logical > entity without tying them to hardware is not practical. > > Both of these formats accomplish the same thing, and we could > make do with just one of them. The only real differences are: > > - OUI is a fixed length naming authority; reverse DNS is variable > and could result in longer names. > > - A human looking at a reverse DNS-based name can easily determine > who the naming authority is; a human looking at an OUI-based > name will usually have to go look it up at IEEE. > > - Only a hardware manufacturer (and a few software manufacters) > will already have an OUI. Others, such as storage service > providers, large end customers, or software driver providers, > would either be out-of-luck for naming their devices, or would > have to register for an OUI, which would otherwise be a waste > of IEEE resources. OTOH, a storage service provider could > provide its own names by using the reverse-DNS authority. > Everyone has a DNS name. This would enable the SSP to provide > its customers with a WWUI that would not change if the back-end > storage device was replaced. After all, it's not the device > that's important to this customer, it's the DATA. > > - If we had to pick one or the other, I would pick the reverse > DNS, since it offers the ability to generate unique names to > a wider variety of users. > > > In any case, the fewer name formats we have to deal with, > > the better. > > Although I probably should have discussed this with the rest > of the NDT first, I would personally be happy with: > > - iscsi - The canonical name. > - eui - For compatibility with existing device naming schemes. > - reverse-dns - For better naming flexibility moving forward. > > Perhaps the IESG would accept the WWUI with just these three. > Or perhaps we could just make the WWUI into a URN? I don't > know if that would help, but perhaps... > > The main point is that we are NOT creating any new name > spaces; we really are just using what's already there. Maybe > by specifying fewer of the options that are already there, we > would be in better shape. > > > I want to try to anticipate an objection to this, which > > would note that from a functional viewpoint the basic > > impact of this is to move some characters from one text > > string to another (e.g., from a WWUI type designator > > to part of an iSCSI text key), and wonder if this is > > a distinction without a difference. One of the reasons > > for the that started this post is that a functional > > view is not sufficient for naming - how things are named, > > the intended usage of names and their scope matter a lot. > > So that leads to the question: Has any other WG come up with > a world-wide-unique-identifier for something else that the > IESG folks deems acceptable? If we can see a few examples, > we would be better able to anticipate what they want. > > > This is particularly true when considering the structure > > of a namespace and how that structure may be extended. > > The upshot is that avoiding introduction of something > > claiming to be yet another global namespace is important > > (i.e., use existing namespaces with global scope in preference > > to inventing new ones). The resulting need to define > > the name spaces/formats in the main iSCSI spec. is > > probably a "feature" as it forces us to pay more > > attention to the sorts of names we use and raises the > > bar for adding additional sorts of names in the future. > > > I will be working with > > the naming and discovery team in my "copious spare time" > > to make sure that we don't lose the valuable work and > > progress they've made to date as a consequence of this > > change. Discussion on the list about what sort > > of names are important (e.g., the Reverse DNS and OUI > > namespaces) and why would be useful. > > From a requirements point of view, please respond if any > of the types of names are important to you (plural). > > Hopefully, we can get this out of the way quickly. > > > > > Thanks, > > --David > > > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 > From owner-ips@ece.cmu.edu Fri Mar 23 16:10:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA03914 for ; Fri, 23 Mar 2001 16:10:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NJ8Ju07417 for ips-outgoing; Fri, 23 Mar 2001 14:08:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NJ7Fr07324 for ; Fri, 23 Mar 2001 14:07:15 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA57876; Fri, 23 Mar 2001 14:07:19 -0500 Received: from f4n49e (d03nm065h.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id MAA152454; Fri, 23 Mar 2001 12:07:05 -0700 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI Naming: WWUIs, URNs, and namespaces To: "Jim Hafner" Cc: "Elliott, Robert" , "'ips@ece.cmu.edu'" , "'Mark Bakke'" , "'Black_David@emc.com'" X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Fri, 23 Mar 2001 11:06:39 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/23/2001 12:07:05 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, You are NOT wrong. Things got a lot simpler after we came up with the WWUIs. Trying to tie ourselves to HW caused all sorts of problems. (in Fibre Channel also.) The ability to have a higher order name permitted us the ability to have Security Authentication against something that made sense. We have all moved or swapped HBAs from one system to another (especially within I/T supported groups), the last thing we would want to do is have to reestablish across some network a new WWUI and Security context, just because we changed our NICs. Besides can you see the additional work that would happen on the admin side. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com Jim Hafner/Almaden/IBM@IBMUS@ece.cmu.edu on 03/23/2001 10:34:06 AM Sent by: owner-ips@ece.cmu.edu To: "Elliott, Robert" cc: "'ips@ece.cmu.edu'" , "'Mark Bakke'" , "'Black_David@emc.com'" Subject: RE: iSCSI Naming: WWUIs, URNs, and namespaces Rob, Admittedly a lofty goal for a common namespace, but the problem we faced in iSCSI is that (primarily) on the initiator side there is NO canonical hardware component from which to derive an EUI-64+extra name. Some systems have motherboard identifiers, some have network mac addresses, some have ... I know of no common hardware component that lives on every single IP device (including my PDA, my phone, my...). Additionally, on the target side, the virtualization of target devices through gateways (e.g., what an SSP might set up) and the like (thinking of the device as a service and not a hardware component) made a more user-manageable (less hardware driven) namespace a better alternative. I could be wrong on this "no canonical hardware component" point, and I'd love to be corrected as it would make this a lot simpler. Jim Hafner "Elliott, Robert" @ece.cmu.edu on 03-23-2001 09:31:32 AM Sent by: owner-ips@ece.cmu.edu To: "'ips@ece.cmu.edu'" , "'Mark Bakke'" , "'Black_David@emc.com'" cc: Subject: RE: iSCSI Naming: WWUIs, URNs, and namespaces The SCSI over RDMA protocol (for InfiniBand) chose: 1. initiator port identifier = 64-bit EUI-64 plus 64-bit identifier extension 2. target port identifier = 64-bit EUI-64 The target port identifier is the same as the InfiniBand IO controller (IOC) globally unique identifier (GUID). (InfiniBand GUIDs are EUI-64s) Targets will often be hardware devices that already have IOC GUIDs. In the cases where they are not, we assume the software provider can provide GUIDs just like it provides license keys. Software could allow customers to specify their own EUI-64s, too. Initiators are more likely to be software entities. We assumed the software could take one of the GUIDs of the hardware on which it is located (e.g. the InfiniBand port GUID) and append some sort of unique identifier to it. It is unspecified how to guarantee software generates unique IDs. If there is a name server on the fabric, it may check if the ID it chooses is already in use and pick another. Alternatively, the software provider could generate its own EUI-64 or the customer could provide one (as with targets implemented in software). I proposed using the iSCSI reverse DNS WWUI format so SRP and iSCSI could share a namespace, but the group preferred the EUI-64 based approach (among other concerns, the 255 byte string is too big for some of the packets). If iSCSI drops the long WWUI format and goes with EUI-64, we may achieve the common namespace goal. --- Rob Elliott, Compaq Server Storage Robert.Elliott@compaq.com > -----Original Message----- > From: Mark Bakke [mailto:mbakke@cisco.com] > Sent: Friday, March 23, 2001 8:36 AM > To: Black_David@emc.com > Cc: ips@ece.cmu.edu > Subject: Re: iSCSI Naming: WWUIs, URNs, and namespaces > > > > David- > > This is starting to seem like work. In case they help, > please see my comments below. We can take this up in > more detail in the N&D team. > > -- > Mark > > Black_David@emc.com wrote: > > > > I don't like naming issues. :-) :-) > > > > After suitable consulting with some members of > > the IESG and IAB, I have some news to convey about > > the current approach to iSCSI naming. > > > > The IESG will not approve another global namespace > > for iSCSI's use - this means that WWUIs as currently > > So I take it that WWUIs are fine, as long as they use > existing global name spaces. > > > designed will need to be revised out of the > > naming and discovery draft, and that it will not be > > possible to proceed with the WWUI URN draft > > as an official IPS WG work item. The best course of > > action would probably be to allow the WWUI URN draft > > to expire without further revision. > > > > To a first approximation, WWUIs are/were a "grand > > unified theory" of naming, in that any namespace could > > be glued into the WWUI world (as several were). > > The reason we glued these in was to avoid creating a new > global name space. We allowed the use of a few currently > available global name spaces. How can this be a problem? > > > The WG is being directed to instead focus on the > > individual namespaces and make sure that the ones that > > are used are in fact necessary. iSCSI can use text > > keys to identify which sort of name is being used > > (one key for each sort of format, for each instance > > in which a name is used), and it may be possible > > to encode the name format in the parse rules for the > > values of iSCSI keys to avoid proliferation of keys. > > > > Taking a look at the namespaces in the current iSCSI > > naming and discovery draft, here's some initial > > guidance from this WG co-chair: > > iscsi - canonical target -- This should be fine. > > eui - WWNs -- The use of these for storage makes eminent > > sense. I don't see a problem here. > > So far, so good. Does that mean that we can keep the above > two, in the current WWUI, as we have it formatted? > > EUI, although it's not that flexible (see later), is required > for interoperating with and adding iSCSI to existing devices. > > > dns - hostnames -- Use of these should be abandoned as > > not only are they not really URNs (as indicated > > in the draft), but their intended usage is straying > > into the tarpit known as "URN resolution". Faster > > progress will made by staying out. A DNS hostname > > can be put into an Alias or something new can be > > invented to carry it as a Location Hint, BUT the > > relevant URN WG RFCs and drafts on URN resolution > > should be reviewed before proceeding too far in this > > direction. > > I don't like this one that much, either, for the same reasons. > > > iscsi - Reverse DNS and oui - Org. Unique Identifier -- > > The rationale for these is not clear to me. > > Assuming that WWNs are going to be available for > > use in naming iSCSI Initiators and Targets, what > > are the problems that these sorts of names solve > > that WWNs don't? It appears that one of the problems > > may be who can get/create them. Discussion of this > > on the list would be appropriate. > > These two formats use existing global name spaces, and allow > the implementor or end user to take a more flexible approach > to naming than is offered by the EUI-64. When dealing with > a large number of logical targets, burning WWNs (essentially, > MAC addresses), and attempting to keep them tied to a logical > entity without tying them to hardware is not practical. > > Both of these formats accomplish the same thing, and we could > make do with just one of them. The only real differences are: > > - OUI is a fixed length naming authority; reverse DNS is variable > and could result in longer names. > > - A human looking at a reverse DNS-based name can easily determine > who the naming authority is; a human looking at an OUI-based > name will usually have to go look it up at IEEE. > > - Only a hardware manufacturer (and a few software manufacters) > will already have an OUI. Others, such as storage service > providers, large end customers, or software driver providers, > would either be out-of-luck for naming their devices, or would > have to register for an OUI, which would otherwise be a waste > of IEEE resources. OTOH, a storage service provider could > provide its own names by using the reverse-DNS authority. > Everyone has a DNS name. This would enable the SSP to provide > its customers with a WWUI that would not change if the back-end > storage device was replaced. After all, it's not the device > that's important to this customer, it's the DATA. > > - If we had to pick one or the other, I would pick the reverse > DNS, since it offers the ability to generate unique names to > a wider variety of users. > > > In any case, the fewer name formats we have to deal with, > > the better. > > Although I probably should have discussed this with the rest > of the NDT first, I would personally be happy with: > > - iscsi - The canonical name. > - eui - For compatibility with existing device naming schemes. > - reverse-dns - For better naming flexibility moving forward. > > Perhaps the IESG would accept the WWUI with just these three. > Or perhaps we could just make the WWUI into a URN? I don't > know if that would help, but perhaps... > > The main point is that we are NOT creating any new name > spaces; we really are just using what's already there. Maybe > by specifying fewer of the options that are already there, we > would be in better shape. > > > I want to try to anticipate an objection to this, which > > would note that from a functional viewpoint the basic > > impact of this is to move some characters from one text > > string to another (e.g., from a WWUI type designator > > to part of an iSCSI text key), and wonder if this is > > a distinction without a difference. One of the reasons > > for the that started this post is that a functional > > view is not sufficient for naming - how things are named, > > the intended usage of names and their scope matter a lot. > > So that leads to the question: Has any other WG come up with > a world-wide-unique-identifier for something else that the > IESG folks deems acceptable? If we can see a few examples, > we would be better able to anticipate what they want. > > > This is particularly true when considering the structure > > of a namespace and how that structure may be extended. > > The upshot is that avoiding introduction of something > > claiming to be yet another global namespace is important > > (i.e., use existing namespaces with global scope in preference > > to inventing new ones). The resulting need to define > > the name spaces/formats in the main iSCSI spec. is > > probably a "feature" as it forces us to pay more > > attention to the sorts of names we use and raises the > > bar for adding additional sorts of names in the future. > > > I will be working with > > the naming and discovery team in my "copious spare time" > > to make sure that we don't lose the valuable work and > > progress they've made to date as a consequence of this > > change. Discussion on the list about what sort > > of names are important (e.g., the Reverse DNS and OUI > > namespaces) and why would be useful. > > From a requirements point of view, please respond if any > of the types of names are important to you (plural). > > Hopefully, we can get this out of the way quickly. > > > > > Thanks, > > --David > > > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 > From owner-ips@ece.cmu.edu Fri Mar 23 16:10:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA03909 for ; Fri, 23 Mar 2001 16:10:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NJ4Pe07100 for ips-outgoing; Fri, 23 Mar 2001 14:04:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NJ4Cr07081 for ; Fri, 23 Mar 2001 14:04:12 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id OAA09017; Fri, 23 Mar 2001 14:04:02 -0500 (EST) Message-ID: <3ABB9E0D.536203C1@cisco.com> Date: Fri, 23 Mar 2001 13:03:41 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Jim Hafner CC: "Elliott, Robert" , "'ips@ece.cmu.edu'" , "'Black_David@emc.com'" Subject: Re: iSCSI Naming: WWUIs, URNs, and namespaces References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Absolutely. The other problem with using the EUI-64+extra is that the naming authority is the OUI within the EUI, which does not hold for OUI-vs-reverse-DNS argument I stated earlier. Also, since the EUI-64+ extra is a binary address, it does not do a very good job of fulfilling the URN "human transcribability" requirement; the reverse-DNS has a chance to do this better. Rob- perhaps the IB folks would re-consider your proposal to use the iSCSI WWUI given these arguments? My guess is that the people currently focussed on IB are the hardware/firmware folks, and that the higher-level software folks are not really involved yet to support what you had requested. If we could specify the size of the naming authority down to some smaller number (say 64+4 bytes "com.whatever64"), and specify that whatever comes after that fits in 128 bytes, it might be possible to get something a little smaller, even though it technically breaks the DNS name maximums. I would also guess that IB only uses these when connecting to a SCSI device, and that these would not appear in normal command transport headers anyway. Is that correct? Hope this helps, Mark Jim Hafner wrote: > > Rob, > > Admittedly a lofty goal for a common namespace, but the problem we faced in > iSCSI is that (primarily) on the initiator side there is NO canonical > hardware component from which to derive an EUI-64+extra name. Some > systems have motherboard identifiers, some have network mac addresses, some > have ... I know of no common hardware component that lives on every single > IP device (including my PDA, my phone, my...). Additionally, on the target > side, the virtualization of target devices through gateways (e.g., what an > SSP might set up) and the like (thinking of the device as a service and not > a hardware component) made a more user-manageable (less hardware driven) > namespace a better alternative. > > I could be wrong on this "no canonical hardware component" point, and I'd > love to be corrected as it would make this a lot simpler. > > Jim Hafner > > "Elliott, Robert" @ece.cmu.edu on 03-23-2001 > 09:31:32 AM > > Sent by: owner-ips@ece.cmu.edu > > To: "'ips@ece.cmu.edu'" , "'Mark Bakke'" > , "'Black_David@emc.com'" > cc: > Subject: RE: iSCSI Naming: WWUIs, URNs, and namespaces > > The SCSI over RDMA protocol (for InfiniBand) chose: > > 1. initiator port identifier = 64-bit EUI-64 plus > 64-bit identifier extension > 2. target port identifier = 64-bit EUI-64 > > The target port identifier is the same as the InfiniBand > IO controller (IOC) globally unique identifier (GUID). > (InfiniBand GUIDs are EUI-64s) > > Targets will often be hardware devices that already > have IOC GUIDs. In the cases where they are not, we > assume the software provider can provide GUIDs just > like it provides license keys. Software could allow > customers to specify their own EUI-64s, too. > > Initiators are more likely to be software entities. > We assumed the software could take one of the GUIDs > of the hardware on which it is located (e.g. the > InfiniBand port GUID) and append some sort of unique > identifier to it. It is unspecified how to guarantee > software generates unique IDs. If there is a name > server on the fabric, it may check if the ID it chooses > is already in use and pick another. Alternatively, > the software provider could generate its own EUI-64 > or the customer could provide one (as with targets > implemented in software). > > I proposed using the iSCSI reverse DNS WWUI format so > SRP and iSCSI could share a namespace, but the group > preferred the EUI-64 based approach (among other > concerns, the 255 byte string is too big for some of > the packets). If iSCSI drops the long WWUI format > and goes with EUI-64, we may achieve the common > namespace goal. > > --- > Rob Elliott, Compaq Server Storage > Robert.Elliott@compaq.com > > > -----Original Message----- > > From: Mark Bakke [mailto:mbakke@cisco.com] > > Sent: Friday, March 23, 2001 8:36 AM > > To: Black_David@emc.com > > Cc: ips@ece.cmu.edu > > Subject: Re: iSCSI Naming: WWUIs, URNs, and namespaces > > > > > > > > David- > > > > This is starting to seem like work. In case they help, > > please see my comments below. We can take this up in > > more detail in the N&D team. > > > > -- > > Mark > > > > Black_David@emc.com wrote: > > > > > > I don't like naming issues. :-) :-) > > > > > > After suitable consulting with some members of > > > the IESG and IAB, I have some news to convey about > > > the current approach to iSCSI naming. > > > > > > The IESG will not approve another global namespace > > > for iSCSI's use - this means that WWUIs as currently > > > > So I take it that WWUIs are fine, as long as they use > > existing global name spaces. > > > > > designed will need to be revised out of the > > > naming and discovery draft, and that it will not be > > > possible to proceed with the WWUI URN draft > > > as an official IPS WG work item. The best course of > > > action would probably be to allow the WWUI URN draft > > > to expire without further revision. > > > > > > To a first approximation, WWUIs are/were a "grand > > > unified theory" of naming, in that any namespace could > > > be glued into the WWUI world (as several were). > > > > The reason we glued these in was to avoid creating a new > > global name space. We allowed the use of a few currently > > available global name spaces. How can this be a problem? > > > > > The WG is being directed to instead focus on the > > > individual namespaces and make sure that the ones that > > > are used are in fact necessary. iSCSI can use text > > > keys to identify which sort of name is being used > > > (one key for each sort of format, for each instance > > > in which a name is used), and it may be possible > > > to encode the name format in the parse rules for the > > > values of iSCSI keys to avoid proliferation of keys. > > > > > > Taking a look at the namespaces in the current iSCSI > > > naming and discovery draft, here's some initial > > > guidance from this WG co-chair: > > > iscsi - canonical target -- This should be fine. > > > eui - WWNs -- The use of these for storage makes eminent > > > sense. I don't see a problem here. > > > > So far, so good. Does that mean that we can keep the above > > two, in the current WWUI, as we have it formatted? > > > > EUI, although it's not that flexible (see later), is required > > for interoperating with and adding iSCSI to existing devices. > > > > > dns - hostnames -- Use of these should be abandoned as > > > not only are they not really URNs (as indicated > > > in the draft), but their intended usage is straying > > > into the tarpit known as "URN resolution". Faster > > > progress will made by staying out. A DNS hostname > > > can be put into an Alias or something new can be > > > invented to carry it as a Location Hint, BUT the > > > relevant URN WG RFCs and drafts on URN resolution > > > should be reviewed before proceeding too far in this > > > direction. > > > > I don't like this one that much, either, for the same reasons. > > > > > iscsi - Reverse DNS and oui - Org. Unique Identifier -- > > > The rationale for these is not clear to me. > > > Assuming that WWNs are going to be available for > > > use in naming iSCSI Initiators and Targets, what > > > are the problems that these sorts of names solve > > > that WWNs don't? It appears that one of the problems > > > may be who can get/create them. Discussion of this > > > on the list would be appropriate. > > > > These two formats use existing global name spaces, and allow > > the implementor or end user to take a more flexible approach > > to naming than is offered by the EUI-64. When dealing with > > a large number of logical targets, burning WWNs (essentially, > > MAC addresses), and attempting to keep them tied to a logical > > entity without tying them to hardware is not practical. > > > > Both of these formats accomplish the same thing, and we could > > make do with just one of them. The only real differences are: > > > > - OUI is a fixed length naming authority; reverse DNS is variable > > and could result in longer names. > > > > - A human looking at a reverse DNS-based name can easily determine > > who the naming authority is; a human looking at an OUI-based > > name will usually have to go look it up at IEEE. > > > > - Only a hardware manufacturer (and a few software manufacters) > > will already have an OUI. Others, such as storage service > > providers, large end customers, or software driver providers, > > would either be out-of-luck for naming their devices, or would > > have to register for an OUI, which would otherwise be a waste > > of IEEE resources. OTOH, a storage service provider could > > provide its own names by using the reverse-DNS authority. > > Everyone has a DNS name. This would enable the SSP to provide > > its customers with a WWUI that would not change if the back-end > > storage device was replaced. After all, it's not the device > > that's important to this customer, it's the DATA. > > > > - If we had to pick one or the other, I would pick the reverse > > DNS, since it offers the ability to generate unique names to > > a wider variety of users. > > > > > In any case, the fewer name formats we have to deal with, > > > the better. > > > > Although I probably should have discussed this with the rest > > of the NDT first, I would personally be happy with: > > > > - iscsi - The canonical name. > > - eui - For compatibility with existing device naming schemes. > > - reverse-dns - For better naming flexibility moving forward. > > > > Perhaps the IESG would accept the WWUI with just these three. > > Or perhaps we could just make the WWUI into a URN? I don't > > know if that would help, but perhaps... > > > > The main point is that we are NOT creating any new name > > spaces; we really are just using what's already there. Maybe > > by specifying fewer of the options that are already there, we > > would be in better shape. > > > > > I want to try to anticipate an objection to this, which > > > would note that from a functional viewpoint the basic > > > impact of this is to move some characters from one text > > > string to another (e.g., from a WWUI type designator > > > to part of an iSCSI text key), and wonder if this is > > > a distinction without a difference. One of the reasons > > > for the that started this post is that a functional > > > view is not sufficient for naming - how things are named, > > > the intended usage of names and their scope matter a lot. > > > > So that leads to the question: Has any other WG come up with > > a world-wide-unique-identifier for something else that the > > IESG folks deems acceptable? If we can see a few examples, > > we would be better able to anticipate what they want. > > > > > This is particularly true when considering the structure > > > of a namespace and how that structure may be extended. > > > The upshot is that avoiding introduction of something > > > claiming to be yet another global namespace is important > > > (i.e., use existing namespaces with global scope in preference > > > to inventing new ones). The resulting need to define > > > the name spaces/formats in the main iSCSI spec. is > > > probably a "feature" as it forces us to pay more > > > attention to the sorts of names we use and raises the > > > bar for adding additional sorts of names in the future. > > > > > I will be working with > > > the naming and discovery team in my "copious spare time" > > > to make sure that we don't lose the valuable work and > > > progress they've made to date as a consequence of this > > > change. Discussion on the list about what sort > > > of names are important (e.g., the Reverse DNS and OUI > > > namespaces) and why would be useful. > > > > From a requirements point of view, please respond if any > > of the types of names are important to you (plural). > > > > Hopefully, we can get this out of the way quickly. > > > > > > > > > Thanks, > > > --David > > > > > > --------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > --------------------------------------------------- > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 > > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Fri Mar 23 16:10:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA03963 for ; Fri, 23 Mar 2001 16:10:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NIqJ706273 for ips-outgoing; Fri, 23 Mar 2001 13:52:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NIq2r06257 for ; Fri, 23 Mar 2001 13:52:05 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id TAA338704 for ; Fri, 23 Mar 2001 19:46:35 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id TAA49662 for ; Fri, 23 Mar 2001 19:44:04 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A18.0066E7B8 ; Fri, 23 Mar 2001 19:43:59 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 23 Mar 2001 20:46:42 +0200 Subject: resend note Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, I sent a reply to you without sending to the list and without keeping a copy. Please redistribute. Thanks, Julo From owner-ips@ece.cmu.edu Fri Mar 23 16:10:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA03964 for ; Fri, 23 Mar 2001 16:10:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NIqNl06276 for ips-outgoing; Fri, 23 Mar 2001 13:52:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NIlXr05917 for ; Fri, 23 Mar 2001 13:48:58 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id TAA420706; Fri, 23 Mar 2001 19:44:30 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id TAA30496; Fri, 23 Mar 2001 19:41:58 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A18.0066B6BA ; Fri, 23 Mar 2001 19:41:54 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Matt Wakeley cc: ips@ece.cmu.edu Message-ID: Date: Fri, 23 Mar 2001 20:44:46 +0200 Subject: Re: iscsi: technical comments to iSCSI 05: Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Answers in text. Regards, Julo Matt Wakeley on 23/03/2001 01:02:14 Please respond to Matt Wakeley To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Re: iscsi: technical comments to iSCSI 05: Julian, Why haven't you responded to my comments? You seem to respond to everyone elses. Does no response mean you agree? -Matt Matt Wakeley wrote: > > > > ------------------------------------------------------------------------------ > technical comments to iSCSI 05: > > Global: (especially in section 6) timeouts are not defined. We went > through a lot of work defining time-out values for FCP-2. I think > that time-out values need to be specified for interoperability. +++ probably. Please note also that we had ages ago some timers and people did not find them usefull. Considering the variability in network diameters we will have to either have large values or set them based or RTT (complex) +++ > > 1.2, page 10: > "The iSCSI protocol is a mapping of the SCSI remote procedure > invocation model on top of the TCP protocol." > is iSCSI limited to TCP? Seems like it could be implemented over > other transports, such as SCTP with no modification... > +++ I am sure we can but on SCTP other mappings would be more effective (numbering, digests, marking etc.). In SCTP many of those are already in. Some others- like security would have been more complex +++ > > 1.2.3, page 14: > After the sentance "Otherwise, it sends a response with > a "login reject" ..." > add that the connection is terminated. > +++ did +++ > 1.2.5, page 16 (middle) and 6.7.1, page 84: > The initiator should not be required ("MUST") to honor R2Ts. > Conditions may have occurred that might prevent the initiator > from processing the R2T, in which case appropriate error recovery > should be performed. I suggest "SHOULD if at all possible...". > +++ did +++ > 1.2.5, page 16 (bottom): > "Unsolicited data MUST be sent on every connection in the same order > in which commands were sent." - must the data be sent immediately after > the command, or can other commands be sent before the data? > +++ "were sent" does not imply that commands precede the data and that is the only ordering required - i.e., other commands can be sent before the data +++ > 1.2.7, page 19, middle of page > must "iSCSI://..." be caps? is "iscsi://..." allowed? > +++ the syntax is the same as for URNs -- see the NDT doc +++ > 2.2.3, page 28: > Are "what's next" fields included in the header digest? It's not > stated that they are. > > +++ will dissapear as a separate entity - and yes they are included +++ > 2.7.2, page 45: > Why must a LUN be associated with a Target Transfer Tag? > +++ I though we had this settled. The reason is to allow targets to set their Tags independent of each other and help the target steer the data without looking at Tags. if the device server is located with the LU. The alternative would be to declare the whole LU+Tag in R2T and Data Out as an extended tag and let every target use it as they wish - no mandatory LU correctness check +++ > 2.7.4, page 45: > It is not clear if the DataSN restarts from zero for each R2T. > +++ will fix wording +++ > 2.7.6, page 46: > It is not stated, but seems implied that a bi-di commands must > send status in a separate response PDU. If this is so, please > state it. > +++ will fix although not very relevant now as Bidi are supposed to be input-followed-by-output - but this may change +++ > 2.10.9, page 52: > What is the default maximum length for login parameters? +++ the default text command length currently defined as one DataPDU and one data PDULength is defaulted to 8K +++ > > 2.15, page 63: > What is the initiator supposed to do when it receives a code of > "cleanup failed"? It seems like it's the target's problem and > should not be reported to the initiator. > +++ It is supposed to do a reset +++ > 2.17, page 66: > *if* as you state in 2.7.2 a LUN must be sent when a target transfer > tag is specified, shouldn't the LUN be also specified in the R2T? > +++ see above - the target may have device servers that do not need to know what they are (they may get this information from the command but they don't need it). On the other hand the initiator has always this information. +++ > 2.20, page 71: > remove "reserved fields not 0". This makes it sound like > reserved fields must be checked, which they should not must > be checked. +++ I did. But this raises an interesting side-question. It was said that checking for reserved fiels being 0 is a good practice. It enables easy migration from version to version +++ > > 2.20, page 71: > add "data digest error" to the list of reject reasons. +++ it is there as cause 3. Is the name confusing? +++ > > 3.1.2 and 3.1.3, page 72: > I don't care what SCSI has defined for it's mode pages, for iSCSI, > these fields should be byte values, not multiples of 512. > +++ the current field size is 16 bit. Do we want to limit ourselves to 64k PDUs? +++ > 4.3, page 78: > It may not be clear to everyone that a target can send a text or > login response to a login or text command. That is, a target could > respond to a text command with a login response. This should be > made more clear. > > 6.2, page 81, last paragraph: > Can a "restart" also be issued in this case? > +++ made it lowercase must -:)+++ > Appendix A: 01, page 94: > I don't think that crc-64 "MUST" be implemented. > I also don't agree with the phrase "Cyclic codes are particularly > well suited for hardware implementations." This is not true if > iSCSI PDUs span TCP segments and arrive out of order. > +++ fine on crc64 - on the rest if you do on the DMA to host it is still decent+++ > Appendix A: 01, page 94: > "Implementations MAY also negotiate digests with security > significance for data authentication and integrity as detailed > in the following table:" needs a lot more description. > +++ what for example? (copy the reference?) +++ > Appendix C: page 104: > change: "will leave at least" to "should leave at least". > +++ fixed +++ > Appendix C: 06, page 105: > change: "When reduced to iSCSI terms markers MUST point to..." > to: "When reduced to iSCSI terms markers MUST indicate the offset to..." > +++ fixed ++ > Appendix C: 07, page 105: > "The marker-less interval is not less than 4 kbytes and the > default is 4 kbytes." It should be whatever the the receiver > needs it to be (not min 4K). > +++ how should a send know? we don't want this to kick in before end of login +++ > Appendix C: 17,18 page 109: > The sender should send markers at whatever interval the reciever > requires them. Not the "larger" of the sender and receiver. > Otherwise, the sender could say 0xffffffff and it would be of no > use to the receiver. Also, the default should be "none", not 2050. +++ imposed by receiver is also not palatable. How about the receiver sending a min and a max and the receiver selecting beween them? as in: 01 RFMarkInt RFMarkInt=[,] This is a connection specific parameter. The receiver indicates the minimum to maximum interval (in 4-byte words) the receiver wants the markers. The sender selects a value within the minimum and maximum the receiver requires or indicates through the FMarker key=value its inability to set markers. The interval is measured from the beginning of a marker to the beginning of the next marker. For example, a value of 1026 means 1026 words (4096 bytes of "pure" payload between markers). Default is 2050. 02 SFMarkInt SFMarkInt= This is a connection specific parameter. Indicates at what interval (in 4-byte words) the sender accepts to send the markers. The number MUST be within the range required by the receiver. The interval is measured from the beginning of a marker to the beginning of the next marker. For example, a value of 1026 means 1026 words (4096 bytes of "pure" payload between markers). Default is 2050. +++ > > Appendix C: 22 page 111: > Indicate what "no,no" and "yes,no" means. > +++ OK +++ > Appendix C: 23 page 111: > This parameter should be in bytes, not multiples of 512, > especially since framing mechanisms may require that a PDU not be > larger than MSS. (perhaps -1 can be used to indicate MSS) > +++ the field length in the mode page is limited +++ > Appendix C: 26 page 112: > This should be deleted, since CmdSN is now mandatory. +++ that is the SAM2 CmdRN +++ > > Appendix C: 27 page 112: > PingMaxReplyLength is both a target and initiator negotiated value. > Therefore, text saying "the target MUST..." should be change to > indicate both target and initiator. +++ fixed - this holds also for textmaxlength +++ From owner-ips@ece.cmu.edu Fri Mar 23 16:39:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA03908 for ; Fri, 23 Mar 2001 16:10:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NIZGU05024 for ips-outgoing; Fri, 23 Mar 2001 13:35:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NIYhr05002 for ; Fri, 23 Mar 2001 13:34:43 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id B1C0CC8 for ; Fri, 23 Mar 2001 10:34:38 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id KAA05037; Fri, 23 Mar 2001 10:34:37 -0800 (PST) Message-ID: <3ABB9750.56D8CA66@hp.com> Date: Fri, 23 Mar 2001 10:34:56 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI: LUs ownership/iSCSI MIB Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mark (bakke), Where ===== In the MIB draft, in the chapter 5.1. Overall MIB structure There is: "Therefore, LUs are "owned" by Targets, and LUNs are owned by Sessions." Problem ====== If a LU is owned by only one target how do you deal with an active-passive configuration such as: active link: portal IP1, LUN=1 passive: portal IP2, LUN=2 (but it is the same LU as the one accessed through the active link) As the various portals of a target provide the same LUN view, IP1 and IP2 can NOT be two portals of the same target. In this case we would need two target names, one for the active link and one for the passive. But if a LU can be owned by only one target we can't do it. I think SAM doesn't allows a LU to be shared by two targets and that seems to be reflected in the chapter "5.1. Overall MIB structure". Solution ====== In the case of an active-passive configuration, the target must guarantee the same view on the two paths (active and passive). Is it a problem for whose who build such targets? Regards, Pierre From owner-ips@ece.cmu.edu Fri Mar 23 16:39:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA03912 for ; Fri, 23 Mar 2001 16:10:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NJ3IU07041 for ips-outgoing; Fri, 23 Mar 2001 14:03:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NJ1cr06926 for ; Fri, 23 Mar 2001 14:01:38 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id NAA41958; Fri, 23 Mar 2001 13:54:19 -0500 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id MAA75506; Fri, 23 Mar 2001 12:01:36 -0700 Importance: Normal Subject: Re: iSCSI: LUs ownership/iSCSI MIB To: Pierre Labat Cc: ips@ece.cmu.edu From: "Jim Hafner" Date: Fri, 23 Mar 2001 11:01:35 -0800 Message-ID: X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/23/2001 11:01:35 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Pierre, I have some comments on your note with respect to SAM/SCSI. 1) The interpretation depends a lot on what one means by "target". The default notion of this term is Target Port (not device, though that is not necessarily well captured by the MIB). Clearly SAM-2 allows multiple target ports to be i/o paths to a common logical unit. 2) the "asymmetric port" proposal from Ken Moe of Sun in T10 (for SPC-3) is designed to deal with SCSI's view of this active/passive picture you describe. 3) I don't believe I've ever seen anything in SAM that precludes even multiple Target Devices from sharing a logical unit. I believe the architecture is silent on this, which means that it's OK. 4) Also what's "viewable" by the MIB interface may NOT be the same as viewable by Report LUNS. The first is a management interface to get a view of the whole target device. The other is a specific host's (more precisely, initiator's) view of the accessible logical units on a particular port. So, the MIB could easily present the same answer on all ports *of the same target device* and a different target device may present a different answer. All this confusion about multiple pathing, etc. is one reason for logical units to have unique identifiers (EVPD page83h) AND (in my opinion) to keep most of this discussion out of iSCSI MIB. Finally, some of us in the N&DT team are wrestling with the mapping of SAM constructs to iSCSI constructs. The resolution of that may clarify the ambiguity of the term "target" in the MIB. Jim Hafner Pierre Labat @ece.cmu.edu on 03-23-2001 10:34:56 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: iSCSI: LUs ownership/iSCSI MIB Mark (bakke), Where ===== In the MIB draft, in the chapter 5.1. Overall MIB structure There is: "Therefore, LUs are "owned" by Targets, and LUNs are owned by Sessions." Problem ====== If a LU is owned by only one target how do you deal with an active-passive configuration such as: active link: portal IP1, LUN=1 passive: portal IP2, LUN=2 (but it is the same LU as the one accessed through the active link) As the various portals of a target provide the same LUN view, IP1 and IP2 can NOT be two portals of the same target. In this case we would need two target names, one for the active link and one for the passive. But if a LU can be owned by only one target we can't do it. I think SAM doesn't allows a LU to be shared by two targets and that seems to be reflected in the chapter "5.1. Overall MIB structure". Solution ====== In the case of an active-passive configuration, the target must guarantee the same view on the two paths (active and passive). Is it a problem for whose who build such targets? Regards, Pierre From owner-ips@ece.cmu.edu Fri Mar 23 16:39:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA03911 for ; Fri, 23 Mar 2001 16:10:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NJHM808056 for ips-outgoing; Fri, 23 Mar 2001 14:17:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NJGNr07954 for ; Fri, 23 Mar 2001 14:16:23 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id AC842A64; Fri, 23 Mar 2001 12:16:22 -0700 (MST) Received: from axcsbh2.cos.agilent.com (axcsbh2.cos.agilent.com [130.29.152.144]) by msgrel1.cos.agilent.com (Postfix) with SMTP id 96F005B; Fri, 23 Mar 2001 12:16:22 -0700 (MST) Received: from 130.29.152.144 by axcsbh2.cos.agilent.com (InterScan E-Mail VirusWall NT); Fri, 23 Mar 2001 12:16:22 -0700 (Mountain Standard Time) Received: by axcsbh2.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Fri, 23 Mar 2001 12:16:22 -0700 Message-ID: From: vince_cavanna@agilent.com To: dotis@sanlight.net, mbakke@cisco.com Cc: ips@ece.cmu.edu, vince_cavanna@agilent.com Subject: RE: CRC vs CHKSUM presentation slides Date: Fri, 23 Mar 2001 12:16:20 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hello Doug, I have inserted my comments below. Vince |-----Original Message----- |From: Douglas Otis [mailto:dotis@sanlight.net] |Sent: Thursday, March 22, 2001 3:16 PM |To: Mark Bakke; CAVANNA,VICENTE V (A-Roseville,ex1) |Cc: ips@ece.cmu.edu; ipsan@rtl.rose.agilent.com |Subject: RE: CRC vs CHKSUM presentation slides | | |Mark, | |One small note on these slides. Although the number of instructions |required to implement CRC-32 with 128K table lookups compares |closely to |that of Adler-32, Adler-32 is not accessing a 128K byte lookup |table. These |slides made a comparison of instructions without considering additional |memory access required or the size of the table required. I am not sure what comparison you are referring to since I did not compare the cost of implementing Checksums against that of implementing CRCs. All I compared was the cost, in gates (not table size), of a complex CRC vs that of a simple CRC. | |If you were to use the same CRC-32, you will find the same levels of |protection as this CRC is to protect against various types of equipment |failures NOT protected by the Ethernet/FDDI/Fibre-Channel CRC, then a |replication of this polynomial should not be a problem in that |sense. For |those that wish to include a different polynomial tend to justify this |effort by indicating this would improve the present physical |layer checks. | |The problem that I see with that conclusion is errors seen |outside of any |potential Ethernet CRC failure are of such orders of magnitude as to |out-weight any replicate benefit derived. In that light, even |Adler-32 may |provide benefit in that case. The next area being looked at |is the number |of bits detected in a burst error. The unprotected equipment |is not likely |using a serial method of delivery, so you are again left |errors that may |provide strange error patterns. The concept supporting a different |polynomial was its greater ability to detect burst errors. If I understand you correctly you are saying that the ethernet polynomial is good enough to protect against the type of errors iSCSI needs to provide protection for. I agree. My main argument for wanting a simple (standard) polynomial is that I would like to take advantage of the lower cost in gate count. | |The large piece of information missing from this presentation was a |characterization of the types of errors seen by intermediate equipment. |There was a bullet that suggested this equipment will produce |burst errors. |If so, what is the nature of this equipment and what are the prevalent |errors? From that perspective a comparisons could be made. |But as we are |familiar with noise distributions of single bit errors, this became our |focus. It is a bit like looking for your keys under the light |post even |though you dropped them in the dark. You wish to look where |the light is |better. I give a lot more detail on small number of single bit errors and small number of bursts because there is theory that I can use to analyze those cases. Like you and others, I have struggled with the claims that it is important for iSCSI to provide protection from small number of errors. I have finally convinced myself that small number of errors are probable even in our environment. In my slides I outlined several important error mechanims that tend to produce few number of individual errors. Here are some I can think of at the moment: 1. Random memory errors. 2. Marginal timing paths in the hardware. 3. Noise (crosstalk, power supply noise). 4. Leakage problems in bistable devices due to manufacturing defects. 5. Hardware degradation. Of course there are also possible error patterns that contain lots of errors and unfortunately I cannot say much about about them other than that 1 in 2^32 (if all errors are equiprobable) will get through both checksums and CRCs and that those that get through the CRC are more random looking than those that get through the Checksums. | |Doug | |> Vicente- |> |> I just took another look through your slides after seeing the |> presentation on Monday. They were very well-done. I have |> one question, though. If the CCITT-CRC32 is considered "good |> enough", then would the Ethernet CRC32 also be good enough? The |> reason I ask is that every hardware vendor involved in building |> iSCSI stuff already has implementations of the Ethernet CRC, |> which is used for both Ethernet and Fibre Channel. |> |> The Ethernet poly has more terms than CCITT, and perhaps is |> not as good as CRC-32C (any thoughts?), but everyone has hardware |> and software for this, with proven interoperability (bit and |> byte order, etc). Performance-wise, it will be there for |> 10Gb Ethernet, so it should be fast enough. |> |> So if the Ethernet poly is deemed good enough (even if it's not |> the best), and fast enough (even if it's not the fastest), why |> not use it? I think we would stand a much better chance of |> achieving interoperability in a short time. |> |> Please let me know what you think of this; I realize that a few |> of my questions were speculative. |> |> Regards, |> |> Mark |> |> "CAVANNA,VICENTE V (A-Roseville,ex1)" wrote: |> > |> > Here are the slides I presented at the IETF-50 in Minneapolis. |> > |> > Vicente Cavanna |> > Agilent Technologies |> > |> > <> |> > |> > |> ------------------------------------------------------------------ |> ---------------------------------- |> > Name: CRCorChksm.pdf |> > CRCorChksm.pdf Type: Portable Document Format |(application/pdf) |> > Encoding: base64 |> |> -- |> Mark A. Bakke |> Cisco Systems |> mbakke@cisco.com |> 763.398.1054 |> | From owner-ips@ece.cmu.edu Fri Mar 23 16:39:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA03910 for ; Fri, 23 Mar 2001 16:10:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NIZJI05029 for ips-outgoing; Fri, 23 Mar 2001 13:35:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NIYRr04985 for ; Fri, 23 Mar 2001 13:34:27 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id BAC67A63; Fri, 23 Mar 2001 11:34:23 -0700 (MST) Received: from axcsbh1.cos.agilent.com (axcsbh1.cos.agilent.com [130.29.152.143]) by msgrel1.cos.agilent.com (Postfix) with SMTP id 6C58D99; Fri, 23 Mar 2001 11:34:23 -0700 (MST) Received: from 130.29.152.143 by axcsbh1.cos.agilent.com (InterScan E-Mail VirusWall NT); Fri, 23 Mar 2001 11:34:22 -0700 (Mountain Standard Time) Received: by axcsbh1.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Fri, 23 Mar 2001 11:34:22 -0700 Message-ID: From: vince_cavanna@agilent.com To: pat_thaler@agilent.com, mbakke@cisco.com, jim.williams@emulex.com, ips@ece.cmu.edu Cc: vince_cavanna@agilent.com Subject: RE: CRC vs CHKSUM presentation slides Date: Fri, 23 Mar 2001 11:34:19 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Pat and others, I have not actually determined for myself the coverage of the ethernet poly for 3 errors. All I know for sure, as I show in my slides, is that the ethernet poly, because it does not have an x^c+1 term, cannot detect ALL occurrences of an odd number of errors (including 3). I have therefore quoted data from the widely referenced paper (which I have refereced many times before) by Fujiware, Kasami and Lin on Error detecting capabilities of hte ethernet polynomial . This paper claims the Ethernet poly has a Hamming distance of 4 for lengths 3007 <= n <= 12144 and larger hamming distance for smaller n. It seems quite a coincidence to me that 12144 happens to be the length of the maximum IEE 802.3 frame (prior to VLAN tagging) and I can only assume that the polynomial was either expressely designed for that coverage or the coverage is actually larger than 12144 and the authors thought 12144 was sufficient and did not bother to give the exact number. In any case, I don't think it is particularly important for the ethernet poly to be able to catch ALL 3 bit errors since, due to the group encoding used, 3 bit errors on the link will manifest themselves as 3 bursts for which the ethernet poly most likely does not have complete coverage anyway. At one time (prior to the use of group encoding) the ability to detect 3 bit errors may have been important but not today. Same is true for Fiberchannel which has even larger frame sizes and also uses group encoding. Vince |-----Original Message----- |From: THALER,PAT (A-Roseville,ex1) |Sent: Friday, March 23, 2001 9:59 AM |To: vince_cavanna@agilent.com; mbakke@cisco.com; |jim.williams@emulex.com; ips@ece.cmu.edu |Subject: RE: CRC vs CHKSUM presentation slides | | |Vince, | |Your statement is correct except for the number 12144. 12144 |bits is the length of the maximum IEEE 802.3 frame (before we |added 4 bytes for VLAN tagging). The polynomial was actually |checked for coverage of 3-bit errors out to the Token Ring and |FDDI maximum packet sizes which are about 3 times as long. I |don't know the length at which the first undetectable 3-bit |error occurs, but it is something beyond 32000 bits. | |I also have trouble with an argument based on leverage. |Hardware design of CRC generators and checkers is just not |that hard. It's a register plus a bunch of XOR's. | |Regards, |Pat | |-----Original Message----- |From: vince_cavanna@agilent.com [mailto:vince_cavanna@agilent.com] |Sent: Thursday, March 22, 2001 4:10 PM |To: vince_cavanna@agilent.com; mbakke@cisco.com; |jim.williams@emulex.com; ips@ece.cmu.edu |Subject: RE: CRC vs CHKSUM presentation slides | | |To save Julian from having to write a memo I would like to add that the |ethernet polynomial has a certain weakness for 3 bit errors |that the others |don't. The ethernet polynomial is only guaranteed to catch all |combination |sof 3 bit errors when the message length is up to 12144 bits. |For larger |message lengths some 3 bit errors may get through. |Vince | | ||-----Original Message----- ||From: CAVANNA,VICENTE V (A-Roseville,ex1) ||Sent: Thursday, March 22, 2001 3:42 PM ||To: 'Mark Bakke'; 'Jim Williams'; ips@ece.cmu.edu ||Cc: CAVANNA,VICENTE V (A-Roseville,ex1) ||Subject: RE: CRC vs CHKSUM presentation slides || || ||Mark and Jim, ||I think any of the 32 bit CRC polynomials that have been ||proposed are more than good enough. My main reason for ||recommending hte CCITT-CRC32 is that it is half the cost of ||the others in terms of gate count - and unless you are doing a ||serial divider, which would be too slow, the gate count is ||very significant. Also, the leverage may not be as high as you ||think unless we are willing to use the datapath width of ||existing implementations. For 10 gig we will probably need to ||have larger than normal datapath widths. If you change the ||datapath width to handle, say, 64 bits at a time you change ||the implementation. I would otherwise have no problem with ||using the ethernet polynomial. I will try to explain later why ||I am not concerned about using the same polynomial and ||potentially giving up some cross-checking (I am not quite sure yet). ||Vince || |||-----Original Message----- |||From: Jim Williams [mailto:jim.williams@emulex.com] |||Sent: Thursday, March 22, 2001 1:30 PM |||To: ips@ece.cmu.edu |||Subject: Re: CRC vs CHKSUM presentation slides ||| ||| |||From: "Mark Bakke" |||To: "CAVANNA,VICENTE V (A-Roseville,ex1)" |||Cc: ; |||Sent: Thursday, March 22, 2001 4:00 PM |||Subject: Re: CRC vs CHKSUM presentation slides ||| ||| |||> Vicente- |||> |||> I just took another look through your slides after seeing the |||> presentation on Monday. They were very well-done. I have |||> one question, though. If the CCITT-CRC32 is considered "good |||> enough", then would the Ethernet CRC32 also be good enough? The |||> reason I ask is that every hardware vendor involved in building |||> iSCSI stuff already has implementations of the Ethernet CRC, |||> which is used for both Ethernet and Fibre Channel. |||> |||> The Ethernet poly has more terms than CCITT, and perhaps is |||> not as good as CRC-32C (any thoughts?), but everyone has hardware |||> and software for this, with proven interoperability (bit and |||> byte order, etc). Performance-wise, it will be there for |||> 10Gb Ethernet, so it should be fast enough. |||> |||> So if the Ethernet poly is deemed good enough (even if it's not |||> the best), and fast enough (even if it's not the fastest), why |||> not use it? I think we would stand a much better chance of |||> achieving interoperability in a short time. |||> |||> Please let me know what you think of this; I realize that a few |||> of my questions were speculative. ||| |||Since the iSCSI messages will often be encapsulated in ethernet |||packets, there is some value to using a different CRC. Link |||errors are double protected with two different CRCs. If |||ethernet and iSCSI use the same polynomial, there is little |||additional coverage against link errors. This point may not |||be decisive, but all other things being equal or almost |||equal, it is worth considering. ||| || | From owner-ips@ece.cmu.edu Fri Mar 23 16:39:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA03913 for ; Fri, 23 Mar 2001 16:10:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NIYJf04980 for ips-outgoing; Fri, 23 Mar 2001 13:34:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.bld.us.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NIYDr04972 for ; Fri, 23 Mar 2001 13:34:13 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e34.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id NAA16360; Fri, 23 Mar 2001 13:17:39 -0500 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id LAA264228; Fri, 23 Mar 2001 11:34:07 -0700 Importance: Normal Subject: RE: iSCSI Naming: WWUIs, URNs, and namespaces To: "Elliott, Robert" Cc: "'ips@ece.cmu.edu'" , "'Mark Bakke'" , "'Black_David@emc.com'" From: "Jim Hafner" Date: Fri, 23 Mar 2001 10:34:06 -0800 Message-ID: X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/23/2001 10:34:06 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Rob, Admittedly a lofty goal for a common namespace, but the problem we faced in iSCSI is that (primarily) on the initiator side there is NO canonical hardware component from which to derive an EUI-64+extra name. Some systems have motherboard identifiers, some have network mac addresses, some have ... I know of no common hardware component that lives on every single IP device (including my PDA, my phone, my...). Additionally, on the target side, the virtualization of target devices through gateways (e.g., what an SSP might set up) and the like (thinking of the device as a service and not a hardware component) made a more user-manageable (less hardware driven) namespace a better alternative. I could be wrong on this "no canonical hardware component" point, and I'd love to be corrected as it would make this a lot simpler. Jim Hafner "Elliott, Robert" @ece.cmu.edu on 03-23-2001 09:31:32 AM Sent by: owner-ips@ece.cmu.edu To: "'ips@ece.cmu.edu'" , "'Mark Bakke'" , "'Black_David@emc.com'" cc: Subject: RE: iSCSI Naming: WWUIs, URNs, and namespaces The SCSI over RDMA protocol (for InfiniBand) chose: 1. initiator port identifier = 64-bit EUI-64 plus 64-bit identifier extension 2. target port identifier = 64-bit EUI-64 The target port identifier is the same as the InfiniBand IO controller (IOC) globally unique identifier (GUID). (InfiniBand GUIDs are EUI-64s) Targets will often be hardware devices that already have IOC GUIDs. In the cases where they are not, we assume the software provider can provide GUIDs just like it provides license keys. Software could allow customers to specify their own EUI-64s, too. Initiators are more likely to be software entities. We assumed the software could take one of the GUIDs of the hardware on which it is located (e.g. the InfiniBand port GUID) and append some sort of unique identifier to it. It is unspecified how to guarantee software generates unique IDs. If there is a name server on the fabric, it may check if the ID it chooses is already in use and pick another. Alternatively, the software provider could generate its own EUI-64 or the customer could provide one (as with targets implemented in software). I proposed using the iSCSI reverse DNS WWUI format so SRP and iSCSI could share a namespace, but the group preferred the EUI-64 based approach (among other concerns, the 255 byte string is too big for some of the packets). If iSCSI drops the long WWUI format and goes with EUI-64, we may achieve the common namespace goal. --- Rob Elliott, Compaq Server Storage Robert.Elliott@compaq.com > -----Original Message----- > From: Mark Bakke [mailto:mbakke@cisco.com] > Sent: Friday, March 23, 2001 8:36 AM > To: Black_David@emc.com > Cc: ips@ece.cmu.edu > Subject: Re: iSCSI Naming: WWUIs, URNs, and namespaces > > > > David- > > This is starting to seem like work. In case they help, > please see my comments below. We can take this up in > more detail in the N&D team. > > -- > Mark > > Black_David@emc.com wrote: > > > > I don't like naming issues. :-) :-) > > > > After suitable consulting with some members of > > the IESG and IAB, I have some news to convey about > > the current approach to iSCSI naming. > > > > The IESG will not approve another global namespace > > for iSCSI's use - this means that WWUIs as currently > > So I take it that WWUIs are fine, as long as they use > existing global name spaces. > > > designed will need to be revised out of the > > naming and discovery draft, and that it will not be > > possible to proceed with the WWUI URN draft > > as an official IPS WG work item. The best course of > > action would probably be to allow the WWUI URN draft > > to expire without further revision. > > > > To a first approximation, WWUIs are/were a "grand > > unified theory" of naming, in that any namespace could > > be glued into the WWUI world (as several were). > > The reason we glued these in was to avoid creating a new > global name space. We allowed the use of a few currently > available global name spaces. How can this be a problem? > > > The WG is being directed to instead focus on the > > individual namespaces and make sure that the ones that > > are used are in fact necessary. iSCSI can use text > > keys to identify which sort of name is being used > > (one key for each sort of format, for each instance > > in which a name is used), and it may be possible > > to encode the name format in the parse rules for the > > values of iSCSI keys to avoid proliferation of keys. > > > > Taking a look at the namespaces in the current iSCSI > > naming and discovery draft, here's some initial > > guidance from this WG co-chair: > > iscsi - canonical target -- This should be fine. > > eui - WWNs -- The use of these for storage makes eminent > > sense. I don't see a problem here. > > So far, so good. Does that mean that we can keep the above > two, in the current WWUI, as we have it formatted? > > EUI, although it's not that flexible (see later), is required > for interoperating with and adding iSCSI to existing devices. > > > dns - hostnames -- Use of these should be abandoned as > > not only are they not really URNs (as indicated > > in the draft), but their intended usage is straying > > into the tarpit known as "URN resolution". Faster > > progress will made by staying out. A DNS hostname > > can be put into an Alias or something new can be > > invented to carry it as a Location Hint, BUT the > > relevant URN WG RFCs and drafts on URN resolution > > should be reviewed before proceeding too far in this > > direction. > > I don't like this one that much, either, for the same reasons. > > > iscsi - Reverse DNS and oui - Org. Unique Identifier -- > > The rationale for these is not clear to me. > > Assuming that WWNs are going to be available for > > use in naming iSCSI Initiators and Targets, what > > are the problems that these sorts of names solve > > that WWNs don't? It appears that one of the problems > > may be who can get/create them. Discussion of this > > on the list would be appropriate. > > These two formats use existing global name spaces, and allow > the implementor or end user to take a more flexible approach > to naming than is offered by the EUI-64. When dealing with > a large number of logical targets, burning WWNs (essentially, > MAC addresses), and attempting to keep them tied to a logical > entity without tying them to hardware is not practical. > > Both of these formats accomplish the same thing, and we could > make do with just one of them. The only real differences are: > > - OUI is a fixed length naming authority; reverse DNS is variable > and could result in longer names. > > - A human looking at a reverse DNS-based name can easily determine > who the naming authority is; a human looking at an OUI-based > name will usually have to go look it up at IEEE. > > - Only a hardware manufacturer (and a few software manufacters) > will already have an OUI. Others, such as storage service > providers, large end customers, or software driver providers, > would either be out-of-luck for naming their devices, or would > have to register for an OUI, which would otherwise be a waste > of IEEE resources. OTOH, a storage service provider could > provide its own names by using the reverse-DNS authority. > Everyone has a DNS name. This would enable the SSP to provide > its customers with a WWUI that would not change if the back-end > storage device was replaced. After all, it's not the device > that's important to this customer, it's the DATA. > > - If we had to pick one or the other, I would pick the reverse > DNS, since it offers the ability to generate unique names to > a wider variety of users. > > > In any case, the fewer name formats we have to deal with, > > the better. > > Although I probably should have discussed this with the rest > of the NDT first, I would personally be happy with: > > - iscsi - The canonical name. > - eui - For compatibility with existing device naming schemes. > - reverse-dns - For better naming flexibility moving forward. > > Perhaps the IESG would accept the WWUI with just these three. > Or perhaps we could just make the WWUI into a URN? I don't > know if that would help, but perhaps... > > The main point is that we are NOT creating any new name > spaces; we really are just using what's already there. Maybe > by specifying fewer of the options that are already there, we > would be in better shape. > > > I want to try to anticipate an objection to this, which > > would note that from a functional viewpoint the basic > > impact of this is to move some characters from one text > > string to another (e.g., from a WWUI type designator > > to part of an iSCSI text key), and wonder if this is > > a distinction without a difference. One of the reasons > > for the that started this post is that a functional > > view is not sufficient for naming - how things are named, > > the intended usage of names and their scope matter a lot. > > So that leads to the question: Has any other WG come up with > a world-wide-unique-identifier for something else that the > IESG folks deems acceptable? If we can see a few examples, > we would be better able to anticipate what they want. > > > This is particularly true when considering the structure > > of a namespace and how that structure may be extended. > > The upshot is that avoiding introduction of something > > claiming to be yet another global namespace is important > > (i.e., use existing namespaces with global scope in preference > > to inventing new ones). The resulting need to define > > the name spaces/formats in the main iSCSI spec. is > > probably a "feature" as it forces us to pay more > > attention to the sorts of names we use and raises the > > bar for adding additional sorts of names in the future. > > > I will be working with > > the naming and discovery team in my "copious spare time" > > to make sure that we don't lose the valuable work and > > progress they've made to date as a consequence of this > > change. Discussion on the list about what sort > > of names are important (e.g., the Reverse DNS and OUI > > namespaces) and why would be useful. > > From a requirements point of view, please respond if any > of the types of names are important to you (plural). > > Hopefully, we can get this out of the way quickly. > > > > > Thanks, > > --David > > > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 > From owner-ips@ece.cmu.edu Fri Mar 23 17:08:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA04972 for ; Fri, 23 Mar 2001 17:08:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NJOPj08563 for ips-outgoing; Fri, 23 Mar 2001 14:24:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NJNXr08515 for ; Fri, 23 Mar 2001 14:23:33 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 7DA9A1484; Fri, 23 Mar 2001 12:23:31 -0700 (MST) Received: from axandbh3.and.agilent.com (axandbh3.and.agilent.com [130.30.32.200]) by msgrel1.and.agilent.com (Postfix) with SMTP id ABB01196; Fri, 23 Mar 2001 14:23:30 -0500 (EST) Received: from 130.30.32.200 by axandbh3.and.agilent.com (InterScan E-Mail VirusWall NT); Fri, 23 Mar 2001 14:23:29 -0500 (Eastern Standard Time) Received: by axandbh3.and.agilent.com with Internet Mail Service (5.5.2653.19) id ; Fri, 23 Mar 2001 14:23:28 -0500 Message-ID: <1BEBA5E8600DD4119A50009027AF54A00568466F@axcs04.cos.agilent.com> From: "THALER,PAT (A-Roseville,ex1)" To: "CAVANNA,VICENTE V (A-Roseville,ex1)" , "'mbakke@cisco.com'" , "'jim.williams@emulex.com'" , "'ips@ece.cmu.edu'" Subject: RE: CRC vs CHKSUM presentation slides Date: Fri, 23 Mar 2001 14:23:27 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Vince, Probably Fujiware, et al only examined the results for the Ethernet packet length and shorter. The Ethernet CRC was also adopted for FDDI and 802.5 Token Ring. I know its protection was checked for 4-bit Hamming distance out to their larger maximum packet sizes. I think Raj Jain would have been one of the authors, but it would have been quite a while ago like early 80's. Actually, the three bit coverage for Ethernet is still important for protection against individual bit errors as might occur in a fiber optic receiver. Ethernet uses two coding schemes at 10 Gbit/s. One is the 8B/10B code. In that case, 3-bit error coverage of the polynomial is not important because the code as it is used will detect any odd number of bit errors in a packet. The other is the 64B/66B code. This code doesn't block code the data. A single bit error in the data field causes a single bit error in the received data. Packet delimiters and sync headers in this code are designed to provide 4-bit Hamming distance for false delimiter creation. The code relies on the CRC for protection against bit hits in the data field. I'm not sure what is most important for protection under the ISCSI CRC (or checksum). I think the supposition is that the data will be protected by transmission system CRCs when it is on the wires and that the main purpose of the ISCSI protection is for coverage while the packets are in switches, routerrs, etc. There are several potential kinds of errors: Random bit errors in data - Hamming distance matters here because Pbe^Hamming_distance is a factor in the Pud calculation. Individual wrong byte/word read/write error in memory - If this is a concern and one expects byte wide structures in the memory, then a CRC like the Ethernet CRC that detects any two errored bytes in a packet would be desireable. If such errors are expected to be wider than a byte, then the various 32-bit CRCs should be about equal. Fletcher is extremely weak for this kind of error because many two byte hits exist that produce undetectable errors. Adler avoids the 2 byte hit problem, but is weak for hits beyond. Random bit errors in buffer structure pointers - A broad class of switch/router/end node architectures store packets in a series of buffers with pointer lists. If there is an error in reading a pointer, the packet error would be similar to the errors analyzed in Performance of Checksums and CRC's over Real Data; Jonathan Stone, et al. Unfortunately, that paper examined only the 16-bit TCP checksum and the AAL5 CRC. Regards, Pat -----Original Message----- From: CAVANNA,VICENTE V (A-Roseville,ex1) Sent: Friday, March 23, 2001 10:34 AM To: THALER,PAT (A-Roseville,ex1); 'mbakke@cisco.com'; 'jim.williams@emulex.com'; 'ips@ece.cmu.edu' Cc: CAVANNA,VICENTE V (A-Roseville,ex1) Subject: RE: CRC vs CHKSUM presentation slides Pat and others, I have not actually determined for myself the coverage of the ethernet poly for 3 errors. All I know for sure, as I show in my slides, is that the ethernet poly, because it does not have an x^c+1 term, cannot detect ALL occurrences of an odd number of errors (including 3). I have therefore quoted data from the widely referenced paper (which I have refereced many times before) by Fujiware, Kasami and Lin on Error detecting capabilities of hte ethernet polynomial . This paper claims the Ethernet poly has a Hamming distance of 4 for lengths 3007 <= n <= 12144 and larger hamming distance for smaller n. It seems quite a coincidence to me that 12144 happens to be the length of the maximum IEE 802.3 frame (prior to VLAN tagging) and I can only assume that the polynomial was either expressely designed for that coverage or the coverage is actually larger than 12144 and the authors thought 12144 was sufficient and did not bother to give the exact number. In any case, I don't think it is particularly important for the ethernet poly to be able to catch ALL 3 bit errors since, due to the group encoding used, 3 bit errors on the link will manifest themselves as 3 bursts for which the ethernet poly most likely does not have complete coverage anyway. At one time (prior to the use of group encoding) the ability to detect 3 bit errors may have been important but not today. Same is true for Fiberchannel which has even larger frame sizes and also uses group encoding. Vince |-----Original Message----- |From: THALER,PAT (A-Roseville,ex1) |Sent: Friday, March 23, 2001 9:59 AM |To: vince_cavanna@agilent.com; mbakke@cisco.com; |jim.williams@emulex.com; ips@ece.cmu.edu |Subject: RE: CRC vs CHKSUM presentation slides | | |Vince, | |Your statement is correct except for the number 12144. 12144 |bits is the length of the maximum IEEE 802.3 frame (before we |added 4 bytes for VLAN tagging). The polynomial was actually |checked for coverage of 3-bit errors out to the Token Ring and |FDDI maximum packet sizes which are about 3 times as long. I |don't know the length at which the first undetectable 3-bit |error occurs, but it is something beyond 32000 bits. | |I also have trouble with an argument based on leverage. |Hardware design of CRC generators and checkers is just not |that hard. It's a register plus a bunch of XOR's. | |Regards, |Pat | |-----Original Message----- |From: vince_cavanna@agilent.com [mailto:vince_cavanna@agilent.com] |Sent: Thursday, March 22, 2001 4:10 PM |To: vince_cavanna@agilent.com; mbakke@cisco.com; |jim.williams@emulex.com; ips@ece.cmu.edu |Subject: RE: CRC vs CHKSUM presentation slides | | |To save Julian from having to write a memo I would like to add that the |ethernet polynomial has a certain weakness for 3 bit errors |that the others |don't. The ethernet polynomial is only guaranteed to catch all |combination |sof 3 bit errors when the message length is up to 12144 bits. |For larger |message lengths some 3 bit errors may get through. |Vince | | ||-----Original Message----- ||From: CAVANNA,VICENTE V (A-Roseville,ex1) ||Sent: Thursday, March 22, 2001 3:42 PM ||To: 'Mark Bakke'; 'Jim Williams'; ips@ece.cmu.edu ||Cc: CAVANNA,VICENTE V (A-Roseville,ex1) ||Subject: RE: CRC vs CHKSUM presentation slides || || ||Mark and Jim, ||I think any of the 32 bit CRC polynomials that have been ||proposed are more than good enough. My main reason for ||recommending hte CCITT-CRC32 is that it is half the cost of ||the others in terms of gate count - and unless you are doing a ||serial divider, which would be too slow, the gate count is ||very significant. Also, the leverage may not be as high as you ||think unless we are willing to use the datapath width of ||existing implementations. For 10 gig we will probably need to ||have larger than normal datapath widths. If you change the ||datapath width to handle, say, 64 bits at a time you change ||the implementation. I would otherwise have no problem with ||using the ethernet polynomial. I will try to explain later why ||I am not concerned about using the same polynomial and ||potentially giving up some cross-checking (I am not quite sure yet). ||Vince || |||-----Original Message----- |||From: Jim Williams [mailto:jim.williams@emulex.com] |||Sent: Thursday, March 22, 2001 1:30 PM |||To: ips@ece.cmu.edu |||Subject: Re: CRC vs CHKSUM presentation slides ||| ||| |||From: "Mark Bakke" |||To: "CAVANNA,VICENTE V (A-Roseville,ex1)" |||Cc: ; |||Sent: Thursday, March 22, 2001 4:00 PM |||Subject: Re: CRC vs CHKSUM presentation slides ||| ||| |||> Vicente- |||> |||> I just took another look through your slides after seeing the |||> presentation on Monday. They were very well-done. I have |||> one question, though. If the CCITT-CRC32 is considered "good |||> enough", then would the Ethernet CRC32 also be good enough? The |||> reason I ask is that every hardware vendor involved in building |||> iSCSI stuff already has implementations of the Ethernet CRC, |||> which is used for both Ethernet and Fibre Channel. |||> |||> The Ethernet poly has more terms than CCITT, and perhaps is |||> not as good as CRC-32C (any thoughts?), but everyone has hardware |||> and software for this, with proven interoperability (bit and |||> byte order, etc). Performance-wise, it will be there for |||> 10Gb Ethernet, so it should be fast enough. |||> |||> So if the Ethernet poly is deemed good enough (even if it's not |||> the best), and fast enough (even if it's not the fastest), why |||> not use it? I think we would stand a much better chance of |||> achieving interoperability in a short time. |||> |||> Please let me know what you think of this; I realize that a few |||> of my questions were speculative. ||| |||Since the iSCSI messages will often be encapsulated in ethernet |||packets, there is some value to using a different CRC. Link |||errors are double protected with two different CRCs. If |||ethernet and iSCSI use the same polynomial, there is little |||additional coverage against link errors. This point may not |||be decisive, but all other things being equal or almost |||equal, it is worth considering. ||| || | From owner-ips@ece.cmu.edu Fri Mar 23 17:08:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA04984 for ; Fri, 23 Mar 2001 17:08:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NJjM310080 for ips-outgoing; Fri, 23 Mar 2001 14:45:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NJitr10061 for ; Fri, 23 Mar 2001 14:44:55 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 2890E133B; Fri, 23 Mar 2001 12:44:55 -0700 (MST) Received: from axcsbh2.cos.agilent.com (axcsbh2.cos.agilent.com [130.29.152.144]) by msgrel1.cos.agilent.com (Postfix) with SMTP id F12CE99; Fri, 23 Mar 2001 12:44:54 -0700 (MST) Received: from 130.29.152.144 by axcsbh2.cos.agilent.com (InterScan E-Mail VirusWall NT); Fri, 23 Mar 2001 12:44:54 -0700 (Mountain Standard Time) Received: by axcsbh2.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Fri, 23 Mar 2001 12:44:54 -0700 Message-ID: From: vince_cavanna@agilent.com To: pat_thaler@agilent.com, mbakke@cisco.com, jim.williams@emulex.com, ips@ece.cmu.edu Subject: RE: CRC vs CHKSUM presentation slides Date: Fri, 23 Mar 2001 12:44:51 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Pat, Good point. I was not thinking about the 64B/66B code. Vince |-----Original Message----- |From: THALER,PAT (A-Roseville,ex1) |Sent: Friday, March 23, 2001 11:23 AM |To: CAVANNA,VICENTE V (A-Roseville,ex1); 'mbakke@cisco.com'; |'jim.williams@emulex.com'; 'ips@ece.cmu.edu' |Subject: RE: CRC vs CHKSUM presentation slides | | |Vince, | |Probably Fujiware, et al only examined the results for the |Ethernet packet length and shorter. The Ethernet CRC was also |adopted for FDDI and 802.5 Token Ring. I know its protection |was checked for 4-bit Hamming distance out to their larger |maximum packet sizes. I think Raj Jain would have been one of |the authors, but it would have been quite a while ago like early 80's. | |Actually, the three bit coverage for Ethernet is still |important for protection against individual bit errors as |might occur in a fiber optic receiver. Ethernet uses two |coding schemes at 10 Gbit/s. One is the 8B/10B code. In that |case, 3-bit error coverage of the polynomial is not important |because the code as it is used will detect any odd number of |bit errors in a packet. The other is the 64B/66B code. This |code doesn't block code the data. A single bit error in the |data field causes a single bit error in the received data. |Packet delimiters and sync headers in this code are designed |to provide 4-bit Hamming distance for false delimiter |creation. The code relies on the CRC for protection against |bit hits in the data field. | |I'm not sure what is most important for protection under the |ISCSI CRC (or checksum). I think the supposition is that the |data will be protected by transmission system CRCs when it is |on the wires and that the main purpose of the ISCSI protection |is for coverage while the packets are in switches, routerrs, |etc. There are several potential kinds of errors: | |Random bit errors in data - Hamming distance matters here |because Pbe^Hamming_distance is a factor in the Pud calculation. | |Individual wrong byte/word read/write error in memory - If |this is a concern and one expects byte wide structures in the |memory, then a CRC like the Ethernet CRC that detects any two |errored bytes in a packet would be desireable. If such errors |are expected to be wider than a byte, then the various 32-bit |CRCs should be about equal. Fletcher is extremely weak for |this kind of error because many two byte hits exist that |produce undetectable errors. Adler avoids the 2 byte hit |problem, but is weak for hits beyond. | |Random bit errors in buffer structure pointers - A broad class |of switch/router/end node architectures store packets in a |series of buffers with pointer lists. If there is an error in |reading a pointer, the packet error would be similar to the |errors analyzed in Performance of Checksums and CRC's over |Real Data; Jonathan Stone, et al. Unfortunately, that paper |examined only the 16-bit TCP checksum and the AAL5 CRC. | |Regards, |Pat |-----Original Message----- |From: CAVANNA,VICENTE V (A-Roseville,ex1) |Sent: Friday, March 23, 2001 10:34 AM |To: THALER,PAT (A-Roseville,ex1); 'mbakke@cisco.com'; |'jim.williams@emulex.com'; 'ips@ece.cmu.edu' |Cc: CAVANNA,VICENTE V (A-Roseville,ex1) |Subject: RE: CRC vs CHKSUM presentation slides | | |Pat and others, | |I have not actually determined for myself the coverage of the |ethernet poly for 3 errors. All I know for sure, as I show in |my slides, is that the ethernet poly, because it does not have |an x^c+1 term, cannot detect ALL occurrences of an odd number |of errors (including 3). I have therefore quoted data from the |widely referenced paper (which I have refereced many times |before) by Fujiware, Kasami and Lin on Error detecting |capabilities of hte ethernet polynomial . This paper claims |the Ethernet poly has a Hamming distance of 4 for lengths 3007 |<= n <= 12144 and larger hamming distance for smaller n. | |It seems quite a coincidence to me that 12144 happens to be |the length of the maximum IEE 802.3 frame (prior to VLAN |tagging) and I can only assume that the polynomial was either |expressely designed for that coverage or the coverage is |actually larger than 12144 and the authors thought 12144 was |sufficient and did not bother to give the exact number. | |In any case, I don't think it is particularly important for |the ethernet poly to be able to catch ALL 3 bit errors since, |due to the group encoding used, 3 bit errors on the link will |manifest themselves as 3 bursts for which the ethernet poly |most likely does not have complete coverage anyway. At one |time (prior to the use of group encoding) the ability to |detect 3 bit errors may have been important but not today. |Same is true for Fiberchannel which has even larger frame |sizes and also uses group encoding. | |Vince | ||-----Original Message----- ||From: THALER,PAT (A-Roseville,ex1) ||Sent: Friday, March 23, 2001 9:59 AM ||To: vince_cavanna@agilent.com; mbakke@cisco.com; ||jim.williams@emulex.com; ips@ece.cmu.edu ||Subject: RE: CRC vs CHKSUM presentation slides || || ||Vince, || ||Your statement is correct except for the number 12144. 12144 ||bits is the length of the maximum IEEE 802.3 frame (before we ||added 4 bytes for VLAN tagging). The polynomial was actually ||checked for coverage of 3-bit errors out to the Token Ring and ||FDDI maximum packet sizes which are about 3 times as long. I ||don't know the length at which the first undetectable 3-bit ||error occurs, but it is something beyond 32000 bits. || ||I also have trouble with an argument based on leverage. ||Hardware design of CRC generators and checkers is just not ||that hard. It's a register plus a bunch of XOR's. || ||Regards, ||Pat || ||-----Original Message----- ||From: vince_cavanna@agilent.com [mailto:vince_cavanna@agilent.com] ||Sent: Thursday, March 22, 2001 4:10 PM ||To: vince_cavanna@agilent.com; mbakke@cisco.com; ||jim.williams@emulex.com; ips@ece.cmu.edu ||Subject: RE: CRC vs CHKSUM presentation slides || || ||To save Julian from having to write a memo I would like to |add that the ||ethernet polynomial has a certain weakness for 3 bit errors ||that the others ||don't. The ethernet polynomial is only guaranteed to catch all ||combination ||sof 3 bit errors when the message length is up to 12144 bits. ||For larger ||message lengths some 3 bit errors may get through. ||Vince || || |||-----Original Message----- |||From: CAVANNA,VICENTE V (A-Roseville,ex1) |||Sent: Thursday, March 22, 2001 3:42 PM |||To: 'Mark Bakke'; 'Jim Williams'; ips@ece.cmu.edu |||Cc: CAVANNA,VICENTE V (A-Roseville,ex1) |||Subject: RE: CRC vs CHKSUM presentation slides ||| ||| |||Mark and Jim, |||I think any of the 32 bit CRC polynomials that have been |||proposed are more than good enough. My main reason for |||recommending hte CCITT-CRC32 is that it is half the cost of |||the others in terms of gate count - and unless you are doing a |||serial divider, which would be too slow, the gate count is |||very significant. Also, the leverage may not be as high as you |||think unless we are willing to use the datapath width of |||existing implementations. For 10 gig we will probably need to |||have larger than normal datapath widths. If you change the |||datapath width to handle, say, 64 bits at a time you change |||the implementation. I would otherwise have no problem with |||using the ethernet polynomial. I will try to explain later why |||I am not concerned about using the same polynomial and |||potentially giving up some cross-checking (I am not quite sure yet). |||Vince ||| ||||-----Original Message----- ||||From: Jim Williams [mailto:jim.williams@emulex.com] ||||Sent: Thursday, March 22, 2001 1:30 PM ||||To: ips@ece.cmu.edu ||||Subject: Re: CRC vs CHKSUM presentation slides |||| |||| ||||From: "Mark Bakke" ||||To: "CAVANNA,VICENTE V (A-Roseville,ex1)" | ||||Cc: ; ||||Sent: Thursday, March 22, 2001 4:00 PM ||||Subject: Re: CRC vs CHKSUM presentation slides |||| |||| ||||> Vicente- ||||> ||||> I just took another look through your slides after seeing the ||||> presentation on Monday. They were very well-done. I have ||||> one question, though. If the CCITT-CRC32 is considered "good ||||> enough", then would the Ethernet CRC32 also be good enough? The ||||> reason I ask is that every hardware vendor involved in building ||||> iSCSI stuff already has implementations of the Ethernet CRC, ||||> which is used for both Ethernet and Fibre Channel. ||||> ||||> The Ethernet poly has more terms than CCITT, and perhaps is ||||> not as good as CRC-32C (any thoughts?), but everyone has hardware ||||> and software for this, with proven interoperability (bit and ||||> byte order, etc). Performance-wise, it will be there for ||||> 10Gb Ethernet, so it should be fast enough. ||||> ||||> So if the Ethernet poly is deemed good enough (even if it's not ||||> the best), and fast enough (even if it's not the fastest), why ||||> not use it? I think we would stand a much better chance of ||||> achieving interoperability in a short time. ||||> ||||> Please let me know what you think of this; I realize that a few ||||> of my questions were speculative. |||| ||||Since the iSCSI messages will often be encapsulated in ethernet ||||packets, there is some value to using a different CRC. Link ||||errors are double protected with two different CRCs. If ||||ethernet and iSCSI use the same polynomial, there is little ||||additional coverage against link errors. This point may not ||||be decisive, but all other things being equal or almost ||||equal, it is worth considering. |||| ||| || | From owner-ips@ece.cmu.edu Fri Mar 23 18:19:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA05752 for ; Fri, 23 Mar 2001 18:19:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NLJQs16989 for ips-outgoing; Fri, 23 Mar 2001 16:19:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NLItr16932 for ; Fri, 23 Mar 2001 16:18:55 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id QAA14761; Fri, 23 Mar 2001 16:18:36 -0500 (EST) Message-ID: <3ABBBD98.93299B8E@cisco.com> Date: Fri, 23 Mar 2001 15:18:16 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Douglas Otis CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: iSCSI Naming: WWUIs, URNs, and namespaces References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Doug- We are using a reversed domain name as a naming authority -- something that you "own" that you can use to guarantee that your locally-generated names are world-wide unique. Users of a WWUI will under no circumstances attempt to turn one of these into an address. Please re-read the naming & discovery draft section on this format; it spells this out quite clearly, since we realized early that this would be a source of confusion. -- Mark Douglas Otis wrote: > > David, > > If Microsoft responds by using a NetBIOS node-status query see: > http://support.microsoft.com/support/kb/articles/Q154/5/53.ASP > perhaps you can see why dependence on reverse DNS is prone to errors and > spoofing in general. > > You have stated at the IETF meeting there is no need for configuration > management tools to be interoperable and as such there is no need for there > to be any defined schema to support these naming conventions or > configuration structures. > > Rather than using Name->IP->Reverse-Name as a means of getting the desired > result, an LDAP schema will allow these types of associations without the > need to invent new or modified name servers. Unless a schema is defined, > each required utility will be based on proprietary schema that will not > interoperate with other pieces of equipment from other vendors. The ability > of this equipment to interoperate without proprietary intervention should be > a requirement and of course LDAP already provides the means of promulgating > changes across multi-vendor environments. Insisting such vital components > remain vendor unique is a major disservice to the needs of all affected > parties. With this schema defined, the naming issues are cleanly closed > where SLP as example can then leverage this information and changes emerge > as LDAP slave notifications. LDAP does not need to be large, can be > embedded, and allowed to contain only a portion of the information relevant > to a switch. > > Doug > > > Mark, > > > > I need to read your note in more detail before > > I can respond completely, but we may be close > > to a solution here. At a high level, the notion > > of the WWUI as yet another globally unique > > namespace (which is what it is, even if it's > > assembled by gluing in existing namespaces) > > needs to go away. The set of {canonical > > iSCSI, eui[WWN], reverse-dns} as usable namespaces > > may be the right way forward, especially as it > > looks like you've made a good start on the > > rationale required to justify reverse DNS. > > > > We need to describe how to uniquely identify > > Initiators and Targets without introducing > > the notion of a WWUI as a new first-class > > global unique identifier abstraction (e.g., > > that some other set of people may try to > > use and/or extend). In other words, we > > need to scrap the current WWUI structure > > without losing the important functionality > > that iSCSI needs in this area. Some amount > > of this may just be word-smithing to avoid > > unnecessarily waving red flags ... > > > > More to come ... > > > > --David > > > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > > > > > -----Original Message----- > > > From: Mark Bakke [SMTP:mbakke@cisco.com] > > > Sent: Friday, March 23, 2001 9:36 AM > > > To: Black_David@emc.com > > > Cc: ips@ece.cmu.edu > > > Subject: Re: iSCSI Naming: WWUIs, URNs, and namespaces > > > > > > > > > David- > > > > > > This is starting to seem like work. In case they help, > > > please see my comments below. We can take this up in > > > more detail in the N&D team. > > > > > > -- > > > Mark > > > > > > Black_David@emc.com wrote: > > > > > > > > I don't like naming issues. :-) :-) > > > > > > > > After suitable consulting with some members of > > > > the IESG and IAB, I have some news to convey about > > > > the current approach to iSCSI naming. > > > > > > > > The IESG will not approve another global namespace > > > > for iSCSI's use - this means that WWUIs as currently > > > > > > So I take it that WWUIs are fine, as long as they use > > > existing global name spaces. > > > > > > > designed will need to be revised out of the > > > > naming and discovery draft, and that it will not be > > > > possible to proceed with the WWUI URN draft > > > > as an official IPS WG work item. The best course of > > > > action would probably be to allow the WWUI URN draft > > > > to expire without further revision. > > > > > > > > To a first approximation, WWUIs are/were a "grand > > > > unified theory" of naming, in that any namespace could > > > > be glued into the WWUI world (as several were). > > > > > > The reason we glued these in was to avoid creating a new > > > global name space. We allowed the use of a few currently > > > available global name spaces. How can this be a problem? > > > > > > > The WG is being directed to instead focus on the > > > > individual namespaces and make sure that the ones that > > > > are used are in fact necessary. iSCSI can use text > > > > keys to identify which sort of name is being used > > > > (one key for each sort of format, for each instance > > > > in which a name is used), and it may be possible > > > > to encode the name format in the parse rules for the > > > > values of iSCSI keys to avoid proliferation of keys. > > > > > > > > Taking a look at the namespaces in the current iSCSI > > > > naming and discovery draft, here's some initial > > > > guidance from this WG co-chair: > > > > iscsi - canonical target -- This should be fine. > > > > eui - WWNs -- The use of these for storage makes eminent > > > > sense. I don't see a problem here. > > > > > > So far, so good. Does that mean that we can keep the above > > > two, in the current WWUI, as we have it formatted? > > > > > > EUI, although it's not that flexible (see later), is required > > > for interoperating with and adding iSCSI to existing devices. > > > > > > > dns - hostnames -- Use of these should be abandoned as > > > > not only are they not really URNs (as indicated > > > > in the draft), but their intended usage is straying > > > > into the tarpit known as "URN resolution". Faster > > > > progress will made by staying out. A DNS hostname > > > > can be put into an Alias or something new can be > > > > invented to carry it as a Location Hint, BUT the > > > > relevant URN WG RFCs and drafts on URN resolution > > > > should be reviewed before proceeding too far in this > > > > direction. > > > > > > I don't like this one that much, either, for the same reasons. > > > > > > > iscsi - Reverse DNS and oui - Org. Unique Identifier -- > > > > The rationale for these is not clear to me. > > > > Assuming that WWNs are going to be available for > > > > use in naming iSCSI Initiators and Targets, what > > > > are the problems that these sorts of names solve > > > > that WWNs don't? It appears that one of the problems > > > > may be who can get/create them. Discussion of this > > > > on the list would be appropriate. > > > > > > These two formats use existing global name spaces, and allow > > > the implementor or end user to take a more flexible approach > > > to naming than is offered by the EUI-64. When dealing with > > > a large number of logical targets, burning WWNs (essentially, > > > MAC addresses), and attempting to keep them tied to a logical > > > entity without tying them to hardware is not practical. > > > > > > Both of these formats accomplish the same thing, and we could > > > make do with just one of them. The only real differences are: > > > > > > - OUI is a fixed length naming authority; reverse DNS is variable > > > and could result in longer names. > > > > > > - A human looking at a reverse DNS-based name can easily determine > > > who the naming authority is; a human looking at an OUI-based > > > name will usually have to go look it up at IEEE. > > > > > > - Only a hardware manufacturer (and a few software manufacters) > > > will already have an OUI. Others, such as storage service > > > providers, large end customers, or software driver providers, > > > would either be out-of-luck for naming their devices, or would > > > have to register for an OUI, which would otherwise be a waste > > > of IEEE resources. OTOH, a storage service provider could > > > provide its own names by using the reverse-DNS authority. > > > Everyone has a DNS name. This would enable the SSP to provide > > > its customers with a WWUI that would not change if the back-end > > > storage device was replaced. After all, it's not the device > > > that's important to this customer, it's the DATA. > > > > > > - If we had to pick one or the other, I would pick the reverse > > > DNS, since it offers the ability to generate unique names to > > > a wider variety of users. > > > > > > > In any case, the fewer name formats we have to deal with, > > > > the better. > > > > > > Although I probably should have discussed this with the rest > > > of the NDT first, I would personally be happy with: > > > > > > - iscsi - The canonical name. > > > - eui - For compatibility with existing device naming schemes. > > > - reverse-dns - For better naming flexibility moving forward. > > > > > > Perhaps the IESG would accept the WWUI with just these three. > > > Or perhaps we could just make the WWUI into a URN? I don't > > > know if that would help, but perhaps... > > > > > > The main point is that we are NOT creating any new name > > > spaces; we really are just using what's already there. Maybe > > > by specifying fewer of the options that are already there, we > > > would be in better shape. > > > > > > > I want to try to anticipate an objection to this, which > > > > would note that from a functional viewpoint the basic > > > > impact of this is to move some characters from one text > > > > string to another (e.g., from a WWUI type designator > > > > to part of an iSCSI text key), and wonder if this is > > > > a distinction without a difference. One of the reasons > > > > for the that started this post is that a functional > > > > view is not sufficient for naming - how things are named, > > > > the intended usage of names and their scope matter a lot. > > > > > > So that leads to the question: Has any other WG come up with > > > a world-wide-unique-identifier for something else that the > > > IESG folks deems acceptable? If we can see a few examples, > > > we would be better able to anticipate what they want. > > > > > > > This is particularly true when considering the structure > > > > of a namespace and how that structure may be extended. > > > > The upshot is that avoiding introduction of something > > > > claiming to be yet another global namespace is important > > > > (i.e., use existing namespaces with global scope in preference > > > > to inventing new ones). The resulting need to define > > > > the name spaces/formats in the main iSCSI spec. is > > > > probably a "feature" as it forces us to pay more > > > > attention to the sorts of names we use and raises the > > > > bar for adding additional sorts of names in the future. > > > > > > > I will be working with > > > > the naming and discovery team in my "copious spare time" > > > > to make sure that we don't lose the valuable work and > > > > progress they've made to date as a consequence of this > > > > change. Discussion on the list about what sort > > > > of names are important (e.g., the Reverse DNS and OUI > > > > namespaces) and why would be useful. > > > > > > From a requirements point of view, please respond if any > > > of the types of names are important to you (plural). > > > > > > Hopefully, we can get this out of the way quickly. > > > > > > > > > > > > > Thanks, > > > > --David > > > > > > > > --------------------------------------------------- > > > > David L. Black, Senior Technologist > > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > > --------------------------------------------------- > > > > > > -- > > > Mark A. Bakke > > > Cisco Systems > > > mbakke@cisco.com > > > 763.398.1054 > > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Fri Mar 23 18:19:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA05763 for ; Fri, 23 Mar 2001 18:19:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NKMJt12787 for ips-outgoing; Fri, 23 Mar 2001 15:22:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NKLLr12707 for ; Fri, 23 Mar 2001 15:21:21 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GAO00G143721G@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Fri, 23 Mar 2001 12:21:05 -0800 (PST) Date: Fri, 23 Mar 2001 12:19:41 -0800 From: Douglas Otis Subject: RE: iSCSI Naming: WWUIs, URNs, and namespaces In-reply-to: <0F31E5C394DAD311B60C00E029101A070801531D@corpmx9.isus.emc.com> To: Black_David@emc.com, mbakke@cisco.com Cc: ips@ece.cmu.edu Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, If Microsoft responds by using a NetBIOS node-status query see: http://support.microsoft.com/support/kb/articles/Q154/5/53.ASP perhaps you can see why dependence on reverse DNS is prone to errors and spoofing in general. You have stated at the IETF meeting there is no need for configuration management tools to be interoperable and as such there is no need for there to be any defined schema to support these naming conventions or configuration structures. Rather than using Name->IP->Reverse-Name as a means of getting the desired result, an LDAP schema will allow these types of associations without the need to invent new or modified name servers. Unless a schema is defined, each required utility will be based on proprietary schema that will not interoperate with other pieces of equipment from other vendors. The ability of this equipment to interoperate without proprietary intervention should be a requirement and of course LDAP already provides the means of promulgating changes across multi-vendor environments. Insisting such vital components remain vendor unique is a major disservice to the needs of all affected parties. With this schema defined, the naming issues are cleanly closed where SLP as example can then leverage this information and changes emerge as LDAP slave notifications. LDAP does not need to be large, can be embedded, and allowed to contain only a portion of the information relevant to a switch. Doug > Mark, > > I need to read your note in more detail before > I can respond completely, but we may be close > to a solution here. At a high level, the notion > of the WWUI as yet another globally unique > namespace (which is what it is, even if it's > assembled by gluing in existing namespaces) > needs to go away. The set of {canonical > iSCSI, eui[WWN], reverse-dns} as usable namespaces > may be the right way forward, especially as it > looks like you've made a good start on the > rationale required to justify reverse DNS. > > We need to describe how to uniquely identify > Initiators and Targets without introducing > the notion of a WWUI as a new first-class > global unique identifier abstraction (e.g., > that some other set of people may try to > use and/or extend). In other words, we > need to scrap the current WWUI structure > without losing the important functionality > that iSCSI needs in this area. Some amount > of this may just be word-smithing to avoid > unnecessarily waving red flags ... > > More to come ... > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > > -----Original Message----- > > From: Mark Bakke [SMTP:mbakke@cisco.com] > > Sent: Friday, March 23, 2001 9:36 AM > > To: Black_David@emc.com > > Cc: ips@ece.cmu.edu > > Subject: Re: iSCSI Naming: WWUIs, URNs, and namespaces > > > > > > David- > > > > This is starting to seem like work. In case they help, > > please see my comments below. We can take this up in > > more detail in the N&D team. > > > > -- > > Mark > > > > Black_David@emc.com wrote: > > > > > > I don't like naming issues. :-) :-) > > > > > > After suitable consulting with some members of > > > the IESG and IAB, I have some news to convey about > > > the current approach to iSCSI naming. > > > > > > The IESG will not approve another global namespace > > > for iSCSI's use - this means that WWUIs as currently > > > > So I take it that WWUIs are fine, as long as they use > > existing global name spaces. > > > > > designed will need to be revised out of the > > > naming and discovery draft, and that it will not be > > > possible to proceed with the WWUI URN draft > > > as an official IPS WG work item. The best course of > > > action would probably be to allow the WWUI URN draft > > > to expire without further revision. > > > > > > To a first approximation, WWUIs are/were a "grand > > > unified theory" of naming, in that any namespace could > > > be glued into the WWUI world (as several were). > > > > The reason we glued these in was to avoid creating a new > > global name space. We allowed the use of a few currently > > available global name spaces. How can this be a problem? > > > > > The WG is being directed to instead focus on the > > > individual namespaces and make sure that the ones that > > > are used are in fact necessary. iSCSI can use text > > > keys to identify which sort of name is being used > > > (one key for each sort of format, for each instance > > > in which a name is used), and it may be possible > > > to encode the name format in the parse rules for the > > > values of iSCSI keys to avoid proliferation of keys. > > > > > > Taking a look at the namespaces in the current iSCSI > > > naming and discovery draft, here's some initial > > > guidance from this WG co-chair: > > > iscsi - canonical target -- This should be fine. > > > eui - WWNs -- The use of these for storage makes eminent > > > sense. I don't see a problem here. > > > > So far, so good. Does that mean that we can keep the above > > two, in the current WWUI, as we have it formatted? > > > > EUI, although it's not that flexible (see later), is required > > for interoperating with and adding iSCSI to existing devices. > > > > > dns - hostnames -- Use of these should be abandoned as > > > not only are they not really URNs (as indicated > > > in the draft), but their intended usage is straying > > > into the tarpit known as "URN resolution". Faster > > > progress will made by staying out. A DNS hostname > > > can be put into an Alias or something new can be > > > invented to carry it as a Location Hint, BUT the > > > relevant URN WG RFCs and drafts on URN resolution > > > should be reviewed before proceeding too far in this > > > direction. > > > > I don't like this one that much, either, for the same reasons. > > > > > iscsi - Reverse DNS and oui - Org. Unique Identifier -- > > > The rationale for these is not clear to me. > > > Assuming that WWNs are going to be available for > > > use in naming iSCSI Initiators and Targets, what > > > are the problems that these sorts of names solve > > > that WWNs don't? It appears that one of the problems > > > may be who can get/create them. Discussion of this > > > on the list would be appropriate. > > > > These two formats use existing global name spaces, and allow > > the implementor or end user to take a more flexible approach > > to naming than is offered by the EUI-64. When dealing with > > a large number of logical targets, burning WWNs (essentially, > > MAC addresses), and attempting to keep them tied to a logical > > entity without tying them to hardware is not practical. > > > > Both of these formats accomplish the same thing, and we could > > make do with just one of them. The only real differences are: > > > > - OUI is a fixed length naming authority; reverse DNS is variable > > and could result in longer names. > > > > - A human looking at a reverse DNS-based name can easily determine > > who the naming authority is; a human looking at an OUI-based > > name will usually have to go look it up at IEEE. > > > > - Only a hardware manufacturer (and a few software manufacters) > > will already have an OUI. Others, such as storage service > > providers, large end customers, or software driver providers, > > would either be out-of-luck for naming their devices, or would > > have to register for an OUI, which would otherwise be a waste > > of IEEE resources. OTOH, a storage service provider could > > provide its own names by using the reverse-DNS authority. > > Everyone has a DNS name. This would enable the SSP to provide > > its customers with a WWUI that would not change if the back-end > > storage device was replaced. After all, it's not the device > > that's important to this customer, it's the DATA. > > > > - If we had to pick one or the other, I would pick the reverse > > DNS, since it offers the ability to generate unique names to > > a wider variety of users. > > > > > In any case, the fewer name formats we have to deal with, > > > the better. > > > > Although I probably should have discussed this with the rest > > of the NDT first, I would personally be happy with: > > > > - iscsi - The canonical name. > > - eui - For compatibility with existing device naming schemes. > > - reverse-dns - For better naming flexibility moving forward. > > > > Perhaps the IESG would accept the WWUI with just these three. > > Or perhaps we could just make the WWUI into a URN? I don't > > know if that would help, but perhaps... > > > > The main point is that we are NOT creating any new name > > spaces; we really are just using what's already there. Maybe > > by specifying fewer of the options that are already there, we > > would be in better shape. > > > > > I want to try to anticipate an objection to this, which > > > would note that from a functional viewpoint the basic > > > impact of this is to move some characters from one text > > > string to another (e.g., from a WWUI type designator > > > to part of an iSCSI text key), and wonder if this is > > > a distinction without a difference. One of the reasons > > > for the that started this post is that a functional > > > view is not sufficient for naming - how things are named, > > > the intended usage of names and their scope matter a lot. > > > > So that leads to the question: Has any other WG come up with > > a world-wide-unique-identifier for something else that the > > IESG folks deems acceptable? If we can see a few examples, > > we would be better able to anticipate what they want. > > > > > This is particularly true when considering the structure > > > of a namespace and how that structure may be extended. > > > The upshot is that avoiding introduction of something > > > claiming to be yet another global namespace is important > > > (i.e., use existing namespaces with global scope in preference > > > to inventing new ones). The resulting need to define > > > the name spaces/formats in the main iSCSI spec. is > > > probably a "feature" as it forces us to pay more > > > attention to the sorts of names we use and raises the > > > bar for adding additional sorts of names in the future. > > > > > I will be working with > > > the naming and discovery team in my "copious spare time" > > > to make sure that we don't lose the valuable work and > > > progress they've made to date as a consequence of this > > > change. Discussion on the list about what sort > > > of names are important (e.g., the Reverse DNS and OUI > > > namespaces) and why would be useful. > > > > From a requirements point of view, please respond if any > > of the types of names are important to you (plural). > > > > Hopefully, we can get this out of the way quickly. > > > > > > > > > Thanks, > > > --David > > > > > > --------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > --------------------------------------------------- > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 > From owner-ips@ece.cmu.edu Fri Mar 23 20:13:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA07057 for ; Fri, 23 Mar 2001 20:13:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NN6MO24190 for ips-outgoing; Fri, 23 Mar 2001 18:06:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NN5qr24137 for ; Fri, 23 Mar 2001 18:05:52 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Fri, 23 Mar 2001 15:05:45 -0800 Message-ID: From: Joshua Tseng To: ips@ece.cmu.edu Subject: iFCP & iSNS Presentations Date: Fri, 23 Mar 2001 15:05:45 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, Here are links to the iFCP and iSNS presentations from Minneapolis: iFCP Status: http://www.nishansystems.com/ietf/ifcp-plan.pdf iFCP Encapsulation Requirements: http://www.nishansystems.com/ietf/ifcp_encapsulation_requirements.pdf iSNS: http://www.nishansystems.com/ietf/iSNS-ietf-minn.ppt Regards, Josh Tseng From owner-ips@ece.cmu.edu Fri Mar 23 20:15:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA07076 for ; Fri, 23 Mar 2001 20:15:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NMxNE23724 for ips-outgoing; Fri, 23 Mar 2001 17:59:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NMwPr23669 for ; Fri, 23 Mar 2001 17:58:25 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GAO004VVAGP17@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Fri, 23 Mar 2001 14:58:04 -0800 (PST) Date: Fri, 23 Mar 2001 14:56:40 -0800 From: Douglas Otis Subject: RE: iSCSI Naming: WWUIs, URNs, and namespaces In-reply-to: <3ABBBD98.93299B8E@cisco.com> To: mbakke@cisco.com Cc: Black_David@emc.com, ips@ece.cmu.edu Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mark, No, I was not confused nor did I expect clients to turn a WWUI into an address as obviously you already have an address to do the reverse DNS. I indicated the process as Name->IP->Reverse-Name and illustrated this process by pointing to the technique that Microsoft used to obtain names from machines. Their technique is prone to errors and spoofing. Secure management and promulgation was my concern. The level of effort to extract secure information from DNS is not simpler than that from LDAP. The mayhem from each vendor finding a unique method of obtaining and promulgating information would not be pleasant. Doug > Doug- > > We are using a reversed domain name as a naming authority -- something > that you "own" that you can use to guarantee that your locally-generated > names are world-wide unique. > > Users of a WWUI will under no circumstances attempt to turn one of these > into an address. > > Please re-read the naming & discovery draft section on this format; it > spells this out quite clearly, since we realized early that this would > be a source of confusion. > > -- > Mark > > > Douglas Otis wrote: > > > > David, > > > > If Microsoft responds by using a NetBIOS node-status query see: > > http://support.microsoft.com/support/kb/articles/Q154/5/53.ASP > > perhaps you can see why dependence on reverse DNS is prone to errors and > > spoofing in general. > > > > You have stated at the IETF meeting there is no need for configuration > > management tools to be interoperable and as such there is no > need for there > > to be any defined schema to support these naming conventions or > > configuration structures. > > > > Rather than using Name->IP->Reverse-Name as a means of getting > the desired > > result, an LDAP schema will allow these types of associations > without the > > need to invent new or modified name servers. Unless a schema > is defined, > > each required utility will be based on proprietary schema that will not > > interoperate with other pieces of equipment from other vendors. > The ability > > of this equipment to interoperate without proprietary > intervention should be > > a requirement and of course LDAP already provides the means of > promulgating > > changes across multi-vendor environments. Insisting such vital > components > > remain vendor unique is a major disservice to the needs of all affected > > parties. With this schema defined, the naming issues are cleanly closed > > where SLP as example can then leverage this information and > changes emerge > > as LDAP slave notifications. LDAP does not need to be large, can be > > embedded, and allowed to contain only a portion of the > information relevant > > to a switch. > > > > Doug > > > > > Mark, > > > > > > I need to read your note in more detail before > > > I can respond completely, but we may be close > > > to a solution here. At a high level, the notion > > > of the WWUI as yet another globally unique > > > namespace (which is what it is, even if it's > > > assembled by gluing in existing namespaces) > > > needs to go away. The set of {canonical > > > iSCSI, eui[WWN], reverse-dns} as usable namespaces > > > may be the right way forward, especially as it > > > looks like you've made a good start on the > > > rationale required to justify reverse DNS. > > > > > > We need to describe how to uniquely identify > > > Initiators and Targets without introducing > > > the notion of a WWUI as a new first-class > > > global unique identifier abstraction (e.g., > > > that some other set of people may try to > > > use and/or extend). In other words, we > > > need to scrap the current WWUI structure > > > without losing the important functionality > > > that iSCSI needs in this area. Some amount > > > of this may just be word-smithing to avoid > > > unnecessarily waving red flags ... > > > > > > More to come ... > > > > > > --David > > > > > > --------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > --------------------------------------------------- > > > > > > > -----Original Message----- > > > > From: Mark Bakke [SMTP:mbakke@cisco.com] > > > > Sent: Friday, March 23, 2001 9:36 AM > > > > To: Black_David@emc.com > > > > Cc: ips@ece.cmu.edu > > > > Subject: Re: iSCSI Naming: WWUIs, URNs, and namespaces > > > > > > > > > > > > David- > > > > > > > > This is starting to seem like work. In case they help, > > > > please see my comments below. We can take this up in > > > > more detail in the N&D team. > > > > > > > > -- > > > > Mark > > > > > > > > Black_David@emc.com wrote: > > > > > > > > > > I don't like naming issues. :-) :-) > > > > > > > > > > After suitable consulting with some members of > > > > > the IESG and IAB, I have some news to convey about > > > > > the current approach to iSCSI naming. > > > > > > > > > > The IESG will not approve another global namespace > > > > > for iSCSI's use - this means that WWUIs as currently > > > > > > > > So I take it that WWUIs are fine, as long as they use > > > > existing global name spaces. > > > > > > > > > designed will need to be revised out of the > > > > > naming and discovery draft, and that it will not be > > > > > possible to proceed with the WWUI URN draft > > > > > as an official IPS WG work item. The best course of > > > > > action would probably be to allow the WWUI URN draft > > > > > to expire without further revision. > > > > > > > > > > To a first approximation, WWUIs are/were a "grand > > > > > unified theory" of naming, in that any namespace could > > > > > be glued into the WWUI world (as several were). > > > > > > > > The reason we glued these in was to avoid creating a new > > > > global name space. We allowed the use of a few currently > > > > available global name spaces. How can this be a problem? > > > > > > > > > The WG is being directed to instead focus on the > > > > > individual namespaces and make sure that the ones that > > > > > are used are in fact necessary. iSCSI can use text > > > > > keys to identify which sort of name is being used > > > > > (one key for each sort of format, for each instance > > > > > in which a name is used), and it may be possible > > > > > to encode the name format in the parse rules for the > > > > > values of iSCSI keys to avoid proliferation of keys. > > > > > > > > > > Taking a look at the namespaces in the current iSCSI > > > > > naming and discovery draft, here's some initial > > > > > guidance from this WG co-chair: > > > > > iscsi - canonical target -- This should be fine. > > > > > eui - WWNs -- The use of these for storage makes eminent > > > > > sense. I don't see a problem here. > > > > > > > > So far, so good. Does that mean that we can keep the above > > > > two, in the current WWUI, as we have it formatted? > > > > > > > > EUI, although it's not that flexible (see later), is required > > > > for interoperating with and adding iSCSI to existing devices. > > > > > > > > > dns - hostnames -- Use of these should be abandoned as > > > > > not only are they not really URNs (as indicated > > > > > in the draft), but their intended usage is straying > > > > > into the tarpit known as "URN resolution". Faster > > > > > progress will made by staying out. A DNS hostname > > > > > can be put into an Alias or something new can be > > > > > invented to carry it as a Location Hint, BUT the > > > > > relevant URN WG RFCs and drafts on URN resolution > > > > > should be reviewed before proceeding too far in this > > > > > direction. > > > > > > > > I don't like this one that much, either, for the same reasons. > > > > > > > > > iscsi - Reverse DNS and oui - Org. Unique Identifier -- > > > > > The rationale for these is not clear to me. > > > > > Assuming that WWNs are going to be available for > > > > > use in naming iSCSI Initiators and Targets, what > > > > > are the problems that these sorts of names solve > > > > > that WWNs don't? It appears that one of the problems > > > > > may be who can get/create them. Discussion of this > > > > > on the list would be appropriate. > > > > > > > > These two formats use existing global name spaces, and allow > > > > the implementor or end user to take a more flexible approach > > > > to naming than is offered by the EUI-64. When dealing with > > > > a large number of logical targets, burning WWNs (essentially, > > > > MAC addresses), and attempting to keep them tied to a logical > > > > entity without tying them to hardware is not practical. > > > > > > > > Both of these formats accomplish the same thing, and we could > > > > make do with just one of them. The only real differences are: > > > > > > > > - OUI is a fixed length naming authority; reverse DNS is variable > > > > and could result in longer names. > > > > > > > > - A human looking at a reverse DNS-based name can easily determine > > > > who the naming authority is; a human looking at an OUI-based > > > > name will usually have to go look it up at IEEE. > > > > > > > > - Only a hardware manufacturer (and a few software manufacters) > > > > will already have an OUI. Others, such as storage service > > > > providers, large end customers, or software driver providers, > > > > would either be out-of-luck for naming their devices, or would > > > > have to register for an OUI, which would otherwise be a waste > > > > of IEEE resources. OTOH, a storage service provider could > > > > provide its own names by using the reverse-DNS authority. > > > > Everyone has a DNS name. This would enable the SSP to provide > > > > its customers with a WWUI that would not change if the back-end > > > > storage device was replaced. After all, it's not the device > > > > that's important to this customer, it's the DATA. > > > > > > > > - If we had to pick one or the other, I would pick the reverse > > > > DNS, since it offers the ability to generate unique names to > > > > a wider variety of users. > > > > > > > > > In any case, the fewer name formats we have to deal with, > > > > > the better. > > > > > > > > Although I probably should have discussed this with the rest > > > > of the NDT first, I would personally be happy with: > > > > > > > > - iscsi - The canonical name. > > > > - eui - For compatibility with existing device naming schemes. > > > > - reverse-dns - For better naming flexibility moving forward. > > > > > > > > Perhaps the IESG would accept the WWUI with just these three. > > > > Or perhaps we could just make the WWUI into a URN? I don't > > > > know if that would help, but perhaps... > > > > > > > > The main point is that we are NOT creating any new name > > > > spaces; we really are just using what's already there. Maybe > > > > by specifying fewer of the options that are already there, we > > > > would be in better shape. > > > > > > > > > I want to try to anticipate an objection to this, which > > > > > would note that from a functional viewpoint the basic > > > > > impact of this is to move some characters from one text > > > > > string to another (e.g., from a WWUI type designator > > > > > to part of an iSCSI text key), and wonder if this is > > > > > a distinction without a difference. One of the reasons > > > > > for the that started this post is that a functional > > > > > view is not sufficient for naming - how things are named, > > > > > the intended usage of names and their scope matter a lot. > > > > > > > > So that leads to the question: Has any other WG come up with > > > > a world-wide-unique-identifier for something else that the > > > > IESG folks deems acceptable? If we can see a few examples, > > > > we would be better able to anticipate what they want. > > > > > > > > > This is particularly true when considering the structure > > > > > of a namespace and how that structure may be extended. > > > > > The upshot is that avoiding introduction of something > > > > > claiming to be yet another global namespace is important > > > > > (i.e., use existing namespaces with global scope in preference > > > > > to inventing new ones). The resulting need to define > > > > > the name spaces/formats in the main iSCSI spec. is > > > > > probably a "feature" as it forces us to pay more > > > > > attention to the sorts of names we use and raises the > > > > > bar for adding additional sorts of names in the future. > > > > > > > > > I will be working with > > > > > the naming and discovery team in my "copious spare time" > > > > > to make sure that we don't lose the valuable work and > > > > > progress they've made to date as a consequence of this > > > > > change. Discussion on the list about what sort > > > > > of names are important (e.g., the Reverse DNS and OUI > > > > > namespaces) and why would be useful. > > > > > > > > From a requirements point of view, please respond if any > > > > of the types of names are important to you (plural). > > > > > > > > Hopefully, we can get this out of the way quickly. > > > > > > > > > > > > > > > > > Thanks, > > > > > --David > > > > > > > > > > --------------------------------------------------- > > > > > David L. Black, Senior Technologist > > > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > > > --------------------------------------------------- > > > > > > > > -- > > > > Mark A. Bakke > > > > Cisco Systems > > > > mbakke@cisco.com > > > > 763.398.1054 > > > > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 > From owner-ips@ece.cmu.edu Fri Mar 23 20:16:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA07113 for ; Fri, 23 Mar 2001 20:16:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2NN9Ml24387 for ips-outgoing; Fri, 23 Mar 2001 18:09:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2NN9Br24378 for ; Fri, 23 Mar 2001 18:09:11 -0500 (EST) Received: from ljoy ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0GAO00HTJAV0W2@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Fri, 23 Mar 2001 15:07:51 -0800 (PST) Date: Fri, 23 Mar 2001 15:05:15 -0800 From: Douglas Otis Subject: RE: CRC vs CHKSUM presentation slides In-reply-to: To: vince_cavanna@agilent.com, mbakke@cisco.com Cc: ips@ece.cmu.edu Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Vince, Sorry, I had confused a comparison made by Julian with yours. There they compared the number of instructions without considering expense of accessing a large lookup. How do you view the TCP checksum related to various CRC susceptible errors? Doug > Hello Doug, > I have inserted my comments below. > Vince > > > |-----Original Message----- > |From: Douglas Otis [mailto:dotis@sanlight.net] > |Sent: Thursday, March 22, 2001 3:16 PM > |To: Mark Bakke; CAVANNA,VICENTE V (A-Roseville,ex1) > |Cc: ips@ece.cmu.edu; ipsan@rtl.rose.agilent.com > |Subject: RE: CRC vs CHKSUM presentation slides > | > | > |Mark, > | > |One small note on these slides. Although the number of instructions > |required to implement CRC-32 with 128K table lookups compares > |closely to > |that of Adler-32, Adler-32 is not accessing a 128K byte lookup > |table. These > |slides made a comparison of instructions without considering additional > |memory access required or the size of the table required. > > I am not sure what comparison you are referring to since I did not compare > the cost of implementing Checksums against that of implementing > CRCs. All I > compared was the cost, in gates (not table size), of a complex CRC vs that > of a simple CRC. > > | > |If you were to use the same CRC-32, you will find the same levels of > |protection as this CRC is to protect against various types of equipment > |failures NOT protected by the Ethernet/FDDI/Fibre-Channel CRC, then a > |replication of this polynomial should not be a problem in that > |sense. For > |those that wish to include a different polynomial tend to justify this > |effort by indicating this would improve the present physical > |layer checks. > | > |The problem that I see with that conclusion is errors seen > |outside of any > |potential Ethernet CRC failure are of such orders of magnitude as to > |out-weight any replicate benefit derived. In that light, even > |Adler-32 may > |provide benefit in that case. The next area being looked at > |is the number > |of bits detected in a burst error. The unprotected equipment > |is not likely > |using a serial method of delivery, so you are again left > |errors that may > |provide strange error patterns. The concept supporting a different > |polynomial was its greater ability to detect burst errors. > > If I understand you correctly you are saying that the ethernet > polynomial is > good enough to protect against the type of errors iSCSI needs to provide > protection for. I agree. My main argument for wanting a simple (standard) > polynomial is that I would like to take advantage of the lower > cost in gate > count. > > | > |The large piece of information missing from this presentation was a > |characterization of the types of errors seen by intermediate equipment. > |There was a bullet that suggested this equipment will produce > |burst errors. > |If so, what is the nature of this equipment and what are the prevalent > |errors? From that perspective a comparisons could be made. > |But as we are > |familiar with noise distributions of single bit errors, this became our > |focus. It is a bit like looking for your keys under the light > |post even > |though you dropped them in the dark. You wish to look where > |the light is > |better. > > I give a lot more detail on small number of single bit errors and small > number of bursts because there is theory that I can use to analyze those > cases. Like you and others, I have struggled with the claims that it is > important for iSCSI to provide protection from small number of errors. I > have finally convinced myself that small number of errors are > probable even > in our environment. In my slides I outlined several important error > mechanims that tend to produce few number of individual errors. Here are > some I can think of at the moment: > > 1. Random memory errors. > 2. Marginal timing paths in the hardware. > 3. Noise (crosstalk, power supply noise). > 4. Leakage problems in bistable devices due to manufacturing defects. > 5. Hardware degradation. > > > Of course there are also possible error patterns that contain > lots of errors > and unfortunately I cannot say much about about them other than that 1 in > 2^32 (if all errors are equiprobable) will get through both checksums and > CRCs and that those that get through the CRC are more random looking than > those that get through the Checksums. > > | > |Doug > | > |> Vicente- > |> > |> I just took another look through your slides after seeing the > |> presentation on Monday. They were very well-done. I have > |> one question, though. If the CCITT-CRC32 is considered "good > |> enough", then would the Ethernet CRC32 also be good enough? The > |> reason I ask is that every hardware vendor involved in building > |> iSCSI stuff already has implementations of the Ethernet CRC, > |> which is used for both Ethernet and Fibre Channel. > |> > |> The Ethernet poly has more terms than CCITT, and perhaps is > |> not as good as CRC-32C (any thoughts?), but everyone has hardware > |> and software for this, with proven interoperability (bit and > |> byte order, etc). Performance-wise, it will be there for > |> 10Gb Ethernet, so it should be fast enough. > |> > |> So if the Ethernet poly is deemed good enough (even if it's not > |> the best), and fast enough (even if it's not the fastest), why > |> not use it? I think we would stand a much better chance of > |> achieving interoperability in a short time. > |> > |> Please let me know what you think of this; I realize that a few > |> of my questions were speculative. > |> > |> Regards, > |> > |> Mark > |> > |> "CAVANNA,VICENTE V (A-Roseville,ex1)" wrote: > |> > > |> > Here are the slides I presented at the IETF-50 in Minneapolis. > |> > > |> > Vicente Cavanna > |> > Agilent Technologies > |> > > |> > <> > |> > > |> > > |> ------------------------------------------------------------------ > |> ---------------------------------- > |> > Name: CRCorChksm.pdf > |> > CRCorChksm.pdf Type: Portable Document Format > |(application/pdf) > |> > Encoding: base64 > |> > |> -- > |> Mark A. Bakke > |> Cisco Systems > |> mbakke@cisco.com > |> 763.398.1054 > |> > | > From owner-ips@ece.cmu.edu Sat Mar 24 00:06:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA12442 for ; Sat, 24 Mar 2001 00:06:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2O2pS007679 for ips-outgoing; Fri, 23 Mar 2001 21:51:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2O2err07083 for ; Fri, 23 Mar 2001 21:40:59 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id DAA207160; Sat, 24 Mar 2001 03:36:27 +0100 From: julian_satran@il.ibm.com Received: from d12mta05.de.ibm.com (d12mta06_cs0 [9.165.222.255]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id DAA188540; Sat, 24 Mar 2001 03:34:58 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A19.000E507F ; Sat, 24 Mar 2001 03:36:21 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: vince_cavanna@agilent.com cc: pat_thaler@agilent.com, mbakke@cisco.com, jim.williams@emulex.com, ips@ece.cmu.edu, vince_cavanna@agilent.com Message-ID: Date: Sat, 24 Mar 2001 02:38:42 +0200 Subject: RE: CRC vs CHKSUM presentation slides Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Vince, In the references I gave you you have the coverage for the polynomials we are taking about. Regards, Julo vince_cavanna@agilent.com on 23/03/2001 20:34:19 Please respond to vince_cavanna@agilent.com To: pat_thaler@agilent.com, mbakke@cisco.com, jim.williams@emulex.com, ips@ece.cmu.edu cc: vince_cavanna@agilent.com Subject: RE: CRC vs CHKSUM presentation slides Pat and others, I have not actually determined for myself the coverage of the ethernet poly for 3 errors. All I know for sure, as I show in my slides, is that the ethernet poly, because it does not have an x^c+1 term, cannot detect ALL occurrences of an odd number of errors (including 3). I have therefore quoted data from the widely referenced paper (which I have refereced many times before) by Fujiware, Kasami and Lin on Error detecting capabilities of hte ethernet polynomial . This paper claims the Ethernet poly has a Hamming distance of 4 for lengths 3007 <= n <= 12144 and larger hamming distance for smaller n. It seems quite a coincidence to me that 12144 happens to be the length of the maximum IEE 802.3 frame (prior to VLAN tagging) and I can only assume that the polynomial was either expressely designed for that coverage or the coverage is actually larger than 12144 and the authors thought 12144 was sufficient and did not bother to give the exact number. In any case, I don't think it is particularly important for the ethernet poly to be able to catch ALL 3 bit errors since, due to the group encoding used, 3 bit errors on the link will manifest themselves as 3 bursts for which the ethernet poly most likely does not have complete coverage anyway. At one time (prior to the use of group encoding) the ability to detect 3 bit errors may have been important but not today. Same is true for Fiberchannel which has even larger frame sizes and also uses group encoding. Vince |-----Original Message----- |From: THALER,PAT (A-Roseville,ex1) |Sent: Friday, March 23, 2001 9:59 AM |To: vince_cavanna@agilent.com; mbakke@cisco.com; |jim.williams@emulex.com; ips@ece.cmu.edu |Subject: RE: CRC vs CHKSUM presentation slides | | |Vince, | |Your statement is correct except for the number 12144. 12144 |bits is the length of the maximum IEEE 802.3 frame (before we |added 4 bytes for VLAN tagging). The polynomial was actually |checked for coverage of 3-bit errors out to the Token Ring and |FDDI maximum packet sizes which are about 3 times as long. I |don't know the length at which the first undetectable 3-bit |error occurs, but it is something beyond 32000 bits. | |I also have trouble with an argument based on leverage. |Hardware design of CRC generators and checkers is just not |that hard. It's a register plus a bunch of XOR's. | |Regards, |Pat | |-----Original Message----- |From: vince_cavanna@agilent.com [mailto:vince_cavanna@agilent.com] |Sent: Thursday, March 22, 2001 4:10 PM |To: vince_cavanna@agilent.com; mbakke@cisco.com; |jim.williams@emulex.com; ips@ece.cmu.edu |Subject: RE: CRC vs CHKSUM presentation slides | | |To save Julian from having to write a memo I would like to add that the |ethernet polynomial has a certain weakness for 3 bit errors |that the others |don't. The ethernet polynomial is only guaranteed to catch all |combination |sof 3 bit errors when the message length is up to 12144 bits. |For larger |message lengths some 3 bit errors may get through. |Vince | | ||-----Original Message----- ||From: CAVANNA,VICENTE V (A-Roseville,ex1) ||Sent: Thursday, March 22, 2001 3:42 PM ||To: 'Mark Bakke'; 'Jim Williams'; ips@ece.cmu.edu ||Cc: CAVANNA,VICENTE V (A-Roseville,ex1) ||Subject: RE: CRC vs CHKSUM presentation slides || || ||Mark and Jim, ||I think any of the 32 bit CRC polynomials that have been ||proposed are more than good enough. My main reason for ||recommending hte CCITT-CRC32 is that it is half the cost of ||the others in terms of gate count - and unless you are doing a ||serial divider, which would be too slow, the gate count is ||very significant. Also, the leverage may not be as high as you ||think unless we are willing to use the datapath width of ||existing implementations. For 10 gig we will probably need to ||have larger than normal datapath widths. If you change the ||datapath width to handle, say, 64 bits at a time you change ||the implementation. I would otherwise have no problem with ||using the ethernet polynomial. I will try to explain later why ||I am not concerned about using the same polynomial and ||potentially giving up some cross-checking (I am not quite sure yet). ||Vince || |||-----Original Message----- |||From: Jim Williams [mailto:jim.williams@emulex.com] |||Sent: Thursday, March 22, 2001 1:30 PM |||To: ips@ece.cmu.edu |||Subject: Re: CRC vs CHKSUM presentation slides ||| ||| |||From: "Mark Bakke" |||To: "CAVANNA,VICENTE V (A-Roseville,ex1)" |||Cc: ; |||Sent: Thursday, March 22, 2001 4:00 PM |||Subject: Re: CRC vs CHKSUM presentation slides ||| ||| |||> Vicente- |||> |||> I just took another look through your slides after seeing the |||> presentation on Monday. They were very well-done. I have |||> one question, though. If the CCITT-CRC32 is considered "good |||> enough", then would the Ethernet CRC32 also be good enough? The |||> reason I ask is that every hardware vendor involved in building |||> iSCSI stuff already has implementations of the Ethernet CRC, |||> which is used for both Ethernet and Fibre Channel. |||> |||> The Ethernet poly has more terms than CCITT, and perhaps is |||> not as good as CRC-32C (any thoughts?), but everyone has hardware |||> and software for this, with proven interoperability (bit and |||> byte order, etc). Performance-wise, it will be there for |||> 10Gb Ethernet, so it should be fast enough. |||> |||> So if the Ethernet poly is deemed good enough (even if it's not |||> the best), and fast enough (even if it's not the fastest), why |||> not use it? I think we would stand a much better chance of |||> achieving interoperability in a short time. |||> |||> Please let me know what you think of this; I realize that a few |||> of my questions were speculative. ||| |||Since the iSCSI messages will often be encapsulated in ethernet |||packets, there is some value to using a different CRC. Link |||errors are double protected with two different CRCs. If |||ethernet and iSCSI use the same polynomial, there is little |||additional coverage against link errors. This point may not |||be decisive, but all other things being equal or almost |||equal, it is worth considering. ||| || | From owner-ips@ece.cmu.edu Sat Mar 24 00:06:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA12453 for ; Sat, 24 Mar 2001 00:06:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2O1oQ004158 for ips-outgoing; Fri, 23 Mar 2001 20:50:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2O1nYr04130 for ; Fri, 23 Mar 2001 20:49:34 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id A514C29A for ; Fri, 23 Mar 2001 18:49:19 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id DFCBA1F7 for ; Fri, 23 Mar 2001 20:49:18 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id RAA16385 for ; Fri, 23 Mar 2001 17:49:13 -0800 (PST) Message-ID: <3ABBFD46.9CC69A83@agilent.com> Date: Fri, 23 Mar 2001 17:49:58 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iscsi: technical comments to iSCSI 05: References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > > technical comments to iSCSI 05: > > > > Global: (especially in section 6) timeouts are not defined. We went > > through a lot of work defining time-out values for FCP-2. I think > > that time-out values need to be specified for interoperability. > > +++ probably. Please note also that we had ages ago some timers and people > did not find them usefull. Considering the variability in network > diameters we will have to either have large values or set them based or > RTT (complex) > +++ The timers you had ages ago were overkill and per I/O to support gateways to unreliable networks. In 6.6, you need to specify the amount of time after connection failure that a session is terminated. > > 1.2.7, page 19, middle of page > > must "iSCSI://..." be caps? is "iscsi://..." allowed? > > > +++ the syntax is the same as for URNs -- see the NDT doc +++ How about a simple yes or no... > > 2.7.2, page 45: > > Why must a LUN be associated with a Target Transfer Tag? > > > +++ I though we had this settled. The reason is to allow targets to set > their Tags independent of each other and help the target steer the data > without looking at Tags. That's the point of a tag - to help steer data. Fibre Channel doesn't have a LUN field in the headers of data transferred, why does iSCSI require it? > if the device server is located with the LU. The alternative would be to > declare the whole LU+Tag in R2T and Data Out as an extended tag and let > every target use it as they wish - no mandatory LU correctness check +++ No - just use the target task tag and nothing else, just like fibre channel. > > 2.17, page 66: > > *if* as you state in 2.7.2 a LUN must be sent when a target transfer > > tag is specified, shouldn't the LUN be also specified in the R2T? > > > +++ see above - the target may have device servers that do not need to know > what they are (they may get this information from the command but they > don't need it). On the other hand the initiator has always this > information. +++ I restate that I don't think LUN should be used as a steering mechanism. > > 2.20, page 71: > > remove "reserved fields not 0". This makes it sound like > > reserved fields must be checked, which they should not must > > be checked. > > +++ I did. But this raises an interesting side-question. It was said that > checking for reserved fiels being 0 is a good practice. It enables easy > migration from version to version +++ No it doesn't. If some future version makes use of a reserved field, it makes older versions puke on the new messages. > > 2.20, page 71: > > add "data digest error" to the list of reject reasons. > > +++ it is there as cause 3. Is the name confusing? +++ Ok I see it - you should call it data digest error. > > 3.1.2 and 3.1.3, page 72: > > I don't care what SCSI has defined for it's mode pages, for iSCSI, > > these fields should be byte values, not multiples of 512. > > > +++ the current field size is 16 bit. Do we want to limit ourselves to 64k > PDUs? > +++ See my comment below on the max PDU size. > > 4.3, page 78: > > It may not be clear to everyone that a target can send a text or > > login response to a login or text command. That is, a target could > > respond to a text command with a login response. This should be > > made more clear. You didn't reply to this. > > > > 6.2, page 81, last paragraph: > > Can a "restart" also be issued in this case? > > > > +++ made it lowercase must -:)+++ good. do you allow a restart? > > Appendix A: 01, page 94: > > I don't think that crc-64 "MUST" be implemented. > > I also don't agree with the phrase "Cyclic codes are particularly > > well suited for hardware implementations." This is not true if > > iSCSI PDUs span TCP segments and arrive out of order. > > > +++ fine on crc64 - on the rest if you do on the DMA to host it is still > decent+++ My point was that I don't think that phrase should be in the document. It adds no value. > > > Appendix A: 01, page 94: > > "Implementations MAY also negotiate digests with security > > significance for data authentication and integrity as detailed > > in the following table:" needs a lot more description. > > > +++ what for example? (copy the reference?) +++ Well for example, the "what's next" field doesn't seem to have any encoding for a security digest. There is no format of the digest (what it contains) and how it's validated. > > Appendix C: 07, page 105: > > "The marker-less interval is not less than 4 kbytes and the > > default is 4 kbytes." It should be whatever the the receiver > > needs it to be (not min 4K). > > > +++ how should a send know? we don't want this to kick in before end of > login +++ I agree - but if login only takes 1K, then the above statement says that I can't have a marker for another 3k. It should simply say that the markers are disabled until after login. And then the interval will be whatever was negotiated. > > Appendix C: 23 page 111: > > This parameter should be in bytes, not multiples of 512, > > especially since framing mechanisms may require that a PDU not be > > larger than MSS. (perhaps -1 can be used to indicate MSS) > > > +++ the field length in the mode page is limited +++ The login parameter should be able to negotiate any value, including some indication to use the current MSS. The value stored in the mode page can be an approximation of the real value. (This raises the question, why does iSCSI need mode pages when the values can be negotiated and discovered using the key:value pairs?) The whole purpose of the WARP proposal is to be able to have a fully self describing iSCSI PDU per TCP segment. Therefore, there must be a way to specify that the max sized iSCSI PDU is the MSS less whatever size is required for the WARP header. -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Sat Mar 24 14:38:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA08838 for ; Sat, 24 Mar 2001 14:38:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2OGqnp01792 for ips-outgoing; Sat, 24 Mar 2001 11:52:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2OGpqr01742 for ; Sat, 24 Mar 2001 11:51:52 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA420114 for ; Sat, 24 Mar 2001 17:51:22 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA174944 for ; Sat, 24 Mar 2001 17:49:52 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A19.005C5E23 ; Sat, 24 Mar 2001 17:48:54 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 24 Mar 2001 18:51:46 +0200 Subject: Re: iscsi: technical comments to iSCSI 05: Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, This tag story keeps coming up again and again. There are two camps: - camp 1 wants them to be the only thing that steers data - camp 2 wants every LUN to allocate tags regardless of others and they need another element to differentiate the streams (to which you argued that they should divide the space) I don't see why we should not carry the LUN. Timers will be in 6 as needed. On the reserved field opinions vary - mandating them 0 makes sure that nobody makes any assumption or use of them. We took the T10 approach that checking for 0 is at the discretion of the implementer. Answering you with yes or no would have meant me checking the naming draft and I thought that you may want to do this by yourself. AS for the initial interval I wanted markers at regular intervals and just saying after login is not a good reference. Julo Matt Wakeley on 24/03/2001 03:49:58 Please respond to Matt Wakeley To: IPS Reflector cc: Subject: Re: iscsi: technical comments to iSCSI 05: > > technical comments to iSCSI 05: > > > > Global: (especially in section 6) timeouts are not defined. We went > > through a lot of work defining time-out values for FCP-2. I think > > that time-out values need to be specified for interoperability. > > +++ probably. Please note also that we had ages ago some timers and people > did not find them usefull. Considering the variability in network > diameters we will have to either have large values or set them based or > RTT (complex) > +++ The timers you had ages ago were overkill and per I/O to support gateways to unreliable networks. In 6.6, you need to specify the amount of time after connection failure that a session is terminated. > > 1.2.7, page 19, middle of page > > must "iSCSI://..." be caps? is "iscsi://..." allowed? > > > +++ the syntax is the same as for URNs -- see the NDT doc +++ How about a simple yes or no... > > 2.7.2, page 45: > > Why must a LUN be associated with a Target Transfer Tag? > > > +++ I though we had this settled. The reason is to allow targets to set > their Tags independent of each other and help the target steer the data > without looking at Tags. That's the point of a tag - to help steer data. Fibre Channel doesn't have a LUN field in the headers of data transferred, why does iSCSI require it? > if the device server is located with the LU. The alternative would be to > declare the whole LU+Tag in R2T and Data Out as an extended tag and let > every target use it as they wish - no mandatory LU correctness check +++ No - just use the target task tag and nothing else, just like fibre channel. > > 2.17, page 66: > > *if* as you state in 2.7.2 a LUN must be sent when a target transfer > > tag is specified, shouldn't the LUN be also specified in the R2T? > > > +++ see above - the target may have device servers that do not need to know > what they are (they may get this information from the command but they > don't need it). On the other hand the initiator has always this > information. +++ I restate that I don't think LUN should be used as a steering mechanism. > > 2.20, page 71: > > remove "reserved fields not 0". This makes it sound like > > reserved fields must be checked, which they should not must > > be checked. > > +++ I did. But this raises an interesting side-question. It was said that > checking for reserved fiels being 0 is a good practice. It enables easy > migration from version to version +++ No it doesn't. If some future version makes use of a reserved field, it makes older versions puke on the new messages. > > 2.20, page 71: > > add "data digest error" to the list of reject reasons. > > +++ it is there as cause 3. Is the name confusing? +++ Ok I see it - you should call it data digest error. > > 3.1.2 and 3.1.3, page 72: > > I don't care what SCSI has defined for it's mode pages, for iSCSI, > > these fields should be byte values, not multiples of 512. > > > +++ the current field size is 16 bit. Do we want to limit ourselves to 64k > PDUs? > +++ See my comment below on the max PDU size. > > 4.3, page 78: > > It may not be clear to everyone that a target can send a text or > > login response to a login or text command. That is, a target could > > respond to a text command with a login response. This should be > > made more clear. You didn't reply to this. > > > > 6.2, page 81, last paragraph: > > Can a "restart" also be issued in this case? > > > > +++ made it lowercase must -:)+++ good. do you allow a restart? > > Appendix A: 01, page 94: > > I don't think that crc-64 "MUST" be implemented. > > I also don't agree with the phrase "Cyclic codes are particularly > > well suited for hardware implementations." This is not true if > > iSCSI PDUs span TCP segments and arrive out of order. > > > +++ fine on crc64 - on the rest if you do on the DMA to host it is still > decent+++ My point was that I don't think that phrase should be in the document. It adds no value. > > > Appendix A: 01, page 94: > > "Implementations MAY also negotiate digests with security > > significance for data authentication and integrity as detailed > > in the following table:" needs a lot more description. > > > +++ what for example? (copy the reference?) +++ Well for example, the "what's next" field doesn't seem to have any encoding for a security digest. There is no format of the digest (what it contains) and how it's validated. > > Appendix C: 07, page 105: > > "The marker-less interval is not less than 4 kbytes and the > > default is 4 kbytes." It should be whatever the the receiver > > needs it to be (not min 4K). > > > +++ how should a send know? we don't want this to kick in before end of > login +++ I agree - but if login only takes 1K, then the above statement says that I can't have a marker for another 3k. It should simply say that the markers are disabled until after login. And then the interval will be whatever was negotiated. > > Appendix C: 23 page 111: > > This parameter should be in bytes, not multiples of 512, > > especially since framing mechanisms may require that a PDU not be > > larger than MSS. (perhaps -1 can be used to indicate MSS) > > > +++ the field length in the mode page is limited +++ The login parameter should be able to negotiate any value, including some indication to use the current MSS. The value stored in the mode page can be an approximation of the real value. (This raises the question, why does iSCSI need mode pages when the values can be negotiated and discovered using the key:value pairs?) The whole purpose of the WARP proposal is to be able to have a fully self describing iSCSI PDU per TCP segment. Therefore, there must be a way to specify that the max sized iSCSI PDU is the MSS less whatever size is required for the WARP header. -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Sun Mar 25 06:49:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA11940 for ; Sun, 25 Mar 2001 06:49:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2P9NAR24254 for ips-outgoing; Sun, 25 Mar 2001 04:23:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.bld.us.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2P9MWr24202 for ; Sun, 25 Mar 2001 04:22:32 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e34.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id EAA72174; Sun, 25 Mar 2001 04:05:57 -0500 Received: from f4n49e (d03nm065h.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id CAA206262; Sun, 25 Mar 2001 02:21:11 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: Target Resets are Management Functions To: Matt Wakeley Cc: IPS Reflector X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Sun, 25 Mar 2001 01:20:50 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/25/2001 02:21:10 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk It should be removed. T10 removed it as a required function. Management should be done via a management interface. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com Matt Wakeley @ece.cmu.edu on 03/22/2001 02:45:54 PM Please respond to Matt Wakeley Sent by: owner-ips@ece.cmu.edu To: IPS Reflector cc: Subject: Re: iSCSI: Target Resets are Management Functions John, Are you saying that iSCSI allows the function, but does not specify how to use it? Or that target reset is to be removed from the list of management functions? John Hufferd wrote: > > Julian, > Target Resets are management functions, that is where they belong, not as > an iSCSI or SCSI action/command. It is not like this is going to be part > of some automatic error recovery function. Target Resets need and deserve > this function as part of administration management, it is up the vendors' > products to perform, or not, this function, with information they get (or > not) from an Admin interaction). I do not think we should support it in > the normal iSCSI Protocols. And we should not have to specify the problem > avoidance approaches that the implementer SHOULD/MUST take to support this > function. You will probably say that it s an implementation decision as to > how they avoid the problems, and that is what I am saying, and it belongs > to the vendors' Admin function to implement or not. > > If it is a big enough problem for FC to take it out, with their limited > network domain, we certainly should do that also. The reset function is still a management function in FCP-2 (table 3), and it's actions are described in FCP-2, tables 4 & 5 (and elsewhere). -Matt From owner-ips@ece.cmu.edu Sun Mar 25 14:44:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA06636 for ; Sun, 25 Mar 2001 14:44:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2PHdLW28685 for ips-outgoing; Sun, 25 Mar 2001 12:39:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2PHcpr28668 for ; Sun, 25 Mar 2001 12:38:54 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id TAA379978 for ; Sun, 25 Mar 2001 19:38:38 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id TAA127004 for ; Sun, 25 Mar 2001 19:37:05 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1A.0060B05B ; Sun, 25 Mar 2001 19:36:06 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 25 Mar 2001 19:39:07 +0200 Subject: Re: iSCSI: Target Resets are Management Functions Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk To be correct - T10 removed it from the list of function a target is mandated to implement in every protocol. We have to decide. John is ready to shoot it down. I am not ready yet. Julo "John Hufferd" on 25/03/2001 11:20:50 Please respond to "John Hufferd" To: Matt Wakeley cc: IPS Reflector Subject: Re: iSCSI: Target Resets are Management Functions It should be removed. T10 removed it as a required function. Management should be done via a management interface. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com Matt Wakeley @ece.cmu.edu on 03/22/2001 02:45:54 PM Please respond to Matt Wakeley Sent by: owner-ips@ece.cmu.edu To: IPS Reflector cc: Subject: Re: iSCSI: Target Resets are Management Functions John, Are you saying that iSCSI allows the function, but does not specify how to use it? Or that target reset is to be removed from the list of management functions? John Hufferd wrote: > > Julian, > Target Resets are management functions, that is where they belong, not as > an iSCSI or SCSI action/command. It is not like this is going to be part > of some automatic error recovery function. Target Resets need and deserve > this function as part of administration management, it is up the vendors' > products to perform, or not, this function, with information they get (or > not) from an Admin interaction). I do not think we should support it in > the normal iSCSI Protocols. And we should not have to specify the problem > avoidance approaches that the implementer SHOULD/MUST take to support this > function. You will probably say that it s an implementation decision as to > how they avoid the problems, and that is what I am saying, and it belongs > to the vendors' Admin function to implement or not. > > If it is a big enough problem for FC to take it out, with their limited > network domain, we certainly should do that also. The reset function is still a management function in FCP-2 (table 3), and it's actions are described in FCP-2, tables 4 & 5 (and elsewhere). -Matt From owner-ips@ece.cmu.edu Sun Mar 25 17:16:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA18153 for ; Sun, 25 Mar 2001 17:16:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2PK4Uv06770 for ips-outgoing; Sun, 25 Mar 2001 15:04:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.wrs.com (unknown-1-11.wrs.com [147.11.1.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2PK3br06738 for ; Sun, 25 Mar 2001 15:03:37 -0500 (EST) Received: from london ([192.168.1.25]) by mail.wrs.com (8.9.3/8.9.1) with SMTP id MAA15055 for ; Sun, 25 Mar 2001 12:03:05 -0800 (PST) From: "Rod Harrison" To: Subject: iSCSI: LUN field in PDUs Date: Sun, 25 Mar 2001 21:06:07 +0100 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Carrying the LUN in some of the iSCSI messages can be a little awkward for the initiator since it has to find the LUN. This means the initiator has to understand the CDB and know whether we are in the SCSI 1/2 3 bit LUN or a more modern world. I understand the argument that some targets might want to steer based on the LUN but I also think Matt Wakeley has a valid point when he says the target should be able to steer based on the target task tag alone. I have a proposal that might get us the best of both worlds. I believe in practice most targets will want to use solicited data mode, perhaps with a fairly small amount of immediate data [reasoning at the end of this message to avoid cluttering things]. This being so, I think we could get away with leaving all the messages as they are except the SCSI command PDU. Let's take the LUN field out of there (make it reserved), since the LUN is in the CDB anyway. This would avoid the initiator having to grub around inside a protocol layer's data it might not want to understand. If the target wants to be a 'pure tunnel' and not crack open the CDB it can do so, and if the target wants to steer on the LUN it still has that information at the cost of understanding the CDB. This doesn't seem to bad from the target point of view. If the target wants to steer on the LUN we can have it include the LUN in the R2T, and perhaps a flag bit to say this has happened. The initiator should always return the LUN in subsequent messages if the target has included it in the R2T. The overall effect of this change would be relatively minor. Command PDU has a simple change, data PDUs remain the same. I'm not sure why we are carrying LUN in the NOPs since they are iSCSI protocol management messages and are not really related to any underlying SCSI device. I assume the LUN in NOP-OUT is there to carry whatever the target might set in the LUN field of NOP-IN. If this is so then we need no changes to NOPs. My main sticking point with this is the task management PDU. The initiator might want to send a task management PDU before the target has responded with an R2T. Indeed, there might never be an R2T if the initial burst satisfies the entire data transfer. Do we really need the LUN in task management PDUs? No matter where the target thinks it should steer the PDU based on the LUN it still has to go find the task associated with the initiator task tag. - Rod Harrison [Target use of R2T reasoning] The reason I believe most targets will prefer solicited data is simply one of buffer management. If the target is controlling many SCSI devices, and thus running many iSCSI sessions to many initiators I think the only viable way to control buffer usage is via R2T. Since the target can never know how many sessions it might have to manage, and how many commands will be outstanding in each session it would have to be very conservative in negotiations for the amount of unsolicited data. If some sessions are much more active than others, this could lead to the active sessions being throttled because of negotiated limits whilst only a small amount of buffers are being used on the target overall. The tighter feedback loop afforded by R2T would seem to be the way to go. From owner-ips@ece.cmu.edu Sun Mar 25 20:35:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA04537 for ; Sun, 25 Mar 2001 20:35:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2PNdSR18029 for ips-outgoing; Sun, 25 Mar 2001 18:39:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.wrs.com (unknown-1-11.wrs.com [147.11.1.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2PNcgr18016 for ; Sun, 25 Mar 2001 18:38:43 -0500 (EST) Received: from london ([192.168.1.60]) by mail.wrs.com (8.9.3/8.9.1) with SMTP id PAA03826 for ; Sun, 25 Mar 2001 15:38:11 -0800 (PST) From: "Rod Harrison" To: Subject: Re: iSCSI: LUN field in PDUs Date: Mon, 26 Mar 2001 00:41:11 +0100 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Folks, I've just realised that my last message with the same subject was rubbish. We do, of course, need the LUN field in the command PDU. Apologies for the wasted time and bandwidth. - Rod From owner-ips@ece.cmu.edu Mon Mar 26 00:19:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA19947 for ; Mon, 26 Mar 2001 00:19:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2Q3JXl27464 for ips-outgoing; Sun, 25 Mar 2001 22:19:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2Q3JVr27459 for ; Sun, 25 Mar 2001 22:19:31 -0500 (EST) Received: from ahganemw2k (sj-dial-4-25.cisco.com [171.68.181.154]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with SMTP id WAA20324 for ; Sun, 25 Mar 2001 22:19:24 -0500 (EST) From: "Ayman Ghanem" To: Subject: iSCSI login question Date: Sun, 25 Mar 2001 21:18:13 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit 1) N&D draft (pp. 8): "The initiator MUST send an InitiatorWWUI and a TargetWWUI within the login request". What should a target return for a missing InitiatorWWUI in a login request?. I think we need an error code for that similar to "TargetWWUI required", or perhaps combine both as a "I/T WWUI required". 2) iSCSI draft section 4.2 mentions that initiator MUST NOT send operational parameters in a login until after security has been established. Later it says "any operational parameters sent before establishing a secure context MUST be reset ...". I suggest changing this so that operational parameters sent before security is established be ignored. -Ayman From owner-ips@ece.cmu.edu Mon Mar 26 00:20:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA20283 for ; Mon, 26 Mar 2001 00:20:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2Q36Xq26839 for ips-outgoing; Sun, 25 Mar 2001 22:06:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dirty.research.bell-labs.com (dirty.research.bell-labs.com [204.178.16.6]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2Q365r26823 for ; Sun, 25 Mar 2001 22:06:05 -0500 (EST) Received: from scummy.research.bell-labs.com ([135.104.2.10]) by dirty; Sun Mar 25 22:04:49 EST 2001 Received: from aura.research.bell-labs.com ([135.104.46.10]) by scummy; Sun Mar 25 22:04:48 EST 2001 Received: from research.bell-labs.com (IDENT:sandeepj@sandeepj-pcmh.research.bell-labs.com [135.104.47.90]) by aura.research.bell-labs.com (8.9.1/8.9.1) with ESMTP id WAA19325 for ; Sun, 25 Mar 2001 22:04:48 -0500 (EST) Message-ID: <3ABEB1D0.B97B6986@research.bell-labs.com> Date: Sun, 25 Mar 2001 22:04:48 -0500 From: Sandeep Joshi X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.2.16-3 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: SNMP traps Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mark, There seem to be no traps/notifications defined in the latest MIB doc. I do see that "notification-group" has been imported but not "notification-type" so perhaps there were discussions on adding something later ? A trap for login/auth failures atleast would be nice. Failure statistics are already being kept in the MIB. regards, -Sandeep From owner-ips@ece.cmu.edu Mon Mar 26 04:03:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA11863 for ; Mon, 26 Mar 2001 04:02:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2Q6ua806972 for ips-outgoing; Mon, 26 Mar 2001 01:56:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2Q6qXr06837 for ; Mon, 26 Mar 2001 01:52:33 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id IAA104242 for ; Mon, 26 Mar 2001 08:51:40 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA142404 for ; Mon, 26 Mar 2001 08:50:06 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1B.0025731F ; Mon, 26 Mar 2001 08:49:03 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 26 Mar 2001 08:52:05 +0200 Subject: Re: iSCSI: LUN field in PDUs Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Rod, The LUN field inside the CDB is not the "real" LUN - it is there only for compatibility with SCSI-1/2. Julo "Rod Harrison" on 25/03/2001 22:06:07 Please respond to "Rod Harrison" To: ips@ece.cmu.edu cc: Subject: iSCSI: LUN field in PDUs Carrying the LUN in some of the iSCSI messages can be a little awkward for the initiator since it has to find the LUN. This means the initiator has to understand the CDB and know whether we are in the SCSI 1/2 3 bit LUN or a more modern world. I understand the argument that some targets might want to steer based on the LUN but I also think Matt Wakeley has a valid point when he says the target should be able to steer based on the target task tag alone. I have a proposal that might get us the best of both worlds. I believe in practice most targets will want to use solicited data mode, perhaps with a fairly small amount of immediate data [reasoning at the end of this message to avoid cluttering things]. This being so, I think we could get away with leaving all the messages as they are except the SCSI command PDU. Let's take the LUN field out of there (make it reserved), since the LUN is in the CDB anyway. This would avoid the initiator having to grub around inside a protocol layer's data it might not want to understand. If the target wants to be a 'pure tunnel' and not crack open the CDB it can do so, and if the target wants to steer on the LUN it still has that information at the cost of understanding the CDB. This doesn't seem to bad from the target point of view. If the target wants to steer on the LUN we can have it include the LUN in the R2T, and perhaps a flag bit to say this has happened. The initiator should always return the LUN in subsequent messages if the target has included it in the R2T. The overall effect of this change would be relatively minor. Command PDU has a simple change, data PDUs remain the same. I'm not sure why we are carrying LUN in the NOPs since they are iSCSI protocol management messages and are not really related to any underlying SCSI device. I assume the LUN in NOP-OUT is there to carry whatever the target might set in the LUN field of NOP-IN. If this is so then we need no changes to NOPs. My main sticking point with this is the task management PDU. The initiator might want to send a task management PDU before the target has responded with an R2T. Indeed, there might never be an R2T if the initial burst satisfies the entire data transfer. Do we really need the LUN in task management PDUs? No matter where the target thinks it should steer the PDU based on the LUN it still has to go find the task associated with the initiator task tag. - Rod Harrison [Target use of R2T reasoning] The reason I believe most targets will prefer solicited data is simply one of buffer management. If the target is controlling many SCSI devices, and thus running many iSCSI sessions to many initiators I think the only viable way to control buffer usage is via R2T. Since the target can never know how many sessions it might have to manage, and how many commands will be outstanding in each session it would have to be very conservative in negotiations for the amount of unsolicited data. If some sessions are much more active than others, this could lead to the active sessions being throttled because of negotiated limits whilst only a small amount of buffers are being used on the target overall. The tighter feedback loop afforded by R2T would seem to be the way to go. From owner-ips@ece.cmu.edu Mon Mar 26 11:04:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA10412 for ; Mon, 26 Mar 2001 11:04:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2QDagQ01978 for ips-outgoing; Mon, 26 Mar 2001 08:36:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2QDaTr01935 for ; Mon, 26 Mar 2001 08:36:29 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2650.21) id ; Mon, 26 Mar 2001 08:27:30 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A070823CEB5@corpmx9.isus.emc.com> From: Black_David@emc.com To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: SACK (was RE:Async Message PDU) Date: Mon, 26 Mar 2001 08:36:11 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Let's see ... that's to report that something ate some of the data that the receiver was expecting, right :-) ?? Sounds like a reasonable suggestion. --David > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Friday, March 23, 2001 10:43 AM > To: ips@ece.cmu.edu > Subject: RE: SACK (was RE:Async Message PDU) > > > > how about SNACK? > > Julo > > Black_David@emc.com on 23/03/2001 06:52:20 > > Please respond to Black_David@emc.com > > To: marjorie_krueger@hp.com, ips@ece.cmu.edu > cc: > Subject: RE: SACK (was RE:Async Message PDU) > > > > > I think picking a new name for SACK to avoid confusion > with TCP is an excellent idea. Any suggestions for a > new name? > > --David > > > -----Original Message----- > > From: KRUEGER,MARJORIE (HP-Roseville,ex1) > [SMTP:marjorie_krueger@hp.com] > > Sent: Thursday, March 22, 2001 7:35 PM > > To: 'Black_David@emc.com'; ips@ece.cmu.edu > > Subject: RE: SACK (was RE:Async Message PDU) > > > > David, > > Sorry for the delayed reaction, I'm just catching up - You stated that a > > decision was made at the Orlando interim meeting to "change some of the > > terminology with the goal of each term and acronym being unambiguously > > owned > > by a single layer in the protocol stack". I think that's the right > thing > > to > > do, so can we agree to rename SACK to something that doesn't cause > > confusion > > with the TCP construct? > > > > Marjorie Krueger > > Networked Storage Architecture > > Networked Storage Solutions Org. > > Hewlett-Packard > > tel: +1 916 785 2656 > > fax: +1 916 785 0391 > > email: marjorie_krueger@hp.com > > > > > -----Original Message----- > > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > > Sent: Thursday, March 01, 2001 11:37 AM > > > To: cbm@rose.hp.com; ips@ece.cmu.edu > > > Subject: RE: Async Message PDU > > > > > > > > > Julian will doubtless pick up the rest of these, > > > but I thought I'd take this issue, since it was > > > part of what we did in Orlando. > > > > > > > 4. Somehow, "Asynchronous Message" seems at odds with the > > > rest of the > > > > usage in the draft in regards to PDUs (since a message is > > > introduced as > > > > PDU in section 1.2). Should we may be just call it an AEN > > > PDU? Section > > > > 2.14.3 calls this so. (I realize that it may not always > > > result in a > > > > SCSI-level AER.) > > > > > > One of the things we did in Orlando was to change > > > some of the terminology with the goal of each term and > > > acronym being unambiguously owned by a single layer > > > in the protocol stack, in particular making a sharp > > > distinction between SCSI concepts and iSCSI mechanisms. > > > > > > All the *RN entities changed to *SN for this reason > > > (SCSI has a CmdRN, so all *RN entities are SCSI, and > > > all *SN entities are iSCSI). Similarly, all AE* > > > entities are SCSI, and we went with "Asynchronous > > > Message <*>" to name the iSCSI entities. This matters > > > here because there are circumstances in which iSCSI > > > will send Asynchronous Message PDUs for its own use > > > even when SCSI has disabled AER. The usage of "AEN > > > PDU" in 2.14.3 is probably an editing oversight. > > > Suggestions for words to use instead of "Message" > > > are welcome, but they must not start with the letter > > > "E" ;-). > > > > > > Thanks, > > > --David > > > --------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > --------------------------------------------------- > > > > > From owner-ips@ece.cmu.edu Mon Mar 26 11:04:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA10428 for ; Mon, 26 Mar 2001 11:04:56 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2QEdi705109 for ips-outgoing; Mon, 26 Mar 2001 09:39:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2QEdfr05104 for ; Mon, 26 Mar 2001 09:39:41 -0500 (EST) Received: from ahganemw2k (dhcp-161-44-68-139.cisco.com [161.44.68.139]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with SMTP id JAA06762 for ; Mon, 26 Mar 2001 09:39:36 -0500 (EST) From: "Ayman Ghanem" To: Subject: RE: iSCSI login question Date: Mon, 26 Mar 2001 08:38:24 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I would think this should be a 0x02 class code (Initiator Error). -Ayman -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Monday, March 26, 2001 1:09 AM To: Ayman Ghanem Subject: Re: iSCSI login question Ayman,The wwui have to be sent during the Login Phase - not necessarily the login request. I will add a 0303 code for missing parameters. Regards, Julo "Ayman Ghanem" on 26/03/2001 05:18:13 Please respond to "Ayman Ghanem" To: ips@ece.cmu.edu cc: Subject: iSCSI login question 1) N&D draft (pp. 8): "The initiator MUST send an InitiatorWWUI and a TargetWWUI within the login request". What should a target return for a missing InitiatorWWUI in a login request?. I think we need an error code for that similar to "TargetWWUI required", or perhaps combine both as a "I/T WWUI required". 2) iSCSI draft section 4.2 mentions that initiator MUST NOT send operational parameters in a login until after security has been established. Later it says "any operational parameters sent before establishing a secure context MUST be reset ...". I suggest changing this so that operational parameters sent before security is established be ignored. -Ayman From owner-ips@ece.cmu.edu Mon Mar 26 12:07:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA12289 for ; Mon, 26 Mar 2001 12:07:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2QFPiZ07887 for ips-outgoing; Mon, 26 Mar 2001 10:25:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mcmail.mcdata.com (mcgate.mcdata.com [144.49.1.5]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2QFPLr07854 for ; Mon, 26 Mar 2001 10:25:21 -0500 (EST) Received: from exchange5.mcdata.com (exchange5.mcdata.com [144.49.33.200]) by mcmail.mcdata.com (8.8.6 (PHNE_14041)/8.8.6) with ESMTP id IAA16022; Mon, 26 Mar 2001 08:12:22 -0700 (MST) Received: by exchange5.mcdata.com with Internet Mail Service (5.5.2653.19) id ; Mon, 26 Mar 2001 08:25:13 -0700 Message-ID: From: "Mike O'Donnell" To: "'Elizabeth Rodriguez'" , "IPS Mailing List (E-mail)" Subject: RE: Fibre Channel Design Team forming Date: Mon, 26 Mar 2001 08:25:08 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I assume you received my business card along with Anil's? Mike O ===================== Michael E. O'Donnell McDATA Corporation 310 Interlocken Parkway Broomfield, Colorado 80021 303.460.4142 modonnell@mcdata.com ===================== -----Original Message----- From: Elizabeth Rodriguez [mailto:egrodriguez@lucent.com] Sent: Wednesday, March 21, 2001 1:32 PM To: IPS Mailing List (E-mail) Subject: Fibre Channel Design Team forming Hello all, A small Fibre Channel common encapsulation design team is being formed, to define a common header encapsulation for the Fibre Channel drafts -- FCIP and iFCP. Anyone interested in participating on this design team should notify either myself or David Black by Friday, March 23. Thank you, Elizabeth Rodriguez From owner-ips@ece.cmu.edu Mon Mar 26 12:08:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA12316 for ; Mon, 26 Mar 2001 12:08:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2QEhgq05300 for ips-outgoing; Mon, 26 Mar 2001 09:43:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sandmail.sandburst.com (sandburst-gw.bstn-gw02.ma.us.intelilink.net [216.57.129.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2QEhBr05287 for ; Mon, 26 Mar 2001 09:43:11 -0500 (EST) Received: from cs.uchicago.edu (dynamite-38.sandburst.com [172.16.5.38]) by sandmail.sandburst.com (Postfix) with ESMTP id 71B8394006 for ; Mon, 26 Mar 2001 09:42:56 -0500 (EST) To: ips@ece.cmu.edu Subject: Re: SACK (was RE:Async Message PDU) In-Reply-To: Message from Black_David@emc.com of "Thu, 22 Mar 2001 23:52:20 EST." <0F31E5C394DAD311B60C00E029101A0708015314@corpmx9.isus.emc.com> References: <0F31E5C394DAD311B60C00E029101A0708015314@corpmx9.isus.emc.com> Date: Mon, 26 Mar 2001 09:41:47 -0500 From: Stephen Bailey Message-Id: <20010326144256.71B8394006@sandmail.sandburst.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk > Any suggestions for a new name? Ah, finally my opportunity. I would suggest the string between the quotes: "" Then maybe we can have it wither away from the document and architecture, since we can no longer really call it anything (sure, in computers the empty string is just another string, but try pronouncing it in any language), and eventually its existence will fade from memory completely. :^) Steph From owner-ips@ece.cmu.edu Mon Mar 26 13:17:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA13607 for ; Mon, 26 Mar 2001 13:17:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2QGIlK12239 for ips-outgoing; Mon, 26 Mar 2001 11:18:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ihemail2.firewall.lucent.com (ihemail2.lucent.com [192.11.222.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2QGITr12203 for ; Mon, 26 Mar 2001 11:18:29 -0500 (EST) Received: from ihemail2.firewall.lucent.com (localhost [127.0.0.1]) by ihemail2.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id LAA26435 for ; Mon, 26 Mar 2001 11:18:28 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by ihemail2.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id LAA26428 for ; Mon, 26 Mar 2001 11:18:28 -0500 (EST) Subject: FC Encapsulation Design Team Formed MIME-Version: 1.0 Date: Mon, 26 Mar 2001 11:18:24 -0500 Content-Type: text/plain; charset="iso-8859-1" Message-ID: X-MimeOLE: Produced By Microsoft Exchange V6.0.4417.0 content-class: urn:content-classes:message Thread-Topic: FC Encapsulation Design Team Formed Thread-Index: AcC2EEyLBEiGvf52RSWIJ8eMxyv9XQ== From: "Elizabeth Rodriguez" To: "IPS Mailing List (E-mail)" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id f2QGIhs12228 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Hello all, As mentioned at the IPS WG meeting last week, a Fibre Channel Common Encapsulation design team has been formed, with the task of coming up with an initial draft common encapsulation for both iFCP and FCIP drafts to use. This draft, targeted for completion by the Interim Meeting on April 30, is subject to review by the IPS WG, and to modification should the IPS WG direct such. We had numerous qualified volunteers express an interest in participating on this design team; far more than could reasonable be accommodated on the design team. As mentioned last week, wish to keep this group small so that it can be effective in producing this initial draft. The design team is comprised of the following 7 individuals: Franco Travostino, Nortel Murali Rajagopal, LightSand Communications Ralph Weber, representing Brocade Charles Monia, Nishan Vi Chau, Gadzoox Mike O'Donnell, McData Milan Merhar, Pirus Our thanks to all who volunteered, Elizabeth Rodriguez & David Black From owner-ips@ece.cmu.edu Mon Mar 26 13:19:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA13649 for ; Mon, 26 Mar 2001 13:19:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2QG7mN11209 for ips-outgoing; Mon, 26 Mar 2001 11:07:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com (42s3082.isus.emc.com [168.159.211.82] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2QG7Nr11180 for ; Mon, 26 Mar 2001 11:07:23 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 26 Mar 2001 11:08:43 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A070801532A@corpmx9.isus.emc.com> From: Black_David@emc.com To: dotis@sanlight.net Cc: ips@ece.cmu.edu Subject: RE: iSCSI Naming: WWUIs, URNs, and namespaces Date: Mon, 26 Mar 2001 11:07:17 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > No, I was not confused nor did I expect clients to turn a WWUI into an > address as obviously you already have an address to do the reverse DNS. I > indicated the process as Name->IP->Reverse-Name and illustrated this process > by pointing to the technique that Microsoft used to obtain names from > machines. I'm afraid Doug is still confused. For connection to a specific target identified using a Reverse DNS name, the sequence is: Reverse Name --> DNS name --> IP where the first association is handled by some sort of nameservice (such as the LDAP server that Doug favors) and/or static configuration and the second is the usual DNS lookup. The first association has to be some sort of table, directory or database lookup, as extracting the DNS component of the reverse DNS name and trying to connect to it will not work in general. The only way to get from IP to the reverse name is to connect to the canonical iSCSI Target ("iscsi") and use the "Send Targets" command to get a list of targets visible to the initiator. Spoofing concerns get handled at the stage where the Initiator connects to the Target (there's a full iSCSI login that may be riding on a secure IPSec or TLS connection, depending on how we decide to do this). The Reverse Name is NOT being extracted from DNS. This mechanism is analogous to the SCSI REPORT_LUNS command and the way that a Fibre Channel initiator downloads a set of targets from the FC nameserver as part of FC discovery -- both of these mechanisms work fine in appropriate environments, and neither has anything to do with NetBIOS. As for LDAP - writing a draft on how to use it would be a far more productive approach than complaining that the WG is ignoring LDAP. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Mar 26 14:13:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA14523 for ; Mon, 26 Mar 2001 14:13:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2QGwlQ14465 for ips-outgoing; Mon, 26 Mar 2001 11:58:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2QGw0r14435 for ; Mon, 26 Mar 2001 11:58:00 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2650.21) id ; Mon, 26 Mar 2001 11:49:00 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A070801532D@corpmx9.isus.emc.com> From: Black_David@emc.com To: dotis@sanlight.net Cc: ips@ece.cmu.edu Subject: RE: iSCSI: Out Of Sequence due to null sequence with multipleconn ections. Date: Mon, 26 Mar 2001 11:57:48 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Let me apologize for unintentionally stepping on Doug in the meeting. Due to the time squeeze, I neglected to ask for other issues at the end of the iSCSI discussion - sorry about that. I'm going to back off and try to take a high level view of this and see what sort of observations emerge. When a SCSI task management command pops out of an iSCSI TCP connection at the target, there are four places that the SCSI operations it affects could be: (1) Executing in SCSI. (2) Queued to SCSI for execution, but not executing. (3) Queued in iSCSI waiting for command sequencing. (4) In-flight. (2) includes the "resource limitations between the sequencer and the target that may lead to a stall or a long term delay". (1) and (2) are the easy cases - the SCSI implementation must apply the task management command to executing tasks, and should perform the obvious "peephole optimization" to the commands queued for execution (i.e., if they're to be aborted, abort them and send the response(s) without starting execution). In essence, this models a command as crossing the boundary from iSCSI to SCSI the moment that iSCSI is prepared to give it to SCSI (i.e., any queue and related resource limitations are on SCSI's side of the line). (4) is hard. One SCSI task management command generates one response. That response can either be generated immediately (command arrives, is passed to SCSI, SCSI does its thing) or at the right point in the sequence (command arrives, is sequenced by iSCSI, passed to SCSI at the right point in the sequence, and SCSI does its thing), but NOT both. As things currently stand, having a task management command apply to in-flight commands requires sending the task management command for ordered delivery - so if it's desired to have the task management command take immediate effect and also catch everything in flight, it's going to have to be sent twice. I'm not enthusiastic about the idea of the task management command taking immediate effect but delaying the response until everything in flight that might be affected arrives, as I suspect the Initiator would like to know what happened sooner rather than later. (3) is "interesting". The results of applying a SCSI task management command to a SCSI operation are known only to SCSI, and hence asking that a command stuck in the iSCSI sequencer be affected immediately by a task management command is asking that the task management command have the side effect of changing some of the commands it affects to immediate delivery so that it can immediately do its (SCSI) thing to them. I wouldn't want to mandate this, nor would I want to prohibit it, BUT ... if the above discussion of in-flight commands is correct, I would observe that the application on the Initiator side can't tell the difference between commands that are in-flight vs. waiting for something in-flight on another connection, and hence is going to have to issue the task management command for ordered delivery if it wants to affect operations in either place (and issue a second copy if it wants immediate action). The upshot is that, aside from a longer discussion of this issue, I'm not sure anything needs to be changed. Comments? Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Mar 26 17:13:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA18894 for ; Mon, 26 Mar 2001 17:13:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2QJpqh23662 for ips-outgoing; Mon, 26 Mar 2001 14:51:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2QJpdr23632 for ; Mon, 26 Mar 2001 14:51:39 -0500 (EST) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 766BE2B0E; Mon, 26 Mar 2001 11:27:06 -0800 (PST) Received: from mk731913.cup.hp.com (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id LAA17658; Mon, 26 Mar 2001 11:30:47 -0800 (PST) Message-Id: <5.0.2.1.2.20010326100600.020c8ea8@esalpha2.cup.hp.com> X-Sender: krause@esalpha2.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Mon, 26 Mar 2001 11:11:23 -0800 To: "Jim Williams" From: Michael Krause Subject: Re: CRC vs CHKSUM presentation slides Cc: In-Reply-To: <008f01c0b317$33a58880$710e10ac@giganet.com> References: <3ABA67EB.77CFF4AF@cisco.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 04:29 PM 3/22/2001 -0500, Jim Williams wrote: >From: "Mark Bakke" >To: "CAVANNA,VICENTE V (A-Roseville,ex1)" >Cc: ; >Sent: Thursday, March 22, 2001 4:00 PM >Subject: Re: CRC vs CHKSUM presentation slides > > > > Vicente- > > > > I just took another look through your slides after seeing the > > presentation on Monday. They were very well-done. I have > > one question, though. If the CCITT-CRC32 is considered "good > > enough", then would the Ethernet CRC32 also be good enough? The > > reason I ask is that every hardware vendor involved in building > > iSCSI stuff already has implementations of the Ethernet CRC, > > which is used for both Ethernet and Fibre Channel. > > > > The Ethernet poly has more terms than CCITT, and perhaps is > > not as good as CRC-32C (any thoughts?), but everyone has hardware > > and software for this, with proven interoperability (bit and > > byte order, etc). Performance-wise, it will be there for > > 10Gb Ethernet, so it should be fast enough. > > > > So if the Ethernet poly is deemed good enough (even if it's not > > the best), and fast enough (even if it's not the fastest), why > > not use it? I think we would stand a much better chance of > > achieving interoperability in a short time. > > > > Please let me know what you think of this; I realize that a few > > of my questions were speculative. > >Since the iSCSI messages will often be encapsulated in ethernet >packets, there is some value to using a different CRC. Link >errors are double protected with two different CRCs. If >ethernet and iSCSI use the same polynomial, there is little >additional coverage against link errors. This point may not >be decisive, but all other things being equal or almost >equal, it is worth considering. Ethernet link (layer 2) protection only applies within a subnet; iSCSI CRC protection is end-to-end and is layer 5. The amount of bytes within a given packet covered by each is different as well. The arguments for using two different polynomials may be found in the selection proof used for GSN (HIPPI6400) which used two 16-bit CRCs to provide overall data protection. It would be interesting to see a mathematical analysis showing the data protection provided by two 32-bit CRCs using the same polynomial by across two over-lapping but not identical data sets versus two different CRCs but I doubt that there is that much improvement and there is clearly benefit from being able to use the same cell for both purposes. Mike Mike From owner-ips@ece.cmu.edu Mon Mar 26 18:57:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA21259 for ; Mon, 26 Mar 2001 18:57:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2QMZtP02304 for ips-outgoing; Mon, 26 Mar 2001 17:35:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2QMZjr02253 for ; Mon, 26 Mar 2001 17:35:45 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 26 Mar 2001 17:35:39 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A070801533A@corpmx9.isus.emc.com> From: Black_David@emc.com To: dotis@sanlight.net Cc: ips@ece.cmu.edu Subject: RE: iSCSI Naming: WWUIs, URNs, and namespaces Date: Mon, 26 Mar 2001 17:35:35 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > So from what I see from your example is iSCSI.Example.Com returns the WWUI > via SCSI that is then inserted into the domain to obtain the desired IP. No for a couple of reasons: - The WWUI is not inserted into the domain. The WWUI is an identifier, not an address, and extracting its DNS portion and contacting that host will not generally result in anything useful happening. - The IP is the *same* as iscsi.example.com, because the Send Targets command instructs whatever machine has been contacted to send a list of targets that are exported from that machine. Note that the machine contacted could be foo.example.com (it doesn't have to be iscsi.example.com) as the "iscsi" is the iSCSI target name sent in the iSCSI login command - it's not part of the DNS hostname. > As example [WWUI].example.com which is not World Wide Unique Identification but > at least unique to example.com. Why are you describing this process as > reverse DNS? If this is the process then I find the term reverse DNS > confusing. Because the actual name returned would be something like com.example.example7 - since the DNS components are reversed, the naming authority is referred to as "reverse DNS". Note that the name need not contain the name of the machine returning them - com.example.example7 could be returned from iscsi.example.com, foo.example.com, or bar.example.com, BUT if more than one of them returns it, it MUST be the same target every time. So, one could connect to foo.example.com, do an iSCSI login to the "iscsi" target at foo.example.com, send it the "Send Targets" command, and get back com.example.example1, com.example.example2, etc. All of these are separate targets behind the same communication endpoint at foo.example.com. > You are referring to a conventional alias. No, it's not alias. "iscsi" is a canonical target used by iSCSI - none of the aliases listed in your message could be used in its place with the expectation that something useful would happen (e.g., logging into the "archie" iSCSI target will not yield access to the Archie service - why on earth would one want to tunnel Archie over iSCSI?). It's a unique name for one of the targets behind this communication endpoint. > As this one machine > may not be able to carry the load for all clients or connect to all targets, > would you then break connection to the iSCSI machine and then make direct > connections to the LUNs? Break the connection -Yes. For communication load reasons - not necessarily, as the connection can only be used to retrieve target identifiers and can't access LUs. To connect to a LUN - No, the names returned name iSCSI targets, all of which share this communication endpoint. The session over which "Send Targets" was sent is only useful for "Send Targets" because it's logged into the canonical iSCSI target, which is only good for that purpose. To access the actual targets, a new connection to foo.example.com and a login to the target (one for each target) would be necessary. > Would a domain support multiple SCSI portals with > other names like iscsi1 or iscsi2? Is that also to be a standard > convention? Sure, and I don't see the need to standardize that convention. If someone wants to use blue-suede-shoes.example.com as a storage server that speaks iSCSI, I don't see any problem. Hiding behind this is the question of how foo.example.com (or blue-sued-shoes. example.com) was discovered in the first place - the answers to that are to be found in the naming and discovery slides Mark used in Minneapolis - static configuration, SLP, and iSNS can be used for this purpose. Please read the naming and discovery draft carefully before continuing this thread. Much of the above is contained in it. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Mar 26 18:58:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA21272 for ; Mon, 26 Mar 2001 18:58:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2QLWsS28979 for ips-outgoing; Mon, 26 Mar 2001 16:32:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2QLWfr28960 for ; Mon, 26 Mar 2001 16:32:41 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f2QMfH077658; Mon, 26 Mar 2001 14:41:18 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: Cc: Subject: RE: iSCSI Naming: WWUIs, URNs, and namespaces Date: Mon, 26 Mar 2001 13:31:03 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 In-Reply-To: <0F31E5C394DAD311B60C00E029101A070801532A@corpmx9.isus.emc.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, Perhaps we are having a problem with semantics. A reverse DNS is perhaps not the best name for this function if it has nothing to do with a reverse lookup of an IP. You are taking a domain and attempting to find related services. Alias Service ----------------------------------------------------------- archie archie [ARCHIE] finger Finger [RFC-1288] ftp File Transfer Protocol [RFC-959] gopher Internet Gopher Protocol [RFC-1436] ldap Lightweight Directory Access Protocol [RFC-1777] mail SMTP mail [RFC-821] news Usenet News via NNTP [RFC-977] ntp Network Time Protocol [RFC-1305] ph CCSO nameserver [PH] pop Post Office Protocol [RFC-1939] rwhois Referral WHOIS [RFC-1714] wais Wide Area Information Server [RFC-1625] whois NICNAME/WHOIS [RFC-954] www World-Wide Web HTTP [RFC-1945] iscsi iSCSI Server [RFC-xxxx] (Name Server via SCSI) So from what I see from your example is iSCSI.Example.Com returns the WWUI via SCSI that is then inserted into the domain to obtain the desired IP. As example [WWUI].example.com which is not World Wide Unique Identification but at least unique to example.com. Why are you describing this process as reverse DNS? If this is the process then I find the term reverse DNS confusing. You are referring to a conventional alias. As this one machine may not be able to carry the load for all clients or connect to all targets, would you then break connection to the iSCSI machine and then make direct connections to the LUNs? Would a domain support multiple SCSI portals with other names like iscsi1 or iscsi2? Is that also to be a standard convention? Doug > > No, I was not confused nor did I expect clients to turn a WWUI into an > > address as obviously you already have an address to do the > reverse DNS. I > > indicated the process as Name->IP->Reverse-Name and illustrated this > process > > by pointing to the technique that Microsoft used to obtain names from > > machines. > > I'm afraid Doug is still confused. For connection to a specific target > identified > using a Reverse DNS name, the sequence is: > > Reverse Name --> DNS name --> IP > > where the first association is handled by some sort of > nameservice (such as > the > LDAP server that Doug favors) and/or static configuration and the > second is > the usual DNS lookup. The first association has to be some sort of table, > directory or database lookup, as extracting the DNS component of > the reverse > DNS name and trying to connect to it will not work in general. > > The only way to get from IP to the reverse name is to connect to the > canonical iSCSI Target ("iscsi") and use the "Send Targets" command to > get a list of targets visible to the initiator. Spoofing concerns > get handled at the stage where the Initiator connects to the Target > (there's a full iSCSI login that may be riding on a secure IPSec or TLS > connection, depending on how we decide to do this). The Reverse > Name is NOT being extracted from DNS. This mechanism > is analogous to the SCSI REPORT_LUNS command and the way > that a Fibre Channel initiator downloads a set of targets from the FC > nameserver as part of FC discovery -- both of these mechanisms > work fine in appropriate environments, and neither has anything to > do with NetBIOS. > > As for LDAP - writing a draft on how to use it would be a far more > productive approach than complaining that the WG is ignoring LDAP. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Tue Mar 27 00:28:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA00150 for ; Tue, 27 Mar 2001 00:28:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2R2vxu15610 for ips-outgoing; Mon, 26 Mar 2001 21:57:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2R2vgr15571 for ; Mon, 26 Mar 2001 21:57:42 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f2R46K077890; Mon, 26 Mar 2001 20:06:20 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: Cc: Subject: RE: iSCSI Naming: WWUIs, URNs, and namespaces Date: Mon, 26 Mar 2001 18:56:05 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 In-Reply-To: <0F31E5C394DAD311B60C00E029101A070801533A@corpmx9.isus.emc.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, Thank you for your explanation. As a small side note, the list I gave was a list of service related aliases. I intended this to be a comprehensive list of all the typical services located in this manner. The machine that is identified by an alias would use this for some other name assigned to allow reassignment. Aliases are a convention to allow outsiders a likely means of locating services. Mail, News, or WWW are perhaps the most common uses of this convention. From what you are saying, iSNS would make sense as a conventional alias and not iSCSI. One is likely to have a need to expose several such machines so a single service locator would not be appropriate for an iSCSI server. You could think of this as just another service found by means of a convention. In this case, it is a SCSI Name Service that should be found by an alias convention. I was trying to interpret the statement made by Mark Bakke at the origin of this thread- "So I take it that WWUIs are fine, as long as they use existing global name spaces" ... "OTOH, a storage service provider could provide its own names by using the reverse-DNS authority. Everyone has a DNS name. This would enable the SSP to provide its customers with a WWUI that would not change if the back-end storage device was replaced. After all, it's not the device that's important to this customer, it's the DATA." You saw my extrapolation of what changes were implied. From your statements, I admit I do not understand the meaning of these statements as it does imply WWUI is used as a naming convention. As we are discussing changes to existing specifications, you can understand why I am seeking guidance on the reflector. Doug > > > So from what I see from your example is iSCSI.Example.Com > returns the WWUI > > via SCSI that is then inserted into the domain to obtain the desired IP. > > No for a couple of reasons: > - The WWUI is not inserted into the domain. The WWUI is an identifier, > not an address, and extracting its DNS portion and contacting that > host will not generally result in anything useful happening. > - The IP is the *same* as iscsi.example.com, because the Send Targets > command instructs whatever machine has been contacted to send > a list of targets that are exported from that machine. > Note that the machine contacted could be foo.example.com (it doesn't > have to be iscsi.example.com) as the "iscsi" is the iSCSI target name > sent in the iSCSI login command - it's not part of the DNS hostname. > > > As example [WWUI].example.com which is not World Wide Unique > Identification but > > at least unique to example.com. Why are you describing this process as > > reverse DNS? If this is the process then I find the term reverse DNS > > confusing. > > Because the actual name returned would be something like > com.example.example7 - since the DNS components are reversed, the > naming authority is referred to as "reverse DNS". Note that the name need > not contain the name of the machine returning them - com.example.example7 > could be returned from iscsi.example.com, foo.example.com, or > bar.example.com, BUT if more than one of them returns it, it MUST be the > same target every time. So, one could connect to foo.example.com, do > an iSCSI login to the "iscsi" target at foo.example.com, send it the > "Send Targets" command, and get back com.example.example1, > com.example.example2, etc. All of these are separate targets > behind the same communication endpoint at foo.example.com. > > > You are referring to a conventional alias. > > No, it's not alias. "iscsi" is a canonical target used by iSCSI - > none of the aliases listed in your message could be used in its place > with the expectation that something useful would happen (e.g., > logging into the "archie" iSCSI target will not yield access > to the Archie service - why on earth would one want to tunnel > Archie over iSCSI?). It's a unique name for one of the targets behind > this communication endpoint. > > > As this one machine > > may not be able to carry the load for all clients or connect to all > targets, > > would you then break connection to the iSCSI machine and then > make direct > > connections to the LUNs? > > Break the connection -Yes. For communication load reasons - > not necessarily, as the connection can only be used to retrieve > target identifiers and can't access LUs. To connect to a LUN - No, > the names returned name iSCSI targets, all of which share this > communication endpoint. The session over which "Send Targets" > was sent is only useful for "Send Targets" because it's logged into > the canonical iSCSI target, which is only good for that purpose. > To access the actual targets, a new connection to foo.example.com > and a login to the target (one for each target) would be necessary. > > > Would a domain support multiple SCSI portals with > > other names like iscsi1 or iscsi2? Is that also to be a standard > > convention? > > Sure, and I don't see the need to standardize that convention. If someone > wants > to use blue-suede-shoes.example.com as a storage server that speaks iSCSI, > I don't see any problem. > > Hiding behind this is the question of how foo.example.com (or > blue-sued-shoes. > example.com) was discovered in the first place - the answers to > that are to > be > found in the naming and discovery slides Mark used in Minneapolis - static > configuration, SLP, and iSNS can be used for this purpose. > > Please read the naming and discovery draft carefully before > continuing this > thread. Much of the above is contained in it. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Tue Mar 27 03:54:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA15480 for ; Tue, 27 Mar 2001 03:54:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2R6g2a25950 for ips-outgoing; Tue, 27 Mar 2001 01:42:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d06lmsgate.uk.ibm.COM (d06lmsgate.uk.ibm.com [195.212.29.1]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2R6fQr25906 for ; Tue, 27 Mar 2001 01:41:26 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d06lmsgate.uk.ibm.COM (1.0.0) with ESMTP id HAA135968 for ; Tue, 27 Mar 2001 07:24:37 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA81560 for ; Tue, 27 Mar 2001 08:38:11 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1C.00245B84 ; Tue, 27 Mar 2001 08:37:07 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 27 Mar 2001 08:40:10 +0200 Subject: RE: iSCSI: Out Of Sequence due to null sequence with multipleconn ections. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Your summary is correct. And (except for a minor point) it is all a matter of target implementation. SCSI is not a "completely layered" stack and others have gone so far as to do task cancellation by an action at link layer (parallel and fiber). There might be 2 funny side-effects though if an implementer chooses to cancel "holes" (commands in flight on other connections): 1)the cancelled command is a another task management command (there can be only one active but what it the one active gets stucked?) 2)(academic I admit) the cancelled command arrives after a wrap around in command sequencing; this is a bit harder (although not impossible) to fix in the implementation Implementers should be aware of those side effects. Regards, Julo Black_David@emc.com on 26/03/2001 18:57:48 Please respond to Black_David@emc.com To: dotis@sanlight.net cc: ips@ece.cmu.edu Subject: RE: iSCSI: Out Of Sequence due to null sequence with multipleconn ections. Let me apologize for unintentionally stepping on Doug in the meeting. Due to the time squeeze, I neglected to ask for other issues at the end of the iSCSI discussion - sorry about that. I'm going to back off and try to take a high level view of this and see what sort of observations emerge. When a SCSI task management command pops out of an iSCSI TCP connection at the target, there are four places that the SCSI operations it affects could be: (1) Executing in SCSI. (2) Queued to SCSI for execution, but not executing. (3) Queued in iSCSI waiting for command sequencing. (4) In-flight. (2) includes the "resource limitations between the sequencer and the target that may lead to a stall or a long term delay". (1) and (2) are the easy cases - the SCSI implementation must apply the task management command to executing tasks, and should perform the obvious "peephole optimization" to the commands queued for execution (i.e., if they're to be aborted, abort them and send the response(s) without starting execution). In essence, this models a command as crossing the boundary from iSCSI to SCSI the moment that iSCSI is prepared to give it to SCSI (i.e., any queue and related resource limitations are on SCSI's side of the line). (4) is hard. One SCSI task management command generates one response. That response can either be generated immediately (command arrives, is passed to SCSI, SCSI does its thing) or at the right point in the sequence (command arrives, is sequenced by iSCSI, passed to SCSI at the right point in the sequence, and SCSI does its thing), but NOT both. As things currently stand, having a task management command apply to in-flight commands requires sending the task management command for ordered delivery - so if it's desired to have the task management command take immediate effect and also catch everything in flight, it's going to have to be sent twice. I'm not enthusiastic about the idea of the task management command taking immediate effect but delaying the response until everything in flight that might be affected arrives, as I suspect the Initiator would like to know what happened sooner rather than later. (3) is "interesting". The results of applying a SCSI task management command to a SCSI operation are known only to SCSI, and hence asking that a command stuck in the iSCSI sequencer be affected immediately by a task management command is asking that the task management command have the side effect of changing some of the commands it affects to immediate delivery so that it can immediately do its (SCSI) thing to them. I wouldn't want to mandate this, nor would I want to prohibit it, BUT ... if the above discussion of in-flight commands is correct, I would observe that the application on the Initiator side can't tell the difference between commands that are in-flight vs. waiting for something in-flight on another connection, and hence is going to have to issue the task management command for ordered delivery if it wants to affect operations in either place (and issue a second copy if it wants immediate action). The upshot is that, aside from a longer discussion of this issue, I'm not sure anything needs to be changed. Comments? Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Tue Mar 27 09:00:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id JAA18146 for ; Tue, 27 Mar 2001 09:00:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2RBh8q18095 for ips-outgoing; Tue, 27 Mar 2001 06:43:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2RBfwr18037 for ; Tue, 27 Mar 2001 06:42:44 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id NAA428428; Tue, 27 Mar 2001 13:41:44 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id NAA174986; Tue, 27 Mar 2001 13:39:58 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1C.003FFDD1 ; Tue, 27 Mar 2001 13:38:57 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Michael Krause cc: ips@ece.cmu.edu Message-ID: Date: Tue, 27 Mar 2001 13:42:03 +0200 Subject: Re: CRC vs CHKSUM presentation slides Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mike, A cell for CRC is really not the issue - every-one is the same (taht is true for the serial implementations and for all the parallel implementations that I know about). The improvement stems not from using a different CRC but from the protection level the CRC offers you. The research about Pud on CRCs postdated the IEEE-CRC32 and the CIITT-CRC32 and considering that we should use the best solution available. Julo Michael Krause on 26/03/2001 21:11:23 Please respond to Michael Krause To: "Jim Williams" cc: ips@ece.cmu.edu Subject: Re: CRC vs CHKSUM presentation slides At 04:29 PM 3/22/2001 -0500, Jim Williams wrote: >From: "Mark Bakke" >To: "CAVANNA,VICENTE V (A-Roseville,ex1)" >Cc: ; >Sent: Thursday, March 22, 2001 4:00 PM >Subject: Re: CRC vs CHKSUM presentation slides > > > > Vicente- > > > > I just took another look through your slides after seeing the > > presentation on Monday. They were very well-done. I have > > one question, though. If the CCITT-CRC32 is considered "good > > enough", then would the Ethernet CRC32 also be good enough? The > > reason I ask is that every hardware vendor involved in building > > iSCSI stuff already has implementations of the Ethernet CRC, > > which is used for both Ethernet and Fibre Channel. > > > > The Ethernet poly has more terms than CCITT, and perhaps is > > not as good as CRC-32C (any thoughts?), but everyone has hardware > > and software for this, with proven interoperability (bit and > > byte order, etc). Performance-wise, it will be there for > > 10Gb Ethernet, so it should be fast enough. > > > > So if the Ethernet poly is deemed good enough (even if it's not > > the best), and fast enough (even if it's not the fastest), why > > not use it? I think we would stand a much better chance of > > achieving interoperability in a short time. > > > > Please let me know what you think of this; I realize that a few > > of my questions were speculative. > >Since the iSCSI messages will often be encapsulated in ethernet >packets, there is some value to using a different CRC. Link >errors are double protected with two different CRCs. If >ethernet and iSCSI use the same polynomial, there is little >additional coverage against link errors. This point may not >be decisive, but all other things being equal or almost >equal, it is worth considering. Ethernet link (layer 2) protection only applies within a subnet; iSCSI CRC protection is end-to-end and is layer 5. The amount of bytes within a given packet covered by each is different as well. The arguments for using two different polynomials may be found in the selection proof used for GSN (HIPPI6400) which used two 16-bit CRCs to provide overall data protection. It would be interesting to see a mathematical analysis showing the data protection provided by two 32-bit CRCs using the same polynomial by across two over-lapping but not identical data sets versus two different CRCs but I doubt that there is that much improvement and there is clearly benefit from being able to use the same cell for both purposes. Mike Mike From owner-ips@ece.cmu.edu Tue Mar 27 11:29:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA22066 for ; Tue, 27 Mar 2001 11:29:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2RDsB723952 for ips-outgoing; Tue, 27 Mar 2001 08:54:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2RDrCr23920 for ; Tue, 27 Mar 2001 08:53:15 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id PAA257938 for ; Tue, 27 Mar 2001 15:52:28 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id PAA16416 for ; Tue, 27 Mar 2001 15:48:29 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1C.004BD8CE ; Tue, 27 Mar 2001 15:48:27 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 27 Mar 2001 15:51:31 +0200 Subject: Re: Frame Formats Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk As we have mentioned in our presentation - we need a decision on frame format soon - to get to do our work. Barry and myself are OK with both formats. Please have again a look at them and voice your opinion (with some words to indicate why if you feel like it). Thanks, Julo "Barry Reinhold" on 27/03/2001 15:32:14 Please respond to "Barry Reinhold" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Frame Formats Julian, There seems to be no additional discussion about frame formats. What happens now? Barry Reinhold Principal Architect Trebia Networks barry.reinhold@trebia.com 603-868-5144/603-659-0885/978-929-0830 x138 From owner-ips@ece.cmu.edu Tue Mar 27 12:25:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA23571 for ; Tue, 27 Mar 2001 12:25:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2REwD927261 for ips-outgoing; Tue, 27 Mar 2001 09:58:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2REvUr27200 for ; Tue, 27 Mar 2001 09:57:30 -0500 (EST) Received: from ahganemw2k (dhcp-161-44-68-139.cisco.com [161.44.68.139]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with SMTP id JAA06442 for ; Tue, 27 Mar 2001 09:57:24 -0500 (EST) From: "Ayman Ghanem" To: Subject: RE: Frame Formats Date: Tue, 27 Mar 2001 08:56:12 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I would vote in favor of new format-2 (AHS-length in first byte). I find it simpler and good enough, assuming the 1020 byte limit on AHSs length is not an issue for other applications. -Ayman -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Tuesday, March 27, 2001 7:52 AM To: ips@ece.cmu.edu Subject: Re: Frame Formats As we have mentioned in our presentation - we need a decision on frame format soon - to get to do our work. Barry and myself are OK with both formats. Please have again a look at them and voice your opinion (with some words to indicate why if you feel like it). Thanks, Julo "Barry Reinhold" on 27/03/2001 15:32:14 Please respond to "Barry Reinhold" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Frame Formats Julian, There seems to be no additional discussion about frame formats. What happens now? Barry Reinhold Principal Architect Trebia Networks barry.reinhold@trebia.com 603-868-5144/603-659-0885/978-929-0830 x138 From owner-ips@ece.cmu.edu Tue Mar 27 13:20:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA25038 for ; Tue, 27 Mar 2001 13:20:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2RGeGC03141 for ips-outgoing; Tue, 27 Mar 2001 11:40:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2RGcSr03078 for ; Tue, 27 Mar 2001 11:38:28 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA98494 for ; Tue, 27 Mar 2001 18:37:43 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA50124 for ; Tue, 27 Mar 2001 18:36:06 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1C.005B19E4 ; Tue, 27 Mar 2001 18:35:04 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 27 Mar 2001 18:38:10 +0200 Subject: Re: Out of order data delivery in response to R2T. Mime-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=r5IE3RnUixsxDbFcExMxaLD24H8jurBJldUrnscnlnD77USrcBlkk5pJ" Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=r5IE3RnUixsxDbFcExMxaLD24H8jurBJldUrnscnlnD77USrcBlkk5pJ Content-type: text/plain; charset=us-ascii Content-Disposition: inline Barry, You are right. It was I changed it by mistake while fixing some other text. It reads again like in 03. Regards, Julo "Barry Reinhold" on 27/03/2001 18:11:13 Please respond to "Barry Reinhold" To: Julian Satran/Haifa/IBM@IBMIL cc: "Jon Sreekanth" , "James Smart" Subject: Out of order data delivery in response to R2T. --0__=r5IE3RnUixsxDbFcExMxaLD24H8jurBJldUrnscnlnD77USrcBlkk5pJ Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Julian, Early versions of iSCSI allowed out of order responses to an R2T. = This was fixed in rev 03 after some discussion. (See below) I have just noticed = upon review that it was taken out at 04 (see below) . Was this by error or design. I have attached the email conversation related to that change f= or your reference. (Rev 03 - 2.16) "An R2T MAY be answered with one or more iSCSI Data-out PDU with a matching Target Transfer Tag. If an R2T is answered with a single Data PDU the Buffer Offset in the Data PDU MUST be the same as the one specified by the R2T and the data length of the Data PDU must no= t exceed the Desired Data Length specified in R2T. If the R2T is answered with a sequence of Data PDUs the Buffer Offset and Length must be within the range of those specified by R2T, the last PDU should have the F bit set to 1, the Buffer Offsets and Lengths for consecutive PDUs SHOULD form a continuous non-overlapping range and the PDUs should be sent in increasing offset order." (Rev 05 - 2.17) "An R2T MAY be answered with one or more iSCSI Data-out PDU with a matching Target Transfer Tag. If an R2T is answered with a single Data PDU, the Buffer Offset in the Data PDU MUST be the same as the one specified by the R2T. The data length of the Data PDU must not exceed the Desired Data Length specified in R2T. If the R2T is answered with a sequence of Data PDUs, the Buffer Offset and Length MUST be within the range of those specified by R2T. The last PDU should have the F bit set to 1." Email discussion of topic (follow up from San Diego meetings) Barry, I can understand the rationale and the new text is strict (see attached= ). However a bad digest can result in the need to redo part or the whole R= 2T (or to reclaim all the data after the failure). If the R2T is answered with a sequence of Data PDUs the Buffer Offse= t and Length must be within the range of those specified by R2T, the last PDU should have the F bit set to 1, the Buffer Offsets and Lengths for consecutive PDUs SHOULD form a contin= uous non-overlapping range and the PDUs should be sent in increasing offs= et order. Regards, Julo "Barry Reinhold" on 29/12/2000 23:48:30 Please respond to "Barry Reinhold" To: Julian Satran/Haifa/IBM@IBMIL cc: "Jon Sreekanth" , "James Smart" Subject: Out of order response to R2T Julian, This is a bit of a delayed follow up to a conversation we had in S= an Diego. The issue has to do with how an initiator is allowed to respond to an R= 2T. Right now the iSCSI draft says: "If the R2T is answered with a sequence of Data PDUs the Buffer Offset and Length must be within the range of those specified by R2T and the last PDU should have the F bit set to 1; th= e Buffer Offsets and Lengths for consecutive PDUs SHOULD form a continuous range. " Based on previous conversations this means that an initiator can break = up the delivery of the data into 4 segments and deliver the segments in an= y order. I would like to argue that this should be restricted such that when responding to an R2T the data is delivered in order. I understand the l= ogic behind allowing the target to request data out of order based on the R2= T. This is consistent with Fibre Channel protocol and disk drive needs. However, it is not clear to me why we should allow the iSCSI data PDUs = sent in response to the R2T to be out of order. I have three observations on= this: 1. The concept behind the target sending the R2T (based on analogy to t= he FCP_XFER_RDY) is that the target is ready to receive data starting at "offset" of a given length. This does not happen when the data is deliv= ered out of order forcing the end device to reassemble the information and making the check process more complicated. 2. This is specifically diallowed by Fibre Channel. Fibre Channel requi= res that the data be delivered in one IU (sequence) and that the first fram= e of the sequence must have the data offset set to the value sent in the FCP_XFER_RDY frame. All the rest of the frames in the sequence must del= iver data in order by FC-FS rules. For reference - FCP-2 clause 9.3 "If an FCP_XFER_RDY IU is used to describe a data transfer and the first frame of the requeste= d FCP_DATA IU has a relative offset that differs from the value in the FCP_DATA_RO field of the FCP_XFER_RDY IU,= the target shall post the error code =F4FCP_DATA Parameter mismatch with FCP_DATA_RO =F6 in the FCP_RSP_INFO field of the FCP_RSP IU." I contacted Bob Snively (author of FCP-2) to confirm this. Bob is willi= ng to discuss the semantics of the FCP_XFER_RDY data transfer with you if you= wish to pick up that thread. 3. We have not seen this behavior in the field. If some device actually= took advantage of this flexibility I suspect a number of implementation issu= es would come up. In my experience these types of options hinder interoperabiliy. Also note that if a device is going to translate from iSCSI to FC, it i= s going to have a difficult time with this. It must buffer up all the iSC= SI frames until it finds the first piece. It the data is sent in reverse o= rder and the transfer is large the buffering could be significant. Testing t= his is a difficult process, and when testing is difficult interoperability problems creep in. In summary I would like to suggest that out of order data delivery in response to an R2T is not a helpful feature, hinders ineroperability, i= s not compatible with Fibre Channel, and therefore makes interconnecting with= Fibre Channel legacy devices more difficult. I would like to see us ado= pt the same semantics for iSCSI in this requard as Fibre Channel has. Thanks for your time on this, Barry Reinhold Barry Reinhold Principal Architect Trebia Networks barry.reinhold@trebia.com 603-868-5144/603-659-0885/978-929-0830 x138 = --0__=r5IE3RnUixsxDbFcExMxaLD24H8jurBJldUrnscnlnD77USrcBlkk5pJ-- From owner-ips@ece.cmu.edu Tue Mar 27 13:23:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA25130 for ; Tue, 27 Mar 2001 13:23:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2RGFFU01688 for ips-outgoing; Tue, 27 Mar 2001 11:15:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from chmls20.mediaone.net (chmls20.mediaone.net [24.147.1.156]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2RGErr01673 for ; Tue, 27 Mar 2001 11:14:53 -0500 (EST) Received: from breinhold (p-34.ts-6.qnc.ma.ttlc.net [140.186.41.125]) by chmls20.mediaone.net (8.11.1/8.11.1) with SMTP id f2RGCna12215; Tue, 27 Mar 2001 11:12:49 -0500 (EST) From: "Barry Reinhold" To: , Subject: RE: Frame Formats Date: Tue, 27 Mar 2001 11:11:16 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I can accept either format, but I would rather have FORMAT-2 with the lengths always in a fixed location. >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >julian_satran@il.ibm.com >Sent: Tuesday, March 27, 2001 8:52 AM >To: ips@ece.cmu.edu >Subject: Re: Frame Formats > > > > >As we have mentioned in our presentation - we need a decision on frame >format soon - to get to do our work. >Barry and myself are OK with both formats. > >Please have again a look at them and voice your opinion (with some words to >indicate why if you feel like it). > > >Thanks, >Julo > > > >"Barry Reinhold" on 27/03/2001 15:32:14 > >Please respond to "Barry Reinhold" > >To: Julian Satran/Haifa/IBM@IBMIL >cc: >Subject: Frame Formats > > > > >Julian, > There seems to be no additional discussion about frame formats. What >happens now? > >Barry Reinhold >Principal Architect >Trebia Networks >barry.reinhold@trebia.com >603-868-5144/603-659-0885/978-929-0830 x138 > > > > From owner-ips@ece.cmu.edu Tue Mar 27 13:23:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA25151 for ; Tue, 27 Mar 2001 13:23:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2RGWHr02735 for ips-outgoing; Tue, 27 Mar 2001 11:32:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from c007.snv.cp.net (c007-h008.c007.snv.cp.net [209.228.33.214]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2RGVxr02721 for ; Tue, 27 Mar 2001 11:31:59 -0500 (EST) Received: (cpmta 9514 invoked from network); 27 Mar 2001 07:43:05 -0800 Received: from dsl-64-130-130-105.telocity.com (HELO ljoy) (64.130.130.105) by smtp.telocity.com (209.228.33.214) with SMTP; 27 Mar 2001 07:43:05 -0800 X-Sent: 27 Mar 2001 15:43:05 GMT From: "Douglas Otis" To: , Subject: RE: iSCSI: Out Of Sequence due to null sequence with multipleconnections. Date: Tue, 27 Mar 2001 07:41:38 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-reply-to: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, It could be that a "stuck" command in flight or at the sequencer is the reason for the task management which is made problematic by the null sequence. Even if implementers are aware of this problem, there is not a good solution until null sequence is removed. Making a flag to indicate "immediate" or perhaps "reject prior pending iSCSI commands" is a means of ensuring the initiator remains in control with respect to all situations. This ensures the initiator is aware of the state of the target and sequence of commands can be maintained if required. Be careful about being too disk centric. Treating these SN numbers as unsigned allows a simple means of tracking. Here is an example explaination: Comparisons and arithmetic on SNs in this document SHOULD use Serial Number Arithmetic as defined in [RFC1982] where SERIAL_BITS = 32. Doug > David, > > Your summary is correct. And (except for a minor point) it is all a matter > of target implementation. > SCSI is not a "completely layered" stack and others have gone so far as to > do task cancellation by an action at link layer (parallel and fiber). > There might be 2 funny side-effects though if an implementer chooses to > cancel "holes" (commands in flight on other connections): > > 1)the cancelled command is a another task management command (there can be > only one active but what it the one active gets stucked?) > 2)(academic I admit) the cancelled command arrives after a wrap around in > command sequencing; this is a bit harder (although not impossible) to fix > in the implementation > > Implementers should be aware of those side effects. > > Regards, > Julo > > > > Black_David@emc.com on 26/03/2001 18:57:48 > > Please respond to Black_David@emc.com > > To: dotis@sanlight.net > cc: ips@ece.cmu.edu > Subject: RE: iSCSI: Out Of Sequence due to null sequence with > multipleconn > ections. > > > > > Let me apologize for unintentionally stepping on Doug > in the meeting. Due to the time squeeze, I neglected > to ask for other issues at the end of the iSCSI discussion > - sorry about that. > > I'm going to back off and try to take a high level view of > this and see what sort of observations emerge. When a SCSI > task management command pops out of an iSCSI TCP connection > at the target, there are four places that the SCSI operations > it affects could be: > > (1) Executing in SCSI. > (2) Queued to SCSI for execution, but not executing. > (3) Queued in iSCSI waiting for command sequencing. > (4) In-flight. > > (2) includes the "resource limitations between the > sequencer and the target that may lead to a stall or > a long term delay". > > (1) and (2) are the easy cases - the SCSI implementation > must apply the task management command to executing tasks, > and should perform the obvious "peephole optimization" to > the commands queued for execution (i.e., if they're to be > aborted, abort them and send the response(s) without starting > execution). In essence, this models a command as crossing > the boundary from iSCSI to SCSI the moment that iSCSI is > prepared to give it to SCSI (i.e., any queue and related > resource limitations are on SCSI's side of the line). > > (4) is hard. One SCSI task management command generates one > response. That response can either be generated immediately > (command arrives, is passed to SCSI, SCSI does its thing) or > at the right point in the sequence (command arrives, is > sequenced by iSCSI, passed to SCSI at the right point in the > sequence, and SCSI does its thing), but NOT both. As things > currently stand, having a task management command apply to > in-flight commands requires sending the task management > command for ordered delivery - so if it's desired to have > the task management command take immediate effect and also > catch everything in flight, it's going to have to be sent > twice. I'm not enthusiastic about the idea of the task > management command taking immediate effect but delaying the > response until everything in flight that might be affected > arrives, as I suspect the Initiator would like to know what > happened sooner rather than later. > > (3) is "interesting". The results of applying a SCSI task > management command to a SCSI operation are known only to > SCSI, and hence asking that a command stuck in the iSCSI > sequencer be affected immediately by a task management > command is asking that the task management command have > the side effect of changing some of the commands it affects > to immediate delivery so that it can immediately do its > (SCSI) thing to them. I wouldn't want to mandate this, > nor would I want to prohibit it, BUT ... if the above > discussion of in-flight commands is correct, I would > observe that the application on the Initiator side > can't tell the difference between commands that are in-flight > vs. waiting for something in-flight on another connection, > and hence is going to have to issue the task management > command for ordered delivery if it wants to affect operations > in either place (and issue a second copy if it wants > immediate action). > > The upshot is that, aside from a longer discussion of this > issue, I'm not sure anything needs to be changed. Comments? > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > > > > From owner-ips@ece.cmu.edu Tue Mar 27 14:21:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA26913 for ; Tue, 27 Mar 2001 14:21:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2RH4GW04492 for ips-outgoing; Tue, 27 Mar 2001 12:04:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from homer.qlogic.com (216-231-2-2.qlc.com [216.231.2.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2RH34r04457 for ; Tue, 27 Mar 2001 12:03:04 -0500 (EST) Received: by HOMER with Internet Mail Service (5.5.2653.19) id ; Tue, 27 Mar 2001 09:05:44 -0800 Message-ID: From: Mike Thompson To: "'ips@ece.cmu.edu'" Subject: Re: Frame Formats Date: Tue, 27 Mar 2001 09:05:36 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk As an implementer of a hardware implementation of iSCSI/TCP, I would like to see format 2 from the slide set presented at IETF. The fixed location of the total data length and AHS length will make out of order data placement reasonable. With these two fields at the beginning of the PDU, hardware will immediately know how much data needs to be checked to verify the header digest and if the digest is valid, it can go on to process the next PDU, looking for data PDUs that can be processed. In previous formats, the hardware has to process too many fields to get to the digest. Ideally, I would like to see a slight modification to this format where the header digest just covers the BHS. My understanding of the header digest is to allow for iSCSI routers to be able to modify a PDU header if it is acting as a proxy of some type. It seems that in this case the only thing that would be modified would be the BHS and not the AHS. With this change, I would envision the AHS be covered by the data digest. Again, this makes hardware processing easier, since the header that the digest covers is always a fixed length. I also think that the 24 bit total data length is more than adequate for the total PDU length. In order to be able to efficiently/reliably process PDUs, the PDU length should be on the order of 8-64kBytes in length. PDUs of 4 GBytes will require 4 Gbytes of reassembly memory in out of order cases. This is not reasonable. From owner-ips@ece.cmu.edu Tue Mar 27 15:41:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA28993 for ; Tue, 27 Mar 2001 15:41:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2RHuIB07941 for ips-outgoing; Tue, 27 Mar 2001 12:56:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2RHu8r07932 for ; Tue, 27 Mar 2001 12:56:08 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel1.hp.com (Postfix) with ESMTP id BB63F2AF1 for ; Tue, 27 Mar 2001 09:43:44 -0800 (PST) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id JAA23661 for ips@ece.cmu.edu; Tue, 27 Mar 2001 09:44:44 -0800 (PST) Message-Id: <200103271744.JAA23661@core.rose.hp.com> Subject: iSCSI: synch and steering comments To: ips@ece.cmu.edu Date: Tue, 27 Mar 2001 9:44:43 PST Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, Following are the comments I have on Synch and Steering in rev05. -Suggest replacing "it" in the following sentence in section 1.2.8.2, page 23 (first sentence in a new para): "According to our model of layering, iSCSI considers the information it delivers (headers and payloads) as a contiguous stream of bytes mapped to the positive integers from 0 to infinity." with "that a Synch and Steering layer" to make it clear. The sentence is ambiguous about the direction of delivery with the way it is. -Suggest adding the following statement to section 1.2.8.2. All conventional, in-order data arrival notifications generated by TCP are passed through to iSCSI by the Synch and Steering layer after appropriate data placements while none of the out-of-order data placements that it performs are communicated to upper layers. -Section 1.2.8.2 states that a Synch and Steering layer is optional. It has to be qualifed that it is optional only for those iSCSI devices which perform connection recovery on header digest errors, since that's how they cope with loss of framing. (I guess this may change in next rev?) -It appears to me that at least one Synch and Steering layer must be defined/referred to as the minimal implementation in the main draft to enable interoperability, when implementations do implement Synch and Steering. -I am somewhat confused about the following statement in section 1.2.8.2: "The Synch and Steering Layer is required to add to every sent data item (IP packet, TCP packet or some other superstructure) enough information to enable the receiver to steer it to a memory location independent of any other piece. " Clearly from the way I understood the markers in Appendix.C, it doesn't comply with this requirement. A more generic statement would be: "The Synch and Steering Layer is required to add adequate information to the data stream to enable the receiver to quickly steer the stream to its final memory location, even in the face of discontiguities in the stream. " -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Tue Mar 27 17:35:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA02079 for ; Tue, 27 Mar 2001 17:35:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2RLAQQ19900 for ips-outgoing; Tue, 27 Mar 2001 16:10:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dirty.research.bell-labs.com (dirty.research.bell-labs.com [204.178.16.6]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2RLA2r19877 for ; Tue, 27 Mar 2001 16:10:03 -0500 (EST) Received: from grubby.research.bell-labs.com ([135.104.2.9]) by dirty; Tue Mar 27 16:09:57 EST 2001 Received: from aura.research.bell-labs.com ([135.104.46.10]) by grubby; Tue Mar 27 16:09:57 EST 2001 Received: from research.bell-labs.com (IDENT:sandeepj@sandeepj-pcmh.research.bell-labs.com [135.104.47.90]) by aura.research.bell-labs.com (8.9.1/8.9.1) with ESMTP id QAA26545 for ; Tue, 27 Mar 2001 16:09:57 -0500 (EST) Message-ID: <3AC101A5.D5A2581B@research.bell-labs.com> Date: Tue, 27 Mar 2001 16:09:57 -0500 From: Sandeep Joshi X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.2.16-3 i686) X-Accept-Language: en MIME-Version: 1.0 To: "'ips@ece.cmu.edu'" Subject: Re: Frame Formats References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Maybe this helps expand the solution space or maybe it doesnt.. See http://ips.pdl.cs.cmu.edu/mail/msg03098.html for another scheme to keep header digests limited to BHS with Julian's comments on it. -sandeep P.S. my vote is for format-2. "Robert D. Russell" wrote: > > Mike: > > I agree with all your comments. > > Earlier I had proposed that the BHS be covered by a digest that > was always after the 48th byte (i.e., at a fixed offset), and that > the AHS, if present, would be covered by a second digest of its own. > (not by the data digest, which would be after the end of the data). > By having 3 digests, 1 for the BHS, 1 for the AHS, 1 for the data, > every read is fixed length and the digest is at the end of each > of the reads. > However, there was opposition to having a second header digest > that just covered the AHS, so now there is only 1 header digest > which covers both the BHS and the AHS and therefore, as you pointed > out, it is at a position that is unknown when you start to read the > header. > > Since you are a hardware implementer, let me ask the obvious question -- > wouldn't it simplify everything to have a 2nd digest that covered just > the AHS? The steps to receive a PDU would then be: > 1. Read the fixed length BHS with its own digest and check it. > 2. Extract the length of the AHS from the BHS > If this length is 0 there is no AHS. > If this length is positive, read that many bytes of AHS plus > the AHS digest and check it. > 3. Extract the length of the data from the BHS > If this length is 0 there is no data. > If this length is positive, read that many bytes of data plus > the data digest and check it. > > >From a software point of view this looks a lot simpler, since every > read is for a known number of bytes and every read ends with a digest. > However, I am not sure if it simplifies the hardware significantly. > > Thanks, > Bob Russell > > On Tue, 27 Mar 2001, Mike Thompson wrote: > > > As an implementer of a hardware implementation of iSCSI/TCP, I would like to > > see format 2 from the slide set presented at IETF. The fixed location of the > > total data length and AHS length will make out of order data placement > > reasonable. With these two fields at the beginning of the PDU, hardware will > > immediately know how much data needs to be checked to verify the header > > digest and if the digest is valid, it can go on to process the next PDU, > > looking for data PDUs that can be processed. In previous formats, the > > hardware has to process too many fields to get to the digest. > > > > Ideally, I would like to see a slight modification to this format where the > > header digest just covers the BHS. My understanding of the header digest is > > to allow for iSCSI routers to be able to modify a PDU header if it is acting > > as a proxy of some type. It seems that in this case the only thing that > > would be modified would be the BHS and not the AHS. With this change, I > > would envision the AHS be covered by the data digest. Again, this makes > > hardware processing easier, since the header that the digest covers > > is always a fixed length. > > > > I also think that the 24 bit total data length is more than adequate for the > > total PDU length. In order to be able to efficiently/reliably process PDUs, > > the PDU length should be on the order of 8-64kBytes in length. PDUs of 4 > > GBytes will require 4 Gbytes of reassembly memory in out of order cases. > > This is not reasonable. > > From owner-ips@ece.cmu.edu Tue Mar 27 17:36:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA02098 for ; Tue, 27 Mar 2001 17:36:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2RKPNo16761 for ips-outgoing; Tue, 27 Mar 2001 15:25:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (mars.iol.unh.edu [132.177.121.222]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2RKP5r16746 for ; Tue, 27 Mar 2001 15:25:05 -0500 (EST) Received: from localhost (rdr@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id PAA09412; Tue, 27 Mar 2001 15:30:06 -0500 (EST) Date: Tue, 27 Mar 2001 15:30:06 -0500 From: "Robert D. Russell" To: Mike Thompson cc: "'ips@ece.cmu.edu'" Subject: Re: Frame Formats In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Mike: I agree with all your comments. Earlier I had proposed that the BHS be covered by a digest that was always after the 48th byte (i.e., at a fixed offset), and that the AHS, if present, would be covered by a second digest of its own. (not by the data digest, which would be after the end of the data). By having 3 digests, 1 for the BHS, 1 for the AHS, 1 for the data, every read is fixed length and the digest is at the end of each of the reads. However, there was opposition to having a second header digest that just covered the AHS, so now there is only 1 header digest which covers both the BHS and the AHS and therefore, as you pointed out, it is at a position that is unknown when you start to read the header. Since you are a hardware implementer, let me ask the obvious question -- wouldn't it simplify everything to have a 2nd digest that covered just the AHS? The steps to receive a PDU would then be: 1. Read the fixed length BHS with its own digest and check it. 2. Extract the length of the AHS from the BHS If this length is 0 there is no AHS. If this length is positive, read that many bytes of AHS plus the AHS digest and check it. 3. Extract the length of the data from the BHS If this length is 0 there is no data. If this length is positive, read that many bytes of data plus the data digest and check it. From a software point of view this looks a lot simpler, since every read is for a known number of bytes and every read ends with a digest. However, I am not sure if it simplifies the hardware significantly. Thanks, Bob Russell On Tue, 27 Mar 2001, Mike Thompson wrote: > As an implementer of a hardware implementation of iSCSI/TCP, I would like to > see format 2 from the slide set presented at IETF. The fixed location of the > total data length and AHS length will make out of order data placement > reasonable. With these two fields at the beginning of the PDU, hardware will > immediately know how much data needs to be checked to verify the header > digest and if the digest is valid, it can go on to process the next PDU, > looking for data PDUs that can be processed. In previous formats, the > hardware has to process too many fields to get to the digest. > > Ideally, I would like to see a slight modification to this format where the > header digest just covers the BHS. My understanding of the header digest is > to allow for iSCSI routers to be able to modify a PDU header if it is acting > as a proxy of some type. It seems that in this case the only thing that > would be modified would be the BHS and not the AHS. With this change, I > would envision the AHS be covered by the data digest. Again, this makes > hardware processing easier, since the header that the digest covers > is always a fixed length. > > I also think that the 24 bit total data length is more than adequate for the > total PDU length. In order to be able to efficiently/reliably process PDUs, > the PDU length should be on the order of 8-64kBytes in length. PDUs of 4 > GBytes will require 4 Gbytes of reassembly memory in out of order cases. > This is not reasonable. > From owner-ips@ece.cmu.edu Tue Mar 27 18:25:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA02876 for ; Tue, 27 Mar 2001 18:25:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2RLHKr20398 for ips-outgoing; Tue, 27 Mar 2001 16:17:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from PIKACHU.makesans.com (makesans.com [63.203.52.243]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2RLGRr20323 for ; Tue, 27 Mar 2001 16:16:28 -0500 (EST) Received: by PIKACHU.makesans.com with Internet Mail Service (5.5.2650.21) id ; Tue, 27 Mar 2001 13:12:10 -0800 Message-ID: <76EEBF2230B8F64F9053D8292A24B0DF0ADACC@PIKACHU.makesans.com> From: Gary Kotzur To: ips@ece.cmu.edu Subject: RE: Frame Formats Date: Tue, 27 Mar 2001 13:12:08 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I vote for Format-2. Looking at it from a hardware perspective, having the fixed placement for the AHS and data length would make it easier for data allocation and placement in buffer space. From owner-ips@ece.cmu.edu Wed Mar 28 06:13:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA22048 for ; Wed, 28 Mar 2001 06:13:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2S9DVs15277 for ips-outgoing; Wed, 28 Mar 2001 04:13:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2S9D9r15264 for ; Wed, 28 Mar 2001 04:13:09 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA221148 for ; Wed, 28 Mar 2001 09:11:44 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id JAA176496 for ; Wed, 28 Mar 2001 09:09:55 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1D.002742D8 ; Wed, 28 Mar 2001 09:08:50 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 28 Mar 2001 09:11:57 +0200 Subject: Re: Frame Formats Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mike, I understand your concerns (as my hearth is still divided between hardware and software -:)) Regarding digests including only BHS - it has crossed my mind too several times. The argument against it is that it is not future proof - yes todays AHSs don't get modified by a proxy but how long will it hold? In addition I have some trouble understanding what you find objectionable on format 1 (remember I am neutral). With both 1 and 2 you know where the digest is and you don't have to use the data length until the header is checked. With 1 you have the additional benefit of a parity check for the length. In 99.99% of the cases you have only one length to care about and in format 1 this is additionally protected with parity (makes resynch easier in case of a header digest failure). As for the size of the PDU - I am older and more cautious on this as I recall the days when 64k was more that you would ever need and I built a large machine with an ALGOL compiler on 24k -:). But I agree that for now 24 bit should suffice and for later we can use an AHS if need arises. Julo Mike Thompson on 27/03/2001 19:05:36 Please respond to Mike Thompson To: "'ips@ece.cmu.edu'" cc: Subject: Re: Frame Formats As an implementer of a hardware implementation of iSCSI/TCP, I would like to see format 2 from the slide set presented at IETF. The fixed location of the total data length and AHS length will make out of order data placement reasonable. With these two fields at the beginning of the PDU, hardware will immediately know how much data needs to be checked to verify the header digest and if the digest is valid, it can go on to process the next PDU, looking for data PDUs that can be processed. In previous formats, the hardware has to process too many fields to get to the digest. Ideally, I would like to see a slight modification to this format where the header digest just covers the BHS. My understanding of the header digest is to allow for iSCSI routers to be able to modify a PDU header if it is acting as a proxy of some type. It seems that in this case the only thing that would be modified would be the BHS and not the AHS. With this change, I would envision the AHS be covered by the data digest. Again, this makes hardware processing easier, since the header that the digest covers is always a fixed length. I also think that the 24 bit total data length is more than adequate for the total PDU length. In order to be able to efficiently/reliably process PDUs, the PDU length should be on the order of 8-64kBytes in length. PDUs of 4 GBytes will require 4 Gbytes of reassembly memory in out of order cases. This is not reasonable. From owner-ips@ece.cmu.edu Wed Mar 28 12:14:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA05211 for ; Wed, 28 Mar 2001 12:14:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2SEwaI04659 for ips-outgoing; Wed, 28 Mar 2001 09:58:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2SEwRr04652 for ; Wed, 28 Mar 2001 09:58:27 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id JAA08542; Wed, 28 Mar 2001 09:58:20 -0500 (EST) Message-ID: <3AC1FBEB.4A8158E@cisco.com> Date: Wed, 28 Mar 2001 08:57:47 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Sandeep Joshi CC: ips@ece.cmu.edu Subject: Re: SNMP traps References: <3ABEB1D0.B97B6986@research.bell-labs.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Sandeep- We have been looking for requirements for traps, and have not addressed them yet. You are the first to really ask for them. Trapping the login and authentication failures seems like a good idea; do you have any other ideas or requirements? Thanks, Mark Sandeep Joshi wrote: > > Mark, > > There seem to be no traps/notifications defined in the > latest MIB doc. > > I do see that "notification-group" has been imported but > not "notification-type" so perhaps there were discussions > on adding something later ? > > A trap for login/auth failures atleast would be nice. > Failure statistics are already being kept in the MIB. > > regards, > -Sandeep -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Wed Mar 28 12:19:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA05393 for ; Wed, 28 Mar 2001 12:19:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2SEvcO04589 for ips-outgoing; Wed, 28 Mar 2001 09:57:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d06lmsgate-3.uk.ibm.com (d06lmsgate-3.uk.ibm.com [195.212.29.3]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2SEuvr04558 for ; Wed, 28 Mar 2001 09:56:57 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d06lmsgate-3.uk.ibm.com (1.0.0) with ESMTP id PAA57148 for ; Wed, 28 Mar 2001 15:47:39 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA208770 for ; Wed, 28 Mar 2001 16:54:23 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1D.0051CA4B ; Wed, 28 Mar 2001 16:53:22 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 28 Mar 2001 16:56:30 +0200 Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport" Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun, Last summer I thought that recovery within a connection should be left to TCP. It is simple and could be made available through IPsec (if no new option of any form can be added). Two things killed this: The requirement to have a data encapsulation that can pass through application proxies (like a storage router) The "NO WAY" message we got from IESG-Security on a CRC only IPSec header As for the ACK - I am very much in favor of it (it is a no brainer) and implementations are in fact allowed to drop even unacked data. I am bound by the Orlando meeting decision to drop it. Except the regular "oppose everything" crowd the two vocal opponents where Somesh Gupta and Matt Wakeley. David may want or not to re-open the issue - I am not going to ask for it. Regards, Julo "Mallikarjun C." on 28/03/2001 00:45:02 Please respond to cbm@rose.hp.com To: Black_David@emc.com cc: Julian Satran/Haifa/IBM@IBMIL, cbm@rose.hp.com, someshg@yahoo.com, steph@cs.uchicago.edu, John Hufferd/San Jose/IBM@IBMUS, ldalleore@snapserver.com, venkat@rhapsodynetworks.com Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport" David and Julian, I appreciate both your views, and should I say that they're along predicted lines :-) - David's right in saying that the situation is akin to FC's. However, I would like to point out that FC is an unreliable transport, and hence is forced to pick up a lot of the transport baggage (at least in FCP-2, as I understand), in addition to being a SCSI encapsulation layer. Unfortunately, even with TCP being the "reliable" transport, iSCSI is going along the same lines - ie. transport baggage + SCSI encapsulation. My point is - if this is indeed a necessary evil, why don't we complete iSCSI's transport functionality by data-ACKs? - If data SACK is introduced mostly to make up for TCP's shortcomings, we're making its usage (and implementation) drastically less appealing since the only way error recovery algorithms can *rely* on data SACK is when replay is supported (or, "ReplaySupport=yes" in my proposal), which is extremely expensive. IOW, we're defining data SACK in the draft and not providing any incentives to implement and use it! - I submit that since iSCSI is being hailed as the ideal SCSI Transport protocol in its definition so far (and I believe, rightly so - mandating command ordering, bi-di support, SCSI CRN support to name a few examples), the perfectly SCSI-legal R/W interactions that break in other transports *do not* have to break in iSCSI. - A last idea (may seem radical at this point) in regards to iSCSI being a "full transport". This provides us an opportunity to "cast off" the transport baggage in future when we truly move to a "reliable" transport (perhaps TCP with CRCs/SCTP ?) - if we do a good job of keeping the encapsulation stuff separate from the transport stuff. (Julian, I heard from Randy that ideas similar to this were explored in your Haifa meeting. And yes, he recalls they were given up since TCP was supposed to be reliable and granularity of recovery was deemed one I/O.) With that said, may I request David (with his co-chair hat on, :-)) to add some binding comments/observations on this discussion? If we decide to leave data SACKs as unattractive to implement, the draft should in the least add a statement like - "Note that satisfying all possible data SACK requests for a task with an unacknowledged status implies implementing the I/O replay buffer on the part of targets." -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com >I think Julian's basically right -- I would point >out that any case of write after read that breaks >over iSCSI will also break over Fibre Channel. >On FC, the scenario starts with a frame CRC failure >on read data at the Initiator, so applications >have to cope and typically do so by enforcing >ordering at the app rather than using SCSI task >ordering. > >While SCSI has clever tools like ACA and task >ordering that appear to allow dependent operations >to be sent to the target concurrently, in practice >they don't work and/or aren't used (funny thing, >those two reinforce each other ;-) ). Hence >a minimal approach to them is in order: >- Make sure the result will interoperate. >- Make sure T10 doesn't ding us for leaving something > completely out. >- Don't specify anything not needed for the above. > >My 0.02, >--David > >> -----Original Message----- >> From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] >> Sent: Tuesday, March 27, 2001 9:23 AM >> To: cbm@rose.hp.com >> Cc: someshg@yahoo.com; steph@cs.uchicago.edu; hufferd@us.ibm.com; >> cbm@rose.hp.com; ldalleore@snapserver.com; Venkat Rangan; >> Black_David@emc.com >> Subject: Re: iSCSI ERT: data SACK/replay buffer/"semi-transport" >> >> >> >> Mallikarjun, >> >> I commiserate with you at the lack of ack for data but the Orlando meeting >> stated - no. Recall that I kept the number only as a mechanism to detect >> missing packets. >> >> You can achieve the effect you want by keeping around data for a while >> (you >> determine how long and then discard). >> >> If a SACK comes and you can recover - fine. If not you either reaccess the >> media (if you know how) or reject >> and let the initiator retry. >> >> You should not worry about R/W conflicts as programs bound to have such >> conflicts either: >> >> 1)can live with them or >> 2)protect themselves through some locks and rely on "operation-end-status" >> to keep results deterministic. >> >> Regards, >> Julo >> >> >> >> "Mallikarjun C." on 27/03/2001 03:34:16 >> >> Please respond to cbm@rose.hp.com >> >> To: cbm@rose.hp.com, someshg@yahoo.com, steph@cs.uchicago.edu, Julian >> Satran/Haifa/IBM@IBMIL, John Hufferd/San Jose/IBM@IBMUS >> cc: Black_David@emc.com >> Subject: iSCSI ERT: data SACK/replay buffer/"semi-transport" >> >> >> >> >> Hi Error Recovery Team, >> >> iSCSI can discard PDUs because of digest errors and request >> retransmissions using the iSCSI data SACK. To deal with such >> an eventuality, targets that want to support data SACK have >> the following options: >> >> (A) maintain a complete "replay" buffer for the entire I/O since >> a SACK could come anytime before the status is ack'ed by the >> initiator. [ simple, but extremely expensive in memory resources] >> >> (B) (re-introduce data-ACKs into the draft, and) implement data-ACKs. >> Thus enables keeping only those I/O buffers that haven't been ack'ed >> by the initiator. IOW, become a real full transport! [ everyone disliked >> it earlier...] >> >> (C) re-access the medium for data retransmission requests. Now there >> are 3 sub-cases in this to handle the changed data on the medium in a >> write-after-read scenario. (SEE NOTE.1 at the bottom on how it is >> legal.) >> (1) On seeing any write, stall till status is ack'ed for all the >> previous reads (basically drain the pipe). [simple, but incurs >> an additional roundtrip delay for all writes]. >> (2) A variation of the above, keep an eye only on the prior >> overlapping reads. [more BW efficient, but complicated to >> resolve the block dependencies in a stream of reads followed >> by writes] >> (3) Document the caveat and leave it upto the applications >> to avoid this case since this leads to data integrity issues. >> [pushing to apps since the transport can't get it right!] >> >> My first preference is (B), followed by (A), and I suggest we not go >> to (C) at all with its inherent dangers. >> >> Doing (B) naturally completes the transport job that iSCSI has taken >> on itself in view of TCP's claimed unreliable checksum. That is the >> right thing to do architecturally instead of being a "semi-transport"! >> >> Comments? >> -- >> Mallikarjun >> >> >> Mallikarjun Chadalapaka >> Networked Storage Architecture >> Network Storage Solutions Organization >> MS 5668 Hewlett-Packard, Roseville. >> cbm@rose.hp.com >> >> __________________________________________________________________________ >> Note.1: A Read followed by a Write (to the same blocks) is perfectly legal >> if SCSI sets the ORDERED task attribute on both the commands AND >> sets the NACA bit to one to indicate that Write shall be executed >> only if the Read did not fail (result in a Check Condition). >> >> In the current case, since Read completed just fine from SCSI's >> point of view, SCSI is moving on to execute Write. Those read >> buffers >> had been freed up since iSCSI received an ACK at the TCP level, >> and >> since iSCSI has no other way to have the data ack'ed! >> >> >> >> > From owner-ips@ece.cmu.edu Wed Mar 28 15:32:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA10577 for ; Wed, 28 Mar 2001 15:32:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2SIKeO18258 for ips-outgoing; Wed, 28 Mar 2001 13:20:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2SIKPr18248 for ; Wed, 28 Mar 2001 13:20:25 -0500 (EST) Received: from xparelay2.corp.hp.com (xparelay2.corp.hp.com [15.58.137.112]) by palrel3.hp.com (Postfix) with ESMTP id A09B56AE for ; Wed, 28 Mar 2001 10:20:24 -0800 (PST) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by xparelay2.corp.hp.com (Postfix) with ESMTP id C98271F511 for ; Wed, 28 Mar 2001 13:19:07 -0500 (EST) Received: by xboibrg2.boi.hp.com with Internet Mail Service (5.5.2653.19) id ; Wed, 28 Mar 2001 11:20:23 -0700 Message-ID: <6BD67FFB937FD411A04F00D0B74FE87802A08F43@xrose06.rose.hp.com> From: "KRUEGER,MARJORIE (HP-Roseville,ex1)" To: "Ips Reflector (E-mail)" Subject: April 6 deadline for comments to Requirements-01 Date: Wed, 28 Mar 2001 11:20:21 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk In anticipation of submitting the iSCSI Requirements document for Informational RFC status, please submit review comments on draft-ietf-ips-iscsi-reqmts-01.txt to me by April 6, 2001. I will incorporate changes and submit a final document for group review on April 13, 2001. Marjorie Krueger Networked Storage Architecture Networked Storage Solutions Org. Hewlett-Packard tel: +1 916 785 2656 fax: +1 916 785 0391 email: marjorie_krueger@hp.com From owner-ips@ece.cmu.edu Wed Mar 28 15:35:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA10672 for ; Wed, 28 Mar 2001 15:35:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2SIwgG20981 for ips-outgoing; Wed, 28 Mar 2001 13:58:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from homer.qlogic.com (216-231-2-2.qlc.com [216.231.2.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2SIwQr20963 for ; Wed, 28 Mar 2001 13:58:26 -0500 (EST) Received: by HOMER with Internet Mail Service (5.5.2653.19) id ; Wed, 28 Mar 2001 11:01:04 -0800 Message-ID: From: Mike Thompson To: "'ips@ece.cmu.edu'" Subject: RE: Frame Formats Date: Wed, 28 Mar 2001 11:01:03 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, My major objection to Format 1 is that, as I interpret it, to determine the total length of the PDU requires a more processing. In format 2, I only have to interpret the first word (of course after verifying the digest). This allows hardware to quickly scan for data PDUs to do data placement when in an out of order condition. In format 1, the first word sometimes has the data length and I have to interpret the DL bits and conditionally look in a couple of locations to gather up the lengths to do the processing. Mike > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Tuesday, March 27, 2001 11:12 PM > To: ips@ece.cmu.edu > Subject: Re: Frame Formats > > > > > Mike, > > I understand your concerns (as my hearth is still divided > between hardware > and software -:)) > > Regarding digests including only BHS - it has crossed my mind > too several > times. The argument against it is that it is not future > proof - yes todays > AHSs don't get modified by a proxy but how long will it hold? > > In addition I have some trouble understanding what you find > objectionable > on format 1 (remember I am neutral). With both 1 and 2 you > know where the > digest is and you don't have to use the data length until the > header is > checked. With 1 you have the additional benefit of a parity > check for the > length. In 99.99% of the cases > you have only one length to care about and in format 1 this > is additionally > protected with parity (makes resynch easier in case of a header digest > failure). > > > As for the size of the PDU - I am older and more cautious on this as I > recall the days when 64k was more that you would ever need > and I built a > large machine with an ALGOL compiler on 24k -:). But I agree > that for now > 24 bit should suffice and for later we can use an AHS if need arises. > > Julo > > Mike Thompson on 27/03/2001 19:05:36 > > Please respond to Mike Thompson > > To: "'ips@ece.cmu.edu'" > cc: > Subject: Re: Frame Formats > > > > > As an implementer of a hardware implementation of iSCSI/TCP, > I would like > to > see format 2 from the slide set presented at IETF. The fixed > location of > the > total data length and AHS length will make out of order data placement > reasonable. With these two fields at the beginning of the > PDU, hardware > will > immediately know how much data needs to be checked to verify > the header > digest and if the digest is valid, it can go on to process > the next PDU, > looking for data PDUs that can be processed. In previous formats, the > hardware has to process too many fields to get to the digest. > > Ideally, I would like to see a slight modification to this > format where the > header digest just covers the BHS. My understanding of the > header digest is > to allow for iSCSI routers to be able to modify a PDU header if it is > acting > as a proxy of some type. It seems that in this case the only > thing that > would be modified would be the BHS and not the AHS. With this > change, I > would envision the AHS be covered by the data digest. Again, > this makes > hardware processing easier, since the header that the digest covers > is always a fixed length. > > I also think that the 24 bit total data length is more than > adequate for > the > total PDU length. In order to be able to efficiently/reliably > process PDUs, > the PDU length should be on the order of 8-64kBytes in > length. PDUs of 4 > GBytes will require 4 Gbytes of reassembly memory in out of > order cases. > This is not reasonable. > > > > From owner-ips@ece.cmu.edu Wed Mar 28 15:35:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA10688 for ; Wed, 28 Mar 2001 15:35:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2SIoCu20331 for ips-outgoing; Wed, 28 Mar 2001 13:50:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from homer.qlogic.com (216-231-2-2.qlc.com [216.231.2.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2SImxr20283 for ; Wed, 28 Mar 2001 13:49:00 -0500 (EST) Received: by HOMER with Internet Mail Service (5.5.2653.19) id ; Wed, 28 Mar 2001 10:51:36 -0800 Message-ID: From: Mike Thompson To: "'ips@ece.cmu.edu'" Subject: RE: Frame Formats Date: Wed, 28 Mar 2001 10:51:36 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Robert, First off let me state the obvious, more digests create more complication, hardware or software. That said, the approach that you describe is relatively painless because all the information is in the BHS about how long each section of the PDU is. This definitely makes it easier to process PDUs, especially in an out of order case. When you have to process one header to find the next header to find the data, this causes a lot of grief. In the case of out of order processing, the hardware wants to just find the data PDUs and skip over the other ones until they are in order. With your scheme, I could look at just the BHS, find out if it is a data PDU and if not, skip to the next PDU without further processing of numerous headers. Also as you stated, the BHS digest is always in the same location. This definitely makes processing easier. Mike > -----Original Message----- > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > Sent: Tuesday, March 27, 2001 12:30 PM > To: Mike Thompson > Cc: 'ips@ece.cmu.edu' > Subject: Re: Frame Formats > > > Mike: > > I agree with all your comments. > > Earlier I had proposed that the BHS be covered by a digest that > was always after the 48th byte (i.e., at a fixed offset), and that > the AHS, if present, would be covered by a second digest of its own. > (not by the data digest, which would be after the end of the data). > By having 3 digests, 1 for the BHS, 1 for the AHS, 1 for the data, > every read is fixed length and the digest is at the end of each > of the reads. > However, there was opposition to having a second header digest > that just covered the AHS, so now there is only 1 header digest > which covers both the BHS and the AHS and therefore, as you pointed > out, it is at a position that is unknown when you start to read the > header. > > Since you are a hardware implementer, let me ask the obvious > question -- > wouldn't it simplify everything to have a 2nd digest that covered just > the AHS? The steps to receive a PDU would then be: > 1. Read the fixed length BHS with its own digest and check it. > 2. Extract the length of the AHS from the BHS > If this length is 0 there is no AHS. > If this length is positive, read that many bytes of AHS plus > the AHS digest and check it. > 3. Extract the length of the data from the BHS > If this length is 0 there is no data. > If this length is positive, read that many bytes of data plus > the data digest and check it. > > From a software point of view this looks a lot simpler, since every > read is for a known number of bytes and every read ends with a digest. > However, I am not sure if it simplifies the hardware significantly. > > Thanks, > Bob Russell > > > On Tue, 27 Mar 2001, Mike Thompson wrote: > > > As an implementer of a hardware implementation of > iSCSI/TCP, I would like to > > see format 2 from the slide set presented at IETF. The > fixed location of the > > total data length and AHS length will make out of order > data placement > > reasonable. With these two fields at the beginning of the > PDU, hardware will > > immediately know how much data needs to be checked to > verify the header > > digest and if the digest is valid, it can go on to process > the next PDU, > > looking for data PDUs that can be processed. In previous > formats, the > > hardware has to process too many fields to get to the digest. > > > > Ideally, I would like to see a slight modification to this > format where the > > header digest just covers the BHS. My understanding of the > header digest is > > to allow for iSCSI routers to be able to modify a PDU > header if it is acting > > as a proxy of some type. It seems that in this case the > only thing that > > would be modified would be the BHS and not the AHS. With > this change, I > > would envision the AHS be covered by the data digest. > Again, this makes > > hardware processing easier, since the header that the digest covers > > is always a fixed length. > > > > I also think that the 24 bit total data length is more than > adequate for the > > total PDU length. In order to be able to > efficiently/reliably process PDUs, > > the PDU length should be on the order of 8-64kBytes in > length. PDUs of 4 > > GBytes will require 4 Gbytes of reassembly memory in out of > order cases. > > This is not reasonable. > > > From owner-ips@ece.cmu.edu Wed Mar 28 17:13:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA13642 for ; Wed, 28 Mar 2001 17:13:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2SK2g525682 for ips-outgoing; Wed, 28 Mar 2001 15:02:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2SK1xr25652 for ; Wed, 28 Mar 2001 15:01:59 -0500 (EST) Received: from southrelay02.raleigh.ibm.com (southrelay02.raleigh.ibm.com [9.37.3.209]) by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAB31044; Wed, 28 Mar 2001 14:56:49 -0600 Received: from d04nms25.raleigh.ibm.com (d04nms25.raleigh.ibm.com [9.67.228.6]) by southrelay02.raleigh.ibm.com (8.11.2/NCO v4.95) with ESMTP id f2SK1tA69456; Wed, 28 Mar 2001 15:01:55 -0500 Importance: Normal Subject: Re: SNMP traps To: Mark Bakke Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "Thomas McSweeney" Date: Wed, 28 Mar 2001 15:01:55 -0500 X-MIMETrack: Serialize by Router on D04NMS25/04/M/IBM(Release 5.0.6 |December 14, 2000) at 03/28/2001 03:01:56 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Mark, I'm new to the iSCSI MIB effort, but I'll toss out some suggestions... I think we should trap iSCSI Session terminations, except those accomplished via Logout. I know this might result in traps for "normal" events, like when someone powers off their initiator instead of shutting down gracefully. If we want to get fancy, we could send them conditionally, e.g., only trap if there were any incomplete commands or undelivered responses at the time the session terminated. Trapping iSCSI connection terminations would be interesting too, especially to implementations which allow multiple connections per session. Loss of a connection could mean loss of capacity or reduced reliability of a session, which could be reflected on the management station display. This may already be covered by the TCP MIB (I haven't checked yet), but coordinating a TCP trap with iSCSI objects would be more difficult than having an iSCSI trap. It would also be helpful to trap changes to a target's LU list, and/or LUN mappings. Minimally, a simple trap to notify the manager that a change occurred would be enough to prompt the management station to do the GETs to refresh its tables. Tom McSweeney iSCSI Development, IBM Storage Networking Email: rf42tpme@us.ibm.com Phone: (USA) 919-254-5634 (tie line: 444-5634) Fax: (USA) 919-254-0391 (tie line: 444-0391) Mark Bakke @ece.cmu.edu on 03/28/2001 09:57:47 AM Sent by: owner-ips@ece.cmu.edu To: Sandeep Joshi cc: ips@ece.cmu.edu Subject: Re: SNMP traps Sandeep- We have been looking for requirements for traps, and have not addressed them yet. You are the first to really ask for them. Trapping the login and authentication failures seems like a good idea; do you have any other ideas or requirements? Thanks, Mark From owner-ips@ece.cmu.edu Wed Mar 28 18:42:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA15000 for ; Wed, 28 Mar 2001 18:42:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2SJ1ej21154 for ips-outgoing; Wed, 28 Mar 2001 14:01:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d06lmsgate.uk.ibm.COM (d06lmsgate.uk.ibm.com [195.212.29.1]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2SJ1Lr21138 for ; Wed, 28 Mar 2001 14:01:21 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d06lmsgate.uk.ibm.COM (1.0.0) with ESMTP id TAA163460 for ; Wed, 28 Mar 2001 19:44:24 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id PAA184158 for ; Wed, 28 Mar 2001 15:59:24 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1D.004CC059 ; Wed, 28 Mar 2001 15:58:19 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 28 Mar 2001 16:01:29 +0200 Subject: Re: iSCSI: synch and steering comments Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Answers in text. Thanks, Julo "Mallikarjun C." on 27/03/2001 19:44:43 Please respond to cbm@rose.hp.com To: ips@ece.cmu.edu cc: Subject: iSCSI: synch and steering comments Julian, Following are the comments I have on Synch and Steering in rev05. -Suggest replacing "it" in the following sentence in section 1.2.8.2, page 23 (first sentence in a new para): "According to our model of layering, iSCSI considers the information it delivers (headers and payloads) as a contiguous stream of bytes mapped to the positive integers from 0 to infinity." with "that a Synch and Steering layer" to make it clear. The sentence is ambiguous about the direction of delivery with the way it is. +++ I adjusted to clarfy direction ++ -Suggest adding the following statement to section 1.2.8.2. All conventional, in-order data arrival notifications generated by TCP are passed through to iSCSI by the Synch and Steering layer after appropriate data placements while none of the out-of-order data placements that it performs are communicated to upper layers. +++ I have added the following to 1.2.8.2 On the incoming path the Synch and Steering layer does not change the way TCP notifies iSCSI about in-order data arrival. All out-of-order data placements performed by the Synch and Steering layer are hidden from iSCSI. I have aloso changed a bit the figure to convey better the fact that TCP and Synch&Steering are related (not strictly layered +++ ++++ -Section 1.2.8.2 states that a Synch and Steering layer is optional. It has to be qualifed that it is optional only for those iSCSI devices which perform connection recovery on header digest errors, since that's how they cope with loss of framing. (I guess this may change in next rev?) +++ with the new format I think that we have: - one more chance if we go for format 1 or - drop the connection on header error In both cases we can leave synch and steering optional +++ -It appears to me that at least one Synch and Steering layer must be defined/referred to as the minimal implementation in the main draft to enable interoperability, when implementations do implement Synch and Steering. +++ why ? +++ -I am somewhat confused about the following statement in section 1.2.8.2: "The Synch and Steering Layer is required to add to every sent data item (IP packet, TCP packet or some other superstructure) enough information to enable the receiver to steer it to a memory location independent of any other piece. " Clearly from the way I understood the markers in Appendix.C, it doesn't comply with this requirement. A more generic statement would be: "The Synch and Steering Layer is required to add adequate information to the data stream to enable the receiver to quickly steer the stream to its final memory location, even in the face of discontiguities in the stream. " +++ Markers are but one example and have only the Synch component. The statement refers to a full steering (RDMA) scheme +++ -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Wed Mar 28 21:03:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA16968 for ; Wed, 28 Mar 2001 21:03:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2SNfkl11037 for ips-outgoing; Wed, 28 Mar 2001 18:41:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2SNevr10999 for ; Wed, 28 Mar 2001 18:40:57 -0500 (EST) Received: from cisco.com (dhcp-161-44-68-173.cisco.com [161.44.68.173]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id SAA02690 for ; Wed, 28 Mar 2001 18:40:51 -0500 (EST) Message-ID: <3AC275FA.E8D6E1AC@cisco.com> Date: Wed, 28 Mar 2001 17:38:34 -0600 From: Steve Senum X-Mailer: Mozilla 4.73 [en]C-CCK-MCD (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ietf-ips Subject: iSCSI: Parameter Negotiation (draft -05) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian: I have a question on the following text in section 4.3: Operational parameter negotiation MAY involve several request- response exchanges (login and/or text) always driven by the initiator. The initiator MUST indicate its intent to terminate the negotiation by setting the F bit to 1; the target sets the F bit to 1 on the last response. The last response must be the Login Response. If the target responds to a text or Login command with the F bit set to 1, with a text response with the F bit set to 0, or a login response with the text bit set to 0, the initiator must keep sending the text command (even empty) with the F bit set to 1 until it gets the Login Response with the F bit set to 1. Is the target allowed to introduce a parameter into the negotiation process? That is, is the following sequence valid: Text Cmd, F=1 I -> A= B= Text Rsp, F=0 T -> A= B= C= If not, when would the target ever want to response with F=1 to a F=0 Text Command in the operational parameter negotiation phase? I know there was a thread on this before, but it was not clear to me how it was resolved. Thanks, Steve Senum From owner-ips@ece.cmu.edu Wed Mar 28 23:31:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA27114 for ; Wed, 28 Mar 2001 23:31:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2T2Mn420869 for ips-outgoing; Wed, 28 Mar 2001 21:22:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from med.corp.rhapsodynetworks.com (64-160-62-201.rhapsodynetworks.com [64.160.62.201] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2T2M9r20814 for ; Wed, 28 Mar 2001 21:22:09 -0500 (EST) Received: by med.corp.rhapsodynetworks.com with Internet Mail Service (5.5.2653.19) id ; Wed, 28 Mar 2001 18:22:01 -0800 Message-ID: <15851BD69CFCD41186B100B0D0AABE650C1B16@med.corp.rhapsodynetworks.com> From: Venkat Rangan To: "'Mike Thompson'" , "'ips@ece.cmu.edu'" Subject: RE: Frame Formats Date: Wed, 28 Mar 2001 18:22:00 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Mike, We would be in favor of Format 2 with your proposed changes as well: - a AHS length and Data length inside the BHS - a header digest right after the BHS - AHS (if present) after the BHS - Data after the AHS - Data Digest after the data If someone wants protection of the AHS, a single Data Digest can cover everything after the header digest. An oddity with Format 2 in the PDF file is that the AHS Length and Data Length are the first items. We would like to see it in BHS after the first word which describes the OpCode and flags, which would allow for introduction of new OpCode formats which do not have any AHS or Data in the BHS. Another area that hasn't been covered, but mentioned tangentially: - All lengths are in units of four-bytes - All AHS and Data is padded to the nearest four-byte boundary - There is some indication of "fill-bytes" in two-bit fields to tell us how much of the data is actually valid. - Digests cover any fill-bytes also; i.e., the digests are always on quantities that are four-byte aligned. This facilitates word-oriented data movement and is known to be a big win in achieving higher performance. Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com -----Original Message----- From: Mike Thompson [mailto:mike.thompson@qlogic.com] Sent: Wednesday, March 28, 2001 10:52 AM To: 'ips@ece.cmu.edu' Subject: RE: Frame Formats Robert, First off let me state the obvious, more digests create more complication, hardware or software. That said, the approach that you describe is relatively painless because all the information is in the BHS about how long each section of the PDU is. This definitely makes it easier to process PDUs, especially in an out of order case. When you have to process one header to find the next header to find the data, this causes a lot of grief. In the case of out of order processing, the hardware wants to just find the data PDUs and skip over the other ones until they are in order. With your scheme, I could look at just the BHS, find out if it is a data PDU and if not, skip to the next PDU without further processing of numerous headers. Also as you stated, the BHS digest is always in the same location. This definitely makes processing easier. Mike > -----Original Message----- > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > Sent: Tuesday, March 27, 2001 12:30 PM > To: Mike Thompson > Cc: 'ips@ece.cmu.edu' > Subject: Re: Frame Formats > > > Mike: > > I agree with all your comments. > > Earlier I had proposed that the BHS be covered by a digest that > was always after the 48th byte (i.e., at a fixed offset), and that > the AHS, if present, would be covered by a second digest of its own. > (not by the data digest, which would be after the end of the data). > By having 3 digests, 1 for the BHS, 1 for the AHS, 1 for the data, > every read is fixed length and the digest is at the end of each > of the reads. > However, there was opposition to having a second header digest > that just covered the AHS, so now there is only 1 header digest > which covers both the BHS and the AHS and therefore, as you pointed > out, it is at a position that is unknown when you start to read the > header. > > Since you are a hardware implementer, let me ask the obvious > question -- > wouldn't it simplify everything to have a 2nd digest that covered just > the AHS? The steps to receive a PDU would then be: > 1. Read the fixed length BHS with its own digest and check it. > 2. Extract the length of the AHS from the BHS > If this length is 0 there is no AHS. > If this length is positive, read that many bytes of AHS plus > the AHS digest and check it. > 3. Extract the length of the data from the BHS > If this length is 0 there is no data. > If this length is positive, read that many bytes of data plus > the data digest and check it. > > From a software point of view this looks a lot simpler, since every > read is for a known number of bytes and every read ends with a digest. > However, I am not sure if it simplifies the hardware significantly. > > Thanks, > Bob Russell > > > On Tue, 27 Mar 2001, Mike Thompson wrote: > > > As an implementer of a hardware implementation of > iSCSI/TCP, I would like to > > see format 2 from the slide set presented at IETF. The > fixed location of the > > total data length and AHS length will make out of order > data placement > > reasonable. With these two fields at the beginning of the > PDU, hardware will > > immediately know how much data needs to be checked to > verify the header > > digest and if the digest is valid, it can go on to process > the next PDU, > > looking for data PDUs that can be processed. In previous > formats, the > > hardware has to process too many fields to get to the digest. > > > > Ideally, I would like to see a slight modification to this > format where the > > header digest just covers the BHS. My understanding of the > header digest is > > to allow for iSCSI routers to be able to modify a PDU > header if it is acting > > as a proxy of some type. It seems that in this case the > only thing that > > would be modified would be the BHS and not the AHS. With > this change, I > > would envision the AHS be covered by the data digest. > Again, this makes > > hardware processing easier, since the header that the digest covers > > is always a fixed length. > > > > I also think that the 24 bit total data length is more than > adequate for the > > total PDU length. In order to be able to > efficiently/reliably process PDUs, > > the PDU length should be on the order of 8-64kBytes in > length. PDUs of 4 > > GBytes will require 4 Gbytes of reassembly memory in out of > order cases. > > This is not reasonable. > > > From owner-ips@ece.cmu.edu Thu Mar 29 02:56:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA22881 for ; Thu, 29 Mar 2001 02:56:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2T5upl02813 for ips-outgoing; Thu, 29 Mar 2001 00:56:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2T5uDr02786 for ; Thu, 29 Mar 2001 00:56:14 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA267114 for ; Thu, 29 Mar 2001 07:56:03 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id HAA31428 for ; Thu, 29 Mar 2001 07:53:13 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1E.00205834 ; Thu, 29 Mar 2001 07:53:17 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 29 Mar 2001 07:56:31 +0200 Subject: RE: Frame Formats Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Venkat, Thanks. Only one point about lengths - AHS length is in 4 byte units (and all are 4 byte aligned). Data is paded to 4 bytes but the data length covers only real data in bytes. As for digests covering the data padding we could go either way. Regards, Julo Venkat Rangan on 29/03/2001 04:22:00 Please respond to Venkat Rangan To: "'Mike Thompson'" , "'ips@ece.cmu.edu'" cc: Subject: RE: Frame Formats Mike, We would be in favor of Format 2 with your proposed changes as well: - a AHS length and Data length inside the BHS - a header digest right after the BHS - AHS (if present) after the BHS - Data after the AHS - Data Digest after the data If someone wants protection of the AHS, a single Data Digest can cover everything after the header digest. An oddity with Format 2 in the PDF file is that the AHS Length and Data Length are the first items. We would like to see it in BHS after the first word which describes the OpCode and flags, which would allow for introduction of new OpCode formats which do not have any AHS or Data in the BHS. Another area that hasn't been covered, but mentioned tangentially: - All lengths are in units of four-bytes - All AHS and Data is padded to the nearest four-byte boundary - There is some indication of "fill-bytes" in two-bit fields to tell us how much of the data is actually valid. - Digests cover any fill-bytes also; i.e., the digests are always on quantities that are four-byte aligned. This facilitates word-oriented data movement and is known to be a big win in achieving higher performance. Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com -----Original Message----- From: Mike Thompson [mailto:mike.thompson@qlogic.com] Sent: Wednesday, March 28, 2001 10:52 AM To: 'ips@ece.cmu.edu' Subject: RE: Frame Formats Robert, First off let me state the obvious, more digests create more complication, hardware or software. That said, the approach that you describe is relatively painless because all the information is in the BHS about how long each section of the PDU is. This definitely makes it easier to process PDUs, especially in an out of order case. When you have to process one header to find the next header to find the data, this causes a lot of grief. In the case of out of order processing, the hardware wants to just find the data PDUs and skip over the other ones until they are in order. With your scheme, I could look at just the BHS, find out if it is a data PDU and if not, skip to the next PDU without further processing of numerous headers. Also as you stated, the BHS digest is always in the same location. This definitely makes processing easier. Mike > -----Original Message----- > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > Sent: Tuesday, March 27, 2001 12:30 PM > To: Mike Thompson > Cc: 'ips@ece.cmu.edu' > Subject: Re: Frame Formats > > > Mike: > > I agree with all your comments. > > Earlier I had proposed that the BHS be covered by a digest that > was always after the 48th byte (i.e., at a fixed offset), and that > the AHS, if present, would be covered by a second digest of its own. > (not by the data digest, which would be after the end of the data). > By having 3 digests, 1 for the BHS, 1 for the AHS, 1 for the data, > every read is fixed length and the digest is at the end of each > of the reads. > However, there was opposition to having a second header digest > that just covered the AHS, so now there is only 1 header digest > which covers both the BHS and the AHS and therefore, as you pointed > out, it is at a position that is unknown when you start to read the > header. > > Since you are a hardware implementer, let me ask the obvious > question -- > wouldn't it simplify everything to have a 2nd digest that covered just > the AHS? The steps to receive a PDU would then be: > 1. Read the fixed length BHS with its own digest and check it. > 2. Extract the length of the AHS from the BHS > If this length is 0 there is no AHS. > If this length is positive, read that many bytes of AHS plus > the AHS digest and check it. > 3. Extract the length of the data from the BHS > If this length is 0 there is no data. > If this length is positive, read that many bytes of data plus > the data digest and check it. > > From a software point of view this looks a lot simpler, since every > read is for a known number of bytes and every read ends with a digest. > However, I am not sure if it simplifies the hardware significantly. > > Thanks, > Bob Russell > > > On Tue, 27 Mar 2001, Mike Thompson wrote: > > > As an implementer of a hardware implementation of > iSCSI/TCP, I would like to > > see format 2 from the slide set presented at IETF. The > fixed location of the > > total data length and AHS length will make out of order > data placement > > reasonable. With these two fields at the beginning of the > PDU, hardware will > > immediately know how much data needs to be checked to > verify the header > > digest and if the digest is valid, it can go on to process > the next PDU, > > looking for data PDUs that can be processed. In previous > formats, the > > hardware has to process too many fields to get to the digest. > > > > Ideally, I would like to see a slight modification to this > format where the > > header digest just covers the BHS. My understanding of the > header digest is > > to allow for iSCSI routers to be able to modify a PDU > header if it is acting > > as a proxy of some type. It seems that in this case the > only thing that > > would be modified would be the BHS and not the AHS. With > this change, I > > would envision the AHS be covered by the data digest. > Again, this makes > > hardware processing easier, since the header that the digest covers > > is always a fixed length. > > > > I also think that the 24 bit total data length is more than > adequate for the > > total PDU length. In order to be able to > efficiently/reliably process PDUs, > > the PDU length should be on the order of 8-64kBytes in > length. PDUs of 4 > > GBytes will require 4 Gbytes of reassembly memory in out of > order cases. > > This is not reasonable. > > > From owner-ips@ece.cmu.edu Thu Mar 29 04:14:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA28344 for ; Thu, 29 Mar 2001 04:14:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2T7DrN07070 for ips-outgoing; Thu, 29 Mar 2001 02:13:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2T7DZr07054 for ; Thu, 29 Mar 2001 02:13:36 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id JAA221692 for ; Thu, 29 Mar 2001 09:13:14 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id JAA212500 for ; Thu, 29 Mar 2001 09:11:33 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1E.0026FBE2 ; Thu, 29 Mar 2001 09:05:48 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 29 Mar 2001 08:24:09 +0200 Subject: Re: iSCSI: Parameter Negotiation (draft -05) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Steve, Yes a target may introduce a parameter or continue negotiating. This might be the reason it answers with F=0 to an initiator that set F=0. As a response the initiator may resume negotiating and send text with F=0 or 1. In any case a target PDU with F=1 is correct only after an initiator has sent a PDU with F=1 except for errors (like a login reject). Regards, Julo Steve Senum on 29/03/2001 01:38:34 Please respond to Steve Senum To: ietf-ips cc: Subject: iSCSI: Parameter Negotiation (draft -05) Julian: I have a question on the following text in section 4.3: Operational parameter negotiation MAY involve several request- response exchanges (login and/or text) always driven by the initiator. The initiator MUST indicate its intent to terminate the negotiation by setting the F bit to 1; the target sets the F bit to 1 on the last response. The last response must be the Login Response. If the target responds to a text or Login command with the F bit set to 1, with a text response with the F bit set to 0, or a login response with the text bit set to 0, the initiator must keep sending the text command (even empty) with the F bit set to 1 until it gets the Login Response with the F bit set to 1. Is the target allowed to introduce a parameter into the negotiation process? That is, is the following sequence valid: Text Cmd, F=1 I -> A= B= Text Rsp, F=0 T -> A= B= C= If not, when would the target ever want to response with F=1 to a F=0 Text Command in the operational parameter negotiation phase? I know there was a thread on this before, but it was not clear to me how it was resolved. Thanks, Steve Senum From owner-ips@ece.cmu.edu Thu Mar 29 10:28:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA18384 for ; Thu, 29 Mar 2001 10:28:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TDQ2a05118 for ips-outgoing; Thu, 29 Mar 2001 08:26:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TDP4r05027 for ; Thu, 29 Mar 2001 08:25:05 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id IAA00499; Thu, 29 Mar 2001 08:24:51 -0500 (EST) Message-ID: <3AC3377F.1DB8E18C@cisco.com> Date: Thu, 29 Mar 2001 07:24:15 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Venkat Rangan CC: "'Mike Thompson'" , "'ips@ece.cmu.edu'" Subject: Re: Frame Formats References: <15851BD69CFCD41186B100B0D0AABE650C1B16@med.corp.rhapsodynetworks.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Venkat and Mike- I am also in favor of Format 2, as long as we do not include the AHS in the data digest. We had originally separated the header and data digests, so iSCSI gateways or other middle boxes could modify the header, but keep the data CRC end-to-end. By including AHS in the data CRC, this capability will be lost. I agree that we should have a single header digest, but I would like to see it after the AHS. If we must have a BHS digest, then I would rather have two header digests than combine AHS with data. I also agree with your comments on lengths and padding. I would add that any pad bytes MUST be set to zero; it's sort of a nit, but they could contain old data. The security folks seem to get upset whenever someone sends "previously-owned" data around, even though it's no more than three bytes. Regards, Mark Venkat Rangan wrote: > > Mike, > > We would be in favor of Format 2 with your proposed changes as well: > - a AHS length and Data length inside the BHS > - a header digest right after the BHS > - AHS (if present) after the BHS > - Data after the AHS > - Data Digest after the data > > If someone wants protection of the AHS, a single Data Digest can cover > everything after the header digest. > > An oddity with Format 2 in the PDF file is that the AHS Length and Data > Length > are the first items. We would like to see it in BHS after the first word > which > describes the OpCode and flags, which would allow for introduction of new > OpCode formats which do not have any AHS or Data in the BHS. > > Another area that hasn't been covered, but mentioned tangentially: > - All lengths are in units of four-bytes > - All AHS and Data is padded to the nearest four-byte boundary > - There is some indication of "fill-bytes" in two-bit fields to tell > us how much of the data is actually valid. > - Digests cover any fill-bytes also; i.e., the digests are always on > quantities that are four-byte aligned. > > This facilitates word-oriented data movement and is known to be a big > win in achieving higher performance. > > Venkat Rangan > Rhapsody Networks Inc. > http://www.rhapsodynetworks.com > > -----Original Message----- > From: Mike Thompson [mailto:mike.thompson@qlogic.com] > Sent: Wednesday, March 28, 2001 10:52 AM > To: 'ips@ece.cmu.edu' > Subject: RE: Frame Formats > > Robert, > > First off let me state the obvious, more digests create > more complication, hardware or software. That said, the > approach that you describe is relatively painless because > all the information is in the BHS about how long each section > of the PDU is. This definitely makes it easier to process > PDUs, especially in an out of order case. When you have to > process one header to find the next header to find the data, > this causes a lot of grief. > > In the case of out of order processing, the hardware wants to > just find the data PDUs and skip over the other ones until they > are in order. With your scheme, I could look at just the BHS, find > out if it is a data PDU and if not, skip to the next PDU without > further processing of numerous headers. Also as you stated, the BHS > digest is always in the same location. This definitely makes > processing easier. > > Mike > > > -----Original Message----- > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > Sent: Tuesday, March 27, 2001 12:30 PM > > To: Mike Thompson > > Cc: 'ips@ece.cmu.edu' > > Subject: Re: Frame Formats > > > > > > Mike: > > > > I agree with all your comments. > > > > Earlier I had proposed that the BHS be covered by a digest that > > was always after the 48th byte (i.e., at a fixed offset), and that > > the AHS, if present, would be covered by a second digest of its own. > > (not by the data digest, which would be after the end of the data). > > By having 3 digests, 1 for the BHS, 1 for the AHS, 1 for the data, > > every read is fixed length and the digest is at the end of each > > of the reads. > > However, there was opposition to having a second header digest > > that just covered the AHS, so now there is only 1 header digest > > which covers both the BHS and the AHS and therefore, as you pointed > > out, it is at a position that is unknown when you start to read the > > header. > > > > Since you are a hardware implementer, let me ask the obvious > > question -- > > wouldn't it simplify everything to have a 2nd digest that covered just > > the AHS? The steps to receive a PDU would then be: > > 1. Read the fixed length BHS with its own digest and check it. > > 2. Extract the length of the AHS from the BHS > > If this length is 0 there is no AHS. > > If this length is positive, read that many bytes of AHS plus > > the AHS digest and check it. > > 3. Extract the length of the data from the BHS > > If this length is 0 there is no data. > > If this length is positive, read that many bytes of data plus > > the data digest and check it. > > > > From a software point of view this looks a lot simpler, since every > > read is for a known number of bytes and every read ends with a digest. > > However, I am not sure if it simplifies the hardware significantly. > > > > Thanks, > > Bob Russell > > > > > > On Tue, 27 Mar 2001, Mike Thompson wrote: > > > > > As an implementer of a hardware implementation of > > iSCSI/TCP, I would like to > > > see format 2 from the slide set presented at IETF. The > > fixed location of the > > > total data length and AHS length will make out of order > > data placement > > > reasonable. With these two fields at the beginning of the > > PDU, hardware will > > > immediately know how much data needs to be checked to > > verify the header > > > digest and if the digest is valid, it can go on to process > > the next PDU, > > > looking for data PDUs that can be processed. In previous > > formats, the > > > hardware has to process too many fields to get to the digest. > > > > > > Ideally, I would like to see a slight modification to this > > format where the > > > header digest just covers the BHS. My understanding of the > > header digest is > > > to allow for iSCSI routers to be able to modify a PDU > > header if it is acting > > > as a proxy of some type. It seems that in this case the > > only thing that > > > would be modified would be the BHS and not the AHS. With > > this change, I > > > would envision the AHS be covered by the data digest. > > Again, this makes > > > hardware processing easier, since the header that the digest covers > > > is always a fixed length. > > > > > > I also think that the 24 bit total data length is more than > > adequate for the > > > total PDU length. In order to be able to > > efficiently/reliably process PDUs, > > > the PDU length should be on the order of 8-64kBytes in > > length. PDUs of 4 > > > GBytes will require 4 Gbytes of reassembly memory in out of > > order cases. > > > This is not reasonable. > > > > > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Thu Mar 29 10:36:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA18932 for ; Thu, 29 Mar 2001 10:36:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TDw1D07054 for ips-outgoing; Thu, 29 Mar 2001 08:58:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TDuIr06925 for ; Thu, 29 Mar 2001 08:56:18 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id PAA196866 for ; Thu, 29 Mar 2001 15:56:06 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id PAA132252 for ; Thu, 29 Mar 2001 15:53:17 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1E.004C4AEA ; Thu, 29 Mar 2001 15:53:19 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 29 Mar 2001 15:56:30 +0200 Subject: Re: Frame Formats Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mark, Security guys want you to randomize pad. 0s are a weakness. Julo Mark Bakke on 29/03/2001 15:24:15 Please respond to Mark Bakke To: Venkat Rangan cc: "'Mike Thompson'" , "'ips@ece.cmu.edu'" Subject: Re: Frame Formats Venkat and Mike- I am also in favor of Format 2, as long as we do not include the AHS in the data digest. We had originally separated the header and data digests, so iSCSI gateways or other middle boxes could modify the header, but keep the data CRC end-to-end. By including AHS in the data CRC, this capability will be lost. I agree that we should have a single header digest, but I would like to see it after the AHS. If we must have a BHS digest, then I would rather have two header digests than combine AHS with data. I also agree with your comments on lengths and padding. I would add that any pad bytes MUST be set to zero; it's sort of a nit, but they could contain old data. The security folks seem to get upset whenever someone sends "previously-owned" data around, even though it's no more than three bytes. Regards, Mark Venkat Rangan wrote: > > Mike, > > We would be in favor of Format 2 with your proposed changes as well: > - a AHS length and Data length inside the BHS > - a header digest right after the BHS > - AHS (if present) after the BHS > - Data after the AHS > - Data Digest after the data > > If someone wants protection of the AHS, a single Data Digest can cover > everything after the header digest. > > An oddity with Format 2 in the PDF file is that the AHS Length and Data > Length > are the first items. We would like to see it in BHS after the first word > which > describes the OpCode and flags, which would allow for introduction of new > OpCode formats which do not have any AHS or Data in the BHS. > > Another area that hasn't been covered, but mentioned tangentially: > - All lengths are in units of four-bytes > - All AHS and Data is padded to the nearest four-byte boundary > - There is some indication of "fill-bytes" in two-bit fields to tell > us how much of the data is actually valid. > - Digests cover any fill-bytes also; i.e., the digests are always on > quantities that are four-byte aligned. > > This facilitates word-oriented data movement and is known to be a big > win in achieving higher performance. > > Venkat Rangan > Rhapsody Networks Inc. > http://www.rhapsodynetworks.com > > -----Original Message----- > From: Mike Thompson [mailto:mike.thompson@qlogic.com] > Sent: Wednesday, March 28, 2001 10:52 AM > To: 'ips@ece.cmu.edu' > Subject: RE: Frame Formats > > Robert, > > First off let me state the obvious, more digests create > more complication, hardware or software. That said, the > approach that you describe is relatively painless because > all the information is in the BHS about how long each section > of the PDU is. This definitely makes it easier to process > PDUs, especially in an out of order case. When you have to > process one header to find the next header to find the data, > this causes a lot of grief. > > In the case of out of order processing, the hardware wants to > just find the data PDUs and skip over the other ones until they > are in order. With your scheme, I could look at just the BHS, find > out if it is a data PDU and if not, skip to the next PDU without > further processing of numerous headers. Also as you stated, the BHS > digest is always in the same location. This definitely makes > processing easier. > > Mike > > > -----Original Message----- > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > Sent: Tuesday, March 27, 2001 12:30 PM > > To: Mike Thompson > > Cc: 'ips@ece.cmu.edu' > > Subject: Re: Frame Formats > > > > > > Mike: > > > > I agree with all your comments. > > > > Earlier I had proposed that the BHS be covered by a digest that > > was always after the 48th byte (i.e., at a fixed offset), and that > > the AHS, if present, would be covered by a second digest of its own. > > (not by the data digest, which would be after the end of the data). > > By having 3 digests, 1 for the BHS, 1 for the AHS, 1 for the data, > > every read is fixed length and the digest is at the end of each > > of the reads. > > However, there was opposition to having a second header digest > > that just covered the AHS, so now there is only 1 header digest > > which covers both the BHS and the AHS and therefore, as you pointed > > out, it is at a position that is unknown when you start to read the > > header. > > > > Since you are a hardware implementer, let me ask the obvious > > question -- > > wouldn't it simplify everything to have a 2nd digest that covered just > > the AHS? The steps to receive a PDU would then be: > > 1. Read the fixed length BHS with its own digest and check it. > > 2. Extract the length of the AHS from the BHS > > If this length is 0 there is no AHS. > > If this length is positive, read that many bytes of AHS plus > > the AHS digest and check it. > > 3. Extract the length of the data from the BHS > > If this length is 0 there is no data. > > If this length is positive, read that many bytes of data plus > > the data digest and check it. > > > > From a software point of view this looks a lot simpler, since every > > read is for a known number of bytes and every read ends with a digest. > > However, I am not sure if it simplifies the hardware significantly. > > > > Thanks, > > Bob Russell > > > > > > On Tue, 27 Mar 2001, Mike Thompson wrote: > > > > > As an implementer of a hardware implementation of > > iSCSI/TCP, I would like to > > > see format 2 from the slide set presented at IETF. The > > fixed location of the > > > total data length and AHS length will make out of order > > data placement > > > reasonable. With these two fields at the beginning of the > > PDU, hardware will > > > immediately know how much data needs to be checked to > > verify the header > > > digest and if the digest is valid, it can go on to process > > the next PDU, > > > looking for data PDUs that can be processed. In previous > > formats, the > > > hardware has to process too many fields to get to the digest. > > > > > > Ideally, I would like to see a slight modification to this > > format where the > > > header digest just covers the BHS. My understanding of the > > header digest is > > > to allow for iSCSI routers to be able to modify a PDU > > header if it is acting > > > as a proxy of some type. It seems that in this case the > > only thing that > > > would be modified would be the BHS and not the AHS. With > > this change, I > > > would envision the AHS be covered by the data digest. > > Again, this makes > > > hardware processing easier, since the header that the digest covers > > > is always a fixed length. > > > > > > I also think that the 24 bit total data length is more than > > adequate for the > > > total PDU length. In order to be able to > > efficiently/reliably process PDUs, > > > the PDU length should be on the order of 8-64kBytes in > > length. PDUs of 4 > > > GBytes will require 4 Gbytes of reassembly memory in out of > > order cases. > > > This is not reasonable. > > > > > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Thu Mar 29 11:37:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA22826 for ; Thu, 29 Mar 2001 11:37:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TETCC09310 for ips-outgoing; Thu, 29 Mar 2001 09:29:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TESYr09273 for ; Thu, 29 Mar 2001 09:28:34 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id JAA19620; Thu, 29 Mar 2001 09:28:24 -0500 (EST) Message-ID: <3AC34664.97B39166@cisco.com> Date: Thu, 29 Mar 2001 08:27:48 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: julian_satran@il.ibm.com CC: ips@ece.cmu.edu Subject: Re: Frame Formats References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian- If generating random numbers to place in a pad field is too expensive, is the next best thing (from a security point-of-view) to zero the pad, or to leave whatever was there? -- Mark julian_satran@il.ibm.com wrote: > > Mark, > > Security guys want you to randomize pad. 0s are a weakness. > > Julo > > Mark Bakke on 29/03/2001 15:24:15 > > Please respond to Mark Bakke > > To: Venkat Rangan > cc: "'Mike Thompson'" , "'ips@ece.cmu.edu'" > > Subject: Re: Frame Formats > > Venkat and Mike- > > I am also in favor of Format 2, as long as we do not include > the AHS in the data digest. > > We had originally separated the header and data digests, so iSCSI > gateways or other middle boxes could modify the header, but keep > the data CRC end-to-end. By including AHS in the data CRC, this > capability will be lost. > > I agree that we should have a single header digest, but I would > like to see it after the AHS. If we must have a BHS digest, then > I would rather have two header digests than combine AHS with data. > > I also agree with your comments on lengths and padding. I would > add that any pad bytes MUST be set to zero; it's sort of a nit, but > they could contain old data. The security folks seem to get upset > whenever someone sends "previously-owned" data around, even though > it's no more than three bytes. > > Regards, > > Mark > > Venkat Rangan wrote: > > > > Mike, > > > > We would be in favor of Format 2 with your proposed changes as well: > > - a AHS length and Data length inside the BHS > > - a header digest right after the BHS > > - AHS (if present) after the BHS > > - Data after the AHS > > - Data Digest after the data > > > > If someone wants protection of the AHS, a single Data Digest can cover > > everything after the header digest. > > > > An oddity with Format 2 in the PDF file is that the AHS Length and Data > > Length > > are the first items. We would like to see it in BHS after the first word > > which > > describes the OpCode and flags, which would allow for introduction of new > > OpCode formats which do not have any AHS or Data in the BHS. > > > > Another area that hasn't been covered, but mentioned tangentially: > > - All lengths are in units of four-bytes > > - All AHS and Data is padded to the nearest four-byte boundary > > - There is some indication of "fill-bytes" in two-bit fields to tell > > us how much of the data is actually valid. > > - Digests cover any fill-bytes also; i.e., the digests are always on > > quantities that are four-byte aligned. > > > > This facilitates word-oriented data movement and is known to be a big > > win in achieving higher performance. > > > > Venkat Rangan > > Rhapsody Networks Inc. > > http://www.rhapsodynetworks.com > > > > -----Original Message----- > > From: Mike Thompson [mailto:mike.thompson@qlogic.com] > > Sent: Wednesday, March 28, 2001 10:52 AM > > To: 'ips@ece.cmu.edu' > > Subject: RE: Frame Formats > > > > Robert, > > > > First off let me state the obvious, more digests create > > more complication, hardware or software. That said, the > > approach that you describe is relatively painless because > > all the information is in the BHS about how long each section > > of the PDU is. This definitely makes it easier to process > > PDUs, especially in an out of order case. When you have to > > process one header to find the next header to find the data, > > this causes a lot of grief. > > > > In the case of out of order processing, the hardware wants to > > just find the data PDUs and skip over the other ones until they > > are in order. With your scheme, I could look at just the BHS, find > > out if it is a data PDU and if not, skip to the next PDU without > > further processing of numerous headers. Also as you stated, the BHS > > digest is always in the same location. This definitely makes > > processing easier. > > > > Mike > > > > > -----Original Message----- > > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > > Sent: Tuesday, March 27, 2001 12:30 PM > > > To: Mike Thompson > > > Cc: 'ips@ece.cmu.edu' > > > Subject: Re: Frame Formats > > > > > > > > > Mike: > > > > > > I agree with all your comments. > > > > > > Earlier I had proposed that the BHS be covered by a digest that > > > was always after the 48th byte (i.e., at a fixed offset), and that > > > the AHS, if present, would be covered by a second digest of its own. > > > (not by the data digest, which would be after the end of the data). > > > By having 3 digests, 1 for the BHS, 1 for the AHS, 1 for the data, > > > every read is fixed length and the digest is at the end of each > > > of the reads. > > > However, there was opposition to having a second header digest > > > that just covered the AHS, so now there is only 1 header digest > > > which covers both the BHS and the AHS and therefore, as you pointed > > > out, it is at a position that is unknown when you start to read the > > > header. > > > > > > Since you are a hardware implementer, let me ask the obvious > > > question -- > > > wouldn't it simplify everything to have a 2nd digest that covered just > > > the AHS? The steps to receive a PDU would then be: > > > 1. Read the fixed length BHS with its own digest and check it. > > > 2. Extract the length of the AHS from the BHS > > > If this length is 0 there is no AHS. > > > If this length is positive, read that many bytes of AHS plus > > > the AHS digest and check it. > > > 3. Extract the length of the data from the BHS > > > If this length is 0 there is no data. > > > If this length is positive, read that many bytes of data plus > > > the data digest and check it. > > > > > > From a software point of view this looks a lot simpler, since every > > > read is for a known number of bytes and every read ends with a digest. > > > However, I am not sure if it simplifies the hardware significantly. > > > > > > Thanks, > > > Bob Russell > > > > > > > > > On Tue, 27 Mar 2001, Mike Thompson wrote: > > > > > > > As an implementer of a hardware implementation of > > > iSCSI/TCP, I would like to > > > > see format 2 from the slide set presented at IETF. The > > > fixed location of the > > > > total data length and AHS length will make out of order > > > data placement > > > > reasonable. With these two fields at the beginning of the > > > PDU, hardware will > > > > immediately know how much data needs to be checked to > > > verify the header > > > > digest and if the digest is valid, it can go on to process > > > the next PDU, > > > > looking for data PDUs that can be processed. In previous > > > formats, the > > > > hardware has to process too many fields to get to the digest. > > > > > > > > Ideally, I would like to see a slight modification to this > > > format where the > > > > header digest just covers the BHS. My understanding of the > > > header digest is > > > > to allow for iSCSI routers to be able to modify a PDU > > > header if it is acting > > > > as a proxy of some type. It seems that in this case the > > > only thing that > > > > would be modified would be the BHS and not the AHS. With > > > this change, I > > > > would envision the AHS be covered by the data digest. > > > Again, this makes > > > > hardware processing easier, since the header that the digest covers > > > > is always a fixed length. > > > > > > > > I also think that the 24 bit total data length is more than > > > adequate for the > > > > total PDU length. In order to be able to > > > efficiently/reliably process PDUs, > > > > the PDU length should be on the order of 8-64kBytes in > > > length. PDUs of 4 > > > > GBytes will require 4 Gbytes of reassembly memory in out of > > > order cases. > > > > This is not reasonable. > > > > > > > > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Thu Mar 29 11:37:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA22856 for ; Thu, 29 Mar 2001 11:37:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TEr5t10969 for ips-outgoing; Thu, 29 Mar 2001 09:53:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TEqCr10898 for ; Thu, 29 Mar 2001 09:52:12 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 2A4CA639 for ; Thu, 29 Mar 2001 07:52:11 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id CDF91181 for ; Thu, 29 Mar 2001 09:52:09 -0500 (EST) Received: from agilent.com (cos1nai132253.cos.agilent.com [141.184.132.253]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id GAA18476 for ; Thu, 29 Mar 2001 06:52:07 -0800 (PST) Message-ID: <3AC34C15.FADE1A42@agilent.com> Date: Thu, 29 Mar 2001 06:52:05 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: synch and steering comments References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > -Section 1.2.8.2 states that a Synch and Steering layer is optional. > It has to be qualifed that it is optional only for those iSCSI devices > which perform connection recovery on header digest errors, since that's > how they cope with loss of framing. (I guess this may change in next rev?) > > +++ with the new format I think that we have: > > - one more chance if we go for format 1 or > - drop the connection on header error > > In both cases we can leave synch and steering optional If framing is implemented, then you don't have to drop the connection on a header error, since the next frame by definition contains the beginning of the next header. -Matt From owner-ips@ece.cmu.edu Thu Mar 29 11:37:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA22900 for ; Thu, 29 Mar 2001 11:37:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TEp2l10804 for ips-outgoing; Thu, 29 Mar 2001 09:51:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TEoUr10728 for ; Thu, 29 Mar 2001 09:50:31 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2650.21) id ; Thu, 29 Mar 2001 09:41:28 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0708015378@corpmx9.isus.emc.com> From: Black_David@emc.com To: mbakke@cisco.com Cc: ips@ece.cmu.edu Subject: RE: Frame Formats Date: Thu, 29 Mar 2001 09:50:22 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I wouldn't worry about the "security" issue of padding with zeroes vs. padding with random bits. Anyone who cares about this sort of issue will put a compression algorithm in the security implementation, and zeroes tend to compress very nicely :-). The best thing to do with the padding is probably to incorporate it into the AHS header specifications - e.g., rather than tacking 3 bytes of pad onto an AHS, add 3 reserved bytes to the header itself in the AHS, so that all headers that might turn up in the AHS are 4-byte aligned. This is a good thing to do in any case, as dealing with things like a 32-bit quantity that is disaligned by 16 bits tends to be awkward. Reserved bytes are easy to deal with - they MUST be sent as zero, and MUST be ignored at the receiver. This is a variant of the usual admonition to be conservative in what is sent and liberal in what is accepted. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Mark Bakke [SMTP:mbakke@cisco.com] > Sent: Thursday, March 29, 2001 9:28 AM > To: julian_satran@il.ibm.com > Cc: ips@ece.cmu.edu > Subject: Re: Frame Formats > > Julian- > > If generating random numbers to place in a pad field is > too expensive, is the next best thing (from a security > point-of-view) to zero the pad, or to leave whatever was > there? > > -- > Mark > > julian_satran@il.ibm.com wrote: > > > > Mark, > > > > Security guys want you to randomize pad. 0s are a weakness. > > > > Julo > > > > Mark Bakke on 29/03/2001 15:24:15 > > > > Please respond to Mark Bakke > > > > To: Venkat Rangan > > cc: "'Mike Thompson'" , "'ips@ece.cmu.edu'" > > > > Subject: Re: Frame Formats > > > > Venkat and Mike- > > > > I am also in favor of Format 2, as long as we do not include > > the AHS in the data digest. > > > > We had originally separated the header and data digests, so iSCSI > > gateways or other middle boxes could modify the header, but keep > > the data CRC end-to-end. By including AHS in the data CRC, this > > capability will be lost. > > > > I agree that we should have a single header digest, but I would > > like to see it after the AHS. If we must have a BHS digest, then > > I would rather have two header digests than combine AHS with data. > > > > I also agree with your comments on lengths and padding. I would > > add that any pad bytes MUST be set to zero; it's sort of a nit, but > > they could contain old data. The security folks seem to get upset > > whenever someone sends "previously-owned" data around, even though > > it's no more than three bytes. > > > > Regards, > > > > Mark > > > > Venkat Rangan wrote: > > > > > > Mike, > > > > > > We would be in favor of Format 2 with your proposed changes as well: > > > - a AHS length and Data length inside the BHS > > > - a header digest right after the BHS > > > - AHS (if present) after the BHS > > > - Data after the AHS > > > - Data Digest after the data > > > > > > If someone wants protection of the AHS, a single Data Digest can cover > > > everything after the header digest. > > > > > > An oddity with Format 2 in the PDF file is that the AHS Length and > Data > > > Length > > > are the first items. We would like to see it in BHS after the first > word > > > which > > > describes the OpCode and flags, which would allow for introduction of > new > > > OpCode formats which do not have any AHS or Data in the BHS. > > > > > > Another area that hasn't been covered, but mentioned tangentially: > > > - All lengths are in units of four-bytes > > > - All AHS and Data is padded to the nearest four-byte boundary > > > - There is some indication of "fill-bytes" in two-bit fields to tell > > > us how much of the data is actually valid. > > > - Digests cover any fill-bytes also; i.e., the digests are always on > > > quantities that are four-byte aligned. > > > > > > This facilitates word-oriented data movement and is known to be a big > > > win in achieving higher performance. > > > > > > Venkat Rangan > > > Rhapsody Networks Inc. > > > http://www.rhapsodynetworks.com > > > > > > -----Original Message----- > > > From: Mike Thompson [mailto:mike.thompson@qlogic.com] > > > Sent: Wednesday, March 28, 2001 10:52 AM > > > To: 'ips@ece.cmu.edu' > > > Subject: RE: Frame Formats > > > > > > Robert, > > > > > > First off let me state the obvious, more digests create > > > more complication, hardware or software. That said, the > > > approach that you describe is relatively painless because > > > all the information is in the BHS about how long each section > > > of the PDU is. This definitely makes it easier to process > > > PDUs, especially in an out of order case. When you have to > > > process one header to find the next header to find the data, > > > this causes a lot of grief. > > > > > > In the case of out of order processing, the hardware wants to > > > just find the data PDUs and skip over the other ones until they > > > are in order. With your scheme, I could look at just the BHS, find > > > out if it is a data PDU and if not, skip to the next PDU without > > > further processing of numerous headers. Also as you stated, the BHS > > > digest is always in the same location. This definitely makes > > > processing easier. > > > > > > Mike > > > > > > > -----Original Message----- > > > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > > > Sent: Tuesday, March 27, 2001 12:30 PM > > > > To: Mike Thompson > > > > Cc: 'ips@ece.cmu.edu' > > > > Subject: Re: Frame Formats > > > > > > > > > > > > Mike: > > > > > > > > I agree with all your comments. > > > > > > > > Earlier I had proposed that the BHS be covered by a digest that > > > > was always after the 48th byte (i.e., at a fixed offset), and that > > > > the AHS, if present, would be covered by a second digest of its own. > > > > (not by the data digest, which would be after the end of the data). > > > > By having 3 digests, 1 for the BHS, 1 for the AHS, 1 for the data, > > > > every read is fixed length and the digest is at the end of each > > > > of the reads. > > > > However, there was opposition to having a second header digest > > > > that just covered the AHS, so now there is only 1 header digest > > > > which covers both the BHS and the AHS and therefore, as you pointed > > > > out, it is at a position that is unknown when you start to read the > > > > header. > > > > > > > > Since you are a hardware implementer, let me ask the obvious > > > > question -- > > > > wouldn't it simplify everything to have a 2nd digest that covered > just > > > > the AHS? The steps to receive a PDU would then be: > > > > 1. Read the fixed length BHS with its own digest and check it. > > > > 2. Extract the length of the AHS from the BHS > > > > If this length is 0 there is no AHS. > > > > If this length is positive, read that many bytes of AHS > plus > > > > the AHS digest and check it. > > > > 3. Extract the length of the data from the BHS > > > > If this length is 0 there is no data. > > > > If this length is positive, read that many bytes of data > plus > > > > the data digest and check it. > > > > > > > > From a software point of view this looks a lot simpler, since every > > > > read is for a known number of bytes and every read ends with a > digest. > > > > However, I am not sure if it simplifies the hardware significantly. > > > > > > > > Thanks, > > > > Bob Russell > > > > > > > > > > > > On Tue, 27 Mar 2001, Mike Thompson wrote: > > > > > > > > > As an implementer of a hardware implementation of > > > > iSCSI/TCP, I would like to > > > > > see format 2 from the slide set presented at IETF. The > > > > fixed location of the > > > > > total data length and AHS length will make out of order > > > > data placement > > > > > reasonable. With these two fields at the beginning of the > > > > PDU, hardware will > > > > > immediately know how much data needs to be checked to > > > > verify the header > > > > > digest and if the digest is valid, it can go on to process > > > > the next PDU, > > > > > looking for data PDUs that can be processed. In previous > > > > formats, the > > > > > hardware has to process too many fields to get to the digest. > > > > > > > > > > Ideally, I would like to see a slight modification to this > > > > format where the > > > > > header digest just covers the BHS. My understanding of the > > > > header digest is > > > > > to allow for iSCSI routers to be able to modify a PDU > > > > header if it is acting > > > > > as a proxy of some type. It seems that in this case the > > > > only thing that > > > > > would be modified would be the BHS and not the AHS. With > > > > this change, I > > > > > would envision the AHS be covered by the data digest. > > > > Again, this makes > > > > > hardware processing easier, since the header that the digest > covers > > > > > is always a fixed length. > > > > > > > > > > I also think that the 24 bit total data length is more than > > > > adequate for the > > > > > total PDU length. In order to be able to > > > > efficiently/reliably process PDUs, > > > > > the PDU length should be on the order of 8-64kBytes in > > > > length. PDUs of 4 > > > > > GBytes will require 4 Gbytes of reassembly memory in out of > > > > order cases. > > > > > This is not reasonable. > > > > > > > > > > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 From owner-ips@ece.cmu.edu Thu Mar 29 11:38:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA22947 for ; Thu, 29 Mar 2001 11:38:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TF15311475 for ips-outgoing; Thu, 29 Mar 2001 10:01:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TF0pr11421 for ; Thu, 29 Mar 2001 10:00:51 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 0C392662 for ; Thu, 29 Mar 2001 08:00:51 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 3D3B3179 for ; Thu, 29 Mar 2001 10:00:50 -0500 (EST) Received: from agilent.com (cos1nai132253.cos.agilent.com [141.184.132.253]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id HAA19117 for ; Thu, 29 Mar 2001 07:00:47 -0800 (PST) Message-ID: <3AC34E1E.DFBD86AF@agilent.com> Date: Thu, 29 Mar 2001 07:00:46 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: Frame Formats References: <3AC34664.97B39166@cisco.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I would rather the spec be silent on the value of "pad" bytes. The only thing the spec should say is whether or not the pad bytes are included in the data digest. -Matt Mark Bakke wrote: > > Julian- > > If generating random numbers to place in a pad field is > too expensive, is the next best thing (from a security > point-of-view) to zero the pad, or to leave whatever was > there? > > -- > Mark > > julian_satran@il.ibm.com wrote: > > > > Mark, > > > > Security guys want you to randomize pad. 0s are a weakness. > > > > Julo From owner-ips@ece.cmu.edu Thu Mar 29 11:38:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA22967 for ; Thu, 29 Mar 2001 11:38:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TEx3v11338 for ips-outgoing; Thu, 29 Mar 2001 09:59:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TEwer11321 for ; Thu, 29 Mar 2001 09:58:44 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id BAAF5270 for ; Thu, 29 Mar 2001 07:58:39 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id D951E161 for ; Thu, 29 Mar 2001 09:58:38 -0500 (EST) Received: from agilent.com (cos1nai132253.cos.agilent.com [141.184.132.253]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id GAA18945 for ; Thu, 29 Mar 2001 06:58:36 -0800 (PST) Message-ID: <3AC34D9B.9A0B260C@agilent.com> Date: Thu, 29 Mar 2001 06:58:35 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: synch and steering comments References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > -I am somewhat confused about the following statement in section 1.2.8.2: > "The Synch and Steering Layer is required to add to every sent data > item (IP packet, TCP packet or some other superstructure) enough > information to enable the receiver to steer it to a memory location > independent of any other piece. " > > Clearly from the way I understood the markers in Appendix.C, it doesn't > comply with this requirement. A more generic statement would be: > "The Synch and Steering Layer is required to add adequate > information to the data stream to enable the receiver to quickly > steer the stream to its final memory location, even in the face of > discontiguities in the stream. " > > +++ Markers are but one example and have only the Synch component. The > statement refers to a full steering (RDMA) scheme +++ Julian, No "full steering (RDMA) scheme" is required. A Sync mechanism that that implements framing such that the first item in each frame is an iSCSI header is all that is required to enable steering because the iSCSI header contains all the required steering information (task tag, data offset). -Matt From owner-ips@ece.cmu.edu Thu Mar 29 11:38:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA22979 for ; Thu, 29 Mar 2001 11:38:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TFA3w12060 for ips-outgoing; Thu, 29 Mar 2001 10:10:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (mars.iol.unh.edu [132.177.121.222]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TF9er12048 for ; Thu, 29 Mar 2001 10:09:40 -0500 (EST) Received: from localhost (rdr@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id KAA07034; Thu, 29 Mar 2001 10:14:45 -0500 (EST) Date: Thu, 29 Mar 2001 10:14:45 -0500 From: "Robert D. Russell" To: Venkat Rangan cc: "'Mike Thompson'" , "'ips@ece.cmu.edu'" Subject: RE: Frame Formats In-Reply-To: <15851BD69CFCD41186B100B0D0AABE650C1B16@med.corp.rhapsodynetworks.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Venkat: A point about your comment: "which would allow for introduction of new OpCode formats which do not have any AHS or Data in the BHS." I believe the whole point about having the AHS_length and Data_length in FIXED locations in EVERY header is that it always makes it possible to find and decode the PDU, even if the command type is not understood. What you DON't want to do is introduce an irregularity into the BHS such that these fixed fields disappear for some OpCodes, thereby making it impossible to decode new PDU frames and loosing backward compatibility. If a new opcode does not need any AHS or Data, these fields would have a value of 0, but they should still always be there. Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 On Wed, 28 Mar 2001, Venkat Rangan wrote: > Mike, > > We would be in favor of Format 2 with your proposed changes as well: > - a AHS length and Data length inside the BHS > - a header digest right after the BHS > - AHS (if present) after the BHS > - Data after the AHS > - Data Digest after the data > > If someone wants protection of the AHS, a single Data Digest can cover > everything after the header digest. > > An oddity with Format 2 in the PDF file is that the AHS Length and Data > Length > are the first items. We would like to see it in BHS after the first word > which > describes the OpCode and flags, which would allow for introduction of new > OpCode formats which do not have any AHS or Data in the BHS. > > Another area that hasn't been covered, but mentioned tangentially: > - All lengths are in units of four-bytes > - All AHS and Data is padded to the nearest four-byte boundary > - There is some indication of "fill-bytes" in two-bit fields to tell > us how much of the data is actually valid. > - Digests cover any fill-bytes also; i.e., the digests are always on > quantities that are four-byte aligned. > > This facilitates word-oriented data movement and is known to be a big > win in achieving higher performance. > > Venkat Rangan > Rhapsody Networks Inc. > http://www.rhapsodynetworks.com > > > > -----Original Message----- > From: Mike Thompson [mailto:mike.thompson@qlogic.com] > Sent: Wednesday, March 28, 2001 10:52 AM > To: 'ips@ece.cmu.edu' > Subject: RE: Frame Formats > > > Robert, > > First off let me state the obvious, more digests create > more complication, hardware or software. That said, the > approach that you describe is relatively painless because > all the information is in the BHS about how long each section > of the PDU is. This definitely makes it easier to process > PDUs, especially in an out of order case. When you have to > process one header to find the next header to find the data, > this causes a lot of grief. > > In the case of out of order processing, the hardware wants to > just find the data PDUs and skip over the other ones until they > are in order. With your scheme, I could look at just the BHS, find > out if it is a data PDU and if not, skip to the next PDU without > further processing of numerous headers. Also as you stated, the BHS > digest is always in the same location. This definitely makes > processing easier. > > Mike > > > -----Original Message----- > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > Sent: Tuesday, March 27, 2001 12:30 PM > > To: Mike Thompson > > Cc: 'ips@ece.cmu.edu' > > Subject: Re: Frame Formats > > > > > > Mike: > > > > I agree with all your comments. > > > > Earlier I had proposed that the BHS be covered by a digest that > > was always after the 48th byte (i.e., at a fixed offset), and that > > the AHS, if present, would be covered by a second digest of its own. > > (not by the data digest, which would be after the end of the data). > > By having 3 digests, 1 for the BHS, 1 for the AHS, 1 for the data, > > every read is fixed length and the digest is at the end of each > > of the reads. > > However, there was opposition to having a second header digest > > that just covered the AHS, so now there is only 1 header digest > > which covers both the BHS and the AHS and therefore, as you pointed > > out, it is at a position that is unknown when you start to read the > > header. > > > > Since you are a hardware implementer, let me ask the obvious > > question -- > > wouldn't it simplify everything to have a 2nd digest that covered just > > the AHS? The steps to receive a PDU would then be: > > 1. Read the fixed length BHS with its own digest and check it. > > 2. Extract the length of the AHS from the BHS > > If this length is 0 there is no AHS. > > If this length is positive, read that many bytes of AHS plus > > the AHS digest and check it. > > 3. Extract the length of the data from the BHS > > If this length is 0 there is no data. > > If this length is positive, read that many bytes of data plus > > the data digest and check it. > > > > From a software point of view this looks a lot simpler, since every > > read is for a known number of bytes and every read ends with a digest. > > However, I am not sure if it simplifies the hardware significantly. > > > > Thanks, > > Bob Russell > > > > > > On Tue, 27 Mar 2001, Mike Thompson wrote: > > > > > As an implementer of a hardware implementation of > > iSCSI/TCP, I would like to > > > see format 2 from the slide set presented at IETF. The > > fixed location of the > > > total data length and AHS length will make out of order > > data placement > > > reasonable. With these two fields at the beginning of the > > PDU, hardware will > > > immediately know how much data needs to be checked to > > verify the header > > > digest and if the digest is valid, it can go on to process > > the next PDU, > > > looking for data PDUs that can be processed. In previous > > formats, the > > > hardware has to process too many fields to get to the digest. > > > > > > Ideally, I would like to see a slight modification to this > > format where the > > > header digest just covers the BHS. My understanding of the > > header digest is > > > to allow for iSCSI routers to be able to modify a PDU > > header if it is acting > > > as a proxy of some type. It seems that in this case the > > only thing that > > > would be modified would be the BHS and not the AHS. With > > this change, I > > > would envision the AHS be covered by the data digest. > > Again, this makes > > > hardware processing easier, since the header that the digest covers > > > is always a fixed length. > > > > > > I also think that the 24 bit total data length is more than > > adequate for the > > > total PDU length. In order to be able to > > efficiently/reliably process PDUs, > > > the PDU length should be on the order of 8-64kBytes in > > length. PDUs of 4 > > > GBytes will require 4 Gbytes of reassembly memory in out of > > order cases. > > > This is not reasonable. > > > > > > From owner-ips@ece.cmu.edu Thu Mar 29 11:39:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA23014 for ; Thu, 29 Mar 2001 11:39:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TFg5x14244 for ips-outgoing; Thu, 29 Mar 2001 10:42:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TFeqr14116 for ; Thu, 29 Mar 2001 10:41:06 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA59220 for ; Thu, 29 Mar 2001 17:40:41 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA98476 for ; Thu, 29 Mar 2001 17:39:01 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1E.0055DCB2 ; Thu, 29 Mar 2001 17:37:50 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 29 Mar 2001 17:28:45 +0200 Subject: Re: Frame Formats Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I have no idea. I guess that they think random is easy. Julo Mark Bakke on 29/03/2001 16:27:48 Please respond to Mark Bakke To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: Frame Formats Julian- If generating random numbers to place in a pad field is too expensive, is the next best thing (from a security point-of-view) to zero the pad, or to leave whatever was there? -- Mark julian_satran@il.ibm.com wrote: > > Mark, > > Security guys want you to randomize pad. 0s are a weakness. > > Julo > > Mark Bakke on 29/03/2001 15:24:15 > > Please respond to Mark Bakke > > To: Venkat Rangan > cc: "'Mike Thompson'" , "'ips@ece.cmu.edu'" > > Subject: Re: Frame Formats > > Venkat and Mike- > > I am also in favor of Format 2, as long as we do not include > the AHS in the data digest. > > We had originally separated the header and data digests, so iSCSI > gateways or other middle boxes could modify the header, but keep > the data CRC end-to-end. By including AHS in the data CRC, this > capability will be lost. > > I agree that we should have a single header digest, but I would > like to see it after the AHS. If we must have a BHS digest, then > I would rather have two header digests than combine AHS with data. > > I also agree with your comments on lengths and padding. I would > add that any pad bytes MUST be set to zero; it's sort of a nit, but > they could contain old data. The security folks seem to get upset > whenever someone sends "previously-owned" data around, even though > it's no more than three bytes. > > Regards, > > Mark > > Venkat Rangan wrote: > > > > Mike, > > > > We would be in favor of Format 2 with your proposed changes as well: > > - a AHS length and Data length inside the BHS > > - a header digest right after the BHS > > - AHS (if present) after the BHS > > - Data after the AHS > > - Data Digest after the data > > > > If someone wants protection of the AHS, a single Data Digest can cover > > everything after the header digest. > > > > An oddity with Format 2 in the PDF file is that the AHS Length and Data > > Length > > are the first items. We would like to see it in BHS after the first word > > which > > describes the OpCode and flags, which would allow for introduction of new > > OpCode formats which do not have any AHS or Data in the BHS. > > > > Another area that hasn't been covered, but mentioned tangentially: > > - All lengths are in units of four-bytes > > - All AHS and Data is padded to the nearest four-byte boundary > > - There is some indication of "fill-bytes" in two-bit fields to tell > > us how much of the data is actually valid. > > - Digests cover any fill-bytes also; i.e., the digests are always on > > quantities that are four-byte aligned. > > > > This facilitates word-oriented data movement and is known to be a big > > win in achieving higher performance. > > > > Venkat Rangan > > Rhapsody Networks Inc. > > http://www.rhapsodynetworks.com > > > > -----Original Message----- > > From: Mike Thompson [mailto:mike.thompson@qlogic.com] > > Sent: Wednesday, March 28, 2001 10:52 AM > > To: 'ips@ece.cmu.edu' > > Subject: RE: Frame Formats > > > > Robert, > > > > First off let me state the obvious, more digests create > > more complication, hardware or software. That said, the > > approach that you describe is relatively painless because > > all the information is in the BHS about how long each section > > of the PDU is. This definitely makes it easier to process > > PDUs, especially in an out of order case. When you have to > > process one header to find the next header to find the data, > > this causes a lot of grief. > > > > In the case of out of order processing, the hardware wants to > > just find the data PDUs and skip over the other ones until they > > are in order. With your scheme, I could look at just the BHS, find > > out if it is a data PDU and if not, skip to the next PDU without > > further processing of numerous headers. Also as you stated, the BHS > > digest is always in the same location. This definitely makes > > processing easier. > > > > Mike > > > > > -----Original Message----- > > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > > Sent: Tuesday, March 27, 2001 12:30 PM > > > To: Mike Thompson > > > Cc: 'ips@ece.cmu.edu' > > > Subject: Re: Frame Formats > > > > > > > > > Mike: > > > > > > I agree with all your comments. > > > > > > Earlier I had proposed that the BHS be covered by a digest that > > > was always after the 48th byte (i.e., at a fixed offset), and that > > > the AHS, if present, would be covered by a second digest of its own. > > > (not by the data digest, which would be after the end of the data). > > > By having 3 digests, 1 for the BHS, 1 for the AHS, 1 for the data, > > > every read is fixed length and the digest is at the end of each > > > of the reads. > > > However, there was opposition to having a second header digest > > > that just covered the AHS, so now there is only 1 header digest > > > which covers both the BHS and the AHS and therefore, as you pointed > > > out, it is at a position that is unknown when you start to read the > > > header. > > > > > > Since you are a hardware implementer, let me ask the obvious > > > question -- > > > wouldn't it simplify everything to have a 2nd digest that covered just > > > the AHS? The steps to receive a PDU would then be: > > > 1. Read the fixed length BHS with its own digest and check it. > > > 2. Extract the length of the AHS from the BHS > > > If this length is 0 there is no AHS. > > > If this length is positive, read that many bytes of AHS plus > > > the AHS digest and check it. > > > 3. Extract the length of the data from the BHS > > > If this length is 0 there is no data. > > > If this length is positive, read that many bytes of data plus > > > the data digest and check it. > > > > > > From a software point of view this looks a lot simpler, since every > > > read is for a known number of bytes and every read ends with a digest. > > > However, I am not sure if it simplifies the hardware significantly. > > > > > > Thanks, > > > Bob Russell > > > > > > > > > On Tue, 27 Mar 2001, Mike Thompson wrote: > > > > > > > As an implementer of a hardware implementation of > > > iSCSI/TCP, I would like to > > > > see format 2 from the slide set presented at IETF. The > > > fixed location of the > > > > total data length and AHS length will make out of order > > > data placement > > > > reasonable. With these two fields at the beginning of the > > > PDU, hardware will > > > > immediately know how much data needs to be checked to > > > verify the header > > > > digest and if the digest is valid, it can go on to process > > > the next PDU, > > > > looking for data PDUs that can be processed. In previous > > > formats, the > > > > hardware has to process too many fields to get to the digest. > > > > > > > > Ideally, I would like to see a slight modification to this > > > format where the > > > > header digest just covers the BHS. My understanding of the > > > header digest is > > > > to allow for iSCSI routers to be able to modify a PDU > > > header if it is acting > > > > as a proxy of some type. It seems that in this case the > > > only thing that > > > > would be modified would be the BHS and not the AHS. With > > > this change, I > > > > would envision the AHS be covered by the data digest. > > > Again, this makes > > > > hardware processing easier, since the header that the digest covers > > > > is always a fixed length. > > > > > > > > I also think that the 24 bit total data length is more than > > > adequate for the > > > > total PDU length. In order to be able to > > > efficiently/reliably process PDUs, > > > > the PDU length should be on the order of 8-64kBytes in > > > length. PDUs of 4 > > > > GBytes will require 4 Gbytes of reassembly memory in out of > > > order cases. > > > > This is not reasonable. > > > > > > > > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Thu Mar 29 13:22:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA00120 for ; Thu, 29 Mar 2001 13:22:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TG37k15627 for ips-outgoing; Thu, 29 Mar 2001 11:03:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TG2Wr15572 for ; Thu, 29 Mar 2001 11:02:32 -0500 (EST) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id B50D017F2 for ; Thu, 29 Mar 2001 08:01:49 -0800 (PST) Received: from mk731913.cup.hp.com (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id IAA27423; Thu, 29 Mar 2001 08:05:15 -0800 (PST) Message-Id: <5.0.2.1.2.20010329080014.02ec6340@esalpha2.cup.hp.com> X-Sender: krause@esalpha2.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Thu, 29 Mar 2001 08:01:17 -0800 To: Matt Wakeley From: Michael Krause Subject: Re: Frame Formats Cc: IPS Reflector In-Reply-To: <3AC34E1E.DFBD86AF@agilent.com> References: <3AC34664.97B39166@cisco.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 07:00 AM 3/29/2001 -0800, Matt Wakeley wrote: >I would rather the spec be silent on the value of "pad" bytes. The only thing >the spec should say is whether or not the pad bytes are included in the data >digest. In general, pad to zero, ignore on receive is the standard and one includes the pad in the digest to simplify the hardware (minor). Mike From owner-ips@ece.cmu.edu Thu Mar 29 15:58:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA10398 for ; Thu, 29 Mar 2001 15:58:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TIu8d28296 for ips-outgoing; Thu, 29 Mar 2001 13:56:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dirty.research.bell-labs.com (dirty.research.bell-labs.com [204.178.16.6]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2TIu3r28287 for ; Thu, 29 Mar 2001 13:56:03 -0500 (EST) Received: from scummy.research.bell-labs.com ([135.104.2.10]) by dirty; Thu Mar 29 13:55:07 EST 2001 Received: from aura.research.bell-labs.com ([135.104.46.10]) by scummy; Thu Mar 29 13:55:06 EST 2001 Received: from research.bell-labs.com (IDENT:sandeepj@sandeepj-pcmh.research.bell-labs.com [135.104.47.90]) by aura.research.bell-labs.com (8.9.1/8.9.1) with ESMTP id NAA04641 for ; Thu, 29 Mar 2001 13:55:06 -0500 (EST) Message-ID: <3AC3850A.56D628D1@research.bell-labs.com> Date: Thu, 29 Mar 2001 13:55:06 -0500 From: Sandeep Joshi X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.2.16-3 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: SNMP traps References: <3ABEB1D0.B97B6986@research.bell-labs.com> <3AC1FBEB.4A8158E@cisco.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mark, Login & Authentication failure traps came to mind since they would help detect DOS attacks....and prevent them by flooding the network with SNMP messages ;-) Besides what Tom suggested (Abnormal session terminations) and LU list changes), we could also add PortalUp, PortalDown, NetworkEntityReset, etc. We can add the mechanisms for now and determine the policies later. You may also want to raise this question with the T10 folks at the Interim meeting, if part of this is becoming a SCSI MIB. regards, -Sandeep Mark Bakke wrote: > > Sandeep- > > We have been looking for requirements for traps, and have not > addressed them yet. You are the first to really ask for them. > > Trapping the login and authentication failures seems like a > good idea; do you have any other ideas or requirements? > > Thanks, > > Mark > > Sandeep Joshi wrote: > > > > Mark, > > > > There seem to be no traps/notifications defined in the > > latest MIB doc. > > > > I do see that "notification-group" has been imported but > > not "notification-type" so perhaps there were discussions > > on adding something later ? > > > > A trap for login/auth failures atleast would be nice. > > Failure statistics are already being kept in the MIB. > > > > regards, > > -Sandeep > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 From owner-ips@ece.cmu.edu Thu Mar 29 16:00:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA10574 for ; Thu, 29 Mar 2001 16:00:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TJ69d29013 for ips-outgoing; Thu, 29 Mar 2001 14:06:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TJ5kr28949 for ; Thu, 29 Mar 2001 14:05:46 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id OAA25660; Thu, 29 Mar 2001 14:05:37 -0500 (EST) Message-ID: <3AC3875D.BFBEF585@cisco.com> Date: Thu, 29 Mar 2001 13:05:01 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Thomas McSweeney CC: ips@ece.cmu.edu Subject: Re: SNMP traps References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Tom- These are good suggestions. I like the idea of trapping only the abnormal terminations, since generating large numbers of traps is not usually a good thing. -- Mark Thomas McSweeney wrote: > > Mark, > > I'm new to the iSCSI MIB effort, but I'll toss out some suggestions... > > I think we should trap iSCSI Session terminations, except those > accomplished via Logout. I know this might result in traps for "normal" > events, like when someone powers off their initiator instead of shutting > down gracefully. If we want to get fancy, we could send them > conditionally, e.g., only trap if there were any incomplete commands or > undelivered responses at the time the session terminated. > > Trapping iSCSI connection terminations would be interesting too, especially > to implementations which allow multiple connections per session. Loss of a > connection could mean loss of capacity or reduced reliability of a session, > which could be reflected on the management station display. This may > already be covered by the TCP MIB (I haven't checked yet), but coordinating > a TCP trap with iSCSI objects would be more difficult than having an iSCSI > trap. > > It would also be helpful to trap changes to a target's LU list, and/or LUN > mappings. Minimally, a simple trap to notify the manager that a change > occurred would be enough to prompt the management station to do the GETs to > refresh its tables. > > Tom McSweeney > iSCSI Development, IBM Storage Networking > Email: rf42tpme@us.ibm.com > Phone: (USA) 919-254-5634 (tie line: 444-5634) > Fax: (USA) 919-254-0391 (tie line: 444-0391) > > Mark Bakke @ece.cmu.edu on 03/28/2001 09:57:47 AM > > Sent by: owner-ips@ece.cmu.edu > > To: Sandeep Joshi > cc: ips@ece.cmu.edu > Subject: Re: SNMP traps > > Sandeep- > > We have been looking for requirements for traps, and have not > addressed them yet. You are the first to really ask for them. > > Trapping the login and authentication failures seems like a > good idea; do you have any other ideas or requirements? > > Thanks, > > Mark -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Thu Mar 29 16:51:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA12980 for ; Thu, 29 Mar 2001 16:51:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TJxB402845 for ips-outgoing; Thu, 29 Mar 2001 14:59:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TJwUr02814 for ; Thu, 29 Mar 2001 14:58:31 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 2B3D4338 for ; Thu, 29 Mar 2001 12:58:30 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id EA5F6174 for ; Thu, 29 Mar 2001 14:58:28 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id LAA21475 for ; Thu, 29 Mar 2001 11:58:27 -0800 (PST) Message-ID: <3AC39405.FEAF8689@agilent.com> Date: Thu, 29 Mar 2001 11:59:01 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: synch and steering comments References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Doug, You don't have to play your broken record about modifying TCP. What I stated is correct. Just because it could require a "change" to TCP is not relevant. -Matt Douglas Otis wrote: > > Matt, > > If you intend to use TCP, there is no method yet to reliably obtain this > type of header alignment. Unless you are to modify TCP to become a record > based protocol, this type of behavior is a modification of TCP and beyond > the IPS charter. If you wish this type of behavior, you should consider > RFC-2960 instead. > > Doug > > > > -I am somewhat confused about the following statement in > > section 1.2.8.2: > > > "The Synch and Steering Layer is required to add to every sent data > > > item (IP packet, TCP packet or some other superstructure) enough > > > information to enable the receiver to steer it to a memory location > > > independent of any other piece. " > > > > > > Clearly from the way I understood the markers in Appendix.C, it doesn't > > > comply with this requirement. A more generic statement would be: > > > "The Synch and Steering Layer is required to add adequate > > > information to the data stream to enable the receiver to quickly > > > steer the stream to its final memory location, even in the face of > > > discontiguities in the stream. " > > > > > > +++ Markers are but one example and have only the Synch component. The > > > statement refers to a full steering (RDMA) scheme +++ > > > > Julian, > > > > No "full steering (RDMA) scheme" is required. A Sync mechanism that that > > implements framing such that the first item in each frame is an > > iSCSI header > > is all that is required to enable steering because the iSCSI > > header contains > > all the required steering information (task tag, data offset). > > > > -Matt > > From owner-ips@ece.cmu.edu Thu Mar 29 16:53:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA13068 for ; Thu, 29 Mar 2001 16:53:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TJn9602118 for ips-outgoing; Thu, 29 Mar 2001 14:49:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TJmXr02094 for ; Thu, 29 Mar 2001 14:48:33 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f2TKvD082029; Thu, 29 Mar 2001 12:57:14 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Matt Wakeley" , Subject: RE: iSCSI: synch and steering comments Date: Thu, 29 Mar 2001 11:47:02 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <3AC34D9B.9A0B260C@agilent.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt, If you intend to use TCP, there is no method yet to reliably obtain this type of header alignment. Unless you are to modify TCP to become a record based protocol, this type of behavior is a modification of TCP and beyond the IPS charter. If you wish this type of behavior, you should consider RFC-2960 instead. Doug > > -I am somewhat confused about the following statement in > section 1.2.8.2: > > "The Synch and Steering Layer is required to add to every sent data > > item (IP packet, TCP packet or some other superstructure) enough > > information to enable the receiver to steer it to a memory location > > independent of any other piece. " > > > > Clearly from the way I understood the markers in Appendix.C, it doesn't > > comply with this requirement. A more generic statement would be: > > "The Synch and Steering Layer is required to add adequate > > information to the data stream to enable the receiver to quickly > > steer the stream to its final memory location, even in the face of > > discontiguities in the stream. " > > > > +++ Markers are but one example and have only the Synch component. The > > statement refers to a full steering (RDMA) scheme +++ > > Julian, > > No "full steering (RDMA) scheme" is required. A Sync mechanism that that > implements framing such that the first item in each frame is an > iSCSI header > is all that is required to enable steering because the iSCSI > header contains > all the required steering information (task tag, data offset). > > -Matt > From owner-ips@ece.cmu.edu Thu Mar 29 17:52:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA15855 for ; Thu, 29 Mar 2001 17:52:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TKpCX06513 for ips-outgoing; Thu, 29 Mar 2001 15:51:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TKoBr06403 for ; Thu, 29 Mar 2001 15:50:11 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA35320 for ; Thu, 29 Mar 2001 15:42:51 -0500 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id NAA129594 for ; Thu, 29 Mar 2001 13:50:09 -0700 Importance: Normal Subject: iSCSI: value of Session ID To: ips@ece.cmu.edu From: "Jim Hafner" Date: Thu, 29 Mar 2001 12:50:08 -0800 Message-ID: X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/29/2001 12:50:09 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, I know this is bordering on heresy, but I'm trying to understand the value of the Session ID (ISID+TSID) in the iSCSI protocol. My understanding is that this was an attempt to mimic the S_ID and D_ID components of the FCP payloads. Consequently, the ISID would mean the SCSI "initiator identifier" and the TSID would be the "target identifier". Additionally, the SessionID would then uniquely define the SAM-2 I_T nexus object from the perspective of the SCSI domain. The problem I see is a question of scoping. In FCP, the S_ID and D_ID are imposed from the outside (by the network configuration) on the initiator and target. This also means that they are both externally visible to parties other than the initiator and target AND each are unique within a fabric (SCSI domain). As such they suffice both in the context of "initiator and target identifiers" AND for the purpose of identifying a nexus (internal within a target and, as a bonus, externally in the domain itself). In iSCSI, the identifiers are generated internal to the initiator and target. Consequently, they only have meaning in the context of the internal relationship between the initiator and target. They do not provide either external visibility or uniqueness within the "SCSI domain". Additionally, as currently defined, they are not even necessarily unique internal to a target device. As a result, I'm having trouble understanding what they're to be used for. Clearly, between the iSCSI layer and the SCSI layer within a target device (and implicitly to the device server within a logical unit in that target device), there must be some unique "handle" that gets associated to the I_T nexus in order to distinguish associations like "where does the device server return data for a command as it hands the data context down through the layers", who sent the command (for the purposes of checking reservations), etc. In effect the target at least needs an internally unique pointer for the session context that it can pass around between layers. For half the problem (internal uniqueness), one possible solution is to require that the target generate it's TSID in such a way that the resulting SessionID is unique within the target. This seems artificial as it effectively defines the internal algorithm that the target uses for it's pointers. The alternative is to just leave this "pointer" to the implementation -- but then what value is the SessionID? Your collective guidance is welcome! Jim Hafner From owner-ips@ece.cmu.edu Thu Mar 29 19:43:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA21455 for ; Thu, 29 Mar 2001 19:43:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TMHD612535 for ips-outgoing; Thu, 29 Mar 2001 17:17:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TMGMr12462 for ; Thu, 29 Mar 2001 17:16:23 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel3.hp.com (Postfix) with ESMTP id D2A1FF5 for ; Thu, 29 Mar 2001 14:16:21 -0800 (PST) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id OAA11948 for ips@ece.cmu.edu; Thu, 29 Mar 2001 14:17:21 -0800 (PST) Message-Id: <200103292217.OAA11948@core.rose.hp.com> Subject: Re: iSCSI: synch and steering comments To: ips@ece.cmu.edu Date: Thu, 29 Mar 2001 14:17:21 PST In-Reply-To: <3AC34D9B.9A0B260C@agilent.com>; from "Matt Wakeley" at Mar 29, 101 8:00 am Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk >> -I am somewhat confused about the following statement in section 1.2.8.2: >> "The Synch and Steering Layer is required to add to every sent data >> item (IP packet, TCP packet or some other superstructure) enough >> information to enable the receiver to steer it to a memory location >> independent of any other piece. " >> >> Clearly from the way I understood the markers in Appendix.C, it doesn't >> comply with this requirement. A more generic statement would be: >> "The Synch and Steering Layer is required to add adequate >> information to the data stream to enable the receiver to quickly >> steer the stream to its final memory location, even in the face of >> discontiguities in the stream. " >> >> +++ Markers are but one example and have only the Synch component. The >> statement refers to a full steering (RDMA) scheme +++ > >Julian, > >No "full steering (RDMA) scheme" is required. A Sync mechanism that that >implements framing such that the first item in each frame is an iSCSI header >is all that is required to enable steering because the iSCSI header contains >all the required steering information (task tag, data offset). Exactly my thinking. Matt is right on. I think the current statement in the draft I quoted above should be changed to be more generic to include markers (suggested sentence above), since I don't see any reason to exclude markers as a type of synch and steering layer. In fact, Appendix C is (appropriately) titled "Synch and Steering with Fixed Interval Markers". > >-Matt -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Thu Mar 29 19:45:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA21514 for ; Thu, 29 Mar 2001 19:45:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2TMsIv15130 for ips-outgoing; Thu, 29 Mar 2001 17:54:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2TMrPr15056 for ; Thu, 29 Mar 2001 17:53:25 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel1.hp.com (Postfix) with ESMTP id E42C54AB for ; Thu, 29 Mar 2001 17:53:20 -0500 (EST) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id OAA19924 for ips@ece.cmu.edu; Thu, 29 Mar 2001 14:54:20 -0800 (PST) Message-Id: <200103292254.OAA19924@core.rose.hp.com> Subject: Re: iSCSI: synch and steering comments To: ips@ece.cmu.edu Date: Thu, 29 Mar 2001 14:54:20 PST In-Reply-To: ; from "julian_satran@il.ibm.com" at Mar 28, 101 12:35 (noon) Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, Some comments. >Answers in text. Thanks, Julo > > .. >-Suggest adding the following statement to section 1.2.8.2. > > All conventional, in-order data arrival notifications generated by TCP > are passed through to iSCSI by the Synch and Steering layer after > appropriate data placements while none of the out-of-order data placements > that it performs are communicated to upper layers. > >+++ I have added the following to 1.2.8.2 > > On the incoming path the Synch and Steering layer does not change the > way TCP notifies iSCSI about in-order data arrival. All out-of-order > data placements > performed by the Synch and Steering layer are hidden from iSCSI. Okay, I'd however prefer it to imply that in-order data placement is also handled by Synch and Steering in the same sentence, instead of only commenting on in-order notifications, and out-of-order placements. > > I have aloso changed a bit the figure to convey better the fact that TCP > and Synch&Steering are related (not strictly layered +++ That's a good idea. > > ++++ > >-Section 1.2.8.2 states that a Synch and Steering layer is optional. > It has to be qualifed that it is optional only for those iSCSI devices > which perform connection recovery on header digest errors, since that's > how they cope with loss of framing. (I guess this may change in next rev?) > >+++ with the new format I think that we have: > >- one more chance if we go for format 1 or >- drop the connection on header error > >In both cases we can leave synch and steering optional Well, that doesn't address the thrust of my comment. I was implying that the draft should make it clear that those implementations which don't support Synch and Steering should end the connection on a header digest error and/or parity error, and not go into (what Somesh called) a speculative mode. > >+++ > >-It appears to me that at least one Synch and Steering layer must be > defined/referred to as the minimal implementation in the main draft to > enable interoperability, when implementations do implement Synch and >Steering. > >+++ why ? +++ I may be using "interoperability" in a somewhat unconventional sense here. While the draft says that Synch and Steering layer is optional, I don't see that it requires implementations to always support a "no synch & steering" mode, even when they support one type of Synch and Steering layer. Given that there's no mandatory Synch and Steering layer either, I don't see how two iSCSI boxes that a customer buys are guaranteed to interoperate. Please comment if the draft already implies what I am asking for. Thanks. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Fri Mar 30 02:31:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA13468 for ; Fri, 30 Mar 2001 02:31:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2U5WIW09702 for ips-outgoing; Fri, 30 Mar 2001 00:32:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2U5VHr09638 for ; Fri, 30 Mar 2001 00:31:17 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA169510 for ; Fri, 30 Mar 2001 07:31:12 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id HAA46230 for ; Fri, 30 Mar 2001 07:28:21 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1F.001E1145 ; Fri, 30 Mar 2001 07:28:25 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 30 Mar 2001 07:31:39 +0200 Subject: Re: Frame Formats Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Thanks Glen - the online copy is at: http://www.haifa.il.ibm.com/satran/ips Julo Glen Turner on 30/03/2001 06:35:14 Please respond to Glen Turner To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Re: Frame Formats julian_satran@il.ibm.com wrote: > > I have no idea. I guess that they think random is easy. Julo Hi Julo, Ignore the "security people". I've worked on three crypto products and good crypto people know that random numbers are hard. Furthermore, poor pseudo-random numbers by definition leak machiine state. This gives you something to hang your hat on when seeking to defeat the crypto. Zero-fill any unused bytes. Don't leak further machine state by leaving them at pre-existing values. If a cryptographer needs a more random stream then they'll insert their own random header (as this is under the crypto device's control whereas a user-inserted header isn't) and compress the user data. Regards, Glen PS: Where is the online copy of the frame formats presented at the meeting? -- Glen Turner Network Engineer (08) 8303 3936 Australian Academic and Research Network glen.turner@aarnet.edu.au http://www.aarnet.edu.au/ -- The revolution will not be televised, it will be digitised From owner-ips@ece.cmu.edu Fri Mar 30 02:33:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA13732 for ; Fri, 30 Mar 2001 02:33:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2U5RHA09383 for ips-outgoing; Fri, 30 Mar 2001 00:27:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2U5QQr09323 for ; Fri, 30 Mar 2001 00:26:27 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA142364 for ; Fri, 30 Mar 2001 07:26:20 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id HAA46118 for ; Fri, 30 Mar 2001 07:23:28 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1F.001D9D80 ; Fri, 30 Mar 2001 07:23:28 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 30 Mar 2001 07:26:43 +0200 Subject: Re: iSCSI: value of Session ID Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, The 2 IDs where not a mimic of IP. Back in prehistory when we attempted and measured on a "connection-per-LUN" and later one-connection-target the IP addresses where enough to bind initiator and target. When we went for multiple connections we needed something to relate the connections and a mechanism in which each party provided its own piece (an id of its own internal structure) was convenient. The only role it plays now is binding connections into a session. Attaching meaning or external scope to the IDs is not what we had in mind but it is not unconceivable (if it has meaning and if it is not "yet another way" of doing something that we know how to do today). I thought it can be used to create multiple sessions between 2 endpoints but I am not confident that I understand all the implications and where the value proposition is. Julo "Jim Hafner" on 29/03/2001 22:50:08 Please respond to "Jim Hafner" To: ips@ece.cmu.edu cc: Subject: iSCSI: value of Session ID Folks, I know this is bordering on heresy, but I'm trying to understand the value of the Session ID (ISID+TSID) in the iSCSI protocol. My understanding is that this was an attempt to mimic the S_ID and D_ID components of the FCP payloads. Consequently, the ISID would mean the SCSI "initiator identifier" and the TSID would be the "target identifier". Additionally, the SessionID would then uniquely define the SAM-2 I_T nexus object from the perspective of the SCSI domain. The problem I see is a question of scoping. In FCP, the S_ID and D_ID are imposed from the outside (by the network configuration) on the initiator and target. This also means that they are both externally visible to parties other than the initiator and target AND each are unique within a fabric (SCSI domain). As such they suffice both in the context of "initiator and target identifiers" AND for the purpose of identifying a nexus (internal within a target and, as a bonus, externally in the domain itself). In iSCSI, the identifiers are generated internal to the initiator and target. Consequently, they only have meaning in the context of the internal relationship between the initiator and target. They do not provide either external visibility or uniqueness within the "SCSI domain". Additionally, as currently defined, they are not even necessarily unique internal to a target device. As a result, I'm having trouble understanding what they're to be used for. Clearly, between the iSCSI layer and the SCSI layer within a target device (and implicitly to the device server within a logical unit in that target device), there must be some unique "handle" that gets associated to the I_T nexus in order to distinguish associations like "where does the device server return data for a command as it hands the data context down through the layers", who sent the command (for the purposes of checking reservations), etc. In effect the target at least needs an internally unique pointer for the session context that it can pass around between layers. For half the problem (internal uniqueness), one possible solution is to require that the target generate it's TSID in such a way that the resulting SessionID is unique within the target. This seems artificial as it effectively defines the internal algorithm that the target uses for it's pointers. The alternative is to just leave this "pointer" to the implementation -- but then what value is the SessionID? Your collective guidance is welcome! Jim Hafner From owner-ips@ece.cmu.edu Fri Mar 30 05:25:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA03425 for ; Fri, 30 Mar 2001 05:25:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2U8FJm19743 for ips-outgoing; Fri, 30 Mar 2001 03:15:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2U8ERr19712 for ; Fri, 30 Mar 2001 03:14:27 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id KAA277956 for ; Fri, 30 Mar 2001 10:14:21 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id KAA39382 for ; Fri, 30 Mar 2001 10:12:38 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1F.002CFF82 ; Fri, 30 Mar 2001 10:11:29 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 30 Mar 2001 09:52:25 +0200 Subject: Re: iSCSI: synch and steering comments Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, I though the question was about the text. The text of the Synch and Steering reffers to both Synch and Steering mechanisms. Steering to memory when you are in Synch can be accomplished using protocol specific information or a General Purpose RDMA. When you are out of Synch a general purpose RDMA can do the placement while a synch can help you reach the next boundary. Julo Matt Wakeley on 29/03/2001 16:58:35 Please respond to Matt Wakeley To: ips@ece.cmu.edu cc: Subject: Re: iSCSI: synch and steering comments > -I am somewhat confused about the following statement in section 1.2.8.2: > "The Synch and Steering Layer is required to add to every sent data > item (IP packet, TCP packet or some other superstructure) enough > information to enable the receiver to steer it to a memory location > independent of any other piece. " > > Clearly from the way I understood the markers in Appendix.C, it doesn't > comply with this requirement. A more generic statement would be: > "The Synch and Steering Layer is required to add adequate > information to the data stream to enable the receiver to quickly > steer the stream to its final memory location, even in the face of > discontiguities in the stream. " > > +++ Markers are but one example and have only the Synch component. The > statement refers to a full steering (RDMA) scheme +++ Julian, No "full steering (RDMA) scheme" is required. A Sync mechanism that that implements framing such that the first item in each frame is an iSCSI header is all that is required to enable steering because the iSCSI header contains all the required steering information (task tag, data offset). -Matt From owner-ips@ece.cmu.edu Fri Mar 30 10:36:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA21232 for ; Fri, 30 Mar 2001 10:36:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2UCIOv16967 for ips-outgoing; Fri, 30 Mar 2001 07:18:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from bramg1.net.external.hp.com (bramg1.net.external.hp.com [192.6.126.73]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2UCHxr16955 for ; Fri, 30 Mar 2001 07:18:00 -0500 (EST) Received: from quasit.br.itc.hp.com (quasit.br.itc.hp.com [15.145.8.135]) by bramg1.net.external.hp.com (Postfix) with ESMTP id E6A52D0 for ; Fri, 30 Mar 2001 14:17:57 +0200 (METDST) Received: from fowey.BR.ITC.HP.COM (fowey.br.itc.hp.com [15.145.8.186]) by quasit.br.itc.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id NAA23950 for ; Fri, 30 Mar 2001 13:17:57 +0100 (BST) Received: from 15.145.8.186 by fowey.BR.ITC.HP.COM (InterScan E-Mail VirusWall NT); Fri, 30 Mar 2001 13:17:57 +0100 (GMT Daylight Time) Received: by fowey.br.itc.hp.com with Internet Mail Service (5.5.2650.21) id ; Fri, 30 Mar 2001 13:17:57 +0100 Message-ID: <0B9A57FF1D57D411B47500D0B73E5CC101E7A627@dickens.bri.hp.com> From: "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" To: ips@ece.cmu.edu Subject: Unsolicited Data Questions Date: Fri, 30 Mar 2001 13:17:53 +0100 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Julian Sorry if I'm covering old ground... Is it possible to use unsolicited data for the first burst and then request any remaining data using R2T? For example, if the target has a previously allocated buffer available (length defined by FirstBurstSize) for unsolicited data, then once the initiator has sent unsolicited data up to and including this amount then the remaining data (if any) can be requested using R2T once the target has the buffer space available. Section 1.2.5 pertains to this.. "An initiator may send unsolicited data either as immediate (up to the negotiated maximum PDU size - DataPDULength - disconnect-reconnect mode page) or in a separate PDU sequence (up to the negotiated limit - FirstBurstSize - disconnect-reconnect mode page). All subsequent data has to be solicited" However, Appendix D - 22 ImmediateData" appears to contradict this: "If ImmediateData is set to no and UseR2T is set to yes then the initiator MUST NOT send unsolicited data and the target MUST reject them with the corresponding response code." Also I have some comments from my colleagues: Page 15: Last paragraph is unclear from 2nd sentence. The hyphens look like minus signs. All a bit confusing. Page 16: There are several references to initial burst. Should these not be changed to first burst to be consistent with the terminology used subsequently in the spec especially since this it's the "FirstBurstSize" which defines how much data can be accepted. page 111: The spec says that you need to set immediateData to yes so that you can receive a first burst of immediate data, but then "only immediate data are accepted in the first burst". If you have immediate data set to no and R2T set to yes you can't accept any unsolicited data: Are you stating that immediate data and "normal" unsolicited data can not be mixed. 1) Section 2.13.1: Is the response to a NOP-In issues by the target, always a NOP-out? 2) Appendix D 29: KeyValueText: default value is outside of parameter range. Cheers Matthew Burbridge Hewlett Packard, Bristol Telnet: 312 7010 E-mail: matthewb@bri.hp.com From owner-ips@ece.cmu.edu Fri Mar 30 13:16:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA01438 for ; Fri, 30 Mar 2001 13:16:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2UFdTs29161 for ips-outgoing; Fri, 30 Mar 2001 10:39:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2UFcqr29102 for ; Fri, 30 Mar 2001 10:38:52 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA217164 for ; Fri, 30 Mar 2001 17:38:44 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA153738 for ; Fri, 30 Mar 2001 17:37:02 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A1F.0055ADC6 ; Fri, 30 Mar 2001 17:35:50 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 30 Mar 2001 17:39:04 +0200 Subject: Re: iSCSI: synch and steering comments Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun, It is clearly communicated in the paragraph above it - but fine I will add it here too. Julo "Mallikarjun C." on 30/03/2001 00:54:20 Please respond to cbm@rose.hp.com To: ips@ece.cmu.edu cc: Subject: Re: iSCSI: synch and steering comments Julian, Some comments. >Answers in text. Thanks, Julo > > .. >-Suggest adding the following statement to section 1.2.8.2. > > All conventional, in-order data arrival notifications generated by TCP > are passed through to iSCSI by the Synch and Steering layer after > appropriate data placements while none of the out-of-order data placements > that it performs are communicated to upper layers. > >+++ I have added the following to 1.2.8.2 > > On the incoming path the Synch and Steering layer does not change the > way TCP notifies iSCSI about in-order data arrival. All out-of-order > data placements > performed by the Synch and Steering layer are hidden from iSCSI. Okay, I'd however prefer it to imply that in-order data placement is also handled by Synch and Steering in the same sentence, instead of only commenting on in-order notifications, and out-of-order placements. > > I have aloso changed a bit the figure to convey better the fact that TCP > and Synch&Steering are related (not strictly layered +++ That's a good idea. > > ++++ > >-Section 1.2.8.2 states that a Synch and Steering layer is optional. > It has to be qualifed that it is optional only for those iSCSI devices > which perform connection recovery on header digest errors, since that's > how they cope with loss of framing. (I guess this may change in next rev?) > >+++ with the new format I think that we have: > >- one more chance if we go for format 1 or >- drop the connection on header error > >In both cases we can leave synch and steering optional Well, that doesn't address the thrust of my comment. I was implying that the draft should make it clear that those implementations which don't support Synch and Steering should end the connection on a header digest error and/or parity error, and not go into (what Somesh called) a speculative mode. > >+++ > >-It appears to me that at least one Synch and Steering layer must be > defined/referred to as the minimal implementation in the main draft to > enable interoperability, when implementations do implement Synch and >Steering. > >+++ why ? +++ I may be using "interoperability" in a somewhat unconventional sense here. While the draft says that Synch and Steering layer is optional, I don't see that it requires implementations to always support a "no synch & steering" mode, even when they support one type of Synch and Steering layer. Given that there's no mandatory Synch and Steering layer either, I don't see how two iSCSI boxes that a customer buys are guaranteed to interoperate. Please comment if the draft already implies what I am asking for. Thanks. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Fri Mar 30 14:19:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA06912 for ; Fri, 30 Mar 2001 14:19:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2UHbbS07497 for ips-outgoing; Fri, 30 Mar 2001 12:37:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2UHbJr07482 for ; Fri, 30 Mar 2001 12:37:19 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel2.hp.com (Postfix) with ESMTP id 6B5B26A2 for ; Fri, 30 Mar 2001 12:37:18 -0500 (EST) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id JAA14996 for ips@ece.cmu.edu; Fri, 30 Mar 2001 09:38:18 -0800 (PST) Message-Id: <200103301738.JAA14996@core.rose.hp.com> Subject: Re: Unsolicited Data Questions To: ips@ece.cmu.edu Date: Fri, 30 Mar 2001 9:38:18 PST In-Reply-To: <0B9A57FF1D57D411B47500D0B73E5CC101E7A627@dickens.bri.hp.com>; from "BURBRIDGE,MATTHEW" at Mar 30, 101 5:59 am Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, On the subject of unsolicited and immediate data: Perhaps adding a table like this might help understand the intent of the draft better. (InitialR2T = UseR2T, per earlier discussion) +-------------+---------------+----------------------------------------+ | InitialR2T | ImmediateData | Result (upto FirstBurstSize) | +-------------+---------------+----------------------------------------+ | no | no | Unsolicited data in separate PDUs only | +-------------+---------------+----------------------------------------+ | no | yes | Immediate & separate unsolicited data | +-------------+---------------+----------------------------------------+ | yes | no | Unsolicited data disallowed | +-------------+---------------+----------------------------------------+ | yes | yes | Immediate unsolicited data only | +-------------+---------------+----------------------------------------+ -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com >Hi Julian > >Sorry if I'm covering old ground... > >Is it possible to use unsolicited data for the first burst and then request >any remaining data using R2T? For example, if the target has a previously >allocated buffer available (length defined by FirstBurstSize) for >unsolicited data, then once the initiator has sent unsolicited data up to >and including this amount then the remaining data (if any) can be requested >using R2T once the target has the buffer space available. > >Section 1.2.5 pertains to this.. "An initiator may send unsolicited data >either as immediate (up to the negotiated maximum PDU size - DataPDULength >- disconnect-reconnect mode page) or in a separate PDU sequence (up to the >negotiated limit - FirstBurstSize - disconnect-reconnect mode page). All >subsequent data has to be solicited" > >However, Appendix D - 22 ImmediateData" appears to contradict this: "If >ImmediateData is set to no and UseR2T is set to yes then the initiator MUST >NOT send unsolicited data and the target MUST reject them with the >corresponding response code." > >Also I have some comments from my colleagues: > >Page 15: Last paragraph is unclear from 2nd sentence. The hyphens look like >minus signs. All a bit confusing. >Page 16: There are several references to initial burst. Should these not be >changed to first burst to be consistent with the terminology used >subsequently in the spec especially since this it's the "FirstBurstSize" >which defines how much data can be accepted. >page 111: The spec says that you need to set immediateData to yes so that >you can receive a first burst of immediate data, but then "only >immediate data are accepted in the first burst". If you have immediate data >set to no and R2T set to yes you can't accept any unsolicited data: Are you >stating that immediate data and "normal" unsolicited data can not be mixed. > >1) Section 2.13.1: Is the response to a NOP-In issues by the target, always >a NOP-out? > >2) Appendix D 29: KeyValueText: default value is outside of parameter range. > >Cheers > >Matthew Burbridge >Hewlett Packard, Bristol >Telnet: 312 7010 >E-mail: matthewb@bri.hp.com > > From owner-ips@ece.cmu.edu Fri Mar 30 14:21:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA07080 for ; Fri, 30 Mar 2001 14:21:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2UHXXj07187 for ips-outgoing; Fri, 30 Mar 2001 12:33:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2UHXNr07171 for ; Fri, 30 Mar 2001 12:33:23 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id MAA07493; Fri, 30 Mar 2001 12:33:15 -0500 (EST) Message-ID: <3AC4C335.95842096@cisco.com> Date: Fri, 30 Mar 2001 11:32:37 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: julian_satran@il.ibm.com CC: ips@ece.cmu.edu Subject: Re: frame formats References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Excellent. Which header digest positioning method will we choose? 1. Single header digest, after BHS and AHS 2. Two header digests, one for BHS, one for AHS 3. Single header digest for BHS; AHS is added to data digest Option 3 will not work well with iSCSI proxies and gateways that may change header information, but keep the data end-to-end. To me, that leaves option 1 and 2. So, which is easier, having a single header digest in a variable location, or having the potential for two header digests; one in a fixed location, and an optional one in a variable location? I don't believe that we will see AHS on most iSCSI PDUs, so is it OK to have a "slow path" for these? -- Mark julian_satran@il.ibm.com wrote: > > Dear colleagues, > > It look like Format-2 is selected by popular vote. > > Julo -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Fri Mar 30 21:35:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA29196 for ; Fri, 30 Mar 2001 21:35:56 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2V0Veh05345 for ips-outgoing; Fri, 30 Mar 2001 19:31:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp013.mail.yahoo.com (smtp013.mail.yahoo.com [216.136.173.57]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f2V0V9r05320 for ; Fri, 30 Mar 2001 19:31:09 -0500 (EST) Received: from sdsl-216-36-75-164.dsl.sjc.megapath.net (HELO somesh) (216.36.75.164) by smtp.mail.vip.sc5.yahoo.com with SMTP; 31 Mar 2001 00:31:06 -0000 X-Apparently-From: Reply-To: From: "Somesh Gupta" To: , Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport" Date: Fri, 30 Mar 2001 16:25:52 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Sorry to have been missing for a while. Hope you will appreciate my being back in action :-). It was a fairly clear consensus in Orlando that applications broke up their transfers into reasonably small chunks i.e. they did not have very long running transfers. Therefore the consensus was that a command level recovery mechanism was sufficient instead of an ack/sack for each data PDU. The SACK mechanism was a post Orlando invention. Without an ack mechanism (for every data PDU), the SACK mechanism just imposes additional burden on either end of the session, without really much benefit. The benefit of having SACK is of saving bandwidth in case the data part of the data PDU failed an integrity check (but passed TCP checksum). This is a rare enough case that as a percentage, the bandwidth loss from retransmitting all the data associated with a read or write command is very very small. In addition, it avoids the complexity of restarting something from the middle, as compared to from the begining. To me it seems that there is significant simplicity (from implementation, reliability and recovery process) from having smaller data transfer per command. I would really like to get rid of the SACK command. Somesh > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Wednesday, March 28, 2001 6:57 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport" > > > > > Mallikarjun, > > Last summer I thought that recovery within a connection should be left to > TCP. It is simple and could be made available through IPsec (if no new > option of any form can be added). > > Two things killed this: > > The requirement to have a data encapsulation that can pass through > application proxies (like a storage router) > The "NO WAY" message we got from IESG-Security on a CRC only IPSec > header > > > As for the ACK - I am very much in favor of it (it is a no brainer) and > implementations are in fact allowed to drop even unacked data. > > I am bound by the Orlando meeting decision to drop it. Except the regular > "oppose everything" crowd the two vocal opponents where Somesh Gupta and > Matt Wakeley. > > David may want or not to re-open the issue - I am not going to ask for it. > > Regards, > Julo > > "Mallikarjun C." on 28/03/2001 00:45:02 > > Please respond to cbm@rose.hp.com > > To: Black_David@emc.com > cc: Julian Satran/Haifa/IBM@IBMIL, cbm@rose.hp.com, someshg@yahoo.com, > steph@cs.uchicago.edu, John Hufferd/San Jose/IBM@IBMUS, > ldalleore@snapserver.com, venkat@rhapsodynetworks.com > Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport" > > > > > David and Julian, > > I appreciate both your views, and should I say that they're > along predicted lines :-) > > - David's right in saying that the situation is akin to FC's. > However, I would like to point out that FC is an unreliable > transport, and hence is forced to pick up a lot of the transport > baggage (at least in FCP-2, as I understand), in addition > to being a SCSI encapsulation layer. Unfortunately, even with > TCP being the "reliable" transport, iSCSI is going along the > same lines - ie. transport baggage + SCSI encapsulation. My > point is - if this is indeed a necessary evil, why don't we > complete iSCSI's transport functionality by data-ACKs? > > - If data SACK is introduced mostly to make up for TCP's shortcomings, > we're making its usage (and implementation) drastically less appealing > since the only way error recovery algorithms can *rely* on data SACK > is when replay is supported (or, "ReplaySupport=yes" in my proposal), > which is extremely expensive. IOW, we're defining data SACK in the > draft and not providing any incentives to implement and use it! > > - I submit that since iSCSI is being hailed as the ideal SCSI Transport > protocol in its definition so far (and I believe, rightly so - mandating > command ordering, bi-di support, SCSI CRN support to name a few > examples), > the perfectly SCSI-legal R/W interactions that break in other transports > *do not* have to break in iSCSI. > > - A last idea (may seem radical at this point) in regards to iSCSI > being a "full transport". This provides us an opportunity to "cast > off" the transport baggage in future when we truly move to a "reliable" > transport (perhaps TCP with CRCs/SCTP ?) - if we do a good job of > keeping the encapsulation stuff separate from the transport stuff. > (Julian, I heard from Randy that ideas similar to this were explored > in your Haifa meeting. And yes, he recalls they were given up since > TCP was supposed to be reliable and granularity of recovery was deemed > one I/O.) > > With that said, may I request David (with his co-chair hat on, :-)) > to add some binding comments/observations on this discussion? > > If we decide to leave data SACKs as unattractive to implement, the draft > should in the least add a statement like - "Note that satisfying all > possible data SACK requests for a task with an unacknowledged status > implies implementing the I/O replay buffer on the part of targets." > -- > Mallikarjun > > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > > > > >I think Julian's basically right -- I would point > >out that any case of write after read that breaks > >over iSCSI will also break over Fibre Channel. > >On FC, the scenario starts with a frame CRC failure > >on read data at the Initiator, so applications > >have to cope and typically do so by enforcing > >ordering at the app rather than using SCSI task > >ordering. > > > >While SCSI has clever tools like ACA and task > >ordering that appear to allow dependent operations > >to be sent to the target concurrently, in practice > >they don't work and/or aren't used (funny thing, > >those two reinforce each other ;-) ). Hence > >a minimal approach to them is in order: > >- Make sure the result will interoperate. > >- Make sure T10 doesn't ding us for leaving something > > completely out. > >- Don't specify anything not needed for the above. > > > >My 0.02, > >--David > > > >> -----Original Message----- > >> From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > >> Sent: Tuesday, March 27, 2001 9:23 AM > >> To: cbm@rose.hp.com > >> Cc: someshg@yahoo.com; steph@cs.uchicago.edu; hufferd@us.ibm.com; > >> cbm@rose.hp.com; ldalleore@snapserver.com; Venkat Rangan; > >> Black_David@emc.com > >> Subject: Re: iSCSI ERT: data SACK/replay buffer/"semi-transport" > >> > >> > >> > >> Mallikarjun, > >> > >> I commiserate with you at the lack of ack for data but the Orlando > meeting > >> stated - no. Recall that I kept the number only as a mechanism to > detect > >> missing packets. > >> > >> You can achieve the effect you want by keeping around data for a while > >> (you > >> determine how long and then discard). > >> > >> If a SACK comes and you can recover - fine. If not you either reaccess > the > >> media (if you know how) or reject > >> and let the initiator retry. > >> > >> You should not worry about R/W conflicts as programs bound to have such > >> conflicts either: > >> > >> 1)can live with them or > >> 2)protect themselves through some locks and rely on > "operation-end-status" > >> to keep results deterministic. > >> > >> Regards, > >> Julo > >> > >> > >> > >> "Mallikarjun C." on 27/03/2001 03:34:16 > >> > >> Please respond to cbm@rose.hp.com > >> > >> To: cbm@rose.hp.com, someshg@yahoo.com, steph@cs.uchicago.edu, Julian > >> Satran/Haifa/IBM@IBMIL, John Hufferd/San Jose/IBM@IBMUS > >> cc: Black_David@emc.com > >> Subject: iSCSI ERT: data SACK/replay buffer/"semi-transport" > >> > >> > >> > >> > >> Hi Error Recovery Team, > >> > >> iSCSI can discard PDUs because of digest errors and request > >> retransmissions using the iSCSI data SACK. To deal with such > >> an eventuality, targets that want to support data SACK have > >> the following options: > >> > >> (A) maintain a complete "replay" buffer for the entire I/O since > >> a SACK could come anytime before the status is ack'ed by the > >> initiator. [ simple, but extremely expensive in memory resources] > >> > >> (B) (re-introduce data-ACKs into the draft, and) implement data-ACKs. > >> Thus enables keeping only those I/O buffers that haven't been ack'ed > >> by the initiator. IOW, become a real full transport! [ everyone > disliked > >> it earlier...] > >> > >> (C) re-access the medium for data retransmission requests. Now there > >> are 3 sub-cases in this to handle the changed data on the medium in a > >> write-after-read scenario. (SEE NOTE.1 at the bottom on how it is > >> legal.) > >> (1) On seeing any write, stall till status is ack'ed for all the > >> previous reads (basically drain the pipe). [simple, but > incurs > >> an additional roundtrip delay for all writes]. > >> (2) A variation of the above, keep an eye only on the prior > >> overlapping reads. [more BW efficient, but complicated to > >> resolve the block dependencies in a stream of > reads followed > >> by writes] > >> (3) Document the caveat and leave it upto the applications > >> to avoid this case since this leads to data integrity > issues. > >> [pushing to apps since the transport can't get it right!] > >> > >> My first preference is (B), followed by (A), and I suggest we not go > >> to (C) at all with its inherent dangers. > >> > >> Doing (B) naturally completes the transport job that iSCSI has taken > >> on itself in view of TCP's claimed unreliable checksum. That is the > >> right thing to do architecturally instead of being a "semi-transport"! > >> > >> Comments? > >> -- > >> Mallikarjun > >> > >> > >> Mallikarjun Chadalapaka > >> Networked Storage Architecture > >> Network Storage Solutions Organization > >> MS 5668 Hewlett-Packard, Roseville. > >> cbm@rose.hp.com > >> > >> > __________________________________________________________________________ > >> Note.1: A Read followed by a Write (to the same blocks) is perfectly > legal > >> if SCSI sets the ORDERED task attribute on both the > commands AND > >> sets the NACA bit to one to indicate that Write shall be > executed > >> only if the Read did not fail (result in a Check Condition). > >> > >> In the current case, since Read completed just fine from SCSI's > >> point of view, SCSI is moving on to execute Write. Those read > >> buffers > >> had been freed up since iSCSI received an ACK at the TCP level, > >> and > >> since iSCSI has no other way to have the data ack'ed! > >> > >> > >> > >> > > > > > > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Sat Mar 31 02:35:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA07257 for ; Sat, 31 Mar 2001 02:35:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2V62gW23311 for ips-outgoing; Sat, 31 Mar 2001 01:02:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2V62cr23307 for ; Sat, 31 Mar 2001 01:02:38 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA96804 for ; Sat, 31 Mar 2001 08:02:31 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA42948 for ; Sat, 31 Mar 2001 08:00:47 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A20.0020EE06 ; Sat, 31 Mar 2001 07:59:40 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 31 Mar 2001 08:03:00 +0200 Subject: Re: Unsolicited Data Questions Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun, I'll see what I can do. BTW - the second line has also to specify - not in the same command. Regards, Julo "Mallikarjun C." on 30/03/2001 19:38:18 Please respond to cbm@rose.hp.com To: ips@ece.cmu.edu cc: Subject: Re: Unsolicited Data Questions Julian, On the subject of unsolicited and immediate data: Perhaps adding a table like this might help understand the intent of the draft better. (InitialR2T = UseR2T, per earlier discussion) +-------------+---------------+----------------------------------------+ | InitialR2T | ImmediateData | Result (upto FirstBurstSize) | +-------------+---------------+----------------------------------------+ | no | no | Unsolicited data in separate PDUs only | +-------------+---------------+----------------------------------------+ | no | yes | Immediate & separate unsolicited data | +-------------+---------------+----------------------------------------+ | yes | no | Unsolicited data disallowed | +-------------+---------------+----------------------------------------+ | yes | yes | Immediate unsolicited data only | +-------------+---------------+----------------------------------------+ -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com >Hi Julian > >Sorry if I'm covering old ground... > >Is it possible to use unsolicited data for the first burst and then request >any remaining data using R2T? For example, if the target has a previously >allocated buffer available (length defined by FirstBurstSize) for >unsolicited data, then once the initiator has sent unsolicited data up to >and including this amount then the remaining data (if any) can be requested >using R2T once the target has the buffer space available. > >Section 1.2.5 pertains to this.. "An initiator may send unsolicited data >either as immediate (up to the negotiated maximum PDU size - DataPDULength >- disconnect-reconnect mode page) or in a separate PDU sequence (up to the >negotiated limit - FirstBurstSize - disconnect-reconnect mode page). All >subsequent data has to be solicited" > >However, Appendix D - 22 ImmediateData" appears to contradict this: "If >ImmediateData is set to no and UseR2T is set to yes then the initiator MUST >NOT send unsolicited data and the target MUST reject them with the >corresponding response code." > >Also I have some comments from my colleagues: > >Page 15: Last paragraph is unclear from 2nd sentence. The hyphens look like >minus signs. All a bit confusing. >Page 16: There are several references to initial burst. Should these not be >changed to first burst to be consistent with the terminology used >subsequently in the spec especially since this it's the "FirstBurstSize" >which defines how much data can be accepted. >page 111: The spec says that you need to set immediateData to yes so that >you can receive a first burst of immediate data, but then "only >immediate data are accepted in the first burst". If you have immediate data >set to no and R2T set to yes you can't accept any unsolicited data: Are you >stating that immediate data and "normal" unsolicited data can not be mixed. > >1) Section 2.13.1: Is the response to a NOP-In issues by the target, always >a NOP-out? > >2) Appendix D 29: KeyValueText: default value is outside of parameter range. > >Cheers > >Matthew Burbridge >Hewlett Packard, Bristol >Telnet: 312 7010 >E-mail: matthewb@bri.hp.com > > From owner-ips@ece.cmu.edu Sat Mar 31 02:41:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA07331 for ; Sat, 31 Mar 2001 02:41:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2V5nf922584 for ips-outgoing; Sat, 31 Mar 2001 00:49:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2V5nQr22576 for ; Sat, 31 Mar 2001 00:49:27 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA30042 for ; Sat, 31 Mar 2001 07:49:20 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id HAA185866 for ; Sat, 31 Mar 2001 07:47:36 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A20.001FB91A ; Sat, 31 Mar 2001 07:46:30 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 31 Mar 2001 07:49:46 +0200 Subject: Re: frame formats-header digest Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk As in the foils - 1. Julo Mark Bakke on 30/03/2001 19:32:37 Please respond to Mark Bakke To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: frame formats Excellent. Which header digest positioning method will we choose? 1. Single header digest, after BHS and AHS 2. Two header digests, one for BHS, one for AHS 3. Single header digest for BHS; AHS is added to data digest Option 3 will not work well with iSCSI proxies and gateways that may change header information, but keep the data end-to-end. To me, that leaves option 1 and 2. So, which is easier, having a single header digest in a variable location, or having the potential for two header digests; one in a fixed location, and an optional one in a variable location? I don't believe that we will see AHS on most iSCSI PDUs, so is it OK to have a "slow path" for these? -- Mark julian_satran@il.ibm.com wrote: > > Dear colleagues, > > It look like Format-2 is selected by popular vote. > > Julo -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Sat Mar 31 05:12:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA18810 for ; Sat, 31 Mar 2001 05:12:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2V8pjl01992 for ips-outgoing; Sat, 31 Mar 2001 03:51:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2V8our01960 for ; Sat, 31 Mar 2001 03:50:56 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id KAA194464 for ; Sat, 31 Mar 2001 10:50:49 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id KAA69940 for ; Sat, 31 Mar 2001 10:49:06 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A20.0030570C ; Sat, 31 Mar 2001 10:48:00 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 31 Mar 2001 10:51:17 +0200 Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport" Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, As I stated earlier - the DataSN was created to detect missing data PDUs. SNACK is needed to recover missing StatusSN and missing dataSN is only a bonus if the target wants to support it. It is a trivial mechanism and I think it should stay. Julo "Somesh Gupta" on 31/03/2001 02:25:52 Please respond to someshg@yahoo.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport" Sorry to have been missing for a while. Hope you will appreciate my being back in action :-). It was a fairly clear consensus in Orlando that applications broke up their transfers into reasonably small chunks i.e. they did not have very long running transfers. Therefore the consensus was that a command level recovery mechanism was sufficient instead of an ack/sack for each data PDU. The SACK mechanism was a post Orlando invention. Without an ack mechanism (for every data PDU), the SACK mechanism just imposes additional burden on either end of the session, without really much benefit. The benefit of having SACK is of saving bandwidth in case the data part of the data PDU failed an integrity check (but passed TCP checksum). This is a rare enough case that as a percentage, the bandwidth loss from retransmitting all the data associated with a read or write command is very very small. In addition, it avoids the complexity of restarting something from the middle, as compared to from the begining. To me it seems that there is significant simplicity (from implementation, reliability and recovery process) from having smaller data transfer per command. I would really like to get rid of the SACK command. Somesh > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Wednesday, March 28, 2001 6:57 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport" > > > > > Mallikarjun, > > Last summer I thought that recovery within a connection should be left to > TCP. It is simple and could be made available through IPsec (if no new > option of any form can be added). > > Two things killed this: > > The requirement to have a data encapsulation that can pass through > application proxies (like a storage router) > The "NO WAY" message we got from IESG-Security on a CRC only IPSec > header > > > As for the ACK - I am very much in favor of it (it is a no brainer) and > implementations are in fact allowed to drop even unacked data. > > I am bound by the Orlando meeting decision to drop it. Except the regular > "oppose everything" crowd the two vocal opponents where Somesh Gupta and > Matt Wakeley. > > David may want or not to re-open the issue - I am not going to ask for it. > > Regards, > Julo > > "Mallikarjun C." on 28/03/2001 00:45:02 > > Please respond to cbm@rose.hp.com > > To: Black_David@emc.com > cc: Julian Satran/Haifa/IBM@IBMIL, cbm@rose.hp.com, someshg@yahoo.com, > steph@cs.uchicago.edu, John Hufferd/San Jose/IBM@IBMUS, > ldalleore@snapserver.com, venkat@rhapsodynetworks.com > Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport" > > > > > David and Julian, > > I appreciate both your views, and should I say that they're > along predicted lines :-) > > - David's right in saying that the situation is akin to FC's. > However, I would like to point out that FC is an unreliable > transport, and hence is forced to pick up a lot of the transport > baggage (at least in FCP-2, as I understand), in addition > to being a SCSI encapsulation layer. Unfortunately, even with > TCP being the "reliable" transport, iSCSI is going along the > same lines - ie. transport baggage + SCSI encapsulation. My > point is - if this is indeed a necessary evil, why don't we > complete iSCSI's transport functionality by data-ACKs? > > - If data SACK is introduced mostly to make up for TCP's shortcomings, > we're making its usage (and implementation) drastically less appealing > since the only way error recovery algorithms can *rely* on data SACK > is when replay is supported (or, "ReplaySupport=yes" in my proposal), > which is extremely expensive. IOW, we're defining data SACK in the > draft and not providing any incentives to implement and use it! > > - I submit that since iSCSI is being hailed as the ideal SCSI Transport > protocol in its definition so far (and I believe, rightly so - mandating > command ordering, bi-di support, SCSI CRN support to name a few > examples), > the perfectly SCSI-legal R/W interactions that break in other transports > *do not* have to break in iSCSI. > > - A last idea (may seem radical at this point) in regards to iSCSI > being a "full transport". This provides us an opportunity to "cast > off" the transport baggage in future when we truly move to a "reliable" > transport (perhaps TCP with CRCs/SCTP ?) - if we do a good job of > keeping the encapsulation stuff separate from the transport stuff. > (Julian, I heard from Randy that ideas similar to this were explored > in your Haifa meeting. And yes, he recalls they were given up since > TCP was supposed to be reliable and granularity of recovery was deemed > one I/O.) > > With that said, may I request David (with his co-chair hat on, :-)) > to add some binding comments/observations on this discussion? > > If we decide to leave data SACKs as unattractive to implement, the draft > should in the least add a statement like - "Note that satisfying all > possible data SACK requests for a task with an unacknowledged status > implies implementing the I/O replay buffer on the part of targets." > -- > Mallikarjun > > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > > > > >I think Julian's basically right -- I would point > >out that any case of write after read that breaks > >over iSCSI will also break over Fibre Channel. > >On FC, the scenario starts with a frame CRC failure > >on read data at the Initiator, so applications > >have to cope and typically do so by enforcing > >ordering at the app rather than using SCSI task > >ordering. > > > >While SCSI has clever tools like ACA and task > >ordering that appear to allow dependent operations > >to be sent to the target concurrently, in practice > >they don't work and/or aren't used (funny thing, > >those two reinforce each other ;-) ). Hence > >a minimal approach to them is in order: > >- Make sure the result will interoperate. > >- Make sure T10 doesn't ding us for leaving something > > completely out. > >- Don't specify anything not needed for the above. > > > >My 0.02, > >--David > > > >> -----Original Message----- > >> From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > >> Sent: Tuesday, March 27, 2001 9:23 AM > >> To: cbm@rose.hp.com > >> Cc: someshg@yahoo.com; steph@cs.uchicago.edu; hufferd@us.ibm.com; > >> cbm@rose.hp.com; ldalleore@snapserver.com; Venkat Rangan; > >> Black_David@emc.com > >> Subject: Re: iSCSI ERT: data SACK/replay buffer/"semi-transport" > >> > >> > >> > >> Mallikarjun, > >> > >> I commiserate with you at the lack of ack for data but the Orlando > meeting > >> stated - no. Recall that I kept the number only as a mechanism to > detect > >> missing packets. > >> > >> You can achieve the effect you want by keeping around data for a while > >> (you > >> determine how long and then discard). > >> > >> If a SACK comes and you can recover - fine. If not you either reaccess > the > >> media (if you know how) or reject > >> and let the initiator retry. > >> > >> You should not worry about R/W conflicts as programs bound to have such > >> conflicts either: > >> > >> 1)can live with them or > >> 2)protect themselves through some locks and rely on > "operation-end-status" > >> to keep results deterministic. > >> > >> Regards, > >> Julo > >> > >> > >> > >> "Mallikarjun C." on 27/03/2001 03:34:16 > >> > >> Please respond to cbm@rose.hp.com > >> > >> To: cbm@rose.hp.com, someshg@yahoo.com, steph@cs.uchicago.edu, Julian > >> Satran/Haifa/IBM@IBMIL, John Hufferd/San Jose/IBM@IBMUS > >> cc: Black_David@emc.com > >> Subject: iSCSI ERT: data SACK/replay buffer/"semi-transport" > >> > >> > >> > >> > >> Hi Error Recovery Team, > >> > >> iSCSI can discard PDUs because of digest errors and request > >> retransmissions using the iSCSI data SACK. To deal with such > >> an eventuality, targets that want to support data SACK have > >> the following options: > >> > >> (A) maintain a complete "replay" buffer for the entire I/O since > >> a SACK could come anytime before the status is ack'ed by the > >> initiator. [ simple, but extremely expensive in memory resources] > >> > >> (B) (re-introduce data-ACKs into the draft, and) implement data-ACKs. > >> Thus enables keeping only those I/O buffers that haven't been ack'ed > >> by the initiator. IOW, become a real full transport! [ everyone > disliked > >> it earlier...] > >> > >> (C) re-access the medium for data retransmission requests. Now there > >> are 3 sub-cases in this to handle the changed data on the medium in a > >> write-after-read scenario. (SEE NOTE.1 at the bottom on how it is > >> legal.) > >> (1) On seeing any write, stall till status is ack'ed for all the > >> previous reads (basically drain the pipe). [simple, but > incurs > >> an additional roundtrip delay for all writes]. > >> (2) A variation of the above, keep an eye only on the prior > >> overlapping reads. [more BW efficient, but complicated to > >> resolve the block dependencies in a stream of > reads followed > >> by writes] > >> (3) Document the caveat and leave it upto the applications > >> to avoid this case since this leads to data integrity > issues. > >> [pushing to apps since the transport can't get it right!] > >> > >> My first preference is (B), followed by (A), and I suggest we not go > >> to (C) at all with its inherent dangers. > >> > >> Doing (B) naturally completes the transport job that iSCSI has taken > >> on itself in view of TCP's claimed unreliable checksum. That is the > >> right thing to do architecturally instead of being a "semi-transport"! > >> > >> Comments? > >> -- > >> Mallikarjun > >> > >> > >> Mallikarjun Chadalapaka > >> Networked Storage Architecture > >> Network Storage Solutions Organization > >> MS 5668 Hewlett-Packard, Roseville. > >> cbm@rose.hp.com > >> > >> > __________________________________________________________________________ > >> Note.1: A Read followed by a Write (to the same blocks) is perfectly > legal > >> if SCSI sets the ORDERED task attribute on both the > commands AND > >> sets the NACA bit to one to indicate that Write shall be > executed > >> only if the Read did not fail (result in a Check Condition). > >> > >> In the current case, since Read completed just fine from SCSI's > >> point of view, SCSI is moving on to execute Write. Those read > >> buffers > >> had been freed up since iSCSI received an ACK at the TCP level, > >> and > >> since iSCSI has no other way to have the data ack'ed! > >> > >> > >> > >> > > > > > > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Sat Mar 31 05:12:26 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA18821 for ; Sat, 31 Mar 2001 05:12:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2V8vjW02304 for ips-outgoing; Sat, 31 Mar 2001 03:57:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2V8vZr02297 for ; Sat, 31 Mar 2001 03:57:35 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 6DA02116 for ; Sat, 31 Mar 2001 01:57:34 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id A889520F for ; Sat, 31 Mar 2001 03:57:33 -0500 (EST) Received: from agilent.com (cos1nai128138.cos.agilent.com [141.184.128.138]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id AAA23456 for ; Sat, 31 Mar 2001 00:57:31 -0800 (PST) Message-ID: <3AC59BFB.3490C1C4@agilent.com> Date: Sat, 31 Mar 2001 00:57:31 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iscsi: technical comments to iSCSI 05: References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > > Matt, > > This tag story keeps coming up again and again. There are two camps: > > - camp 1 wants them to be the only thing that steers data > - camp 2 wants every LUN to allocate tags regardless of others and they > need another > element to differentiate the streams (to which you argued that they > should divide the space) > > I don't see why we should not carry the LUN. Well, then *require* that the target put the LUN in the R2T that it sends to the initiator so that the initiator does not have to store the LUN in it's data structures. > AS for the initial interval I wanted markers at regular intervals and just > saying after login is not a good reference. You negotiate during login the interval and when to start it. My point is that it's negotiated, and the spec should not specify the initial markerless interval. Can I assume you agree with all the comments you didn't say anything to? -Matt > > Julo > > Matt Wakeley on 24/03/2001 03:49:58 > > Please respond to Matt Wakeley > > To: IPS Reflector > cc: > Subject: Re: iscsi: technical comments to iSCSI 05: > > > > technical comments to iSCSI 05: > > > > > > Global: (especially in section 6) timeouts are not defined. We went > > > through a lot of work defining time-out values for FCP-2. I think > > > that time-out values need to be specified for interoperability. > > > > +++ probably. Please note also that we had ages ago some timers and > people > > did not find them usefull. Considering the variability in network > > diameters we will have to either have large values or set them based or > > RTT (complex) > > +++ > > The timers you had ages ago were overkill and per I/O to support gateways > to > unreliable networks. In 6.6, you need to specify the amount of time after > connection failure that a session is terminated. > > > > 1.2.7, page 19, middle of page > > > must "iSCSI://..." be caps? is "iscsi://..." allowed? > > > > > +++ the syntax is the same as for URNs -- see the NDT doc +++ > > How about a simple yes or no... > > > > 2.7.2, page 45: > > > Why must a LUN be associated with a Target Transfer Tag? > > > > > +++ I though we had this settled. The reason is to allow targets to set > > their Tags independent of each other and help the target steer the data > > without looking at Tags. > > That's the point of a tag - to help steer data. > Fibre Channel doesn't have a LUN field in the headers of data transferred, > why > does iSCSI require it? > > > if the device server is located with the LU. The alternative would be to > > declare the whole LU+Tag in R2T and Data Out as an extended tag and let > > every target use it as they wish - no mandatory LU correctness check +++ > > No - just use the target task tag and nothing else, just like fibre > channel. > > > > 2.17, page 66: > > > *if* as you state in 2.7.2 a LUN must be sent when a target > transfer > > > tag is specified, shouldn't the LUN be also specified in the R2T? > > > > > +++ see above - the target may have device servers that do not need to > know > > what they are (they may get this information from the command but they > > don't need it). On the other hand the initiator has always this > > information. +++ > > I restate that I don't think LUN should be used as a steering mechanism. > > > > 2.20, page 71: > > > remove "reserved fields not 0". This makes it sound like > > > reserved fields must be checked, which they should not must > > > be checked. > > > > +++ I did. But this raises an interesting side-question. It was said that > > checking for reserved fiels being 0 is a good practice. It enables easy > > migration from version to version +++ > > No it doesn't. If some future version makes use of a reserved field, it > makes > older versions puke on the new messages. > > > > 2.20, page 71: > > > add "data digest error" to the list of reject reasons. > > > > +++ it is there as cause 3. Is the name confusing? +++ > > Ok I see it - you should call it data digest error. > > > > 3.1.2 and 3.1.3, page 72: > > > I don't care what SCSI has defined for it's mode pages, for iSCSI, > > > these fields should be byte values, not multiples of 512. > > > > > +++ the current field size is 16 bit. Do we want to limit ourselves to > 64k > > PDUs? > > +++ > > See my comment below on the max PDU size. > > > > 4.3, page 78: > > > It may not be clear to everyone that a target can send a text or > > > login response to a login or text command. That is, a target could > > > respond to a text command with a login response. This should be > > > made more clear. > > You didn't reply to this. > > > > > > > 6.2, page 81, last paragraph: > > > Can a "restart" also be issued in this case? > > > > > > > +++ made it lowercase must -:)+++ > good. do you allow a restart? > > > > Appendix A: 01, page 94: > > > I don't think that crc-64 "MUST" be implemented. > > > I also don't agree with the phrase "Cyclic codes are particularly > > > well suited for hardware implementations." This is not true if > > > iSCSI PDUs span TCP segments and arrive out of order. > > > > > +++ fine on crc64 - on the rest if you do on the DMA to host it is still > > decent+++ > > My point was that I don't think that phrase should be in the document. It > adds no value. > > > > > > Appendix A: 01, page 94: > > > "Implementations MAY also negotiate digests with security > > > significance for data authentication and integrity as detailed > > > in the following table:" needs a lot more description. > > > > > +++ what for example? (copy the reference?) +++ > > Well for example, the "what's next" field doesn't seem to have any encoding > for a security digest. There is no format of the digest (what it contains) > and how it's validated. > > > > Appendix C: 07, page 105: > > > "The marker-less interval is not less than 4 kbytes and the > > > default is 4 kbytes." It should be whatever the the receiver > > > needs it to be (not min 4K). > > > > > +++ how should a send know? we don't want this to kick in before end of > > login +++ > > I agree - but if login only takes 1K, then the above statement says that I > can't have a marker for another 3k. It should simply say that the markers > are > disabled until after login. And then the interval will be whatever was > negotiated. > > > > Appendix C: 23 page 111: > > > This parameter should be in bytes, not multiples of 512, > > > especially since framing mechanisms may require that a PDU not be > > > larger than MSS. (perhaps -1 can be used to indicate MSS) > > > > > +++ the field length in the mode page is limited +++ > > The login parameter should be able to negotiate any value, including some > indication to use the current MSS. The value stored in the mode page can > be > an approximation of the real value. (This raises the question, why does > iSCSI > need mode pages when the values can be negotiated and discovered using the > key:value pairs?) > > The whole purpose of the WARP proposal is to be able to have a fully self > describing iSCSI PDU per TCP segment. Therefore, there must be a way to > specify that the max sized iSCSI PDU is the MSS less whatever size is > required > for the WARP header. > > -Matt Wakeley > Agilent Technologies From owner-ips@ece.cmu.edu Sat Mar 31 05:18:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA19286 for ; Sat, 31 Mar 2001 05:18:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2V8PiZ00647 for ips-outgoing; Sat, 31 Mar 2001 03:25:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2V8PWr00629 for ; Sat, 31 Mar 2001 03:25:32 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 1E4931480 for ; Sat, 31 Mar 2001 01:25:28 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 55EB01C6 for ; Sat, 31 Mar 2001 03:25:27 -0500 (EST) Received: from agilent.com (cos1nai128138.cos.agilent.com [141.184.128.138]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id AAA23367 for ; Sat, 31 Mar 2001 00:25:25 -0800 (PST) Message-ID: <3AC59475.3DD3ED8F@agilent.com> Date: Sat, 31 Mar 2001 00:25:25 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI: synch and steering comments References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, There were many comments in this message. To which comment are you refering to? -Matt julian_satran@il.ibm.com wrote: > > Mallikarjun, > > It is clearly communicated in the paragraph above it - but fine I will add > it here too. > > Julo > > "Mallikarjun C." on 30/03/2001 00:54:20 > > Please respond to cbm@rose.hp.com > > To: ips@ece.cmu.edu > cc: > Subject: Re: iSCSI: synch and steering comments > > Julian, > > Some comments. > > >Answers in text. Thanks, Julo > > > > > .. > > >-Suggest adding the following statement to section 1.2.8.2. > > > > All conventional, in-order data arrival notifications generated by TCP > > are passed through to iSCSI by the Synch and Steering layer after > > appropriate data placements while none of the out-of-order data > placements > > that it performs are communicated to upper layers. > > > >+++ I have added the following to 1.2.8.2 > > > > On the incoming path the Synch and Steering layer does not change the > > way TCP notifies iSCSI about in-order data arrival. All out-of-order > > data placements > > performed by the Synch and Steering layer are hidden from iSCSI. > > Okay, I'd however prefer it to imply that in-order data placement is also > handled by Synch and Steering in the same sentence, instead of only > commenting on in-order notifications, and out-of-order placements. > > > > > I have aloso changed a bit the figure to convey better the fact that > TCP > > and Synch&Steering are related (not strictly layered +++ > > That's a good idea. > > > > > ++++ > > > >-Section 1.2.8.2 states that a Synch and Steering layer is optional. > > It has to be qualifed that it is optional only for those iSCSI devices > > which perform connection recovery on header digest errors, since that's > > how they cope with loss of framing. (I guess this may change in next > rev?) > > > >+++ with the new format I think that we have: > > > >- one more chance if we go for format 1 or > >- drop the connection on header error > > > >In both cases we can leave synch and steering optional > > Well, that doesn't address the thrust of my comment. I was implying > that the draft should make it clear that those implementations which > don't support Synch and Steering should end the connection on a header > digest error and/or parity error, and not go into (what Somesh called) > a speculative mode. > > > > >+++ > > > >-It appears to me that at least one Synch and Steering layer must be > > defined/referred to as the minimal implementation in the main draft to > > enable interoperability, when implementations do implement Synch and > >Steering. > > > >+++ why ? +++ > > I may be using "interoperability" in a somewhat unconventional sense here. > While the draft says that Synch and Steering layer is optional, I don't see > that it requires implementations to always support a "no synch & steering" > mode, even when they support one type of Synch and Steering layer. Given > that > there's no mandatory Synch and Steering layer either, I don't see how two > iSCSI boxes that a customer buys are guaranteed to interoperate. Please > comment if the draft already implies what I am asking for. > > Thanks. > -- > Mallikarjun > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com From owner-ips@ece.cmu.edu Sat Mar 31 05:18:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA19315 for ; Sat, 31 Mar 2001 05:18:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2V8dip01378 for ips-outgoing; Sat, 31 Mar 2001 03:39:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2V8der01373 for ; Sat, 31 Mar 2001 03:39:40 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA65416 for ; Sat, 31 Mar 2001 10:39:33 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id KAA129980 for ; Sat, 31 Mar 2001 10:36:38 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A20.002F4F04 ; Sat, 31 Mar 2001 10:36:44 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 31 Mar 2001 10:40:06 +0200 Subject: Re: iSCSI: synch and steering comments Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I've marked it with --- Matt Wakeley on 31/03/2001 10:25:25 Please respond to Matt Wakeley To: IPS Reflector cc: Subject: Re: iSCSI: synch and steering comments Julian, There were many comments in this message. To which comment are you refering to? -Matt julian_satran@il.ibm.com wrote: > > Mallikarjun, > > It is clearly communicated in the paragraph above it - but fine I will add > it here too. > > Julo > > "Mallikarjun C." on 30/03/2001 00:54:20 > > Please respond to cbm@rose.hp.com > > To: ips@ece.cmu.edu > cc: > Subject: Re: iSCSI: synch and steering comments > > Julian, > > Some comments. > > >Answers in text. Thanks, Julo > > > > > .. > > >-Suggest adding the following statement to section 1.2.8.2. > > > > All conventional, in-order data arrival notifications generated by TCP > > are passed through to iSCSI by the Synch and Steering layer after > > appropriate data placements while none of the out-of-order data > placements > > that it performs are communicated to upper layers. > > > >+++ I have added the following to 1.2.8.2 > > > > On the incoming path the Synch and Steering layer does not change the > > way TCP notifies iSCSI about in-order data arrival. All out-of-order > > data placements > > performed by the Synch and Steering layer are hidden from iSCSI. ------------------------------------------------------------------------------- > > Okay, I'd however prefer it to imply that in-order data placement is also > handled by Synch and Steering in the same sentence, instead of only > commenting on in-order notifications, and out-of-order placements. > ------------------------------------------------------------------------------- > > > > I have aloso changed a bit the figure to convey better the fact that > TCP > > and Synch&Steering are related (not strictly layered +++ > > That's a good idea. > > > > > ++++ > > > >-Section 1.2.8.2 states that a Synch and Steering layer is optional. > > It has to be qualifed that it is optional only for those iSCSI devices > > which perform connection recovery on header digest errors, since that's > > how they cope with loss of framing. (I guess this may change in next > rev?) > > > >+++ with the new format I think that we have: > > > >- one more chance if we go for format 1 or > >- drop the connection on header error > > > >In both cases we can leave synch and steering optional > > Well, that doesn't address the thrust of my comment. I was implying > that the draft should make it clear that those implementations which > don't support Synch and Steering should end the connection on a header > digest error and/or parity error, and not go into (what Somesh called) > a speculative mode. > > > > >+++ > > > >-It appears to me that at least one Synch and Steering layer must be > > defined/referred to as the minimal implementation in the main draft to > > enable interoperability, when implementations do implement Synch and > >Steering. > > > >+++ why ? +++ > > I may be using "interoperability" in a somewhat unconventional sense here. > While the draft says that Synch and Steering layer is optional, I don't see > that it requires implementations to always support a "no synch & steering" > mode, even when they support one type of Synch and Steering layer. Given > that > there's no mandatory Synch and Steering layer either, I don't see how two > iSCSI boxes that a customer buys are guaranteed to interoperate. Please > comment if the draft already implies what I am asking for. > > Thanks. > -- > Mallikarjun > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com From owner-ips@ece.cmu.edu Sat Mar 31 05:42:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA21380 for ; Sat, 31 Mar 2001 05:42:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2V3vfs16708 for ips-outgoing; Fri, 30 Mar 2001 22:57:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from med.corp.rhapsodynetworks.com (64-160-62-201.rhapsodynetworks.com [64.160.62.201] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2V3udr16649 for ; Fri, 30 Mar 2001 22:56:39 -0500 (EST) Received: by med.corp.rhapsodynetworks.com with Internet Mail Service (5.5.2653.19) id ; Fri, 30 Mar 2001 19:56:34 -0800 Message-ID: <15851BD69CFCD41186B100B0D0AABE650C1B28@med.corp.rhapsodynetworks.com> From: Venkat Rangan To: "'someshg@yahoo.com'" , julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport" Date: Fri, 30 Mar 2001 19:56:33 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, Your point is valid for data-SACK. But there is also another benefit that SACK was supposed to provide - the ability to fill StatSN holes and acknowledge timely reception of status by the initiator. This leads to an obvious question: how does a StatSN hole at the initiator created in the first place? Since iSCSI Response PDUs are dominated by the iSCSI header, the most likely cause is a header digest failure of the iSCSI Response PDU, detected at the initiator, but which escapes TCP Checksum. I am not sure that the 'Sense Len' portion of a iSCSI Response is large enough to suffer from CRC-which-escapes-TCP-checksum error condition. Some would argue that if there is an iSCSI header digest failure which escapes TCP Checksum, the entire connection should be reset. If that is the case, header digest error can only be recovered at the session level and not at the connection level. There is another viewpoint that one could keep the connection and advance to the marker/synch point and recover; in this case, the StatSN SACK is useful in allowing the target to quickly send the missing status responses and release its resources associated with unacknowledged status responses. When recovering over to a new connection, you still may have holes in StatSN because you may have received on the old connection several well-formed Response PDUs after the one that had the digest failure. When connection-level recovery occurs, if the initiator throws these away, it will have to request target to resend all responses beyond the acknowledged StatSN. Having the SACK reduces to resending only those that had errors. Simple minded initiators and targets can choose to only do connection level or session level recovery. The Retry bit is then present only to keep the space of sequence numbers consistent in the wake of command recovery on new connection. Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com -----Original Message----- From: Somesh Gupta [mailto:someshg@yahoo.com] Sent: Friday, March 30, 2001 4:26 PM To: julian_satran@il.ibm.com; ips@ece.cmu.edu Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport" Sorry to have been missing for a while. Hope you will appreciate my being back in action :-). It was a fairly clear consensus in Orlando that applications broke up their transfers into reasonably small chunks i.e. they did not have very long running transfers. Therefore the consensus was that a command level recovery mechanism was sufficient instead of an ack/sack for each data PDU. The SACK mechanism was a post Orlando invention. Without an ack mechanism (for every data PDU), the SACK mechanism just imposes additional burden on either end of the session, without really much benefit. The benefit of having SACK is of saving bandwidth in case the data part of the data PDU failed an integrity check (but passed TCP checksum). This is a rare enough case that as a percentage, the bandwidth loss from retransmitting all the data associated with a read or write command is very very small. In addition, it avoids the complexity of restarting something from the middle, as compared to from the begining. To me it seems that there is significant simplicity (from implementation, reliability and recovery process) from having smaller data transfer per command. I would really like to get rid of the SACK command. Somesh > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Wednesday, March 28, 2001 6:57 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport" > > > > > Mallikarjun, > > Last summer I thought that recovery within a connection should be left to > TCP. It is simple and could be made available through IPsec (if no new > option of any form can be added). > > Two things killed this: > > The requirement to have a data encapsulation that can pass through > application proxies (like a storage router) > The "NO WAY" message we got from IESG-Security on a CRC only IPSec > header > > > As for the ACK - I am very much in favor of it (it is a no brainer) and > implementations are in fact allowed to drop even unacked data. > > I am bound by the Orlando meeting decision to drop it. Except the regular > "oppose everything" crowd the two vocal opponents where Somesh Gupta and > Matt Wakeley. > > David may want or not to re-open the issue - I am not going to ask for it. > > Regards, > Julo > > "Mallikarjun C." on 28/03/2001 00:45:02 > > Please respond to cbm@rose.hp.com > > To: Black_David@emc.com > cc: Julian Satran/Haifa/IBM@IBMIL, cbm@rose.hp.com, someshg@yahoo.com, > steph@cs.uchicago.edu, John Hufferd/San Jose/IBM@IBMUS, > ldalleore@snapserver.com, venkat@rhapsodynetworks.com > Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport" > > > > > David and Julian, > > I appreciate both your views, and should I say that they're > along predicted lines :-) > > - David's right in saying that the situation is akin to FC's. > However, I would like to point out that FC is an unreliable > transport, and hence is forced to pick up a lot of the transport > baggage (at least in FCP-2, as I understand), in addition > to being a SCSI encapsulation layer. Unfortunately, even with > TCP being the "reliable" transport, iSCSI is going along the > same lines - ie. transport baggage + SCSI encapsulation. My > point is - if this is indeed a necessary evil, why don't we > complete iSCSI's transport functionality by data-ACKs? > > - If data SACK is introduced mostly to make up for TCP's shortcomings, > we're making its usage (and implementation) drastically less appealing > since the only way error recovery algorithms can *rely* on data SACK > is when replay is supported (or, "ReplaySupport=yes" in my proposal), > which is extremely expensive. IOW, we're defining data SACK in the > draft and not providing any incentives to implement and use it! > > - I submit that since iSCSI is being hailed as the ideal SCSI Transport > protocol in its definition so far (and I believe, rightly so - mandating > command ordering, bi-di support, SCSI CRN support to name a few > examples), > the perfectly SCSI-legal R/W interactions that break in other transports > *do not* have to break in iSCSI. > > - A last idea (may seem radical at this point) in regards to iSCSI > being a "full transport". This provides us an opportunity to "cast > off" the transport baggage in future when we truly move to a "reliable" > transport (perhaps TCP with CRCs/SCTP ?) - if we do a good job of > keeping the encapsulation stuff separate from the transport stuff. > (Julian, I heard from Randy that ideas similar to this were explored > in your Haifa meeting. And yes, he recalls they were given up since > TCP was supposed to be reliable and granularity of recovery was deemed > one I/O.) > > With that said, may I request David (with his co-chair hat on, :-)) > to add some binding comments/observations on this discussion? > > If we decide to leave data SACKs as unattractive to implement, the draft > should in the least add a statement like - "Note that satisfying all > possible data SACK requests for a task with an unacknowledged status > implies implementing the I/O replay buffer on the part of targets." > -- > Mallikarjun > > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > > > > >I think Julian's basically right -- I would point > >out that any case of write after read that breaks > >over iSCSI will also break over Fibre Channel. > >On FC, the scenario starts with a frame CRC failure > >on read data at the Initiator, so applications > >have to cope and typically do so by enforcing > >ordering at the app rather than using SCSI task > >ordering. > > > >While SCSI has clever tools like ACA and task > >ordering that appear to allow dependent operations > >to be sent to the target concurrently, in practice > >they don't work and/or aren't used (funny thing, > >those two reinforce each other ;-) ). Hence > >a minimal approach to them is in order: > >- Make sure the result will interoperate. > >- Make sure T10 doesn't ding us for leaving something > > completely out. > >- Don't specify anything not needed for the above. > > > >My 0.02, > >--David > > > >> -----Original Message----- > >> From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > >> Sent: Tuesday, March 27, 2001 9:23 AM > >> To: cbm@rose.hp.com > >> Cc: someshg@yahoo.com; steph@cs.uchicago.edu; hufferd@us.ibm.com; > >> cbm@rose.hp.com; ldalleore@snapserver.com; Venkat Rangan; > >> Black_David@emc.com > >> Subject: Re: iSCSI ERT: data SACK/replay buffer/"semi-transport" > >> > >> > >> > >> Mallikarjun, > >> > >> I commiserate with you at the lack of ack for data but the Orlando > meeting > >> stated - no. Recall that I kept the number only as a mechanism to > detect > >> missing packets. > >> > >> You can achieve the effect you want by keeping around data for a while > >> (you > >> determine how long and then discard). > >> > >> If a SACK comes and you can recover - fine. If not you either reaccess > the > >> media (if you know how) or reject > >> and let the initiator retry. > >> > >> You should not worry about R/W conflicts as programs bound to have such > >> conflicts either: > >> > >> 1)can live with them or > >> 2)protect themselves through some locks and rely on > "operation-end-status" > >> to keep results deterministic. > >> > >> Regards, > >> Julo > >> > >> > >> > >> "Mallikarjun C." on 27/03/2001 03:34:16 > >> > >> Please respond to cbm@rose.hp.com > >> > >> To: cbm@rose.hp.com, someshg@yahoo.com, steph@cs.uchicago.edu, Julian > >> Satran/Haifa/IBM@IBMIL, John Hufferd/San Jose/IBM@IBMUS > >> cc: Black_David@emc.com > >> Subject: iSCSI ERT: data SACK/replay buffer/"semi-transport" > >> > >> > >> > >> > >> Hi Error Recovery Team, > >> > >> iSCSI can discard PDUs because of digest errors and request > >> retransmissions using the iSCSI data SACK. To deal with such > >> an eventuality, targets that want to support data SACK have > >> the following options: > >> > >> (A) maintain a complete "replay" buffer for the entire I/O since > >> a SACK could come anytime before the status is ack'ed by the > >> initiator. [ simple, but extremely expensive in memory resources] > >> > >> (B) (re-introduce data-ACKs into the draft, and) implement data-ACKs. > >> Thus enables keeping only those I/O buffers that haven't been ack'ed > >> by the initiator. IOW, become a real full transport! [ everyone > disliked > >> it earlier...] > >> > >> (C) re-access the medium for data retransmission requests. Now there > >> are 3 sub-cases in this to handle the changed data on the medium in a > >> write-after-read scenario. (SEE NOTE.1 at the bottom on how it is > >> legal.) > >> (1) On seeing any write, stall till status is ack'ed for all the > >> previous reads (basically drain the pipe). [simple, but > incurs > >> an additional roundtrip delay for all writes]. > >> (2) A variation of the above, keep an eye only on the prior > >> overlapping reads. [more BW efficient, but complicated to > >> resolve the block dependencies in a stream of > reads followed > >> by writes] > >> (3) Document the caveat and leave it upto the applications > >> to avoid this case since this leads to data integrity > issues. > >> [pushing to apps since the transport can't get it right!] > >> > >> My first preference is (B), followed by (A), and I suggest we not go > >> to (C) at all with its inherent dangers. > >> > >> Doing (B) naturally completes the transport job that iSCSI has taken > >> on itself in view of TCP's claimed unreliable checksum. That is the > >> right thing to do architecturally instead of being a "semi-transport"! > >> > >> Comments? > >> -- > >> Mallikarjun > >> > >> > >> Mallikarjun Chadalapaka > >> Networked Storage Architecture > >> Network Storage Solutions Organization > >> MS 5668 Hewlett-Packard, Roseville. > >> cbm@rose.hp.com > >> > >> > __________________________________________________________________________ > >> Note.1: A Read followed by a Write (to the same blocks) is perfectly > legal > >> if SCSI sets the ORDERED task attribute on both the > commands AND > >> sets the NACA bit to one to indicate that Write shall be > executed > >> only if the Read did not fail (result in a Check Condition). > >> > >> In the current case, since Read completed just fine from SCSI's > >> point of view, SCSI is moving on to execute Write. Those read > >> buffers > >> had been freed up since iSCSI received an ACK at the TCP level, > >> and > >> since iSCSI has no other way to have the data ack'ed! > >> > >> > >> > >> > > > > > > _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Sat Mar 31 06:15:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA24010 for ; Sat, 31 Mar 2001 06:15:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2V9GjL03288 for ips-outgoing; Sat, 31 Mar 2001 04:16:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2V9GAr03265 for ; Sat, 31 Mar 2001 04:16:10 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id LAA226432 for ; Sat, 31 Mar 2001 11:16:03 +0200 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA24374 for ; Sat, 31 Mar 2001 11:13:07 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256A20.0032A506 ; Sat, 31 Mar 2001 11:13:10 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 31 Mar 2001 11:16:30 +0200 Subject: Re: iscsi: technical comments to iSCSI 05: Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, We went over this too. There will be targets that do not have their LUN (as seen by the initiator) as the command go through middle boxes (that change LUNs). Is there any problem for the initiator to keep the LUN when it creates the control structure for the task? Julo Matt Wakeley on 31/03/2001 10:57:31 Please respond to Matt Wakeley To: IPS Reflector cc: Subject: Re: iscsi: technical comments to iSCSI 05: julian_satran@il.ibm.com wrote: > > Matt, > > This tag story keeps coming up again and again. There are two camps: > > - camp 1 wants them to be the only thing that steers data > - camp 2 wants every LUN to allocate tags regardless of others and they > need another > element to differentiate the streams (to which you argued that they > should divide the space) > > I don't see why we should not carry the LUN. Well, then *require* that the target put the LUN in the R2T that it sends to the initiator so that the initiator does not have to store the LUN in it's data structures. > AS for the initial interval I wanted markers at regular intervals and just > saying after login is not a good reference. You negotiate during login the interval and when to start it. My point is that it's negotiated, and the spec should not specify the initial markerless interval. Can I assume you agree with all the comments you didn't say anything to? -Matt > > Julo > > Matt Wakeley on 24/03/2001 03:49:58 > > Please respond to Matt Wakeley > > To: IPS Reflector > cc: > Subject: Re: iscsi: technical comments to iSCSI 05: > > > > technical comments to iSCSI 05: > > > > > > Global: (especially in section 6) timeouts are not defined. We went > > > through a lot of work defining time-out values for FCP-2. I think > > > that time-out values need to be specified for interoperability. > > > > +++ probably. Please note also that we had ages ago some timers and > people > > did not find them usefull. Considering the variability in network > > diameters we will have to either have large values or set them based or > > RTT (complex) > > +++ > > The timers you had ages ago were overkill and per I/O to support gateways > to > unreliable networks. In 6.6, you need to specify the amount of time after > connection failure that a session is terminated. > > > > 1.2.7, page 19, middle of page > > > must "iSCSI://..." be caps? is "iscsi://..." allowed? > > > > > +++ the syntax is the same as for URNs -- see the NDT doc +++ > > How about a simple yes or no... > > > > 2.7.2, page 45: > > > Why must a LUN be associated with a Target Transfer Tag? > > > > > +++ I though we had this settled. The reason is to allow targets to set > > their Tags independent of each other and help the target steer the data > > without looking at Tags. > > That's the point of a tag - to help steer data. > Fibre Channel doesn't have a LUN field in the headers of data transferred, > why > does iSCSI require it? > > > if the device server is located with the LU. The alternative would be to > > declare the whole LU+Tag in R2T and Data Out as an extended tag and let > > every target use it as they wish - no mandatory LU correctness check +++ > > No - just use the target task tag and nothing else, just like fibre > channel. > > > > 2.17, page 66: > > > *if* as you state in 2.7.2 a LUN must be sent when a target > transfer > > > tag is specified, shouldn't the LUN be also specified in the R2T? > > > > > +++ see above - the target may have device servers that do not need to > know > > what they are (they may get this information from the command but they > > don't need it). On the other hand the initiator has always this > > information. +++ > > I restate that I don't think LUN should be used as a steering mechanism. > > > > 2.20, page 71: > > > remove "reserved fields not 0". This makes it sound like > > > reserved fields must be checked, which they should not must > > > be checked. > > > > +++ I did. But this raises an interesting side-question. It was said that > > checking for reserved fiels being 0 is a good practice. It enables easy > > migration from version to version +++ > > No it doesn't. If some future version makes use of a reserved field, it > makes > older versions puke on the new messages. > > > > 2.20, page 71: > > > add "data digest error" to the list of reject reasons. > > > > +++ it is there as cause 3. Is the name confusing? +++ > > Ok I see it - you should call it data digest error. > > > > 3.1.2 and 3.1.3, page 72: > > > I don't care what SCSI has defined for it's mode pages, for iSCSI, > > > these fields should be byte values, not multiples of 512. > > > > > +++ the current field size is 16 bit. Do we want to limit ourselves to > 64k > > PDUs? > > +++ > > See my comment below on the max PDU size. > > > > 4.3, page 78: > > > It may not be clear to everyone that a target can send a text or > > > login response to a login or text command. That is, a target could > > > respond to a text command with a login response. This should be > > > made more clear. > > You didn't reply to this. > > > > > > > 6.2, page 81, last paragraph: > > > Can a "restart" also be issued in this case? > > > > > > > +++ made it lowercase must -:)+++ > good. do you allow a restart? > > > > Appendix A: 01, page 94: > > > I don't think that crc-64 "MUST" be implemented. > > > I also don't agree with the phrase "Cyclic codes are particularly > > > well suited for hardware implementations." This is not true if > > > iSCSI PDUs span TCP segments and arrive out of order. > > > > > +++ fine on crc64 - on the rest if you do on the DMA to host it is still > > decent+++ > > My point was that I don't think that phrase should be in the document. It > adds no value. > > > > > > Appendix A: 01, page 94: > > > "Implementations MAY also negotiate digests with security > > > significance for data authentication and integrity as detailed > > > in the following table:" needs a lot more description. > > > > > +++ what for example? (copy the reference?) +++ > > Well for example, the "what's next" field doesn't seem to have any encoding > for a security digest. There is no format of the digest (what it contains) > and how it's validated. > > > > Appendix C: 07, page 105: > > > "The marker-less interval is not less than 4 kbytes and the > > > default is 4 kbytes." It should be whatever the the receiver > > > needs it to be (not min 4K). > > > > > +++ how should a send know? we don't want this to kick in before end of > > login +++ > > I agree - but if login only takes 1K, then the above statement says that I > can't have a marker for another 3k. It should simply say that the markers > are > disabled until after login. And then the interval will be whatever was > negotiated. > > > > Appendix C: 23 page 111: > > > This parameter should be in bytes, not multiples of 512, > > > especially since framing mechanisms may require that a PDU not be > > > larger than MSS. (perhaps -1 can be used to indicate MSS) > > > > > +++ the field length in the mode page is limited +++ > > The login parameter should be able to negotiate any value, including some > indication to use the current MSS. The value stored in the mode page can > be > an approximation of the real value. (This raises the question, why does > iSCSI > need mode pages when the values can be negotiated and discovered using the > key:value pairs?) > > The whole purpose of the WARP proposal is to be able to have a fully self > describing iSCSI PDU per TCP segment. Therefore, there must be a way to > specify that the max sized iSCSI PDU is the MSS less whatever size is > required > for the WARP header. > > -Matt Wakeley > Agilent Technologies From owner-ips@ece.cmu.edu Sat Mar 31 13:57:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA20672 for ; Sat, 31 Mar 2001 13:57:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f2VGYtA06240 for ips-outgoing; Sat, 31 Mar 2001 11:34:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (mars.iol.unh.edu [132.177.121.222]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f2VGYPr06221 for ; Sat, 31 Mar 2001 11:34:25 -0500 (EST) Received: from localhost (rdr@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id LAA02038; Sat, 31 Mar 2001 11:39:37 -0500 (EST) Date: Sat, 31 Mar 2001 11:39:36 -0500 From: "Robert D. Russell" To: Mark Bakke cc: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: Re: frame formats In-Reply-To: <3AC4C335.95842096@cisco.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Mark: There is a potentially important distinction between the 2 choices that is missing from your summary when you indicate that both choices involve a variable length. In option 1, a single header digest after the BHS and AHS, you do not know when you start reading the header how long it will be and therefore you do not know where the digest is. This complicates the reading process, since it will have to either do the read in 2 steps (1st the BHS and then 2nd the AHS (if any) followed by the digest), or 1 read that interprets the data being read "on the fly" to extract the AHS length and extend the length of the read accordingly. In option 2 the first read is always for 48 bytes of header and you always know where the digest is. The second read is needed only if the AHS length field in the BHS is non-zero, and its length is determined by that AHS length field. However, when the 2nd read is started the length IS known and the position of the digest IS known -- you do NOT have the "on the fly" searching needed in option 1. This may (or may not) be a simplification. The other advantage to option 2 is that the input process never has to use unverified data (i.e., the AHS length field) to find the digest (and thus verify the data). Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 On Fri, 30 Mar 2001, Mark Bakke wrote: > > Excellent. Which header digest positioning method will we choose? > > 1. Single header digest, after BHS and AHS > > 2. Two header digests, one for BHS, one for AHS > > 3. Single header digest for BHS; AHS is added to data digest > > Option 3 will not work well with iSCSI proxies and gateways > that may change header information, but keep the data end-to-end. > > To me, that leaves option 1 and 2. So, which is easier, having > a single header digest in a variable location, or having the > potential for two header digests; one in a fixed location, and > an optional one in a variable location? > > I don't believe that we will see AHS on most iSCSI PDUs, so is > it OK to have a "slow path" for these? > > -- > Mark > > julian_satran@il.ibm.com wrote: > > > > Dear colleagues, > > > > It look like Format-2 is selected by popular vote. > > > > Julo > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 > From owner-ips@ece.cmu.edu Sat Mar 31 23:06:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA02163 for ; Sat, 31 Mar 2001 23:06:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f311L3f05626 for ips-outgoing; Sat, 31 Mar 2001 20:21:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f311Ker05558 for ; Sat, 31 Mar 2001 20:20:40 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id UAA88022 for ; Sat, 31 Mar 2001 20:13:18 -0500 Received: from f4n49e (d03nm065h.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id SAA80978 for ; Sat, 31 Mar 2001 18:20:37 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: frame formats To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Sat, 31 Mar 2001 17:20:17 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.6 |December 14, 2000) at 03/31/2001 06:20:37 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Well, perhaps I was just not quick enough. I thought I would let this settle out a bit before I added my two cents. If you all remember, some folks on this reflector gave Julian a hard time because you would have to use a length field that you were not sure was OK, if you had a digest error and wanted to jump forward to the next, etc. etc. etc. I am sure you all remember this. OK, now that Julian proposed a parity way to ensure that you could trust the length field, some of the parties, have now, I think, voted for format #2. Unless you want now to reconsider your vote, we should stop giving Julian a hard time about the length not being ensured correct in the presents of a Digest Error. Either drop the session, or use the length to see if you can get somewhere, search for the next marker etc. All the stuff you said you did not like before. OK, now you have format 2, but lets not go over that old ground now that you have decided against the parity. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 03/30/2001 08:51:50 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: frame formats Dear colleagues, It look like Format-2 is selected by popular vote. Julo