From owner-ips@ece.cmu.edu Tue Jan 2 18:11:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA23922 for ; Tue, 2 Jan 2001 18:11:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f02KKuH20290 for ips-outgoing; Tue, 2 Jan 2001 15:20:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f02KK4P20252 for ; Tue, 2 Jan 2001 15:20:05 -0500 (EST) Received: from IETF ([63.202.160.80]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with SMTP id <0G6J002K2SEA9N@mta5.snfc21.pbi.net> for ips@ece.cmu.edu; Tue, 2 Jan 2001 10:22:59 -0800 (PST) Date: Tue, 02 Jan 2001 10:23:53 -0800 From: Douglas Otis Subject: RE: iFCP: FC-BB exists, why invent something new? In-reply-to: To: Charles Monia , Y P Cheng , IPS Reflector Message-id: MIME-version: 1.0 X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Content-type: text/plain; charset="iso-8859-1" Content-transfer-encoding: 7bit Importance: Normal X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 X-Priority: 3 (Normal) Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Charles, Assurances are made with the fabric timeout but you do not offer a time-stamp to keep this promise. TCP will try well beyond a FC fabric timeout. A cable swap as example could exceed nominal delays. Would you see it possible to make a separate proposal to cover just encapsulation and another proposal to include iFCP specific link services? For those wishing just a tunnel, the encapsulation proposal, with a small (perhaps optional) iFCP specific field, could serve both purposes. Doug > Hi: > > See my remarks below. > > Charles > > -----Original Message----- > > From: Y P Cheng [mailto:ycheng@advansys.com] > > Sent: Friday, December 15, 2000 12:17 PM > > To: IPS Reflector > > Subject: RE: iFCP: FC-BB exists, why invent something new? > > > > > > > > restricted to FCP). What does iFCP bring to the table that > > > > FC-BB does not already provide? > > > > -Matt > > > > > > > iFCP allows routing storage traffic between individual > > > FC devices over an IP network. FC-BB only bridges SAN islands. > > > As you are aware, there is a very significant difference > > > between bridging and routing. > > > > > > The former (FC-BB) requires the routing to be performed by > > > FC switches. The latter (iFCP) has the routing performed by IP > > > switches. Since I work for Nishan, my opinion is the latter is > > > much superior, but I have to admit there may be situations where > > > a simple bridging/tunneling implementation will be sufficient to > > > address a particular requirement. BUT, I also believe > > > there will be situations, especially with increasing storage > > > networking demands, that the latter will be needed to achieve > > > the required scale of storage networking deployments that will > > > be needed in the near future. > > > Josh > > > > Taking my technical hat off, as a business person, may be I can say > > something that Josh did not say. Nishan is in business of > > routing. They > > wish to connect IP networks directly to FC nodes (N, NL, F, > > and E) that do > > not have FC-BB functions. They are in business to replace BSW ports > > Therefore, they propose iFCP, like a NAT device, to map the > > S-ID and D-ID > > into unique fibre channel addresses in other FC domains. > > > > On the other hand, by saying an FCIP device being a bridge > > and tunneling > > device in connecting to the E-port with FC-BB functions, Josh > > simply says > > that FCIP depends on BSW for routing which is not scalable to > > IP network. > > There are companies making FCIP devices. But, the FCIP > > device is inferior > > to iFCP in its capability of routing. Nishan certainly is > > not interested in > > FCIP devices. > > > > From an end user point's of view, should there be a winner > > between FCIP and > > iFCP? I believe they will coexist. As an HBA manufacture, I > > would like to > > be able to connect to either iFCP or FCIP device without > > having two sets of > > microcode. This is why I believe we should have a single > > specification for > > iFCP and FCIP. May be all we need is a common frame format > > between them. > > The N- or NL-nodes can care less about routing or tunneling. > > Just make sure > > FLOGI and name services for discovery staying transparent. > > > > Am I right in this assessment? > > > > In a word, no. With any storage gateway, compatibility at the > storage device > interface is orthogonal to the underlying storage network > implementation to > which the gateway provides access. > > In that regard, iFCP is predicated on the assumption that the > Fibre Channel > to iFCP gateway interface, whether it be N_PORT, E_PORT or whatever, is > fully compliant with all applicable FC standards. An FC adapter, storage > device, or FC switch connecting to such a device will require no change > whatsoever. > > Charles > From owner-ips@ece.cmu.edu Tue Jan 2 19:41:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA24449 for ; Tue, 2 Jan 2001 19:41:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f02MKvp23564 for ips-outgoing; Tue, 2 Jan 2001 17:20:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f02MKTP23553 for ; Tue, 2 Jan 2001 17:20:30 -0500 (EST) Received: from IETF ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f02NVH013087; Tue, 2 Jan 2001 15:31:17 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: , Cc: , Subject: RE: new iSCSI draft 02.txt Date: Tue, 2 Jan 2001 14:18:41 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, My compliments. This appears to be the first version that does not modify TCP. Did I miss something or are you still serious about implementing the IP suffix framing? Doug > The correct URL is : > > http://www.haifa.il.ibm.com/satran/ips/draft-ietf-ips-iSCSI-02.txt > > Sorry for the inconvenience. > > Julo > ---------------------- Forwarded by Julian Satran/Haifa/IBM on 31/12/2000 > 14:39 --------------------------- > > Julian Satran > 30/12/2000 19:05 > > To: internet-drafts@ietf.org > cc: bassoon@yogi.ece.cmu.edu > From: Julian Satran/Haifa/IBM@Haifa/IBM@IBMIL > Subject: new iSCSI draft 02.txt > > > > > On behalf of a group of authors from several organizations as part of the > IPS working group I submit a revision of an IETF-draft for immediate > publication. It specifies iSCSI - a SCSI Over TCP protocol and the file > name is "draft-ietf-ips-iSCSI-02.txt". It completely replaces > "draft-ietf-ips-iSCSI-01.txt". > > The draft can be found at: > > http://www.haifa.il.ibm.com/satran/draft-ietf-ips-iSCSI-02.txt > > Julian Satran - IBM Research Laboratory at Haifa > > > > > > From owner-ips@ece.cmu.edu Tue Jan 2 22:52:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA27300 for ; Tue, 2 Jan 2001 22:52:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f031W0h28445 for ips-outgoing; Tue, 2 Jan 2001 20:32:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f031VVP28428 for ; Tue, 2 Jan 2001 20:31:31 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Tue, 2 Jan 2001 17:41:09 -0800 Message-ID: From: Charles Monia To: "Ips (E-mail)" Subject: RE: iFCP:Timeout and Encapsulation (was FC-BB exists, why invent something new?) Date: Tue, 2 Jan 2001 17:31:06 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Tuesday, January 02, 2001 10:24 AM > To: Charles Monia; Y P Cheng; IPS Reflector > Subject: RE: iFCP: FC-BB exists, why invent something new? > > > Charles, > > Assurances are made with the fabric timeout but you do not offer a > time-stamp to keep this promise. TCP will try well beyond a FC fabric > timeout. A cable swap as example could exceed nominal delays. > We are looking at a number of ways to measure whether or not the underlying network is performing within the parameters of such a performance guarantee. We're also considering a capability that allows the system user to set the required timeouts and establish alarm thresholds and contingency policies. > Would you see it possible to make a separate proposal to cover just > encapsulation and another proposal to include iFCP specific > link services? > For those wishing just a tunnel, the encapsulation proposal, > with a small > (perhaps optional) iFCP specific field, could serve both purposes. > We would need to carefully consider the consequences of such a proposal. I'd be especially concerned about consequences down the road that might impact the ability to make changes to both protocols over time. Although I can't speak for them of course, I suspect the tunneling folks may have a similar view. Charles From owner-ips@ece.cmu.edu Wed Jan 3 12:26:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA29093 for ; Wed, 3 Jan 2001 12:26:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f03Ejud23817 for ips-outgoing; Wed, 3 Jan 2001 09:45:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f03EjN823796 for ; Wed, 3 Jan 2001 09:45:23 -0500 (EST) Received: from sponge.cisco.com (sponge.cisco.com [171.71.61.25]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id GAA01183 for ; Wed, 3 Jan 2001 06:45:25 -0800 (PST) Received: from DAPW2K (dhcp-161-44-68-178.cisco.com [161.44.68.178]) by sponge.cisco.com (Mirapoint) with SMTP id AEO00186; Wed, 3 Jan 2001 08:45:16 -0600 (CST) From: "David Peterson" To: "Ips@Ece. Cmu. Edu" Subject: RE: new iSCSI draft - 02.txt Date: Wed, 3 Jan 2001 08:47:07 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I believe there was agreement to remove the Urgent-Pointer framing mechanism but don't recall any agreement to replace it with an in-stream marker. For a software implementation it would be hard to support this type of framing mechanism. I believe a TCP option indicating the message boundry or a fixed-length PDU at a granularity to minimize overhead are much better solutions and are workable for both software and hardware implementations. I have not seen an agenda but I would hope this issue would be discussed at the upcoming meeting in Orlando. Dave -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Saturday, December 30, 2000 11:11 AM To: ips@ece.cmu.edu Subject: new iSCSI draft - 02.txt Dear colleagues, I've just submitted to the Internet-Drafts repository and to our list archive (at CMU) a new version of the draft. Changes (from 2b!): framing by Urgent-Pointer replaced by framing through in-stream marker editorials and typos (not completed yet) simpler digests digest recovery and some clarifications on iSCSI specific errors (more to come) I will have version 03 with many more editorial changes before the intermediate meeting. You can see the draft at: http://www.haifa.il.ibm.com/satran/draft-ietf-ips-iSCSI-02.txt Regards, Julo From owner-ips@ece.cmu.edu Wed Jan 3 16:23:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA05301 for ; Wed, 3 Jan 2001 16:23:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f03LMCb05391; Wed, 3 Jan 2001 16:22:12 -0500 (EST) Date: Wed, 3 Jan 2001 16:22:12 -0500 (EST) Message-Id: <200101032122.f03LMCb05391@ece.cmu.edu> X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f To: ips-archive@ietf.org From: Majordomo@ece.cmu.edu Subject: Welcome to ips Reply-To: Majordomo@ece.cmu.edu -- Welcome to the ips mailing list! Please save this message for future reference. Thank you. If you ever want to remove yourself from this mailing list, you can send mail to with the following command in the body of your email message: unsubscribe ips or from another account, besides ips-archive@lists.ietf.org: unsubscribe ips ips-archive@lists.ietf.org If you ever need to get in contact with the owner of the list, (if you have trouble unsubscribing, or have questions about the list itself) send email to . This is the general rule for most mailing lists when you need to contact a human. Here's the general information for the list you've subscribed to, in case you don't already have it: No info has been entered From owner-ips@ece.cmu.edu Wed Jan 3 16:25:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA05349 for ; Wed, 3 Jan 2001 16:25:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f03H1vG27947 for ips-outgoing; Wed, 3 Jan 2001 12:01:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcamail05.zca.compaq.com (zcamail05.zca.compaq.com [161.114.32.105]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f03H18827931 for ; Wed, 3 Jan 2001 12:01:08 -0500 (EST) Received: by zcamail05.zca.compaq.com (Postfix, from userid 12345) id 3BB3C806; Wed, 3 Jan 2001 09:02:26 -0800 (PST) Received: from exchou-gh02.cca.cpqcorp.net (exchou-gh02.cca.cpqcorp.net [16.110.248.202]) by zcamail05.zca.compaq.com (Postfix) with ESMTP id B5C5CAB0 for ; Wed, 3 Jan 2001 09:02:25 -0800 (PST) Received: by exchou-gh02.cca.cpqcorp.net with Internet Mail Service (5.5.2652.78) id ; Wed, 3 Jan 2001 11:01:00 -0600 Message-ID: <349DDB0C2ECAD411A15300805FEAE5262629CD@exchou-ca1201.cca.cpqcorp.net> From: "Elliott, Robert" To: "'ips@ece.cmu.edu'" Subject: iSCSI: SCSI Event Indicator Date: Wed, 3 Jan 2001 11:00:58 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2652.78) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk The Asynchronous Event data structure includes an "iSCSI Event Indicator" field and a "SCSI Event Indicator" field. The iSCSI field indicates iSCSI-specific causes, which is fine. The SCSI Event field indicates generic SCSI causes. I suggest removing the SCSI Event Indicator values. The information is contained within the sense data itself. The SEND command in SPC-2, the current mechanism to report Asynchronous Events, does not include an Event field like this. I suggest simplifying the field to: 0 = not a SCSI Event 1 = SCSI Event This is the text in iSCSI revision 02, with comments listing where in the sense data that information is communicated: 2.17.2 SCSI Event Indicator The following values are defined. (See [SAM2] for details): 1 An error condition was encountered after command completion. [Sense data RESPONSE CODE field is DEFERRED ERRORS - 71h] 2 A newly initialized device is available to this initiator. [Sense data ASC/ASCQ is REPORTED LUNS DATA HAS CHANGED] 3 All Task Sets are being Reset by another Initiator [Sense data ASC/ASCQ is COMMANDS CLEARED BY ANOTHER INITIATOR] [where is 4?] 5 Some other type of unit attention condition has occurred. [Sense data SENSE KEY field is UNIT ATTENTION - 6h] 6 An asynchronous event has occurred. [Any other sense data] Sense Data accompanying the report identifies the condition. The Length parameter is set to the length of the Sense Data. For new device identification an iSCSI target MUST support the Device Identification page. The list was probably taken from this list in SAM-2; however, it's not an exact match: Asynchronous Event Reporting is used to signal a device that one of the four events listed below has occurred: a) an error condition was encountered after command completion; b) a newly initialized device is available; c) some other type of unit attention condition has occurred; or d) an asynchronous event has occurred. --- PC: Robert.Elliott@compaq.com UNIX: relliott@unixmail.compaq.com From owner-ips@ece.cmu.edu Wed Jan 3 17:23:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA06505 for ; Wed, 3 Jan 2001 17:23:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f03KMF503315 for ips-outgoing; Wed, 3 Jan 2001 15:22:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f03KLiX03297 for ; Wed, 3 Jan 2001 15:21:44 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f03KaR014453; Wed, 3 Jan 2001 12:36:27 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Charles Monia" , "Ips \(E-mail\)" Subject: RE: iFCP:Timeout and Encapsulation (was FC-BB exists, why invent something new?) Date: Wed, 3 Jan 2001 11:23:52 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Charles, > > Charles, > > > > Assurances are made with the fabric timeout but you do not offer a > > time-stamp to keep this promise. TCP will try well beyond a FC fabric > > timeout. A cable swap as example could exceed nominal delays. > > We are looking at a number of ways to measure whether or not the > underlying > network is performing within the parameters of such a performance > guarantee. > We're also considering a capability that allows the system user to set the > required timeouts and establish alarm thresholds and contingency policies. Should communication be interrupted and restored using various recovery techniques, this recovery may violate the time-out limitations important to FC. In this case, heuristics of nominal communications would be of no use in determining if such an event occurred. Other than placing a time-stamp within the header, I can not see how to police such a policy of squelching late frames. Do you have a better idea? > > Would you see it possible to make a separate proposal to cover just > > encapsulation and another proposal to include iFCP specific > > link services? > > For those wishing just a tunnel, the encapsulation proposal, > > with a small > > (perhaps optional) iFCP specific field, could serve both purposes. > > > > We would need to carefully consider the consequences of such a proposal. > I'd be especially concerned about consequences down the road that might > impact the ability to make changes to both protocols over time. > Although I > can't speak for them of course, I suspect the tunneling folks may have a > similar view. A significant portion of encapsulation would satisfy both requirements. I would recommend adding an option field in the same manner as TCP to include your additional requirements. Do this in a way that allows both uses a level of extensibility as well as a means to negotiate a level of compatibility supported at each end. This would ensure a larger community of products able to interchange and thereby improving your position within the market place. You would still be allowed to add improvements that add value to your products and improve upon this scheme in the future. > > > Charles Doug From owner-ips@ece.cmu.edu Wed Jan 3 17:26:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA06545 for ; Wed, 3 Jan 2001 17:26:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f03HewG28971 for ips-outgoing; Wed, 3 Jan 2001 12:40:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-1.cisco.com (sj-msg-core-1.cisco.com [171.71.163.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f03HeN828952 for ; Wed, 3 Jan 2001 12:40:23 -0500 (EST) Received: from sponge.cisco.com (sponge.cisco.com [171.71.61.25]) by sj-msg-core-1.cisco.com (8.9.3/8.9.1) with ESMTP id JAA13773 for ; Wed, 3 Jan 2001 09:40:24 -0800 (PST) Received: from DAPW2K (dhcp-161-44-68-178.cisco.com [161.44.68.178]) by sponge.cisco.com (Mirapoint) with SMTP id AEQ00956; Wed, 3 Jan 2001 11:40:13 -0600 (CST) From: "David Peterson" To: "Ips@Ece. Cmu. Edu" Subject: iFCP as an IP Storage Work Item Date: Wed, 3 Jan 2001 11:42:04 -0600 Message-ID: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_004D_01C0757A.31A794E0" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------=_NextPart_000_004D_01C0757A.31A794E0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit The iFCP proposal for encapsulating Fibre Channel frames should not be a work item for the IP Storage working group. This view is taken by Cisco Systems and Brocade Communications for the following technical reasons. 1. FCIP is aligned with the existing FC-BB-2 work in progress. 2. FCIP will function with any existing/future FC-4 protocol such as FCP/FCP-2, VI, SRP. 3. A native iSCSI device will provide the same functionality as a device connected to an iFCP gateway. 4. Combining iFCP and FCIP does not make any sense since they are implemented using two different routing planes. In addition, FCIP is a tunneling protocol and requires mimimal processing of the Fibre Channel frame. In contrast, iFCP is a gateway protocol and requires much more processing such as the reading and modification of the Fibre Channel header, augmentation data for specific Fibre Channel Extended Link Services, and special handling of specific Fibre Channel Extended Link Services. 5. Existing Fibre Channel device addressing capability in conjunction with FCIP is viable at this point in time. 6. Manipulation of N_Port ID's as specified in iFCP will make debugging/analysis extremely difficult. In summary, FCIP exists today and is well-defined. FCIP provides all of the features necessary to tunnel Fibre Channel over IP networks. Similarly, iSCSI provides the capability for executing the SCSI command set over IP networks. Creating yet another protocol, such as iFCP, is redundant. David Peterson Cisco Systems, Inc (SRBU) Lead Architect - Standards Development Office: 763-398-1007 Cell: 612-802-3299 e-mail: dap@cisco.com Bob Snively Brocade Communications Systems Principal Engineer Office: 408 487 8135 e-mail: rsnively@brocade.com Mark Bakke Cisco Systems, Inc (SRBU) Chief Architect ------=_NextPart_000_004D_01C0757A.31A794E0 Content-Type: text/x-vcard; name="David A Peterson.vcf" Content-Disposition: attachment; filename="David A Peterson.vcf" Content-Transfer-Encoding: quoted-printable BEGIN:VCARD VERSION:2.1 N:Peterson;David;A FN:David A Peterson ORG:Cisco Systems, Inc TITLE:Lead Architect - Standards Development TEL;WORK;VOICE:763-398-1007 TEL;CELL;VOICE:612-802-3299 TEL;WORK;FAX:763-398-1001 ADR;WORK:;;6450 Wedgwood Road;Maple Grove;MN;55311 LABEL;WORK;ENCODING=3DQUOTED-PRINTABLE:6450 Wedgwood Road=3D0D=3D0AMaple = Grove, MN 55311 EMAIL;PREF;INTERNET:dap@cisco.com REV:20001110T175119Z END:VCARD ------=_NextPart_000_004D_01C0757A.31A794E0-- From owner-ips@ece.cmu.edu Wed Jan 3 17:26:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA06556 for ; Wed, 3 Jan 2001 17:26:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f03KtBF04533 for ips-outgoing; Wed, 3 Jan 2001 15:55:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f03KspX04512 for ; Wed, 3 Jan 2001 15:54:51 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Wed, 3 Jan 2001 15:54:41 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101371@corpmx9.isus.emc.com> From: Black_David@emc.com To: jtseng@NishanSystems.com, Black_David@emc.com, ips@ece.cmu.edu Subject: RE: A couple of iFCP questions Date: Wed, 3 Jan 2001 15:54:37 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > It is safe to discard a translation when there are no active N_PORT > sessions for a remote N_PORT. In that case, the iFCP gateway SHOULD > invalidate and remove the mapping to that remote N_PORT. I don't believe that is always correct. I know of Fibre Channel systems that do something approximating a mainframe-style disconnect/reconnect based on dropping the N_PORT session and opening up a new one if the gap in I/O activity is sufficient. I suspect things may be even worse in the tape world where the gap between sessions may be much longer (e.g., when the tape device is being sequentially shared among a set of systems); is anyone on this list a tape expert who knows for certain? Prematurely discarding a translation will cause unexpected results in these cases. OTOH, there should be a solution along the lines of accumulating a large number of "should be invalid" translations, discarding them in a single batch and issuing the appropriate state change notifications. Care is still required, as this "correct" operation of the (logical/virtual) fabric may still have undesirable higher-level consequences (e.g., backup application aborts). I should note that this approach is unique to Fibre Channel, as the networking analogs (e.g., running UDP through a NAT) don't have a facility like state change notification available, and hence generally can't recover from incorrectly discarding a translation. > Of course, > even if it doesn't, a device establishing a new N_PORT session has the > responsibility of validating any pre-existing mapping by checking WWPN's > of the remote device. If "device" refers to the Fibre Channel device containing the N_PORT, I'm not sure this check is (always) done. OTOH, if a mapping has changed, the state change notification mechanism described in the response to the second question will prevent this sort of problem. The explanation of iSNS's role in change propagation seems fine, although it makes connectivity to iSNS a requirement for operation of an iFCP gateway -- in other words, if the iFCP gateway loses contact with iSNS, it loses the ability to detect that its translations have become invalid. I mostly wanted to note the contrast in availability requirements with DNS - DNS does not have to be available after a connection has been set up. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Wed Jan 3 17:41:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA06865 for ; Wed, 3 Jan 2001 17:41:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f03KOEl03387 for ips-outgoing; Wed, 3 Jan 2001 15:24:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f03KO2X03374 for ; Wed, 3 Jan 2001 15:24:02 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Wed, 3 Jan 2001 15:24:05 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101370@corpmx9.isus.emc.com> From: Black_David@emc.com To: dap@cisco.com, ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Wed, 3 Jan 2001 15:23:51 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > The iFCP proposal for encapsulating Fibre Channel frames should not be a > work item for the IP Storage working group. > This view is taken by Cisco Systems and Brocade Communications for the > following technical reasons. Important procedural note - IETF is an organization of individuals, not companies. Official positions of companies are therefore generally not relevant to decisions like the one being made on iFCP. The technical input is appreciated, and will be treated as the collected technical opinions of the three individuals whose names appeared at the bottom of the email. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Wed Jan 3 18:50:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA09922 for ; Wed, 3 Jan 2001 18:50:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f03LUA905628 for ips-outgoing; Wed, 3 Jan 2001 16:30:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx10.quantum.com (mx10.quantum.com [204.212.103.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f03LTHX05587 for ; Wed, 3 Jan 2001 16:29:17 -0500 (EST) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx10.quantum.com (8.9.3+Sun/8.9.3) with ESMTP id NAA16701; Wed, 3 Jan 2001 13:27:59 -0800 (PST) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Wed, 3 Jan 2001 13:29:13 -0800 Message-ID: <8133266FE373D11190CD00805FA768BF055BD2CA@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: "'David Peterson'" , "Ips@Ece. Cmu. Edu" Cc: "'Black_David@emc.com'" Subject: RE: iFCP as an IP Storage Work Item Date: Wed, 3 Jan 2001 13:29:12 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree with David Peterson, Bob Snively, and Mark Bakke. iFCP is quite different from FCIP; the two are not functionally equivalent. iFCP is functionally equivalent to iSCSI; it's technical merit is that it is cheaper to build a bridge between iFCP and a fibre channel storage device than it to build a bridge between iSCSI and a fibre channel storage device. The downside of this advantage is that native iFCP devices would be burdened with greater complexity and cost. I therefor think iFCP should not be an IP Storage work item. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ece.cmu.edu Wed Jan 3 19:42:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA10742 for ; Wed, 3 Jan 2001 19:42:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f03N6Bb08195 for ips-outgoing; Wed, 3 Jan 2001 18:06:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f03N5lX08189 for ; Wed, 3 Jan 2001 18:05:47 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Wed, 3 Jan 2001 18:05:54 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101375@corpmx9.isus.emc.com> From: Black_David@emc.com To: jtseng@NishanSystems.com, Black_David@emc.com, ips@ece.cmu.edu Subject: RE: A couple of iFCP questions Date: Wed, 3 Jan 2001 18:05:39 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > I need to correct my prior message to state that "It is safe to discard > a translation when there are no N_PORT sessions to the remote N_PORT...". > The translation MAY be discarded when a PLOGO logout message > that terminates the N_PORT session is detected. This is what I meant > in the original message. Sorry about the confusion. And I was a little sloppy in my language. By "dropping the N_PORT session", I meant the use of PLOGO to terminate the session. After doing this, a Fibre Channel device can use PLOGI to initiate a new connection to the same target without checking with the fabric nameserver, because it can rely on state change notification to tell it if a N_PORT ID has become invalid; there are FC devices that behave in this fashion. Probing a remote N_PORT to see if a session is alive won't catch this situation; something like a state change notification to force the originator to go back to the fabric nameserver is necessary. Reliance on state change notification is also why the WWPN of the target may not be checked when the new session is set up. In any case, this is further down in the details than I really wanted to go at this juncture -- I'm satisfied that FC state change notifications can be used to handle these situations. The result may have some "brute force" characteristics to it, and the usual caveats about not using state change notifications unless they're really necessary applies. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Wed Jan 3 19:42:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA10753 for ; Wed, 3 Jan 2001 19:42:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f03M8Ad06702 for ips-outgoing; Wed, 3 Jan 2001 17:08:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f03M7KX06666 for ; Wed, 3 Jan 2001 17:07:20 -0500 (EST) Received: from centralmail2.Central.Sun.COM ([129.147.62.11]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id PAA12919; Wed, 3 Jan 2001 15:07:18 -0700 (MST) Received: from thor.Central.Sun.COM (thor.Central.Sun.COM [129.147.248.69]) by centralmail2.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id PAA10976; Wed, 3 Jan 2001 15:07:18 -0700 (MST) Received: from sun.com (hobo39.Central.Sun.COM [129.147.8.39]) by thor.Central.Sun.COM (8.9.1b+Sun/8.9.1) with ESMTP id PAA06544; Wed, 3 Jan 2001 15:07:13 -0700 (MST) Message-ID: <3A53A43F.DF33335E@sun.com> Date: Wed, 03 Jan 2001 15:14:23 -0700 From: "Mark A. Carlson" Reply-To: mark.carlson@sun.com Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.75 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: "Ips@Ece. Cmu. Edu" CC: "'David Peterson'" , "'Black_David@emc.com'" Subject: Re: iFCP as an IP Storage Work Item References: <8133266FE373D11190CD00805FA768BF055BD2CA@shrcmsg1.tdh.qntm.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit In my mind, the issue of whether this should be a work item should only revolve around two issues: 1) Is there a customer need for this kind of protocol that is not being addressed by this or other working groups? 2) Is there enough interest in the working group (from multiple companies) to support the development of the protocol? On 1), I suspect that we will need to address some sort of "gateway" protocol for customers to use in transitioning from FC Fabrics to IP Networks. Today's reality is that we have FC host OS drivers and HBAs as well as FC devices that will be with us for some time during the transition. If you assume that you need a gateway protocol that does not require the upgrade of either end, I think that 1) is satisfied. There may be other gateway protocols needed as well for when you have an iSCSI HBA and OS Driver, but existing FC devices. Look at the market for the FC/Parallel SCSI routers/bridges to see a similar transitioning market. Whether we standardize it or not, these type of devices will exist due to the market need. On 2), I'd like to see more diversity in the support of iFCP among individuals from more than just the handful of companies that have spoken so far, but the real proof is in the interest in helping to author the related documents. One possibility is to task a subgroup with FC to IP storage transition protocols, and leave the FCIP subgroup to focus just on the bridging and tunneling issues. My thoughts. -- mark From owner-ips@ece.cmu.edu Wed Jan 3 19:42:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA10767 for ; Wed, 3 Jan 2001 19:42:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f03MqBS07851 for ips-outgoing; Wed, 3 Jan 2001 17:52:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f03MpnX07839 for ; Wed, 3 Jan 2001 17:51:49 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0403C014603; Wed, 3 Jan 2001 16:03:12 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "David Peterson" , "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Wed, 3 Jan 2001 14:50:38 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, Your objections stem from added capabilities of iFCP. I would not agree that iSCSI is a replacement for something that is roughly an enhanced tunneling protocol for FC. The question I raised was whether FCIP and iFCP group could agree on an encapsulation structure that would provide needed extensibility to allow both schemes to function. Finding such common ground would be a productive and provide for future developments now excluded by a narrower view of what should be allowed within encapsulation. I would be willing to bet 90% of the transport problems of FCIP and iFCP are shared. iSCSI does not offer even a facsimile of the same structures and would enjoy virtually no commonality to either FCIP or iFCP. How a recovered frame is routed can be considered subject to a different proposal. You should view proposals as transport combined with routing management. In the case of FCIP, the routing will be just a simple physical connector. iFCP expanses this possibility. If you envision being able to cope with optional fields, both proposals can live within the same encapsulation proposal combined with separate definitions of B or N ports and their requisite routing. Finding the ability to cooperate at the encapsulation level of each proposal would allow both groups to share developing a sound structure that allows the extensibility required by both. Think of it as FCE-BB or FCE-NT or something similar where FCE is the encapsulation proposal and BB or NT would be the node definition of the end point. Doug > The iFCP proposal for encapsulating Fibre Channel frames should not be a > work item for the IP Storage working group. > This view is taken by Cisco Systems and Brocade Communications for the > following technical reasons. > > 1. FCIP is aligned with the existing FC-BB-2 work in progress. > 2. FCIP will function with any existing/future FC-4 protocol such as > FCP/FCP-2, VI, SRP. > 3. A native iSCSI device will provide the same functionality as a device > connected to an iFCP gateway. > 4. Combining iFCP and FCIP does not make any sense since they are > implemented using two different routing planes. In addition, FCIP is a > tunneling protocol and requires mimimal processing of the Fibre Channel > frame. In contrast, iFCP is a gateway protocol and requires much more > processing such as the reading and modification of the Fibre > Channel header, > augmentation data for specific Fibre Channel Extended Link Services, and > special handling of specific Fibre Channel Extended Link Services. > 5. Existing Fibre Channel device addressing capability in conjunction with > FCIP is viable at this point in time. > 6. Manipulation of N_Port ID's as specified in iFCP will make > debugging/analysis extremely difficult. > > In summary, FCIP exists today and is well-defined. FCIP provides > all of the > features > necessary to tunnel Fibre Channel over IP networks. Similarly, iSCSI > provides the > capability for executing the SCSI command set over IP networks. Creating > yet another > protocol, such as iFCP, is redundant. > > David Peterson > Cisco Systems, Inc (SRBU) > Lead Architect - Standards Development > Office: 763-398-1007 > Cell: 612-802-3299 > e-mail: dap@cisco.com > > Bob Snively > Brocade Communications Systems > Principal Engineer > Office: 408 487 8135 > e-mail: rsnively@brocade.com > > Mark Bakke > Cisco Systems, Inc (SRBU) > Chief Architect > From owner-ips@ece.cmu.edu Wed Jan 3 19:43:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA10789 for ; Wed, 3 Jan 2001 19:43:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f03NCC908350 for ips-outgoing; Wed, 3 Jan 2001 18:12:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ludwig.troikanetworks.com (host03.troikanetworks.com [12.31.172.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f03NBBX08327 for ; Wed, 3 Jan 2001 18:11:11 -0500 (EST) Received: by host03.troikanetworks.com with Internet Mail Service (5.5.2653.19) id ; Wed, 3 Jan 2001 15:11:09 -0800 Message-ID: From: Bill Terrell To: "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Wed, 3 Jan 2001 15:11:04 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk >The downside of this advantage is that native iFCP devices would be burdened >with greater complexity and cost. I therefor think iFCP should not be an IP >Storage work item. > >Regards, >-Steve How is a native iFCP endpoint (initiator or target) more complex or costly than an iSCSI native endpoint? What are the specific difficulties inherent to native iFCP devices versus native iSCSI devices? Bill From owner-ips@ece.cmu.edu Wed Jan 3 19:43:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA10800 for ; Wed, 3 Jan 2001 19:43:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f03NHDH08500 for ips-outgoing; Wed, 3 Jan 2001 18:17:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f03NGEX08463 for ; Wed, 3 Jan 2001 18:16:14 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Wed, 3 Jan 2001 15:26:02 -0800 Message-ID: From: Charles Monia To: "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Wed, 3 Jan 2001 15:15:44 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Stephen: See my responses below. > -----Original Message----- > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > Sent: Wednesday, January 03, 2001 1:29 PM > To: 'David Peterson'; Ips@Ece. Cmu. Edu > Cc: 'Black_David@emc.com' > Subject: RE: iFCP as an IP Storage Work Item > > > I agree with David Peterson, Bob Snively, and Mark Bakke. > > iFCP is quite different from FCIP; the two are not > functionally equivalent. > > iFCP is functionally equivalent to iSCSI; it's technical > merit is that it is > cheaper to build a bridge between iFCP and a fibre channel > storage device > than it to build a bridge between iSCSI and a fibre channel > storage device. You seem to be saying that iFCP adds value, so I'll grant you that. > The downside of this advantage is that native iFCP devices > would be burdened > with greater complexity and cost. Here's where you lost me. I don't know what you mean by a "native iFCP device" nor do I understand the basis for your cost and complexity comparison? >.......I therefor think iFCP > should not be an IP > Storage work item. > Frankly, I find this conclusion baffling. Charles From owner-ips@ece.cmu.edu Wed Jan 3 19:43:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA10812 for ; Wed, 3 Jan 2001 19:43:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f03MYBH07395 for ips-outgoing; Wed, 3 Jan 2001 17:34:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f03MXgX07383 for ; Wed, 3 Jan 2001 17:33:42 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Wed, 3 Jan 2001 14:43:30 -0800 Message-ID: From: Joshua Tseng To: Black_David@emc.com, jtseng@NishanSystems.com, ips@ece.cmu.edu Subject: RE: A couple of iFCP questions Date: Wed, 3 Jan 2001 14:33:12 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk David, See below: > > > It is safe to discard a translation when there are no active N_PORT > > sessions for a remote N_PORT. In that case, the iFCP gateway SHOULD > > invalidate and remove the mapping to that remote N_PORT. > > I don't believe that is always correct. I know of Fibre > Channel systems > that do something approximating a mainframe-style disconnect/reconnect > based on dropping the N_PORT session and opening up a new one if the > gap in I/O activity is sufficient. I suspect things may be > even worse in > the > tape world where the gap between sessions may be much longer (e.g., > when the tape device is being sequentially shared among a set of > systems); is anyone on this list a tape expert who knows for certain? > Prematurely discarding a translation will cause unexpected results in > these cases. I need to correct my prior message to state that "It is safe to discard a translation when there are no N_PORT sessions to the remote N_PORT...". The translation MAY be discarded when a PLOGO logout message that terminates the N_PORT session is detected. This is what I meant in the original message. Sorry about the confusion. > > OTOH, there should be a solution along the lines of accumulating a > large number of "should be invalid" translations, discarding them in a > single batch and issuing the appropriate state change notifications. It is certainly possible for an implementation to periodically probe N_PORT sessions to determine if they are still valid and alive, and for it to discard mappings and TCP connections supporting N_PORT sessions that no longer exist. > Care is still required, as this "correct" operation of the > (logical/virtual) > fabric may still have undesirable higher-level consequences > (e.g., backup > application aborts). I should note that this approach is unique to > Fibre Channel, as the networking analogs (e.g., running UDP through > a NAT) don't have a facility like state change notification > available, and > hence generally can't recover from incorrectly discarding a > translation. > > > Of course, > > even if it doesn't, a device establishing a new N_PORT > session has the > > responsibility of validating any pre-existing mapping by > checking WWPN's > > of the remote device. > > If "device" refers to the Fibre Channel device containing the > N_PORT, I'm > not > sure this check is (always) done. OTOH, if a mapping has Regardless of whether FC devices actually do this checking, it should be done. Existing FC networks have the exact same issue. If they don't check the WWPN of the device they are talking to, then they run a risk of corrupting data by talking to the wrong device. > changed, the state > change notification mechanism described in the response to the second > question will prevent this sort of problem. > > The explanation of iSNS's role in change propagation seems > fine, although > it makes connectivity to iSNS a requirement for operation of an iFCP > gateway -- in other words, if the iFCP gateway loses contact > with iSNS, > it loses the ability to detect that its translations have > become invalid. > I mostly wanted to note the contrast in availability > requirements with DNS - > DNS does not have to be available after a connection has been set up. > I see the availability requirements for iSNS to be similar to that of DNS for existing IP hosts. Yes, iSNS is required if there is a reconfiguration of the network. But the same is true for IP hosts using DNS for name resolution. A storage device being unplugged and moved around in the storage network is no different from an administrator moving an IP host to a different subnet. In both cases, iSNS/DNS will be needed. In fact, the latter case is more tedious since it requires modification of resource records in the DNS server, while the iSNS has update capability in the protocol itself. Josh > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ece.cmu.edu Wed Jan 3 21:15:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA12272 for ; Wed, 3 Jan 2001 21:15:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f040nWp10610 for ips-outgoing; Wed, 3 Jan 2001 19:49:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail2.atl.bellsouth.net (mail2.atl.bellsouth.net [205.152.0.22]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f040miX10592 for ; Wed, 3 Jan 2001 19:48:44 -0500 (EST) Received: from sukhaghosh (host-216-78-45-156.ath.bellsouth.net [216.78.45.156]) by mail2.atl.bellsouth.net (3.3.5alt/0.75.2) with SMTP id TAA26160; Wed, 3 Jan 2001 19:48:37 -0500 (EST) Reply-To: From: "Sukha Ghosh" To: "David Peterson" , "Ips@Ece. Cmu. Edu" Subject: RE: new iSCSI draft - 02.txt Date: Wed, 3 Jan 2001 19:52:02 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I agree with you fully on this. Also I like the idea of fixed length PDU better than a new TCP option indicating message boundary. -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of David Peterson Sent: Wednesday, January 03, 2001 9:47 AM To: Ips@Ece. Cmu. Edu Subject: RE: new iSCSI draft - 02.txt I believe there was agreement to remove the Urgent-Pointer framing mechanism but don't recall any agreement to replace it with an in-stream marker. For a software implementation it would be hard to support this type of framing mechanism. I believe a TCP option indicating the message boundry or a fixed-length PDU at a granularity to minimize overhead are much better solutions and are workable for both software and hardware implementations. I have not seen an agenda but I would hope this issue would be discussed at the upcoming meeting in Orlando. Dave -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Saturday, December 30, 2000 11:11 AM To: ips@ece.cmu.edu Subject: new iSCSI draft - 02.txt Dear colleagues, I've just submitted to the Internet-Drafts repository and to our list archive (at CMU) a new version of the draft. Changes (from 2b!): framing by Urgent-Pointer replaced by framing through in-stream marker editorials and typos (not completed yet) simpler digests digest recovery and some clarifications on iSCSI specific errors (more to come) I will have version 03 with many more editorial changes before the intermediate meeting. You can see the draft at: http://www.haifa.il.ibm.com/satran/draft-ietf-ips-iSCSI-02.txt Regards, Julo From owner-ips@ece.cmu.edu Wed Jan 3 21:16:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA12300 for ; Wed, 3 Jan 2001 21:16:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f040dDB10382 for ips-outgoing; Wed, 3 Jan 2001 19:39:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f040cLX10358 for ; Wed, 3 Jan 2001 19:38:21 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Wed, 3 Jan 2001 16:32:09 -0800 Message-ID: From: Joshua Tseng To: ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Wed, 3 Jan 2001 16:21:53 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I believe iFCP should be an IPS work item for the following technical reasons: 1) iFCP allows leverage of existing FCP-based driver stacks and preservation of the $$$ and @#$!!% that have been invested in them by vendor companies and their customers. 2) IP switching and routing technology is far more mature, stable, and scalable than the Fibre Channel equivalent. iFCP leverages this aspect of IP technology, while FCIP does not. With regard to your reasons why iFCP should not be permitted: > > The iFCP proposal for encapsulating Fibre Channel frames > should not be a > work item for the IP Storage working group. > This view is taken by Cisco Systems and Brocade Communications for the > following technical reasons. > > 1. FCIP is aligned with the existing FC-BB-2 work in progress. We support independent development of the FCIP separate from iFCP, but this point has nothing to do with iFCP. > 2. FCIP will function with any existing/future FC-4 protocol such as > FCP/FCP-2, VI, SRP. Again, this point says nothing about the technical feasibility of iFCP. Regardless, iFCP can encapsulate any protocol that FCIP does. > 3. A native iSCSI device will provide the same functionality > as a device > connected to an iFCP gateway. Just because iSCSI is an official IPS WG item doesn't mean that FC devices will disappear from the face of the earth. There is still a market need to internetwork FC devices over IP networks. > 4. Combining iFCP and FCIP does not make any sense since they are > implemented using two different routing planes. In addition, FCIP is a > tunneling protocol and requires mimimal processing of the > Fibre Channel > frame. In contrast, iFCP is a gateway protocol and requires much more > processing such as the reading and modification of the Fibre > Channel header, > augmentation data for specific Fibre Channel Extended Link > Services, and > special handling of specific Fibre Channel Extended Link Services. I disagree that the amount of processing is prohibitive in any way. I would like to understand why you feel this way. As iFCP is a gateway protocol, a gateway device typically has more resources available to perform these types of operations than an end device. > 5. Existing Fibre Channel device addressing capability in > conjunction with > FCIP is viable at this point in time. Again, this says nothing about iFCP. > 6. Manipulation of N_Port ID's as specified in iFCP will make > debugging/analysis extremely difficult. I am confused as to why you think debugging/analysis will be difficult. Please explain. I don't think this is true, but even if it is, the same can be said of many technologies that are described in RFC's. Take NAT for example. I and my former network buddies have had a #@!&% of a time working with NAT. Does that mean NAT should be outlawed in the IETF? No, of course not! It is needed to internetwork subnets numbered under the "old" IPv4 regime. Until we reach the promised land of IPv6 everywhere, we will continue to see a lot of NAT. Even after IPv6, I still think there will be a lot NAT around, probably until the year 2654. Same thing for iSCSI. I believe iFCP and iSCSI will coexist into the forseeable future. > > In summary, FCIP exists today and is well-defined. FCIP > provides all of the > features > necessary to tunnel Fibre Channel over IP networks. Similarly, iSCSI Of course FCIP allows tunneling--it IS a tunneling protocol. iFCP is NOT a tunneling protocol. iFCP does NOT tunnel Fibre Channel frames. Rather, it allows FC devices to be natively internetworked over an IP fabric. You can't do that with FCIP. You can't do that with iSCSI. iFCP is NOT redundant. Regards, Josh > provides the > capability for executing the SCSI command set over IP > networks. Creating > yet another > protocol, such as iFCP, is redundant. > > David Peterson > Cisco Systems, Inc (SRBU) > Lead Architect - Standards Development > Office: 763-398-1007 > Cell: 612-802-3299 > e-mail: dap@cisco.com > > Bob Snively > Brocade Communications Systems > Principal Engineer > Office: 408 487 8135 > e-mail: rsnively@brocade.com > > Mark Bakke > Cisco Systems, Inc (SRBU) > Chief Architect > From owner-ips@ece.cmu.edu Wed Jan 3 21:18:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA12319 for ; Wed, 3 Jan 2001 21:18:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f040BD209783 for ips-outgoing; Wed, 3 Jan 2001 19:11:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f040BAX09778 for ; Wed, 3 Jan 2001 19:11:10 -0500 (EST) Message-ID: <618471D08ABDD31188B6009027E5009364565D@apollo.pirus.com> From: "Hall, Howard" To: "'ips@ece.cmu.edu'" Subject: RE: iFCP as an IP Storage Work Item Date: Wed, 3 Jan 2001 19:11:03 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles and Stephen, Considering iFCP as a replacement to iSCSI concerns me as well. In this case I share Stephen's concern that a native iFCP device must always be burdened with applying a FCP header into the iFCP frames, as well as other issues. Neither iFCP or FCIP are conducive to HBA's in my opinion. iFCP as a replacement to FCIP is more palatable. However, in comparing iFCP and FCIP, I can't get past iFCP's interoperating issues with non-FCP frames, like proprietary FC remote mirroring and clustering protocols and VI/FC. -Howard > -----Original Message----- > From: Charles Monia [mailto:cmonia@NishanSystems.com] > Sent: Wednesday, January 03, 2001 6:16 PM > To: Ips@Ece. Cmu. Edu > Subject: RE: iFCP as an IP Storage Work Item > > > Hi Stephen: > > See my responses below. > > > -----Original Message----- > > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > > Sent: Wednesday, January 03, 2001 1:29 PM > > To: 'David Peterson'; Ips@Ece. Cmu. Edu > > Cc: 'Black_David@emc.com' > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > I agree with David Peterson, Bob Snively, and Mark Bakke. > > > > iFCP is quite different from FCIP; the two are not > > functionally equivalent. > > > > iFCP is functionally equivalent to iSCSI; it's technical > > merit is that it is > > cheaper to build a bridge between iFCP and a fibre channel > > storage device > > than it to build a bridge between iSCSI and a fibre channel > > storage device. > > You seem to be saying that iFCP adds value, so I'll grant you that. > > > The downside of this advantage is that native iFCP devices > > would be burdened > > with greater complexity and cost. > > Here's where you lost me. I don't know what you mean by a "native iFCP > device" nor do I understand the basis for your cost and complexity > comparison? > > >.......I therefor think iFCP > > should not be an IP > > Storage work item. > > > > Frankly, I find this conclusion baffling. > > Charles > From owner-ips@ece.cmu.edu Wed Jan 3 22:14:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA13857 for ; Wed, 3 Jan 2001 22:14:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f041QDM11466 for ips-outgoing; Wed, 3 Jan 2001 20:26:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f041PZX11452 for ; Wed, 3 Jan 2001 20:25:35 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id C33EB7ED for ; Wed, 3 Jan 2001 18:25:34 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 1B74D13D for ; Wed, 3 Jan 2001 20:25:34 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id RAA13355 for ; Wed, 3 Jan 2001 17:25:32 -0800 (PST) Message-ID: <3A53D10A.7E8C3467@agilent.com> Date: Wed, 03 Jan 2001 17:25:30 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: new iSCSI draft - 02.txt References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Sukha Ghosh wrote: > > I agree with you fully on this. Also I like the idea of fixed length PDU > better than a new TCP option indicating message boundary. Why do you like fixed length PDUs? SCSI commands are really short. Do you really want to waste a big PDU to send a small payload? Or do you want to use small PDUs and incur a large header vs data ratio? -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Wed Jan 3 22:15:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA13869 for ; Wed, 3 Jan 2001 22:15:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f041MDv11371 for ips-outgoing; Wed, 3 Jan 2001 20:22:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f041LSX11363 for ; Wed, 3 Jan 2001 20:21:28 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 5408E4EF for ; Wed, 3 Jan 2001 18:21:27 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id CDE1F67A for ; Wed, 3 Jan 2001 20:21:25 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id RAA13168 for ; Wed, 3 Jan 2001 17:21:24 -0800 (PST) Message-ID: <3A53D012.D6A20748@agilent.com> Date: Wed, 03 Jan 2001 17:21:22 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: new iSCSI draft - 02.txt References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David Peterson wrote: > > I believe there was agreement to remove the Urgent-Pointer framing mechanism > but don't recall any agreement to replace it with an in-stream marker. For a > software implementation it would be hard to support this type of framing > mechanism. I believe a TCP option indicating the message boundry or a > fixed-length PDU at a granularity to minimize overhead are much better > solutions and are workable for both software and hardware implementations. I > have not seen an agenda but I would hope this issue would be discussed at > the upcoming meeting in Orlando. > Dave What is so difficult for software to insert a few extra bytes in the byte stream? It's simply a layering problem: iSCSI layer <-> marker layer <-> TCP Normally, the marker layer simply transfers the bytes from the iSCSI layer to the TCP. Every X number of bytes, the marker layer inserts the marker into the byte stream. Since your software will probably not benefit from the receipt of the markers, it would negotiate not to receive the markers. It would only send markers *IF* the remote node requested them to be sent. -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Wed Jan 3 22:15:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA13880 for ; Wed, 3 Jan 2001 22:15:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f041jDX11885 for ips-outgoing; Wed, 3 Jan 2001 20:45:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ludwig.troikanetworks.com (host03.troikanetworks.com [12.31.172.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f041j9X11879 for ; Wed, 3 Jan 2001 20:45:09 -0500 (EST) Received: by host03.troikanetworks.com with Internet Mail Service (5.5.2653.19) id ; Wed, 3 Jan 2001 17:45:06 -0800 Message-ID: From: Wayland Jeong To: "'David Peterson'" , "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Wed, 3 Jan 2001 17:45:04 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > 2. FCIP will function with any existing/future FC-4 protocol such as > FCP/FCP-2, VI, SRP. > Even though the name and the current draft focuses on a single FC-4 mapping (FCP) there is nothing inherent in the gateway concept that limits inclusion of other FC-4 mappings. The gateway performs encapsulation at the core and as such, should be suitable for transporting any protocol mapped over FC. -Wayland From owner-ips@ece.cmu.edu Wed Jan 3 22:17:26 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA13896 for ; Wed, 3 Jan 2001 22:17:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f041TEk11539 for ips-outgoing; Wed, 3 Jan 2001 20:29:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f041T3X11532 for ; Wed, 3 Jan 2001 20:29:03 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Wed, 3 Jan 2001 17:38:52 -0800 Message-ID: From: Charles Monia To: Black_David@emc.com Cc: ips@ece.cmu.edu Subject: RE: iFCP: A couple of iFCP questions Date: Wed, 3 Jan 2001 17:28:35 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi David; See below for comments. > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Wednesday, January 03, 2001 3:06 PM > To: jtseng@NishanSystems.com; Black_David@emc.com; ips@ece.cmu.edu > Subject: RE: A couple of iFCP questions > > > > I need to correct my prior message to state that "It is > safe to discard > > a translation when there are no N_PORT sessions to the > remote N_PORT...". > > The translation MAY be discarded when a PLOGO logout message > > that terminates the N_PORT session is detected. This is > what I meant > > in the original message. Sorry about the confusion. > > And I was a little sloppy in my language. By "dropping the > N_PORT session", > I meant the use of PLOGO to terminate the session. After > doing this, a > Fibre Channel device can use PLOGI to initiate a new > connection to the same > target without checking with the fabric nameserver, because > it can rely on > state change notification to tell it if a N_PORT ID has > become invalid; > there > are FC devices that behave in this fashion. Probing a remote N_PORT > to see if a session is alive won't catch this situation; > something like a > state change notification to force the originator to go back > to the fabric > nameserver is necessary. Reliance on state change > notification is also > why the WWPN of the target may not be checked when the new session > is set up. > It's impossible to guarantee that the information returned in response to a directory query won't silently become stale -- in spite of state change notification. Therefore, the requestor must assume that the data returned may be stale by the time it is used to establish a session. It's the responsibility of the session initiator to confirm that the desired endpoint was reached at session creation. I'd venture to say that this is true of any directory-driven storage network in general. Charles Charles Monia Senior Technology Consultant Nishan Systems email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ece.cmu.edu Wed Jan 3 23:30:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA15469 for ; Wed, 3 Jan 2001 23:30:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f042NDw12698 for ips-outgoing; Wed, 3 Jan 2001 21:23:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f042MxX12692 for ; Wed, 3 Jan 2001 21:22:59 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 1C75D54A for ; Wed, 3 Jan 2001 19:22:59 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 36D65F2 for ; Wed, 3 Jan 2001 21:22:58 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id SAA15514 for ; Wed, 3 Jan 2001 18:22:57 -0800 (PST) Message-ID: <3A53DE7E.140D5816@agilent.com> Date: Wed, 03 Jan 2001 18:22:54 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: new iSCSI draft - 02.txt References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David Peterson wrote: > > I believe a TCP option indicating the message boundry ... are much better > solutions and are workable for both software and hardware implementations. Dave, I would *LOVE* to have a TCP option to indicate message boundaries. However, an implementation cannot rely on the new Option being implemented. Besides, the IETF is opposed to any changes to TCP since they've invented the as yet unproven SCTP. The marker scheme is the quickest and cleanest way to achieve the framing requirements iSCSI needs. -Matt From owner-ips@ece.cmu.edu Wed Jan 3 23:30:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA15547 for ; Wed, 3 Jan 2001 23:30:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f042FEf12557 for ips-outgoing; Wed, 3 Jan 2001 21:15:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ludwig.troikanetworks.com (host03.troikanetworks.com [12.31.172.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f042EUX12526 for ; Wed, 3 Jan 2001 21:14:31 -0500 (EST) Received: by host03.troikanetworks.com with Internet Mail Service (5.5.2653.19) id ; Wed, 3 Jan 2001 18:14:28 -0800 Message-ID: From: Wayland Jeong To: "'Hall, Howard'" , "'ips@ece.cmu.edu'" Subject: RE: iFCP as an IP Storage Work Item Date: Wed, 3 Jan 2001 18:14:23 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > Charles and Stephen, > > Considering iFCP as a replacement to iSCSI concerns me as well. In this > case I share Stephen's concern that a native iFCP device must always be > burdened with applying a FCP header into the iFCP frames, as well as other > issues. > I'm not sure that anyone has really defined a "native" iFCP device. I would venture to guess that a "native" iFCP device (or "native" FCIP device for that matter) would perform encapsulation of FC frames in the HBA rather than in a physically separate edge device. Thus, the FC HBA and the iFCP gateway would be collapsed into a single physical realization. With this description of a "native" iFCP device, I would argue that the overhead in implementing FCP protocol, which would amount to header manipulations and context management, is manageable and not entirely different than managing an iSCSI protocol session. You have many analogous things to deal with like exchanges, logins, flow-control, etc. So, I don't quite see the extra burden on such an endpoint product. Now, the issue of replacing iSCSI with iFCP is an entirely different question and I'm not sure I want to get anywhere near that one ;-) > Neither iFCP or FCIP are conducive to HBA's in my opinion. iFCP as > a replacement to FCIP is more palatable. However, in comparing iFCP and > FCIP, I can't get past iFCP's interoperating issues with non-FCP frames, > like proprietary FC remote mirroring and clustering protocols and VI/FC. > I'm not sure I understand your comment regarding remote mirroring, but as far as FC-VI is concerned, a modest extension to the iFCP proposal (support for FARP) would be all that is necessary to support the transport of VI. > -Howard > -Wayland From owner-ips@ece.cmu.edu Thu Jan 4 04:34:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA00830 for ; Thu, 4 Jan 2001 04:34:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f046jHY17563 for ips-outgoing; Thu, 4 Jan 2001 01:45:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f046ihX17533 for ; Thu, 4 Jan 2001 01:44:43 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Wed, 3 Jan 2001 22:54:35 -0800 Message-ID: From: Charles Monia To: David Peterson Cc: "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Wed, 3 Jan 2001 22:44:16 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: David Peterson [mailto:dap@cisco.com] > Sent: Wednesday, January 03, 2001 9:42 AM > To: Ips@Ece. Cmu. Edu > Subject: iFCP as an IP Storage Work Item > > > The iFCP proposal for encapsulating Fibre Channel frames > should not be a > work item for the IP Storage working group. > This view is taken by Cisco Systems and Brocade Communications for the > following technical reasons. > The "technical reasons" given below seem to represent an amalgam of corporate positioning statements, rather than issues posed for technical consideration. In that regard, I don't feel the need to add to Josh's earlier response other than a few anecdotal remarks. > 1. FCIP is aligned with the existing FC-BB-2 work in progress. > 2. FCIP will function with any existing/future FC-4 protocol such as > FCP/FCP-2, VI, SRP. For those who don't follow Fibre Channel, VI refers to a mapping of the VI architecture to Fibre Channel. SRP , is a mapping of SCSI onto VI (I believe this is targeted primarily at InfiniBand). In the context of Fibre Channel, it is functionally equivalent to FCP. At any rate, there is no inherent reason why FC/VI (and hence SRP) or any other upper layer protocol can't be supported by iFCP. > 3. A native iSCSI device will provide the same functionality > as a device > connected to an iFCP gateway. > 4. Combining iFCP and FCIP does not make any sense since they are > implemented using two different routing planes. In addition, FCIP is a > tunneling protocol and requires minimal processing of the > Fibre Channel > frame. In contrast, iFCP is a gateway protocol and requires much more > processing such as the reading and modification of the Fibre > Channel header, We don't see a problem. We expect to keep the wire saturated with flow-through FC frame traffic at Gb speed. > augmentation data for specific Fibre Channel Extended Link > Services, and > special handling of specific Fibre Channel Extended Link Services. From firsthand experience, these are not daunting problems. > 5. Existing Fibre Channel device addressing capability in > conjunction with > FCIP is viable at this point in time. > 6. Manipulation of N_Port ID's as specified in iFCP will make > debugging/analysis extremely difficult. > How did you arrive at that conclusion? Again, this doesn't jibe with our experience. > In summary, FCIP exists today and is well-defined. FCIP > provides all of the > features > necessary to tunnel Fibre Channel over IP networks. > Similarly, iSCSI > provides the > capability for executing the SCSI command set over IP > networks. Creating > yet another > protocol, such as iFCP, is redundant. > In contrast to approaches which keep FC storage devices at arms length from IP, iFCP enables such devices to fully exploit the technology. In that regard, it is definitely not redundant. Charles Charles Monia Senior Technology Consultant Nishan Systems email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ece.cmu.edu Thu Jan 4 04:56:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA01018 for ; Thu, 4 Jan 2001 04:56:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f047UHd18585 for ips-outgoing; Thu, 4 Jan 2001 02:30:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f047TrX18556 for ; Thu, 4 Jan 2001 02:29:53 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA136060 for ; Thu, 4 Jan 2001 08:29:43 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA52232 for ; Thu, 4 Jan 2001 08:29:43 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569CA.00292B0B ; Thu, 4 Jan 2001 08:29:39 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 4 Jan 2001 09:25:23 +0200 Subject: RE: new iSCSI draft - 02.txt Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Several of us thought that we have to have a framing solution in the current draft and since the TCP option (flag, length etc>) got a thumbs down at the last IETF and since in-stream and out-of-stream solution that could be used by others (like the chunking or my IPV6-like proposal) will take a long time to get through the process and mature we where left with two options: the marker having a padding mechanism that will insure that at certain boundaries there is always a PDU boundary I felt that the marker is preferable due to several reasons: It can be implemented "underneath" iSCSI - at the boundary with TCP; the iSCSI code (its main part) will be unaware of it and it can be replaced by another mechanism when such a mechanism becomes available it wastes less than the padding it is easy to implement in software and even in a plain vanilla TCP/IP adapter with hardware assists (and it can be used by many protocols) We had a "low key" consensus that this is a decent solution and even Doug Otis does not object! And yes - we can discuss at any length in Orlando. Regards, Julo "David Peterson" on 03/01/2001 16:47:07 Please respond to "David Peterson" To: "Ips@Ece. Cmu. Edu" cc: Subject: RE: new iSCSI draft - 02.txt I believe there was agreement to remove the Urgent-Pointer framing mechanism but don't recall any agreement to replace it with an in-stream marker. For a software implementation it would be hard to support this type of framing mechanism. I believe a TCP option indicating the message boundry or a fixed-length PDU at a granularity to minimize overhead are much better solutions and are workable for both software and hardware implementations. I have not seen an agenda but I would hope this issue would be discussed at the upcoming meeting in Orlando. Dave -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Saturday, December 30, 2000 11:11 AM To: ips@ece.cmu.edu Subject: new iSCSI draft - 02.txt Dear colleagues, I've just submitted to the Internet-Drafts repository and to our list archive (at CMU) a new version of the draft. Changes (from 2b!): framing by Urgent-Pointer replaced by framing through in-stream marker editorials and typos (not completed yet) simpler digests digest recovery and some clarifications on iSCSI specific errors (more to come) I will have version 03 with many more editorial changes before the intermediate meeting. You can see the draft at: http://www.haifa.il.ibm.com/satran/draft-ietf-ips-iSCSI-02.txt Regards, Julo From owner-ips@ece.cmu.edu Thu Jan 4 05:00:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA01056 for ; Thu, 4 Jan 2001 05:00:56 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0480IV19081 for ips-outgoing; Thu, 4 Jan 2001 03:00:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f047xYX19048 for ; Thu, 4 Jan 2001 02:59:34 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id IAA92836; Thu, 4 Jan 2001 08:59:27 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA188672; Thu, 4 Jan 2001 08:59:27 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569CA.002BE2D9 ; Thu, 4 Jan 2001 08:59:21 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: "Elliott, Robert" cc: ips@ece.cmu.edu Message-ID: Date: Thu, 4 Jan 2001 09:55:06 +0200 Subject: Re: iSCSI: SCSI Event Indicator Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Will do, Thanks, Julo "Elliott, Robert" on 03/01/2001 19:00:58 Please respond to "Elliott, Robert" To: "'ips@ece.cmu.edu'" cc: Subject: iSCSI: SCSI Event Indicator The Asynchronous Event data structure includes an "iSCSI Event Indicator" field and a "SCSI Event Indicator" field. The iSCSI field indicates iSCSI-specific causes, which is fine. The SCSI Event field indicates generic SCSI causes. I suggest removing the SCSI Event Indicator values. The information is contained within the sense data itself. The SEND command in SPC-2, the current mechanism to report Asynchronous Events, does not include an Event field like this. I suggest simplifying the field to: 0 = not a SCSI Event 1 = SCSI Event This is the text in iSCSI revision 02, with comments listing where in the sense data that information is communicated: 2.17.2 SCSI Event Indicator The following values are defined. (See [SAM2] for details): 1 An error condition was encountered after command completion. [Sense data RESPONSE CODE field is DEFERRED ERRORS - 71h] 2 A newly initialized device is available to this initiator. [Sense data ASC/ASCQ is REPORTED LUNS DATA HAS CHANGED] 3 All Task Sets are being Reset by another Initiator [Sense data ASC/ASCQ is COMMANDS CLEARED BY ANOTHER INITIATOR] [where is 4?] 5 Some other type of unit attention condition has occurred. [Sense data SENSE KEY field is UNIT ATTENTION - 6h] 6 An asynchronous event has occurred. [Any other sense data] Sense Data accompanying the report identifies the condition. The Length parameter is set to the length of the Sense Data. For new device identification an iSCSI target MUST support the Device Identification page. The list was probably taken from this list in SAM-2; however, it's not an exact match: Asynchronous Event Reporting is used to signal a device that one of the four events listed below has occurred: a) an error condition was encountered after command completion; b) a newly initialized device is available; c) some other type of unit attention condition has occurred; or d) an asynchronous event has occurred. --- PC: Robert.Elliott@compaq.com UNIX: relliott@unixmail.compaq.com From owner-ips@ece.cmu.edu Thu Jan 4 10:07:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA06348 for ; Thu, 4 Jan 2001 10:07:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04C9Ss04820 for ips-outgoing; Thu, 4 Jan 2001 07:09:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04C8S004808 for ; Thu, 4 Jan 2001 07:08:28 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id NAA272802 for ; Thu, 4 Jan 2001 13:08:22 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id NAA88282 for ; Thu, 4 Jan 2001 13:08:22 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569CA.0042AE2B ; Thu, 4 Jan 2001 13:08:19 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 4 Jan 2001 14:04:05 +0200 Subject: Re: iSCSI: Questions For You Mime-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=bDm1hJtJWBRfYPvw10GkRGFqanwc3oZ0aGwPsQM5SAT5ymTewaixXGRB" Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=bDm1hJtJWBRfYPvw10GkRGFqanwc3oZ0aGwPsQM5SAT5ymTewaixXGRB Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Howard, Here are some answers-- On unsolicited data: (a) Since the iSCSI specification states that the use of R2T is required by default, to clarify R2T message usage we'd like to see= the wording in the last sentence for the description of the UseR2T key= in Appendix C changed to specifically state the following: Before negotiating UseR2T:no, outgoing SCSI data (either immediate= data or data in a SCSI Data PDU) MUST NOT be sent unless solicited= by an R2T message. After negotiating UseR2T:no, the data starting at offset 0 continuing for the number of bytes specified by the initi= al burst size in Mode Page 2 MUST be sent, as either immediate data, = data in a SCSI Data PDU, or some combination thereof, without waiting f= or an R2T. All subsequent outgoing SCSI data for the same SCSI comma= nd MUST be solicited by an R2T message. It might further clarify things to define up front that ?SCSI data= ? means ?data for SCSI commands?, vs. login parameters and other iSC= SI data. Will try to make the text more clear. You are right in your assumptions. Assuming the above is correct, this is equivalent to an implicit R= 2T being sent and received at Offset 0 for ?first burst size? bytes; adding this analogy might clarify the text. Taken with the text f= rom Section 2.15 (2.16 in the 12/30 draft), this implies that ?a targe= t SHOULD NOT issue an R2T which overlaps with (the implied R2T corresponding to) any negotiated unsolicited data.? If the analog= y is incorrect, are there any requirements around overlapping R2Ts with= unsolicited data this is not correct - targets are allowed to send R2T for whenever and whatever they find appropriate and that includes re-requesting data for failed digests (data blocks). To aid understanding, we think it would be useful to have a single= place to go for a clearly worded description of SCSI data transfer= s. Rather than keeping the above text in Appendix C, we suggest combi= ning it with the appropriate text from the descriptions of the ?Ready T= o Transfer? PDU and the ?iSCSI Full Feature Phase? into a new subsec= tion under Section 1.2. Will try - good idea Since R2Ts are always used for SCSI data beyond the initial burst size, could we change ?R2T:no? to ?UnsolicitedData:yes?? (e) Typo in Section 1.2.5: ?no more unsolicited data will not be on th= is connection? should probably be ?no more unsolicited data will be sent o= n this connection will fix. R2T will stay Section 2.2.7 Command data. Since it is being defined specifically for = the SCSI command opcode ( 0x01 ), shouldn?t the statement concerning immedi= ate data be qualified with respect to the setting of the ?UseR2T? key ? will fix Potential Login deadlock ? The iSCSI spec 02b has an example of user-password authentication in appendix A, 05, 1st example. It shows that following successful authentication, when the iSCSI initiator receives the StartSecure: key,= there are a series of ... prior to login accept returned from the iSCSI= target. Section 1.2.3 iSCSI Login, paragraph 7 states: It is expected that iSCSI parameters will be negotiated after the security association protocol is established if there is a security association. The ... in the example represent the Text message negotiation of these iSCSI parameters ( as defined by the receipt of the StartSecure: key ).= Questions: ( a ) If the iSCSI initiator doesn't have any parameters to send after authentication, how long will the target wait for them befor= e giving up and retuning the "accept" Login Response ? ( b ) If the iSCSI target responds with an "accept" Login Respons= e quickly after sending the StartSecure: key, ending the authentication/security exchange, a race condition may occur where= the iSCSI initiator is sending Text message(s) that have the Initiator= Task Tag =3D the Initiator Task Tag used during the Login message = which indicates that these Text messages are considered part of the Logi= n phase. What happens then, the target believes Login is done but it receiv= es a Text message with the same Initiator Task Tag that was used during= the Login message, would this cause an error ? How would the iSCSI tar= get report the error, no iSCSI error bytes defined in the Text Respons= e message. ( c ) The spec allows for iSCSI parameter keys to be sent in mult= iple Text messages. If these are sent within the Login phase, how does = an iSCSI target know when the last Text message has been sent from th= e iSCSI initiator so that it can send the Login "accept" ? Its not a problem if they are sent outside of the Login phase sinc= e the target must wait for iSCSI commands anyway. To prevent these issues, I think the "EndLogin:HERE" iSCSI key should b= e created / defined. It is shown in the example in Apendix A ( iSCSI Secu= rity ). I think it should be specifically defined in Apendix C ( Login / Tex= t keys ). It could be sent with the last authentication Text message if authentication parameters are the only parameters the iSCSI initiator w= ill be sending which would allow an immediate Login Response by the target after it returns the StartSecure: key. or If it is not included in the authentication text message, the target kn= ows that additional Text messages are coming so it will delay its Login response until after the key is seen in a Text message payload. If adopted, the specification should be clear that the EndLogin:HERE ke= y MUST be included in either the Login payload ( no authentication case )= or in the last Text message sent by the iSCSI initiator to prevent a deadl= ock that could occur should the iSCSI initiator not include this key in any= of its Text messages. The whole sections undergoes some cleaning including a "partial" response and a behavior specification (command chaining) ( 5 ) When can an initiator task tag value repeat, after the task completes, or after the entire session completes ? Section 1.2.2.1 paragraph 2 Indicates that it is for the life of the ta= sk, but section 2.1.5 paragraph 1 indicates it is for the life of the sessi= on. I believe the intent is for the life of the task. If it was for the lif= e of the session, the iSCSI initiator would have to Logout and then Login to= create a new session every time its max initiator task tag threshold wa= s hit. With ItagLength negotiation set to 16 bits, this would be quite of= ten. Task tags can be reused when (after) a task is completed and the st= atus is acked Section 2.13 Logout Command. Shouldn?t the Logout command include the I= SID and the TSID ? The same CID may exist in an iSCSI initiator capable of creating multiple iSCSI sessions where the only unique session paramete= r may be the TSID. Although the distinction can be made at the IP layer, shouldn?t it be discernible at the iSCSI layer ? logout can arrive only within a session so why mention it? ( 7 ) Section 2.2.2 AddCDB Additional CDB length is to be added in unit= s of 4 bytes. Does this mean to add 4 CDB bytes that you set the AddCDB fiel= d =3D 1, or 4 ? 4 Section 2.12 NOP-In. Should bit 6 of byte 1 =3D ?P? ( Ping bit ) so tha= t paragraph 2 will work ? The specification has it set =3D ?0?. fixed (typo) Regards, Julo "Hall, Howard" on 03/01/2001 22:27:09 Please respond to "Hall, Howard" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: iSCSI: Questions For You = --0__=bDm1hJtJWBRfYPvw10GkRGFqanwc3oZ0aGwPsQM5SAT5ymTewaixXGRB Content-type: text/plain; charset=us-ascii Content-Disposition: inline Julian, How are you doing? I hope to see you in Orlando! Here are a few question we had for you. They are based on version 02b, but I think most of them are still relevant. Thanks!! -Howard Howard Hall Pirus Networks www.pirus.com <> --0__=bDm1hJtJWBRfYPvw10GkRGFqanwc3oZ0aGwPsQM5SAT5ymTewaixXGRB-- From owner-ips@ece.cmu.edu Thu Jan 4 11:30:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA09325 for ; Thu, 4 Jan 2001 11:30:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04EBUb06958 for ips-outgoing; Thu, 4 Jan 2001 09:11:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04CPx005076 for ; Thu, 4 Jan 2001 07:25:59 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA02155; Thu, 4 Jan 2001 07:25:57 -0500 (EST) Message-Id: <200101041225.HAA02155@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-iscsi-02.txt Date: Thu, 04 Jan 2001 07:25:57 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iSCSI Author(s) : J. Satran et aL. Filename : draft-ietf-ips-iscsi-02.txt Pages : 89 Date : 03-Jan-01 The Small Computer Systems Interface (SCSI) is a popular family of protocols for communicating with I/O devices, especially storage devices. This memo describes a transport protocol for SCSI that operates on top of TCP. The iSCSI protocol aims to be fully compliant with the requirements laid out in the SCSI Architecture Model - 2 [SAM2] document A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-02.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-02.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-02.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20010103133512.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-iscsi-02.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-iscsi-02.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20010103133512.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Thu Jan 4 11:30:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA09338 for ; Thu, 4 Jan 2001 11:30:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04EAUt06932 for ips-outgoing; Thu, 4 Jan 2001 09:10:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f045l6X16446 for ; Thu, 4 Jan 2001 00:47:06 -0500 (EST) Received: from divyaroot.India.Sun.COM ([129.158.226.35]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id VAA14744 for ; Wed, 3 Jan 2001 21:47:04 -0800 (PST) Received: from helix (helix [129.158.226.51]) by divyaroot.India.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.0) with SMTP id LAA06979 for ; Thu, 4 Jan 2001 11:17:02 +0530 (IST) Message-Id: <200101040547.LAA06979@divyaroot.India.Sun.COM> Date: Thu, 4 Jan 2001 11:18:57 -0500 (GMT) From: JP Raghavendra Rao Reply-To: JP Raghavendra Rao Subject: RE: iFCP as an IP Storage Work Item To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: 7+2hLwqr1/9tRYFXUP+W1g== Sender: owner-ips@ece.cmu.edu Precedence: bulk > I believe iFCP should be an IPS work item for the following > technical reasons: > > 1) iFCP allows leverage of existing FCP-based driver stacks and > preservation of the $$$ and @#$!!% that have been invested in them > by vendor companies and their customers. > I think the argument for preserving software doesn't make strong case in the face of a migration to a new mapping/tunneling protocol, new software and new administration challenges in spite of the fact that all of this is likely to mimic FC and contained in one or two edge routers - Today's FC network is difficult to administer and any bridging technology to a different interconnect is only going to compound it. It would be nice if somebody comes up with a stronger case for connecting an iSCSI host to a FC device - Is this attempted for the survival of FC or for a speedier deployment of iSCSI ? -JP From owner-ips@ece.cmu.edu Thu Jan 4 11:33:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA09431 for ; Thu, 4 Jan 2001 11:33:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04EAW706935 for ips-outgoing; Thu, 4 Jan 2001 09:10:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from tiny-teddy.aarnet.edu.au (IDENT:root@tiny-teddy.aarnet.edu.au [203.21.37.30]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0451PX15676 for ; Thu, 4 Jan 2001 00:01:26 -0500 (EST) Received: from aarnet.edu.au (sa005.dialup.csiro.au [144.110.4.5]) by tiny-teddy.aarnet.edu.au (8.9.3/8.9.3) with ESMTP id PAA28986; Thu, 4 Jan 2001 15:31:09 +1030 Message-ID: <3A5403AB.FD26FE27@aarnet.edu.au> Date: Thu, 04 Jan 2001 15:31:31 +1030 From: Glen Turner Organization: Australian Academic and Research Network X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.4.0-prerelease i686) X-Accept-Language: en MIME-Version: 1.0 To: sukha_ghosh@ivivity.com CC: "Ips@Ece. Cmu. Edu" Subject: Re: new iSCSI draft - 02.txt References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Sukha Ghosh wrote: > > Also I like the idea of fixed length PDU better than a new TCP > option indicating message boundary. A fixed length PDU doesn't solve the fundamental problem. In a stream of bytes you still can't tell where the PDU starts unless you assume PDUs pack nicely into packets and frames and the iSCSI process has special knowledge of the data stream (such as TCP options). Without altering TCP (which this IESG is opposed to) the options are an adaption layer containing either a flag byte and escapes (as in SLIP) or a counting scheme (such as computing a CRC which becomes 0 when the PDU is complete). Objections that byte stuffing to mark the PDU boundary discourages host-based implementations are somewhat difficult to understand. Firstly, the feature can, and should, be negotiatable in both directions. A host processor implementation can chose not to negotiate the reception of packets with byte stuffing, at the cost of incurring TCP head-on-line blocking due to late packets. Secondly, the decision to include a iSCSI-level checksum (to avoid surprisingly common corruption by DMA chips and the like, see the SIGCOMM2000 Proceedings) means that the data has to be scanned by the CPU anyway. Any other implementation of the iSCSI-level checksum gives no protection. The real objection is that removing the byte-stuffing requires a buffer copy, not just a scan. The second objection is that the overhead from SLIP-style byte stuffing has a pathological case. The counting schemes do not need a buffer copy (as you can arbitrarily drop, say, the last 8 octets that were required to drive the CRC to 0 and retreive the original data). However, these schemes are unavoidably computationally expensive, perhaps more so than a buffer copy (although RISC processors do about 100 instructions per memory cycle, so this assumption needs testing). A modification of "Consistent Overhead Byte Stuffing" (IEEE/ACM Transactions on Networking, Vol 7, No 2, 1999) seems to hold a neat solution to the pathological case of SLIP. It offers a neat way of identifying packet boundaries that has a bound on the growth in the packet size. However, like SLIP it also requires that a buffer copy occur to undo the detection of the packet boundary. If the WG find it best to deal with TCP as a stream of octets with zero knowledge of the underlying packet boundaries, options and sequence numbers then an adaption layer for TCP with a COBS-based scheme to identify packet boundaries seems to have the most promise. Regards, Glen -- Glen Turner Network Engineer (08) 8303 3936 Australian Academic and Research Network glen.turner@aarnet.edu.au http://www.aarnet.edu.au/ -- The revolution will not be televised, it will be digitised From owner-ips@ece.cmu.edu Thu Jan 4 11:34:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA09445 for ; Thu, 4 Jan 2001 11:34:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04DlTk06473 for ips-outgoing; Thu, 4 Jan 2001 08:47:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx10.quantum.com (mx10.quantum.com [204.212.103.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04Dkq006459 for ; Thu, 4 Jan 2001 08:46:52 -0500 (EST) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx10.quantum.com (8.9.3+Sun/8.9.3) with ESMTP id FAA10842 for ; Thu, 4 Jan 2001 05:45:37 -0800 (PST) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Thu, 4 Jan 2001 05:46:51 -0800 Message-ID: <8133266FE373D11190CD00805FA768BF055BD2CC@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: "'ips@ece.cmu.edu'" Subject: FW: iFCP as an IP Storage Work Item Date: Thu, 4 Jan 2001 05:46:50 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk -----Original Message----- From: Stephen Byan Sent: Thursday, January 04, 2001 8:40 AM To: 'Bill Terrell' Subject: RE: iFCP as an IP Storage Work Item It's all the FC stuff that lets iFCP work over an unreliable data transport like UDP. It's redundant when running over TCP/IP. Regards, -Steve > -----Original Message----- > From: Bill Terrell [mailto:terrell@troikanetworks.com] > Sent: Wednesday, January 03, 2001 6:10 PM > To: 'Stephen Byan' > Subject: RE: iFCP as an IP Storage Work Item > > > >The downside of this advantage is that native iFCP devices would be > burdened > >with greater complexity and cost. I therefor think iFCP > should not be an IP > >Storage work item. > > > >Regards, > >-Steve > > How is a native iFCP endpoint (initiator or target) more > complex or costly > than an iSCSI native endpoint? What are the specific > difficulties inherent > to native iFCP devices versus native iSCSI devices? > > Bill > From owner-ips@ece.cmu.edu Thu Jan 4 13:12:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA11813 for ; Thu, 4 Jan 2001 13:12:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04FnX709605 for ips-outgoing; Thu, 4 Jan 2001 10:49:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ludwig.troikanetworks.com (host03.troikanetworks.com [12.31.172.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04Fn0009585 for ; Thu, 4 Jan 2001 10:49:01 -0500 (EST) Received: by host03.troikanetworks.com with Internet Mail Service (5.5.2653.19) id ; Thu, 4 Jan 2001 07:48:59 -0800 Message-ID: From: Wayland Jeong To: "'Stephen Byan'" , "'ips@ece.cmu.edu'" Subject: RE: iFCP as an IP Storage Work Item Date: Thu, 4 Jan 2001 07:48:57 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I think there are some misconceptions here. As you probably know, the ANSI FC standard is a collection of specifications that are thick enough to choke an elephant. Of that large body of work, only a portion of it has been implemented in the commercial world. I would guess from your comments that you are referring to one of the ways that FC attempts to guarantee delivery. The two most notable mechanisms in FC are buffer-to-buffer credit and Class-2. Now, BB credit is a link-layer concept which ensures that when a FC frame is sent, it will not be dropped on the floor due to congestion. You can achieve the same functionality in Ethernet using 802.3x flow control. That said, BB-credit is a layer below what is encapsulated in iFCP. iFCP encapsulates FC at frame level and largely does not concern itself with an FC primitives (frame delimiters are an exception to this). Fibre Channel has a class of service called Class-2 which provides an ACK mechanism to guarantee delivery to the end station. It uses frame-level acknowledgements of received data as well as end-to-end buffer credit. This level of service ensures that data sent is delivered at the FC peer level. But, Class-2 was not intended to be a congestion management solution. Class-2 is an error recovery concept and it will not work well in a lossy network (i.e. IP networks with congestion management through WRED and such). Moreover, Class-2 is not generally deployed and the FC industry has largely settled on Class-3. Thus, there is nothing within a FC encapsulated in iFCP environment that makes this protocol inherently reliable. iFCP specifies TCP as the transport layer and mFCP uses UDP in conjunction with a well-behaved network. -----Original Message----- From: Stephen Byan [mailto:Stephen.Byan@quantum.com] Sent: Thursday, January 04, 2001 5:47 AM To: 'ips@ece.cmu.edu' Subject: FW: iFCP as an IP Storage Work Item -----Original Message----- From: Stephen Byan Sent: Thursday, January 04, 2001 8:40 AM To: 'Bill Terrell' Subject: RE: iFCP as an IP Storage Work Item It's all the FC stuff that lets iFCP work over an unreliable data transport like UDP. It's redundant when running over TCP/IP. Regards, -Steve > -----Original Message----- > From: Bill Terrell [mailto:terrell@troikanetworks.com] > Sent: Wednesday, January 03, 2001 6:10 PM > To: 'Stephen Byan' > Subject: RE: iFCP as an IP Storage Work Item > > > >The downside of this advantage is that native iFCP devices would be > burdened > >with greater complexity and cost. I therefor think iFCP > should not be an IP > >Storage work item. > > > >Regards, > >-Steve > > How is a native iFCP endpoint (initiator or target) more > complex or costly > than an iSCSI native endpoint? What are the specific > difficulties inherent > to native iFCP devices versus native iSCSI devices? > > Bill > From owner-ips@ece.cmu.edu Thu Jan 4 13:13:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA11864 for ; Thu, 4 Jan 2001 13:13:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04FLUx08788 for ips-outgoing; Thu, 4 Jan 2001 10:21:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04FLL008782 for ; Thu, 4 Jan 2001 10:21:21 -0500 (EST) Message-ID: <618471D08ABDD31188B6009027E50093B021FF@apollo.pirus.com> From: "Ferrari, Stephen" To: "'julian_satran@il.ibm.com'" , ips@ece.cmu.edu Subject: RE: iSCSI: Questions For You Date: Thu, 4 Jan 2001 10:21:22 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian - > Assuming the above is correct, this is equivalent to an implicit R2T > being sent and received at Offset 0 for ?first burst size? bytes; > adding this analogy might clarify the text. Taken with the text from > Section 2.15 (2.16 in the 12/30 draft), this implies that ?a target > SHOULD NOT issue an R2T which overlaps with (the implied R2T > corresponding to) any negotiated unsolicited data.? If the analogy is > incorrect, are there any requirements around overlapping R2Ts with > unsolicited data > this is not correct - targets are allowed to send R2T for > whenever and whatever they find appropriate and that includes > re-requesting data for failed digests (data blocks). Could you please clarify this sentence from Section 2.16 from the new draft: "However, the target SHOULD NOT issue overlapping R2T request (i.e. referring to the same data area)." It seems to conflict with your statement above. Or if you interpret them as not in conflict because of "SHOULD NOT" vs. "MUST NOT", then why isn't it analogous to say that the target SHOULD NOT overlap an R2T with the implied R2T corresponding to the unsolicited data? Thanks, Stephen From owner-ips@ece.cmu.edu Thu Jan 4 13:59:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA13137 for ; Thu, 4 Jan 2001 13:59:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04G7WZ10371 for ips-outgoing; Thu, 4 Jan 2001 11:07:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from eclipse.iphase.com (firewall-user@eclipse.iphase.com [157.175.3.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04G7K010362 for ; Thu, 4 Jan 2001 11:07:20 -0500 (EST) Received: by eclipse.iphase.com; id KAA21242; Thu, 4 Jan 2001 10:07:18 -0600 (CST) Received: from unknown(157.175.85.9) by eclipse.iphase.com via smap (V5.5) id xma021210; Thu, 4 Jan 01 10:06:56 -0600 Received: from localhost (kquick@localhost) by hera.Iphase.COM (8.8.8+Sun/8.8.7) with SMTP id KAA01306; Thu, 4 Jan 2001 10:06:55 -0600 (CST) Date: Thu, 4 Jan 2001 10:06:55 -0600 (CST) From: Kevin Quick X-Sender: kquick@hera To: Matt Wakeley cc: IPS Reflector Subject: Re: new iSCSI draft - 02.txt In-Reply-To: <3A53D012.D6A20748@agilent.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk : From: Matt Wakeley : : David Peterson wrote: : > : > I believe there was agreement to remove the Urgent-Pointer framing mechanism : > but don't recall any agreement to replace it with an in-stream marker. For a : > software implementation it would be hard to support this type of framing : > mechanism. I believe a TCP option indicating the message boundry or a : > fixed-length PDU at a granularity to minimize overhead are much better : > solutions and are workable for both software and hardware implementations. I : > have not seen an agenda but I would hope this issue would be discussed at : > the upcoming meeting in Orlando. : > Dave : : What is so difficult for software to insert a few extra bytes in the byte : stream? It's simply a layering problem: : : iSCSI layer <-> marker layer <-> TCP : : Normally, the marker layer simply transfers the bytes from the iSCSI layer to : the TCP. Every X number of bytes, the marker layer inserts the marker into the : byte stream. : : Since your software will probably not benefit from the receipt of the markers, : it would negotiate not to receive the markers. It would only send markers : *IF* the remote node requested them to be sent. : : -Matt Wakeley : Agilent Technologies I would think that there would have to be a number of changes to existing TCP implementations to utilize TCP-option-based framing: * Modification of the transmit request interface to indicate frame boundaries * Modification of the TCP transmit segmentation algorithm to recognize and preserve frame boundaries * Modification of TCP slow-start window sizing to recognize and preserve frame boundaries * Modification of MTU path discovery, with possible rejection of some frame transmit requests at the transmit request interface if the frame exceeds allowable boundaries. * Possible delivery failures if the path is silently changed or multiplexed after initial discovery? * Possible failures if "legacy" nodes either preserve option data but re-segment the traffic, or discard the "new" option information entirely. ** Firewalls, data-caching nodes, layer-3 routers, and other store-and-forward nodes would be susceptible. * Modification of the receive path socket management to preserve frame boundaries. * If any node re-transmits any portion of the data stream (due to NAK or ACK-timeout) then the retransmit logic must be modified to identify the frame boundaries in the retransmitted portion, recalculate the TCP options for that retransmitted portion (since the headers may occur at a new position relative to the datastream), and then perhaps resynchronize at the original location when the remote ACKs the rest of the previously transmitted window. Some or all of the above may not be typical situations, or even generally "seen", but since we are defining a standard, it's important to define that standard relative to what the existing standards *allow*, not relative to how current sw/hw is *implemented*. [Note: I'm not familiar enough with the details of the standards myself to know if the concerns above are warranted... I'm trying to point out potential issues here for consideration by more knowledgeable experts.] As Matt points out above, in-band markers are unequivocably perserved as a consequence of being part of the data stream, and can be handled easily by a layer above the existing TCP management at the endpoints only. Additionally, an in-band marker can forward reference the id of the next command in the stream, thereby allowing receive hardware to prefetch DMA buffer information for that next command. This usage is congruent with the high-performance considerations of this general effort. Note that this enhances both loss-less and loss-error transfers, since a marker allows resynchronization only at the *next* message anyhow. Regards, Kevin From owner-ips@ece.cmu.edu Thu Jan 4 14:54:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA14546 for ; Thu, 4 Jan 2001 14:54:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04H3Yc12063 for ips-outgoing; Thu, 4 Jan 2001 12:03:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cleitus.hosting.pacbell.net (cleitus.hosting.pacbell.net [216.100.99.17]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04H2x012042 for ; Thu, 4 Jan 2001 12:03:01 -0500 (EST) Received: from VENKAT1 (64-160-62-200.rhapsodynetworks.com [64.160.62.200] (may be forged)) by cleitus.hosting.pacbell.net id MAA07735; Thu, 4 Jan 2001 12:02:43 -0500 (EST) [ConcentricHost SMTP Relay 1.7] From: "Venkat Rangan" To: "David Peterson" , "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Thu, 4 Jan 2001 09:04:43 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit We are also not in support of picking up iFCP as a IP Storage Work Item. As we indicated in earlier posts (FC-BB-2 exists thread and the FCIP support for N_Port etc.) we would not be in support of multiple ways (i.e., FCIP and iFCP) in which FC and IP networks/devices are made to coexist. In particular, our reasons for not supporting iFCP as a work item are the following. 1. Confusion in market place The overall goal of iFCP appears to be to leverage investment in IP networking technology for networking storage. In this respect, it is closer in the goal of iSCSI. In a storage network based on IP networking technology, we see iSCSI as a better alternative. It would be confusing to present both iSCSI and iFCP as two technologies for using the IP network for storage. Since the IP Storage Working Group and its members have already invested heavily in standardizing iSCSI, we do not see much value in diluting the efforts on multiple technologies towards the same end goal. 2. Support for existing FC networks FCIP and its coordinated effort with FC-BB-2 supports existing FC networks better. The interoperability issues between FC and IP networks are restricted to the border switches. Other infrastructure needs such as FC Name Server, Fabric Controller etc. are leveraged from standards work already done in T11 working groups. The argument that iFCP is useful in a world of FC devices is not very strong, since FCIP is a solution that not only preserves investment in FC end nodes but also in FC switches. 3. iFCP does not support FC loops and FC E_Port The drafts as submitted do not handle FC loops and FC switches. There were some email discussions where authors indicated that these are trivial to add to iFCP. We do not think it would be trivial to add E_Port support and work through the interoperability issues between iFCP gateways and FC switches. Adding FC loop support and E_Port support to an iFCP gateway would make that gateway device closer to an FC switch, so one could simply use FC switches with B_Port support and FCIP as the tunneling protocol. 4. iFCP adds cost to deployment End nodes may be forced to support both FC N_Port and iSCSI. It is easier for end nodes to just support iSCSI when storage network is based on an IP network. There is also an additional cost in testing a pure iSCSI end node with a bridged FC-iFCP end node. 5. FCIP has broader coverage iFCP implies that it is only useful for translating FCP traffic. FCIP is able to tunnel all FC traffic through its border switches. It is also possible to add N_Port connectivity using FCIP, although such an attempt would also compete with the goals of iSCSI. 6. iFCP addds processing overhead to frames In contrast to iSCSI, iFCP requires examination of every frame and substitution of addresses at both iFCP Portals. This has the potential to add unwanted latencies as well as consume processing overhead on gateways. Given this, it would be hard to compete with cut-through routing based products in a pure FC or IP based storage network. We also do not believe the NAT like function should really be the top priority for the working group to address. As an example, the first NAT RFC was RFC 1631 (which was an Informational RFC from the one vendor), well after several other higher priority items were addressed. We do not believe that FC networks (when deployed in the context of back-end storage) are running out of address space to warrant this to be addressed immediately. Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of David Peterson Sent: Wednesday, January 03, 2001 9:42 AM To: Ips@Ece. Cmu. Edu Subject: iFCP as an IP Storage Work Item The iFCP proposal for encapsulating Fibre Channel frames should not be a work item for the IP Storage working group. This view is taken by Cisco Systems and Brocade Communications for the following technical reasons. 1. FCIP is aligned with the existing FC-BB-2 work in progress. 2. FCIP will function with any existing/future FC-4 protocol such as FCP/FCP-2, VI, SRP. 3. A native iSCSI device will provide the same functionality as a device connected to an iFCP gateway. 4. Combining iFCP and FCIP does not make any sense since they are implemented using two different routing planes. In addition, FCIP is a tunneling protocol and requires mimimal processing of the Fibre Channel frame. In contrast, iFCP is a gateway protocol and requires much more processing such as the reading and modification of the Fibre Channel header, augmentation data for specific Fibre Channel Extended Link Services, and special handling of specific Fibre Channel Extended Link Services. 5. Existing Fibre Channel device addressing capability in conjunction with FCIP is viable at this point in time. 6. Manipulation of N_Port ID's as specified in iFCP will make debugging/analysis extremely difficult. In summary, FCIP exists today and is well-defined. FCIP provides all of the features necessary to tunnel Fibre Channel over IP networks. Similarly, iSCSI provides the capability for executing the SCSI command set over IP networks. Creating yet another protocol, such as iFCP, is redundant. David Peterson Cisco Systems, Inc (SRBU) Lead Architect - Standards Development Office: 763-398-1007 Cell: 612-802-3299 e-mail: dap@cisco.com Bob Snively Brocade Communications Systems Principal Engineer Office: 408 487 8135 e-mail: rsnively@brocade.com Mark Bakke Cisco Systems, Inc (SRBU) Chief Architect From owner-ips@ece.cmu.edu Thu Jan 4 14:58:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA14697 for ; Thu, 4 Jan 2001 14:58:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04H7YN12208 for ips-outgoing; Thu, 4 Jan 2001 12:07:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from exchange-1.xiotech.com (ftp.xiotech.com [209.46.118.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04H7U012202 for ; Thu, 4 Jan 2001 12:07:30 -0500 (EST) Message-ID: From: "Peglar, Robert" To: "'Charles Monia'" , Black_David@emc.com Cc: ips@ece.cmu.edu Subject: RE: iFCP: A couple of iFCP questions Date: Thu, 4 Jan 2001 11:07:00 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles You stated below... > It's impossible to guarantee that the information returned in > response to a > directory query won't silently become stale -- in spite of > state change > notification. > > Therefore, the requestor must assume that the data returned > may be stale by > the time it is used to establish a session. It's the > responsibility of the > session initiator to confirm that the desired endpoint was reached at > session creation. I agree. However, when does this chase-one's-tail end? If the requestor can make no assumption about validity, i.e. all such responses may be stale - how can unambiguous confirmation be achieved? Does this mean one must establish a class 1 connection between endpoints? If no guarantees can be made about connectionless query responses and their validity, then I think there may be a hole. Thanks Rob Rob Peglar Director, Storage Architecture XIOtech Corporation (314) 308-6983 > -----Original Message----- > From: Charles Monia [mailto:cmonia@NishanSystems.com] > Sent: Wednesday, January 03, 2001 7:29 PM > To: Black_David@emc.com > Cc: ips@ece.cmu.edu > Subject: RE: iFCP: A couple of iFCP questions > > > Hi David; > > See below for comments. > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Wednesday, January 03, 2001 3:06 PM > > To: jtseng@NishanSystems.com; Black_David@emc.com; ips@ece.cmu.edu > > Subject: RE: A couple of iFCP questions > > > > > > > I need to correct my prior message to state that "It is > > safe to discard > > > a translation when there are no N_PORT sessions to the > > remote N_PORT...". > > > The translation MAY be discarded when a PLOGO logout message > > > that terminates the N_PORT session is detected. This is > > what I meant > > > in the original message. Sorry about the confusion. > > > > And I was a little sloppy in my language. By "dropping the > > N_PORT session", > > I meant the use of PLOGO to terminate the session. After > > doing this, a > > Fibre Channel device can use PLOGI to initiate a new > > connection to the same > > target without checking with the fabric nameserver, because > > it can rely on > > state change notification to tell it if a N_PORT ID has > > become invalid; > > there > > are FC devices that behave in this fashion. Probing a remote N_PORT > > to see if a session is alive won't catch this situation; > > something like a > > state change notification to force the originator to go back > > to the fabric > > nameserver is necessary. Reliance on state change > > notification is also > > why the WWPN of the target may not be checked when the new session > > is set up. > > > > It's impossible to guarantee that the information returned in > response to a > directory query won't silently become stale -- in spite of > state change > notification. > > Therefore, the requestor must assume that the data returned > may be stale by > the time it is used to establish a session. It's the > responsibility of the > session initiator to confirm that the desired endpoint was reached at > session creation. > > I'd venture to say that this is true of any directory-driven > storage network > in general. > > > > Charles > Charles Monia > Senior Technology Consultant > Nishan Systems > email: cmonia@nishansystems.com > voice: (408) 519-3986 > fax: (408) 435-8385 > From owner-ips@ece.cmu.edu Thu Jan 4 15:49:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA15775 for ; Thu, 4 Jan 2001 15:49:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04Isai15456 for ips-outgoing; Thu, 4 Jan 2001 13:54:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04Is6015435 for ; Thu, 4 Jan 2001 13:54:06 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Thu, 4 Jan 2001 11:04:03 -0800 Message-ID: From: Joshua Tseng To: "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Thu, 4 Jan 2001 10:53:42 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I don't want to stifle any creative technical discussion here, but I feel the need to remind everybody that iFCP is positioned as a gateway technology only. While the thought of "native" iFCP HBA's might be interesting, this discussion is completely irrelevant with regard to whether iFCP should or should not become an IPS work item. iFCP is being proposed as an IPS work item purely on its merits as a gateway technology. Regards, Josh > -----Original Message----- > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > Sent: Thursday, January 04, 2001 5:47 AM > To: 'ips@ece.cmu.edu' > Subject: FW: iFCP as an IP Storage Work Item > > > > > -----Original Message----- > From: Stephen Byan > Sent: Thursday, January 04, 2001 8:40 AM > To: 'Bill Terrell' > Subject: RE: iFCP as an IP Storage Work Item > > > It's all the FC stuff that lets iFCP work over an unreliable > data transport > like UDP. It's redundant when running over TCP/IP. > > Regards, > -Steve > > > -----Original Message----- > > From: Bill Terrell [mailto:terrell@troikanetworks.com] > > Sent: Wednesday, January 03, 2001 6:10 PM > > To: 'Stephen Byan' > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > >The downside of this advantage is that native iFCP devices would be > > burdened > > >with greater complexity and cost. I therefor think iFCP > > should not be an IP > > >Storage work item. > > > > > >Regards, > > >-Steve > > > > How is a native iFCP endpoint (initiator or target) more > > complex or costly > > than an iSCSI native endpoint? What are the specific > > difficulties inherent > > to native iFCP devices versus native iSCSI devices? > > > > Bill > > > From owner-ips@ece.cmu.edu Thu Jan 4 15:49:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA15797 for ; Thu, 4 Jan 2001 15:49:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04J1aM15682 for ips-outgoing; Thu, 4 Jan 2001 14:01:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04J0x015661 for ; Thu, 4 Jan 2001 14:00:59 -0500 (EST) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id LAA22968 for ; Thu, 4 Jan 2001 11:00:59 -0800 (PST) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.151.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id LAA20155 for ; Thu, 4 Jan 2001 11:00:58 -0800 (PST) Received: from ebay.sun.com by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id LAA13820; Thu, 4 Jan 2001 11:00:57 -0800 (PST) Message-ID: <3A54C868.D86AC90F@ebay.sun.com> Date: Thu, 04 Jan 2001 11:00:56 -0800 From: David Robinson Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.7 [en] (X11; I; SunOS 5.8 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: "Ips@Ece. Cmu. Edu" Subject: Re: iFCP as an IP Storage Work Item References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I am throwing in my hat to have the WG support both iFCP and FCIP. From a business/customer perspective I find believable markets for both approaches (both of which are still speculation). From a technical perspective they are similar enough that having one standard mechanism and one different defacto mechanism will cause more problems than it solves. It is clear that the semantic meaning of the two proposed protocols cannot be merged as they do not operate in the same plane of traditional stacks. However, from my reading of the two proposals the encapsulation mechanisms are remarkably similar even though their semantics are diverge significantly. What I have started (before my two weeks away from work, ahhhh...) but haven't yet finished, is an investigation on standardizing a common encapsulation protocol for FC over TCP/IP. The the FCIP vs iFCP becomes a higher level interpretation of the semantics of the bits instead of also having two completely different stacks. -David From owner-ips@ece.cmu.edu Thu Jan 4 15:51:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA15844 for ; Thu, 4 Jan 2001 15:51:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04I7YV13931 for ips-outgoing; Thu, 4 Jan 2001 13:07:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04I7E013923 for ; Thu, 4 Jan 2001 13:07:14 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id TAA88580 for ; Thu, 4 Jan 2001 19:07:02 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id TAA34456 for ; Thu, 4 Jan 2001 19:07:02 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569CA.00636E24 ; Thu, 4 Jan 2001 19:06:02 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 4 Jan 2001 19:58:38 +0200 Subject: RE: iSCSI: Questions For You Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Sorry - I'll fix the inconsistency. The statement you found is an overlook. (I mean" However, the target SHOULD NOT issue overlapping R2T request (i.e. referring to the same data area).") I'll fix that. Thanks, Julo "Ferrari, Stephen" on 04/01/2001 17:21:22 Please respond to "Ferrari, Stephen" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI: Questions For You Julian - > Assuming the above is correct, this is equivalent to an implicit R2T > being sent and received at Offset 0 for ?first burst size? bytes; > adding this analogy might clarify the text. Taken with the text from > Section 2.15 (2.16 in the 12/30 draft), this implies that ?a target > SHOULD NOT issue an R2T which overlaps with (the implied R2T > corresponding to) any negotiated unsolicited data.? If the analogy is > incorrect, are there any requirements around overlapping R2Ts with > unsolicited data > this is not correct - targets are allowed to send R2T for > whenever and whatever they find appropriate and that includes > re-requesting data for failed digests (data blocks). Could you please clarify this sentence from Section 2.16 from the new draft: "However, the target SHOULD NOT issue overlapping R2T request (i.e. referring to the same data area)." It seems to conflict with your statement above. Or if you interpret them as not in conflict because of "SHOULD NOT" vs. "MUST NOT", then why isn't it analogous to say that the target SHOULD NOT overlap an R2T with the implied R2T corresponding to the unsolicited data? Thanks, Stephen From owner-ips@ece.cmu.edu Thu Jan 4 15:54:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA15911 for ; Thu, 4 Jan 2001 15:54:36 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04Ihko15105 for ips-outgoing; Thu, 4 Jan 2001 13:43:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04IhJ015091 for ; Thu, 4 Jan 2001 13:43:19 -0500 (EST) Received: from omgw2.boi.hp.com (omgw2.boi.hp.com [15.56.8.102]) by palrel3.hp.com (Postfix) with ESMTP id 138F735 for ; Thu, 4 Jan 2001 10:43:18 -0800 (PST) Received: from xrosebh3.rsvl.itc.hp.com (xrosebh3.rsvl.itc.hp.com [15.34.240.67]) by omgw2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with ESMTP id LAA11663 for ; Thu, 4 Jan 2001 11:43:17 -0700 (MST) Received: by xrosebh3.rsvl.itc.hp.com with Internet Mail Service (5.5.2653.19) id ; Thu, 4 Jan 2001 10:43:18 -0800 Message-ID: <670D895CA648D41198EA00A0C9F2D4280384098D@xrose01.rose.hp.com> From: "ELLIOTT,STEPHEN (HP-Roseville,ex1)" To: "'ips@ece.cmu.edu'" Subject: RE: iFCP as an IP Storage Work Item Date: Thu, 4 Jan 2001 10:43:14 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Regarding the statement of JP Raghavendra Rao: It would be nice if somebody comes up with a stronger case for connecting an iSCSI host to a FC device - Is this attempted for the survival of FC or for a speedier deployment of iSCSI ? I offer the following technical consideration. Consider extremely large storage arrays. I think storage arrays tend to be FC devices. Moreover, in some cases there are limitations on the number of SCSI devices that can be accomodated on a single host, which would prevent software RAID with SCSI from scaling to the same degree as FC in these cases. There are other devices as well that are FC-based in order to integrate within the FC-SAN that in some cases includes these extremely large storage arrays. In each of these cases, connectivity to these devices is essential. What SAN devices do you envision ISCSI connecting to? In what timeframe do you think the FC SAN market will disappear? It seems that this issue is separate from the iFCP/iSCSI debate, because an appropriate iSCSI bridge could accomodate these SCSI-over-FC devices. Regards, Stephen Elliott -----Original Message----- From: JP Raghavendra Rao [mailto:jp.raghavendra@india.sun.com] Sent: Thursday, January 04, 2001 8:19 AM To: ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item > I believe iFCP should be an IPS work item for the following > technical reasons: > > 1) iFCP allows leverage of existing FCP-based driver stacks and > preservation of the $$$ and @#$!!% that have been invested in them > by vendor companies and their customers. > I think the argument for preserving software doesn't make strong case in the face of a migration to a new mapping/tunneling protocol, new software and new administration challenges in spite of the fact that all of this is likely to mimic FC and contained in one or two edge routers - Today's FC network is difficult to administer and any bridging technology to a different interconnect is only going to compound it. It would be nice if somebody comes up with a stronger case for connecting an iSCSI host to a FC device - Is this attempted for the survival of FC or for a speedier deployment of iSCSI ? -JP From owner-ips@ece.cmu.edu Thu Jan 4 17:02:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA17219 for ; Thu, 4 Jan 2001 17:02:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04JWac16683 for ips-outgoing; Thu, 4 Jan 2001 14:32:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04JVc016660 for ; Thu, 4 Jan 2001 14:31:38 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA46780 for ; Thu, 4 Jan 2001 14:23:58 -0500 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.95) with ESMTP id f04JTcJ59966 for ; Thu, 4 Jan 2001 12:30:32 -0700 Importance: Normal Subject: Re: iSCSI: Questions For You To: "Julian Satran" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.5 September 22, 2000 Message-ID: From: "Prasenjit Sarkar" Date: Thu, 4 Jan 2001 11:29:21 -0800 X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.5 |September 22, 2000) at 01/04/2001 11:30:33 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk My opinion on this is that a logout outside the context of its session may be useful in certain scenarios (such as failover) - so it may be a good idea to put the ISID and TSID in the logout command. Regards, Prasenjit ------------------------------------------------------------------------- Section 2.13 Logout Command. Shouldn?t the Logout command include the ISID and the TSID ? The same CID may exist in an iSCSI initiator capable of creating multiple iSCSI sessions where the only unique session parameter may be the TSID. Although the distinction can be made at the IP layer, shouldn?t it be discernible at the iSCSI layer ? logout can arrive only within a session so why mention it? From owner-ips@ece.cmu.edu Thu Jan 4 17:42:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA17659 for ; Thu, 4 Jan 2001 17:42:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04Kkbu18987 for ips-outgoing; Thu, 4 Jan 2001 15:46:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04Kk9018971 for ; Thu, 4 Jan 2001 15:46:09 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Thu, 4 Jan 2001 12:56:07 -0800 Message-ID: From: Charles Monia To: "Ips@Ece. Cmu. Edu" Cc: Venkat Rangan Subject: RE: iFCP as an IP Storage Work Item Date: Thu, 4 Jan 2001 12:45:44 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Venkat: The issues you raise have been discussed at length before (but apparently not to your satisfaction). That aside, the general thrust of this note seems to reflect business rather than technical concerns. Within the parameters of the WG charter and IETF ground rules, it's my view that the ips wg needs to be a vehicle for bringing high-quality solutions to the marketplace, not obstructing technologies because they are inconvenient or conflict with someone's business model. See other responses below. Charles > -----Original Message----- > From: Venkat Rangan [mailto:venkat@rhapsodynetworks.com] > Sent: Thursday, January 04, 2001 9:05 AM > To: David Peterson; Ips@Ece. Cmu. Edu > Subject: RE: iFCP as an IP Storage Work Item > > > > We are also not in support of picking up iFCP as a IP Storage > Work Item. As > we indicated in earlier posts (FC-BB-2 exists thread and the > FCIP support > for N_Port etc.) we would not be in support of multiple ways > (i.e., FCIP and > iFCP) in which FC and IP networks/devices are made to coexist. > > In particular, our reasons for not supporting iFCP as a work > item are the > following. > > 1. Confusion in market place > > The overall goal of iFCP appears to be to leverage investment in IP > networking technology for networking storage. In this > respect, it is closer > in the goal of iSCSI. In a storage network based on IP networking > technology, we see iSCSI as a better alternative. It would be > confusing to > present both iSCSI and iFCP as two technologies for using the > IP network for > storage. Since the IP Storage Working Group and its members > have already > invested heavily in standardizing iSCSI, we do not see much value in > diluting the efforts on multiple technologies towards the > same end goal. > Such confusion, if it exists, is best sorted out in the marketplace. In my opinion, it's not the business of standards organizations to anoint any one technology by picking winners and losers (as some would apparently wish). > 2. Support for existing FC networks > > FCIP and its coordinated effort with FC-BB-2 supports > existing FC networks > better. The interoperability issues between FC and IP networks are > restricted to the border switches. Other infrastructure needs > such as FC > Name Server, Fabric Controller etc. are leveraged from standards work > already done in T11 working groups. The argument that iFCP is > useful in a > world of FC devices is not very strong, since FCIP is a > solution that not > only preserves investment in FC end nodes but also in FC switches. > To the contrary, the iFCP gateway model dovetails nicely with existing FC infrastuctures. iFCP's strong point is that the interface to existing FC networks becomes a gateway implementation issue. > 3. iFCP does not support FC loops and FC E_Port > > The drafts as submitted do not handle FC loops and FC > switches. There were > some email discussions where authors indicated that these are > trivial to add > to iFCP. We do not think it would be trivial to add E_Port > support and work > through the interoperability issues between iFCP gateways and > FC switches. > Adding FC loop support and E_Port support to an iFCP gateway > would make that > gateway device closer to an FC switch, so one could simply > use FC switches > with B_Port support and FCIP as the tunneling protocol. > Whether or not such support is trivial is, of course, a matter of opinion. Nonetheless, you seem to have missed the point. iFCP enables the gateway implementation to conceal the FC infrastructure. Given Fibre Channel's historic interoperability problems, some would consider that a benefit. Anyhow, the behavior on the FC side of the device is, of course, governed by the applicable FC standards (or their proprietary equivalents). It's not required for iFCP devices to interoperate with one another and hence doesn't belong in the spec. Since this issue has been mentioned a number of times, an informational appendix giving such implementation examples would apparently be useful. > 4. iFCP adds cost to deployment > > End nodes may be forced to support both FC N_Port and iSCSI. Forced by whom? Presumably, such concerns reflect market pressures and are irrelevant as such. > It is easier > for end nodes to just support iSCSI when storage network is > based on an IP > network. I assume that's a product development choice. >.............There is also an additional cost in testing a pure > iSCSI end node > with a bridged FC-iFCP end node. > Then, I'd suggest that you don't build such products. > 5. FCIP has broader coverage > > iFCP implies that it is only useful for translating FCP > traffic. FCIP is > able to tunnel all FC traffic through its border switches. It is also > possible to add N_Port connectivity using FCIP, although such > an attempt > would also compete with the goals of iSCSI. > > 6. iFCP addds processing overhead to frames > > In contrast to iSCSI, iFCP requires examination of every frame and > substitution of addresses at both iFCP Portals. This has the > potential to > add unwanted latencies as well as consume processing overhead > on gateways. Such assertions about processing overhead are totally without foundation or substance. > Given this, it would be hard to compete with cut-through routing based > products in a pure FC or IP based storage network. We also do > not believe > the NAT like function should really be the top priority for > the working > group to address. As an example, the first NAT RFC was RFC > 1631 (which was > an Informational RFC from the one vendor), well after several > other higher > priority items were addressed. We do not believe that FC > networks (when > deployed in the context of back-end storage) are running out > of address > space to warrant this to be addressed immediately. > > That's nothing more than unsubstantiated opinion and contrary to our experience. Apparently, earlier postings on this matter have gone unread. Charles Charles Monia Senior Technology Consultant Nishan Systems email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ece.cmu.edu Thu Jan 4 17:43:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA17671 for ; Thu, 4 Jan 2001 17:43:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04KEb017963 for ips-outgoing; Thu, 4 Jan 2001 15:14:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcars04e.ca.nortel.com (h56s242a129n47.user.nortelnetworks.com [47.129.242.56]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04KDZ017942 for ; Thu, 4 Jan 2001 15:13:35 -0500 (EST) Received: from zbl6c012.corpeast.baynetworks.com by zcars04e.ca.nortel.com; Thu, 4 Jan 2001 15:12:48 -0500 Received: from zbl6c002.corpeast.baynetworks.com ([132.245.205.52]) by zbl6c012.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id ZLK0A7N0; Thu, 4 Jan 2001 15:12:49 -0500 Received: from ftravost.nortelnetworks.com (dhcp223-248.engeast.baynetworks.com [192.32.223.248]) by zbl6c002.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id CGS4G9PM; Thu, 4 Jan 2001 15:12:49 -0500 Message-Id: <5.0.0.25.2.20010104104242.032b48d0@zbl6c002.corpeast.baynetworks.com> X-Sender: travos@zbl6c002.corpeast.baynetworks.com X-Mailer: QUALCOMM Windows Eudora Version 5.0 Date: Thu, 04 Jan 2001 14:46:50 -0500 To: ips@ece.cmu.edu From: "Franco Travostino" Subject: RE: iFCP as an IP Storage Work Item Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="=====================_86018963==_.ALT" X-Orig: Sender: owner-ips@ece.cmu.edu Precedence: bulk --=====================_86018963==_.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed I support further development of iFCP within the IPS Storage WG. It breaks new grounds in the FC-to-IP transition realm. I'm not satisfied with just iSCSI and FCIP. I often think of FCIP as a steep precinct wall that is being erected too high too fast. With a fair amount of risks lurking (one-fits-all TCP tunnel, dual routing planes, FC dialects' autobahn, manual administration). As for public scrutiny: Out of 3000 messages in my ips@ece.cmu.edu mailbox, I only count about 30 messages with technical comments to the FCIP drafts. Some of us can already give iFCP credit for making the FC-to-IP adaptation a more mundane subject. FCIP and the industry can as well benefit from this FCIP/iFCP constructive dualism in the WG (via interesting byproducts such as iSNS, or common framing techniques). -franco ----- Franco Travostino, Director Content Internetworking Lab Technology Center Nortel Networks, Inc. 600 Technology Park Billerica, MA 01821 USA Tel: 978 288 7708 Fax: 978 288 4690 email: travos@nortelnetworks.com --=====================_86018963==_.ALT Content-Type: text/html; charset="us-ascii"
I support further development of iFCP within the IPS Storage WG. It breaks new grounds in the FC-to-IP transition realm.

I'm not satisfied with just iSCSI and FCIP. I often think of FCIP as a steep precinct wall that is being erected too high too fast. With a fair amount of risks lurking (one-fits-all TCP tunnel, dual routing planes, FC dialects' autobahn, manual administration). As for public scrutiny: Out of 3000 messages in my ips@ece.cmu.edu mailbox, I only count about 30 messages with technical comments to the FCIP drafts.

Some of us can already give iFCP credit for making the FC-to-IP adaptation a more mundane subject. FCIP and the industry can as well benefit from this FCIP/iFCP constructive dualism in the WG (via interesting byproducts such as iSNS, or common framing techniques).

-franco

-----
Franco Travostino, Director Content Internetworking Lab
Technology Center
Nortel Networks, Inc.
600 Technology Park
Billerica, MA 01821 USA
Tel: 978 288 7708 Fax: 978 288 4690
email: travos@nortelnetworks.com
 --=====================_86018963==_.ALT-- From owner-ips@ece.cmu.edu Thu Jan 4 19:08:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA18728 for ; Thu, 4 Jan 2001 19:08:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04Lnc720928 for ips-outgoing; Thu, 4 Jan 2001 16:49:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04Ln8020920 for ; Thu, 4 Jan 2001 16:49:08 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 4 Jan 2001 16:49:02 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101382@corpmx9.isus.emc.com> From: Black_David@emc.com To: cmonia@NishanSystems.com, ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Thu, 4 Jan 2001 16:49:00 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, Sorry to jump on this one, but I must be misinterpreting what you wrote: > Anyhow, the behavior on the FC side of the device is, of course, governed by > the applicable FC standards (or their proprietary equivalents). It's not > required for iFCP devices to interoperate with one another and hence doesn't > belong in the spec. I hope you meant something along the lines of: The FC interfaces on iFCP devices are governed by the appropriate FC standards (which require no modifications for iFCP) and hence must interoperate to the degree required by those standards. This may depend on implementation choices, for example, an E port on a iFCP device would not be expected to communicate with an N port. If iFCP were to somehow make Fibre Channel interoperability worse (e.g., an F port that failed to work properly with N ports), that would be a severe problem. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Thu Jan 4 19:10:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA18749 for ; Thu, 4 Jan 2001 19:10:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04MNeE21889 for ips-outgoing; Thu, 4 Jan 2001 17:23:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04MMw021870 for ; Thu, 4 Jan 2001 17:22:58 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Thu, 4 Jan 2001 14:32:57 -0800 Message-ID: From: Joshua Tseng To: Camden Ford , "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Thu, 4 Jan 2001 14:22:33 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Cam, I'm glad you find our web site interesting. SoIP is Nishan's brand name for its IP-based storage solutions. This includes our products that use iSCSI, iFCP, iSNS and other protocols that we are currently implementing. Josh > > That is an interesting statement considering that Nishan's > web site clearly states its purpose to develop SoIP as an > "...end to end storage networking solution" > > Regards, > > Cam Ford > > -----Original Message----- > From: Joshua Tseng [mailto:jtseng@NishanSystems.com] > Sent: Thursday, January 04, 2001 10:54 AM > To: Ips@Ece. Cmu. Edu > Subject: RE: iFCP as an IP Storage Work Item > > > I don't want to stifle any creative technical discussion here, > but I feel the need to remind everybody that iFCP is positioned > as a gateway technology only. While the thought of "native" > iFCP HBA's might be interesting, this discussion is > completely irrelevant with regard to whether iFCP should > or should not become an IPS work item. iFCP is being proposed > as an IPS work item purely on its merits as a gateway technology. > > Regards, > Josh > > > -----Original Message----- > > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > > Sent: Thursday, January 04, 2001 5:47 AM > > To: 'ips@ece.cmu.edu' > > Subject: FW: iFCP as an IP Storage Work Item > > > > > > > > > > -----Original Message----- > > From: Stephen Byan > > Sent: Thursday, January 04, 2001 8:40 AM > > To: 'Bill Terrell' > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > It's all the FC stuff that lets iFCP work over an unreliable > > data transport > > like UDP. It's redundant when running over TCP/IP. > > > > Regards, > > -Steve > > > > > -----Original Message----- > > > From: Bill Terrell [mailto:terrell@troikanetworks.com] > > > Sent: Wednesday, January 03, 2001 6:10 PM > > > To: 'Stephen Byan' > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > >The downside of this advantage is that native iFCP > devices would be > > > burdened > > > >with greater complexity and cost. I therefor think iFCP > > > should not be an IP > > > >Storage work item. > > > > > > > >Regards, > > > >-Steve > > > > > > How is a native iFCP endpoint (initiator or target) more > > > complex or costly > > > than an iSCSI native endpoint? What are the specific > > > difficulties inherent > > > to native iFCP devices versus native iSCSI devices? > > > > > > Bill > > > > > > From owner-ips@ece.cmu.edu Thu Jan 4 19:10:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA18760 for ; Thu, 4 Jan 2001 19:10:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04MWed22132 for ips-outgoing; Thu, 4 Jan 2001 17:32:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04MWP022123 for ; Thu, 4 Jan 2001 17:32:25 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Thu, 4 Jan 2001 14:42:24 -0800 Message-ID: From: Charles Monia To: Black_David@emc.com, Charles Monia , ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Thu, 4 Jan 2001 14:32:01 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Thursday, January 04, 2001 1:49 PM > To: cmonia@nishansystems.com; ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > Charles, > > Sorry to jump on this one, but I must be misinterpreting what > you wrote: > > > Anyhow, the behavior on the FC side of the device is, of > course, governed > by > > the applicable FC standards (or their proprietary > equivalents). It's not > > required for iFCP devices to interoperate with one another and hence > doesn't > > belong in the spec. > > I hope you meant something along the lines of: > > The FC interfaces on iFCP devices are governed by the appropriate FC > standards > (which require no modifications for iFCP) and hence must > interoperate to > the degree > required by those standards. This may depend on > implementation choices, > for example, > an E port on a iFCP device would not be expected to > communicate with an N > port. > Yes -- your interpretation is correct. Charles Charles Monia Senior Technology Consultant Nishan Systems email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ece.cmu.edu Thu Jan 4 19:11:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA18775 for ; Thu, 4 Jan 2001 19:11:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04Lxdx21229 for ips-outgoing; Thu, 4 Jan 2001 16:59:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.com (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04Lwn021208 for ; Thu, 4 Jan 2001 16:58:49 -0500 (EST) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Thu, 4 Jan 2001 13:58:43 -0800 Message-ID: <176B85B0D4E0D411BA5200508B692E0A35D18A@hq-ex-1.brocade.com> From: Camden Ford To: "'Joshua Tseng'" , "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Thu, 4 Jan 2001 13:58:41 -0800 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk That is an interesting statement considering that Nishan's web site clearly states its purpose to develop SoIP as an "...end to end storage networking solution" Regards, Cam Ford -----Original Message----- From: Joshua Tseng [mailto:jtseng@NishanSystems.com] Sent: Thursday, January 04, 2001 10:54 AM To: Ips@Ece. Cmu. Edu Subject: RE: iFCP as an IP Storage Work Item I don't want to stifle any creative technical discussion here, but I feel the need to remind everybody that iFCP is positioned as a gateway technology only. While the thought of "native" iFCP HBA's might be interesting, this discussion is completely irrelevant with regard to whether iFCP should or should not become an IPS work item. iFCP is being proposed as an IPS work item purely on its merits as a gateway technology. Regards, Josh > -----Original Message----- > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > Sent: Thursday, January 04, 2001 5:47 AM > To: 'ips@ece.cmu.edu' > Subject: FW: iFCP as an IP Storage Work Item > > > > > -----Original Message----- > From: Stephen Byan > Sent: Thursday, January 04, 2001 8:40 AM > To: 'Bill Terrell' > Subject: RE: iFCP as an IP Storage Work Item > > > It's all the FC stuff that lets iFCP work over an unreliable > data transport > like UDP. It's redundant when running over TCP/IP. > > Regards, > -Steve > > > -----Original Message----- > > From: Bill Terrell [mailto:terrell@troikanetworks.com] > > Sent: Wednesday, January 03, 2001 6:10 PM > > To: 'Stephen Byan' > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > >The downside of this advantage is that native iFCP devices would be > > burdened > > >with greater complexity and cost. I therefor think iFCP > > should not be an IP > > >Storage work item. > > > > > >Regards, > > >-Steve > > > > How is a native iFCP endpoint (initiator or target) more > > complex or costly > > than an iSCSI native endpoint? What are the specific > > difficulties inherent > > to native iFCP devices versus native iSCSI devices? > > > > Bill > > > From owner-ips@ece.cmu.edu Thu Jan 4 20:02:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA19052 for ; Thu, 4 Jan 2001 20:02:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04Mt0322764 for ips-outgoing; Thu, 4 Jan 2001 17:55:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thalia.fm.intel.com (thalia.fm.intel.com [132.233.247.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04MsB022740 for ; Thu, 4 Jan 2001 17:54:11 -0500 (EST) Received: from SMTP (fmsmsxvs03-1.fm.intel.com [132.233.42.203]) by thalia.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.33 2000/11/21 19:27:27 smothers Exp $) with SMTP id WAA10346; Thu, 4 Jan 2001 22:55:30 GMT Received: from fmsmsx19.fm.intel.com ([132.233.48.19]) by 132.233.48.203 (Norton AntiVirus for Internet Email Gateways 1.0) ; Thu, 04 Jan 2001 22:54:08 0000 (GMT) Received: by fmsmsx19.fm.intel.com with Internet Mail Service (5.5.2650.21) id ; Thu, 4 Jan 2001 14:54:06 -0800 Message-ID: <31E3E2A9D8DDD311AC43009027C68052C8015F@orsmsx59.jf.intel.com> From: "Grun, Paul" To: "'Black_David@emc.com'" , cmonia@NishanSystems.com, ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Thu, 4 Jan 2001 14:53:59 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id f04Msc122752 Sender: owner-ips@ece.cmu.edu Precedence: bulk X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id f04Mt0422764 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ietf.org id UAA19052 Here's the (crude) way I interpreted what Charles wrote: an iFCP device has two sides; an FC side and an IP side. On its FC side, it better conform to the applicable FC standards, including interoperability with other iFCP devices. I agree with Charles that interoperability between iFCP devices on their "FC side" doesn't seem like an iFCP problem per se; rather it is a function of whatever interoperability exists between FC devices today. Charles? -paul Paul Grun                                              Intel Corporation Enterprise Platform Group Fabric Components Division (503) 677-6768 paul.grun@intel.com > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Thursday, January 04, 2001 1:49 PM > To: cmonia@NishanSystems.com; ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > Charles, > > Sorry to jump on this one, but I must be misinterpreting what > you wrote: > > > Anyhow, the behavior on the FC side of the device is, of > course, governed > by > > the applicable FC standards (or their proprietary > equivalents). It's not > > required for iFCP devices to interoperate with one another and hence > doesn't > > belong in the spec. > > I hope you meant something along the lines of: > > The FC interfaces on iFCP devices are governed by the appropriate FC > standards > (which require no modifications for iFCP) and hence must > interoperate to > the degree > required by those standards. This may depend on > implementation choices, > for example, > an E port on a iFCP device would not be expected to > communicate with an N > port. > > If iFCP were to somehow make Fibre Channel interoperability > worse (e.g., an > F port > that failed to work properly with N ports), that would be a > severe problem. > > Thanks, > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Thu Jan 4 21:17:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA19569 for ; Thu, 4 Jan 2001 21:17:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04NnhR24152 for ips-outgoing; Thu, 4 Jan 2001 18:49:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04Nlj024107 for ; Thu, 4 Jan 2001 18:47:45 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Thu, 4 Jan 2001 15:57:39 -0800 Message-ID: From: Charles Monia To: "Peglar, Robert" Cc: ips@ece.cmu.edu, Charles Monia , Black_David@emc.com Subject: RE: iFCP: A couple of iFCP questions Date: Thu, 4 Jan 2001 15:47:15 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Peglar, Robert [mailto:robert_peglar@xiotech.com] > Sent: Thursday, January 04, 2001 9:07 AM > To: 'Charles Monia'; Black_David@emc.com > Cc: ips@ece.cmu.edu > Subject: RE: iFCP: A couple of iFCP questions > > > Charles > > You stated below... > > It's impossible to guarantee that the information returned in > > response to a > > directory query won't silently become stale -- in spite of > > state change > > notification. > > > > Therefore, the requestor must assume that the data returned > > may be stale by > > the time it is used to establish a session. It's the > > responsibility of the > > session initiator to confirm that the desired endpoint was > reached at > > session creation. > > I agree. > > However, when does this chase-one's-tail end? If the requestor > can make no assumption about validity, i.e. all such responses > may be stale - how can unambiguous confirmation be achieved? Does > this mean one must establish a class 1 connection between endpoints? > > If no guarantees can be made about connectionless query responses > and their validity, then I think there may be a hole. > Hi Rob: With any directory-driven application, there is always a time skew between when the information is received and when it is used. To address that unavoidable window, the session initiator can confirm the device I/D at session startup. As part of the login handshake the session initiator is given the WWUI of the responder, which can be checked against the original value. If there's a difference, the initiator can reissue the directory query. State change notifications can, of course, help this by speeding the discovery of such configuration changes. In sum, while it may take some finite time for device configuration changes to propagate, the process should naturally converge. In a practical case, this should not be an issue. Charles > Rob Peglar > Director, Storage Architecture > XIOtech Corporation > (314) 308-6983 > > > > -----Original Message----- > > From: Charles Monia [mailto:cmonia@NishanSystems.com] > > Sent: Wednesday, January 03, 2001 7:29 PM > > To: Black_David@emc.com > > Cc: ips@ece.cmu.edu > > Subject: RE: iFCP: A couple of iFCP questions > > > > > > Hi David; > > > > See below for comments. > > > > > -----Original Message----- > > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > > Sent: Wednesday, January 03, 2001 3:06 PM > > > To: jtseng@NishanSystems.com; Black_David@emc.com; ips@ece.cmu.edu > > > Subject: RE: A couple of iFCP questions > > > > > > > > > > I need to correct my prior message to state that "It is > > > safe to discard > > > > a translation when there are no N_PORT sessions to the > > > remote N_PORT...". > > > > The translation MAY be discarded when a PLOGO logout message > > > > that terminates the N_PORT session is detected. This is > > > what I meant > > > > in the original message. Sorry about the confusion. > > > > > > And I was a little sloppy in my language. By "dropping the > > > N_PORT session", > > > I meant the use of PLOGO to terminate the session. After > > > doing this, a > > > Fibre Channel device can use PLOGI to initiate a new > > > connection to the same > > > target without checking with the fabric nameserver, because > > > it can rely on > > > state change notification to tell it if a N_PORT ID has > > > become invalid; > > > there > > > are FC devices that behave in this fashion. Probing a > remote N_PORT > > > to see if a session is alive won't catch this situation; > > > something like a > > > state change notification to force the originator to go back > > > to the fabric > > > nameserver is necessary. Reliance on state change > > > notification is also > > > why the WWPN of the target may not be checked when the new session > > > is set up. > > > > > > > It's impossible to guarantee that the information returned in > > response to a > > directory query won't silently become stale -- in spite of > > state change > > notification. > > > > Therefore, the requestor must assume that the data returned > > may be stale by > > the time it is used to establish a session. It's the > > responsibility of the > > session initiator to confirm that the desired endpoint was > reached at > > session creation. > > > > I'd venture to say that this is true of any directory-driven > > storage network > > in general. > > > > > > > > Charles > > Charles Monia > > Senior Technology Consultant > > Nishan Systems > > email: cmonia@nishansystems.com > > voice: (408) 519-3986 > > fax: (408) 435-8385 > > > From owner-ips@ece.cmu.edu Thu Jan 4 21:17:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA19587 for ; Thu, 4 Jan 2001 21:17:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f04NxgG24343 for ips-outgoing; Thu, 4 Jan 2001 18:59:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f04NxM024332 for ; Thu, 4 Jan 2001 18:59:22 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Thu, 4 Jan 2001 16:09:21 -0800 Message-ID: From: Charles Monia To: "Grun, Paul" Cc: ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Thu, 4 Jan 2001 15:58:56 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Grun, Paul [mailto:paul.grun@intel.com] > Sent: Thursday, January 04, 2001 2:54 PM > To: 'Black_David@emc.com'; cmonia@NishanSystems.com; ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > Here's the (crude) way I interpreted what Charles wrote: > an iFCP device has two sides; an FC side and an IP side. On > its FC side, it > better conform to the applicable FC standards, including > interoperability > with other iFCP devices. I agree with Charles that > interoperability between > iFCP devices on their "FC side" doesn't seem like an iFCP > problem per se; > rather it is a function of whatever interoperability exists between FC > devices today. > Charles? > Hi Paul: Basically, I agree with the above. The job of an iFCP gateway is to present FC-side N_PORTs to the IP-side (and vice-versa). Everything else behind the gateway is otherwise opaque (as far as the IP-side is concerned). The FC side of the gateway may, of course, support interfaces to other kinds of FC devices or topologies, such as FC switch elements, arbitrated loops, you-name-it. Although the gateway must comply with the standards or specs that apply in such cases, these are not within the scope of iFCP since these interfaces and topologies are not presented on the IP side of the gateway and are therefore not a factor in gateway-to-gateway interoperability. Charles > -paul > > Paul Grun > Intel Corporation > Enterprise Platform Group > Fabric Components Division > (503) 677-6768 > paul.grun@intel.com > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Thursday, January 04, 2001 1:49 PM > > To: cmonia@NishanSystems.com; ips@ece.cmu.edu > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > Charles, > > > > Sorry to jump on this one, but I must be misinterpreting what > > you wrote: > > > > > Anyhow, the behavior on the FC side of the device is, of > > course, governed > > by > > > the applicable FC standards (or their proprietary > > equivalents). It's not > > > required for iFCP devices to interoperate with one > another and hence > > doesn't > > > belong in the spec. > > > > I hope you meant something along the lines of: > > > > The FC interfaces on iFCP devices are governed by the > appropriate FC > > standards > > (which require no modifications for iFCP) and hence must > > interoperate to > > the degree > > required by those standards. This may depend on > > implementation choices, > > for example, > > an E port on a iFCP device would not be expected to > > communicate with an N > > port. > > > > If iFCP were to somehow make Fibre Channel interoperability > > worse (e.g., an > > F port > > that failed to work properly with N ports), that would be a > > severe problem. > > > > Thanks, > > --David > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > > > > > From owner-ips@ece.cmu.edu Thu Jan 4 21:53:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA20717 for ; Thu, 4 Jan 2001 21:53:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f050ZgF25104 for ips-outgoing; Thu, 4 Jan 2001 19:35:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from exchange-1.xiotech.com (ftp.xiotech.com [209.46.118.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f050Z2025071 for ; Thu, 4 Jan 2001 19:35:02 -0500 (EST) Message-ID: From: "Peglar, Robert" To: "'Charles Monia'" Cc: ips@ece.cmu.edu Subject: RE: iFCP: A couple of iFCP questions Date: Thu, 4 Jan 2001 18:34:31 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles said: > Hi Rob: > > With any directory-driven application, there is always a time > skew between > when the information is received and when it is used. To address that > unavoidable window, the session initiator can confirm the > device I/D at > session startup. As part of the login handshake the session > initiator is > given the WWUI of the responder, which can be checked against > the original > value. If there's a difference, the initiator can reissue > the directory > query. State change notifications can, of course, help this > by speeding the > discovery of such configuration changes. > > In sum, while it may take some finite time for device > configuration changes > to propagate, the process should naturally converge. In a > practical case, > this should not be an issue. OK. If, in the case where the WWUI confirmed at session startup is different than the WWUI seen at login, should there be a built-in delay to allow (some, better, complete, insert word here) convergence to occur, before re-issuing the directory query? For example, typical FC switches might consume 500 to 700 ms while converging the mesh. Just a thought. Rob > From owner-ips@ece.cmu.edu Thu Jan 4 21:53:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA20728 for ; Thu, 4 Jan 2001 21:53:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f050ogL25418 for ips-outgoing; Thu, 4 Jan 2001 19:50:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f050oW025409 for ; Thu, 4 Jan 2001 19:50:32 -0500 (EST) Received: from redfish.adaptec.com (redfish.adaptec.com [162.62.50.11]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id QAA26761; Thu, 4 Jan 2001 16:50:26 -0800 (PST) Received: from aimexc06.corp.adaptec.com ([162.62.62.46]) by redfish.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id QAA20538; Thu, 4 Jan 2001 16:44:17 -0800 (PST) Received: by aimexc06.corp.adaptec.com.62.62.162.in-addr.arpa with Internet Mail Service (5.5.2650.21) id ; Thu, 4 Jan 2001 16:50:26 -0800 Message-ID: From: "Sperry, Todd" To: "'Charles Monia'" Cc: ips@ece.cmu.edu Subject: RE: iFCP: A couple of iFCP questions Date: Thu, 4 Jan 2001 16:50:25 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, One quick question. You mentioned checking the WWUI as part of the login sequence. By this, do you mean the 64-bit WWN (Port, Node, Fabric Port Name) that's listed in iSNS (3.2.4). Or is it something else? Thanks, Todd. -----Original Message----- From: Charles Monia [mailto:cmonia@NishanSystems.com] Sent: Thursday, January 04, 2001 3:47 PM To: Peglar, Robert Cc: ips@ece.cmu.edu; Charles Monia; Black_David@emc.com Subject: RE: iFCP: A couple of iFCP questions > -----Original Message----- > From: Peglar, Robert [mailto:robert_peglar@xiotech.com] > Sent: Thursday, January 04, 2001 9:07 AM > To: 'Charles Monia'; Black_David@emc.com > Cc: ips@ece.cmu.edu > Subject: RE: iFCP: A couple of iFCP questions > > > Charles > > You stated below... > > It's impossible to guarantee that the information returned in > > response to a > > directory query won't silently become stale -- in spite of > > state change > > notification. > > > > Therefore, the requestor must assume that the data returned > > may be stale by > > the time it is used to establish a session. It's the > > responsibility of the > > session initiator to confirm that the desired endpoint was > reached at > > session creation. > > I agree. > > However, when does this chase-one's-tail end? If the requestor > can make no assumption about validity, i.e. all such responses > may be stale - how can unambiguous confirmation be achieved? Does > this mean one must establish a class 1 connection between endpoints? > > If no guarantees can be made about connectionless query responses > and their validity, then I think there may be a hole. > Hi Rob: With any directory-driven application, there is always a time skew between when the information is received and when it is used. To address that unavoidable window, the session initiator can confirm the device I/D at session startup. As part of the login handshake the session initiator is given the WWUI of the responder, which can be checked against the original value. If there's a difference, the initiator can reissue the directory query. State change notifications can, of course, help this by speeding the discovery of such configuration changes. In sum, while it may take some finite time for device configuration changes to propagate, the process should naturally converge. In a practical case, this should not be an issue. Charles > Rob Peglar > Director, Storage Architecture > XIOtech Corporation > (314) 308-6983 > > > > -----Original Message----- > > From: Charles Monia [mailto:cmonia@NishanSystems.com] > > Sent: Wednesday, January 03, 2001 7:29 PM > > To: Black_David@emc.com > > Cc: ips@ece.cmu.edu > > Subject: RE: iFCP: A couple of iFCP questions > > > > > > Hi David; > > > > See below for comments. > > > > > -----Original Message----- > > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > > Sent: Wednesday, January 03, 2001 3:06 PM > > > To: jtseng@NishanSystems.com; Black_David@emc.com; ips@ece.cmu.edu > > > Subject: RE: A couple of iFCP questions > > > > > > > > > > I need to correct my prior message to state that "It is > > > safe to discard > > > > a translation when there are no N_PORT sessions to the > > > remote N_PORT...". > > > > The translation MAY be discarded when a PLOGO logout message > > > > that terminates the N_PORT session is detected. This is > > > what I meant > > > > in the original message. Sorry about the confusion. > > > > > > And I was a little sloppy in my language. By "dropping the > > > N_PORT session", > > > I meant the use of PLOGO to terminate the session. After > > > doing this, a > > > Fibre Channel device can use PLOGI to initiate a new > > > connection to the same > > > target without checking with the fabric nameserver, because > > > it can rely on > > > state change notification to tell it if a N_PORT ID has > > > become invalid; > > > there > > > are FC devices that behave in this fashion. Probing a > remote N_PORT > > > to see if a session is alive won't catch this situation; > > > something like a > > > state change notification to force the originator to go back > > > to the fabric > > > nameserver is necessary. Reliance on state change > > > notification is also > > > why the WWPN of the target may not be checked when the new session > > > is set up. > > > > > > > It's impossible to guarantee that the information returned in > > response to a > > directory query won't silently become stale -- in spite of > > state change > > notification. > > > > Therefore, the requestor must assume that the data returned > > may be stale by > > the time it is used to establish a session. It's the > > responsibility of the > > session initiator to confirm that the desired endpoint was > reached at > > session creation. > > > > I'd venture to say that this is true of any directory-driven > > storage network > > in general. > > > > > > > > Charles > > Charles Monia > > Senior Technology Consultant > > Nishan Systems > > email: cmonia@nishansystems.com > > voice: (408) 519-3986 > > fax: (408) 435-8385 > > > From owner-ips@ece.cmu.edu Thu Jan 4 23:19:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA22599 for ; Thu, 4 Jan 2001 23:19:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f051thb26700 for ips-outgoing; Thu, 4 Jan 2001 20:55:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f051tZ026692 for ; Thu, 4 Jan 2001 20:55:35 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Thu, 4 Jan 2001 18:05:35 -0800 Message-ID: From: Charles Monia To: "Sperry, Todd" Cc: ips@ece.cmu.edu Subject: RE: iFCP: A couple of iFCP questions Date: Thu, 4 Jan 2001 17:55:10 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Todd: In the case of an FC N_PORT login, I was referring to the 64-bit Port_Name (or alternatively the Node_Name) field returned in the ACC frame. Analogous considerations would also apply to an iSCSI login -- although, of course, the handshake and WWUI format is different. Charles > -----Original Message----- > From: Sperry, Todd [mailto:Todd_Sperry@adaptec.com] > Sent: Thursday, January 04, 2001 4:50 PM > To: 'Charles Monia' > Cc: ips@ece.cmu.edu > Subject: RE: iFCP: A couple of iFCP questions > > > > Charles, > > One quick question. You mentioned checking the > WWUI as part of the login sequence. By this, do > you mean the 64-bit WWN (Port, Node, Fabric Port Name) > that's listed in iSNS (3.2.4). Or is it something > else? > > Thanks, > Todd. > > -----Original Message----- > From: Charles Monia [mailto:cmonia@NishanSystems.com] > Sent: Thursday, January 04, 2001 3:47 PM > To: Peglar, Robert > Cc: ips@ece.cmu.edu; Charles Monia; Black_David@emc.com > Subject: RE: iFCP: A couple of iFCP questions > > > > > > -----Original Message----- > > From: Peglar, Robert [mailto:robert_peglar@xiotech.com] > > Sent: Thursday, January 04, 2001 9:07 AM > > To: 'Charles Monia'; Black_David@emc.com > > Cc: ips@ece.cmu.edu > > Subject: RE: iFCP: A couple of iFCP questions > > > > > > Charles > > > > You stated below... > > > It's impossible to guarantee that the information returned in > > > response to a > > > directory query won't silently become stale -- in spite of > > > state change > > > notification. > > > > > > Therefore, the requestor must assume that the data returned > > > may be stale by > > > the time it is used to establish a session. It's the > > > responsibility of the > > > session initiator to confirm that the desired endpoint was > > reached at > > > session creation. > > > > I agree. > > > > However, when does this chase-one's-tail end? If the requestor > > can make no assumption about validity, i.e. all such responses > > may be stale - how can unambiguous confirmation be achieved? Does > > this mean one must establish a class 1 connection between endpoints? > > > > If no guarantees can be made about connectionless query responses > > and their validity, then I think there may be a hole. > > > > Hi Rob: > > With any directory-driven application, there is always a time > skew between > when the information is received and when it is used. To address that > unavoidable window, the session initiator can confirm the > device I/D at > session startup. As part of the login handshake the session > initiator is > given the WWUI of the responder, which can be checked against > the original > value. If there's a difference, the initiator can reissue > the directory > query. State change notifications can, of course, help this > by speeding the > discovery of such configuration changes. > > In sum, while it may take some finite time for device > configuration changes > to propagate, the process should naturally converge. In a > practical case, > this should not be an issue. > > Charles > > > > Rob Peglar > > Director, Storage Architecture > > XIOtech Corporation > > (314) 308-6983 > > > > > > > -----Original Message----- > > > From: Charles Monia [mailto:cmonia@NishanSystems.com] > > > Sent: Wednesday, January 03, 2001 7:29 PM > > > To: Black_David@emc.com > > > Cc: ips@ece.cmu.edu > > > Subject: RE: iFCP: A couple of iFCP questions > > > > > > > > > Hi David; > > > > > > See below for comments. > > > > > > > -----Original Message----- > > > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > > > Sent: Wednesday, January 03, 2001 3:06 PM > > > > To: jtseng@NishanSystems.com; Black_David@emc.com; > ips@ece.cmu.edu > > > > Subject: RE: A couple of iFCP questions > > > > > > > > > > > > > I need to correct my prior message to state that "It is > > > > safe to discard > > > > > a translation when there are no N_PORT sessions to the > > > > remote N_PORT...". > > > > > The translation MAY be discarded when a PLOGO logout message > > > > > that terminates the N_PORT session is detected. This is > > > > what I meant > > > > > in the original message. Sorry about the confusion. > > > > > > > > And I was a little sloppy in my language. By "dropping the > > > > N_PORT session", > > > > I meant the use of PLOGO to terminate the session. After > > > > doing this, a > > > > Fibre Channel device can use PLOGI to initiate a new > > > > connection to the same > > > > target without checking with the fabric nameserver, because > > > > it can rely on > > > > state change notification to tell it if a N_PORT ID has > > > > become invalid; > > > > there > > > > are FC devices that behave in this fashion. Probing a > > remote N_PORT > > > > to see if a session is alive won't catch this situation; > > > > something like a > > > > state change notification to force the originator to go back > > > > to the fabric > > > > nameserver is necessary. Reliance on state change > > > > notification is also > > > > why the WWPN of the target may not be checked when the > new session > > > > is set up. > > > > > > > > > > It's impossible to guarantee that the information returned in > > > response to a > > > directory query won't silently become stale -- in spite of > > > state change > > > notification. > > > > > > Therefore, the requestor must assume that the data returned > > > may be stale by > > > the time it is used to establish a session. It's the > > > responsibility of the > > > session initiator to confirm that the desired endpoint was > > reached at > > > session creation. > > > > > > I'd venture to say that this is true of any directory-driven > > > storage network > > > in general. > > > > > > > > > > > > Charles > > > Charles Monia > > > Senior Technology Consultant > > > Nishan Systems > > > email: cmonia@nishansystems.com > > > voice: (408) 519-3986 > > > fax: (408) 435-8385 > > > > > > From owner-ips@ece.cmu.edu Fri Jan 5 00:02:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA23286 for ; Fri, 5 Jan 2001 00:02:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f052rkA27779 for ips-outgoing; Thu, 4 Jan 2001 21:53:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from falcon.vixel.com (mail.vixel.com [207.115.190.210]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f052ob027715 for ; Thu, 4 Jan 2001 21:50:37 -0500 (EST) Received: from vixel.com ([192.168.1.152]) by falcon.vixel.com (Netscape Messaging Server 4.15) with ESMTP id G6O58A00.VDO for ; Thu, 4 Jan 2001 18:50:34 -0800 Message-ID: <3A5535C0.A6EDCB35@vixel.com> Date: Thu, 04 Jan 2001 18:47:28 -0800 From: "Ken Hirata" Organization: Vixel Corporation X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en,pdf MIME-Version: 1.0 To: "Ips@Ece. Cmu. Edu" Subject: Re: iFCP as an IP Storage Work Item References: <3A54C868.D86AC90F@ebay.sun.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, Why do you want to standardize a common encapsulation protocol for FCIP and iFCP if their semantics and behavior are completely different? Would you want tunneling protocol implementations to also augment certain ELSs even though it isn't necessary for tunneling protocol operation? If a common encapsulation protocol was defined, I believe a negotiation protocol would be necessary to distinguish between usage as a gateway or tunneling protocol. After that behavior would be completely different; the tunneling protocol doesn't need to augment ELSs, resolve Fibre Channel Address Identifiers to IP/N_Port IDs, or do anything else the gateway protocol would do. Ken David Robinson wrote: > I am throwing in my hat to have the WG support both iFCP and > FCIP. From a business/customer perspective I find believable > markets for both approaches (both of which are still speculation). > >From a technical perspective they are similar enough that having > one standard mechanism and one different defacto mechanism will > cause more problems than it solves. > > It is clear that the semantic meaning of the two proposed protocols > cannot be merged as they do not operate in the same plane of > traditional stacks. However, from my reading of the two proposals > the encapsulation mechanisms are remarkably similar even though > their semantics are diverge significantly. What I have started > (before my two weeks away from work, ahhhh...) but haven't > yet finished, is an investigation on standardizing a common > encapsulation protocol for FC over TCP/IP. The the FCIP vs iFCP > becomes a higher level interpretation of the semantics of the > bits instead of also having two completely different stacks. > > -David -- Kenneth Hirata Vixel Corporation Irvine, CA 92618 Phone: (949) 450-6100 Email: khirata@vixel.com From owner-ips@ece.cmu.edu Fri Jan 5 00:03:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA23301 for ; Fri, 5 Jan 2001 00:03:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f053DjV28143 for ips-outgoing; Thu, 4 Jan 2001 22:13:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from falcon.vixel.com (mail.vixel.com [207.115.190.210]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0530O027896 for ; Thu, 4 Jan 2001 22:00:24 -0500 (EST) Received: from vixel.com ([192.168.1.152]) by falcon.vixel.com (Netscape Messaging Server 4.15) with ESMTP id G6O5OL00.UCM for ; Thu, 4 Jan 2001 19:00:21 -0800 Message-ID: <3A55380B.C21CA7AC@vixel.com> Date: Thu, 04 Jan 2001 18:57:15 -0800 From: "Ken Hirata" Organization: Vixel Corporation X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en,pdf MIME-Version: 1.0 To: "Ips@Ece. Cmu. Edu" Subject: Re: iFCP as an IP Storage Work Item References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Charles, There's one item with your reply that I have a question about. With regards to processing overhead, do you agree that FCIP as a tunneling protocol would not have to look for ELSs in the Fibre Channel datastream as iFCP must do? I'm not arguing that iFCP should not be a IP Storage work item; I'm concerned about the attempts to merge FCIP and iFCP. Ken Charles Monia wrote: > Hi Venkat: > > The issues you raise have been discussed at length before (but apparently > not to your satisfaction). That aside, the general thrust of this note seems > to reflect business rather than technical concerns. > > Within the parameters of the WG charter and IETF ground rules, it's my view > that the ips wg needs to be a vehicle for bringing high-quality solutions to > the marketplace, not obstructing technologies because they are inconvenient > or conflict with someone's business model. > > See other responses below. > > Charles > > > -----Original Message----- > > From: Venkat Rangan [mailto:venkat@rhapsodynetworks.com] > > Sent: Thursday, January 04, 2001 9:05 AM > > To: David Peterson; Ips@Ece. Cmu. Edu > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > We are also not in support of picking up iFCP as a IP Storage > > Work Item. snip . . . > > 6. iFCP addds processing overhead to frames > > > > In contrast to iSCSI, iFCP requires examination of every frame and > > substitution of addresses at both iFCP Portals. This has the > > potential to > > add unwanted latencies as well as consume processing overhead > > on gateways. > > Such assertions about processing overhead are totally without foundation or > substance. > > > Given this, it would be hard to compete with cut-through routing based > > products in a pure FC or IP based storage network. snip . . . -- Kenneth Hirata Vixel Corporation Irvine, CA 92618 Phone: (949) 450-6100 Email: khirata@vixel.com From owner-ips@ece.cmu.edu Fri Jan 5 00:03:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA23312 for ; Fri, 5 Jan 2001 00:03:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f052riR27775 for ips-outgoing; Thu, 4 Jan 2001 21:53:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f052n9027663 for ; Thu, 4 Jan 2001 21:49:09 -0500 (EST) Received: from redfish.adaptec.com (redfish.adaptec.com [162.62.50.11]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id SAA09385; Thu, 4 Jan 2001 18:49:04 -0800 (PST) Received: from aimexc03.corp.adaptec.com ([162.62.62.43]) by redfish.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id SAA00537; Thu, 4 Jan 2001 18:42:53 -0800 (PST) Received: by aimexc03.corp.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Thu, 4 Jan 2001 18:49:03 -0800 Message-ID: <268DBFF7D2A3D411A37400D0B72E345F01265497@aimexc03.corp.adaptec.com> From: "Lee, WanHuiHendra" To: "'julian_satran@il.ibm.com'" Cc: ips@ece.cmu.edu Subject: iSCSI: Byte padding requirement question Date: Thu, 4 Jan 2001 18:49:00 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, In the length field description (section 2.2.3 Length): "... The length field accounts for proper iSCSI PDU content; whatever padding is required to reach a 4 byte boundary in the TCP stream is implied by the protocol but not accounted for in the length field." Is this a side effect requirement due to the added "framing using marker at fixed interval" mechanism ? If yes, since framing is a negotiated feature, would it make sense to make the byte padding only a requirement if framing is enabled ? If this is not due to the framing feature, could you let us know why it is a requirement (I did not see it in previous drafts) ? Thanks, Wan-Hui From owner-ips@ece.cmu.edu Fri Jan 5 00:46:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA23601 for ; Fri, 5 Jan 2001 00:46:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f053Sjq28398 for ips-outgoing; Thu, 4 Jan 2001 22:28:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f053SD028391 for ; Thu, 4 Jan 2001 22:28:13 -0500 (EST) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id TAA23137 for ; Thu, 4 Jan 2001 19:28:12 -0800 (PST) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.142.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id TAA22519 for ; Thu, 4 Jan 2001 19:28:12 -0800 (PST) Received: from ebay.sun.com by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id TAA23370; Thu, 4 Jan 2001 19:28:11 -0800 (PST) Message-ID: <3A553F2E.FC01C978@ebay.sun.com> Date: Thu, 04 Jan 2001 19:27:42 -0800 From: David Robinson X-Mailer: Mozilla 4.74 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: "Ips@Ece. Cmu. Edu" Subject: Re: iFCP as an IP Storage Work Item References: <3A54C868.D86AC90F@ebay.sun.com> <3A5535C0.A6EDCB35@vixel.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Ken Hirata wrote: > Why do you want to standardize a common encapsulation protocol > for FCIP and iFCP if their semantics and behavior are completely > different? Would you want tunneling protocol implementations to > also augment certain ELSs even though it isn't necessary for tunneling > protocol operation? If I were to build hardware that either assisted or completely processed both iFCP and FCIP, it sure would be easier to do header parsing and other common processing if there was just one format. > If a common encapsulation protocol was defined, I believe a > negotiation protocol would be necessary to distinguish between > usage as a gateway or tunneling protocol. Yes, either negotiation of a flag bit in the encapsulating header used to choose which algorithm to use. I just don't understand why you would want to make them different. -David From owner-ips@ece.cmu.edu Fri Jan 5 06:29:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA08694 for ; Fri, 5 Jan 2001 06:29:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f058pqS04054 for ips-outgoing; Fri, 5 Jan 2001 03:51:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f058pe004047 for ; Fri, 5 Jan 2001 03:51:41 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id JAA146002 for ; Fri, 5 Jan 2001 09:51:34 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id JAA275758 for ; Fri, 5 Jan 2001 09:51:34 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569CB.0030A80A ; Fri, 5 Jan 2001 09:51:27 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 5 Jan 2001 10:46:11 +0200 Subject: Re: iSCSI: Byte padding requirement question Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Wan-Hui, Yes it is a side effect of the marker and we could have made it optional but it ends up being messy if you admit non-symmetric situations in which you send markers but do not receive them or viceversa. As padding does not practically add too much I thought it would be simpler for the implementers to have it always (and I've resisted previous requests to include padding!). Regards, Julo "Lee, WanHuiHendra" on 05/01/2001 04:49:00 Please respond to "Lee, WanHuiHendra" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: iSCSI: Byte padding requirement question Julian, In the length field description (section 2.2.3 Length): "... The length field accounts for proper iSCSI PDU content; whatever padding is required to reach a 4 byte boundary in the TCP stream is implied by the protocol but not accounted for in the length field." Is this a side effect requirement due to the added "framing using marker at fixed interval" mechanism ? If yes, since framing is a negotiated feature, would it make sense to make the byte padding only a requirement if framing is enabled ? If this is not due to the framing feature, could you let us know why it is a requirement (I did not see it in previous drafts) ? Thanks, Wan-Hui From owner-ips@ece.cmu.edu Fri Jan 5 07:03:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA08797 for ; Fri, 5 Jan 2001 07:03:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f059srO13753 for ips-outgoing; Fri, 5 Jan 2001 04:54:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f059rr013739 for ; Fri, 5 Jan 2001 04:53:53 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA37038 for ; Fri, 5 Jan 2001 10:53:48 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id KAA96286 for ; Fri, 5 Jan 2001 10:53:48 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569CB.00365C79 ; Fri, 5 Jan 2001 10:53:46 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 5 Jan 2001 11:49:37 +0200 Subject: Re: new iSCSI draft - 02.txt Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Glen, I can hardly agree with your assumption that CRCs will have to be recomputed in the end-nodes as the only means for real protection. Our design assumptions was that we don't want to mandate that - it is impractical at high speed. The end node will get data fit to its quality. If it has a bad DMA it will get bad data from local disk too! You are right about the major objection with regard to byte stuffing being copy - and I don't see how COBS is better than other schemes. An iSCSI hardware adapter could do it efficiently while using a plain (even accelerated) TCP adapter will make it very inefficient. The marker scheme suggested does not require any TCP internal knowledge - sequence numbers are not used and markers are relative. Yes you have to count every byte but you can do it independent of the TCP count. Buffer copies become either rare or are eliminated if your DMA mechanism is sophisticated enough. Julo Glen Turner on 04/01/2001 07:01:31 Please respond to Glen Turner To: sukha_ghosh@ivivity.com cc: "Ips@Ece. Cmu. Edu" Subject: Re: new iSCSI draft - 02.txt Sukha Ghosh wrote: > > Also I like the idea of fixed length PDU better than a new TCP > option indicating message boundary. A fixed length PDU doesn't solve the fundamental problem. In a stream of bytes you still can't tell where the PDU starts unless you assume PDUs pack nicely into packets and frames and the iSCSI process has special knowledge of the data stream (such as TCP options). Without altering TCP (which this IESG is opposed to) the options are an adaption layer containing either a flag byte and escapes (as in SLIP) or a counting scheme (such as computing a CRC which becomes 0 when the PDU is complete). Objections that byte stuffing to mark the PDU boundary discourages host-based implementations are somewhat difficult to understand. Firstly, the feature can, and should, be negotiatable in both directions. A host processor implementation can chose not to negotiate the reception of packets with byte stuffing, at the cost of incurring TCP head-on-line blocking due to late packets. Secondly, the decision to include a iSCSI-level checksum (to avoid surprisingly common corruption by DMA chips and the like, see the SIGCOMM2000 Proceedings) means that the data has to be scanned by the CPU anyway. Any other implementation of the iSCSI-level checksum gives no protection. The real objection is that removing the byte-stuffing requires a buffer copy, not just a scan. The second objection is that the overhead from SLIP-style byte stuffing has a pathological case. The counting schemes do not need a buffer copy (as you can arbitrarily drop, say, the last 8 octets that were required to drive the CRC to 0 and retreive the original data). However, these schemes are unavoidably computationally expensive, perhaps more so than a buffer copy (although RISC processors do about 100 instructions per memory cycle, so this assumption needs testing). A modification of "Consistent Overhead Byte Stuffing" (IEEE/ACM Transactions on Networking, Vol 7, No 2, 1999) seems to hold a neat solution to the pathological case of SLIP. It offers a neat way of identifying packet boundaries that has a bound on the growth in the packet size. However, like SLIP it also requires that a buffer copy occur to undo the detection of the packet boundary. If the WG find it best to deal with TCP as a stream of octets with zero knowledge of the underlying packet boundaries, options and sequence numbers then an adaption layer for TCP with a COBS-based scheme to identify packet boundaries seems to have the most promise. Regards, Glen -- Glen Turner Network Engineer (08) 8303 3936 Australian Academic and Research Network glen.turner@aarnet.edu.au http://www.aarnet.edu.au/ -- The revolution will not be televised, it will be digitised From owner-ips@ece.cmu.edu Fri Jan 5 13:50:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA19513 for ; Fri, 5 Jan 2001 13:50:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f05Fi3820540 for ips-outgoing; Fri, 5 Jan 2001 10:44:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ludwig.troikanetworks.com (host03.troikanetworks.com [12.31.172.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f05Fh8020508 for ; Fri, 5 Jan 2001 10:43:08 -0500 (EST) Received: by host03.troikanetworks.com with Internet Mail Service (5.5.2653.19) id ; Fri, 5 Jan 2001 07:43:08 -0800 Message-ID: From: Wayland Jeong To: "'Ken Hirata'" , "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 5 Jan 2001 07:43:06 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Yes. I am a bit curious as well about the merit of having a common encapsulation method. If the thought is to standardize on a common TCP encapsulation method (i.e. hardware assist mechanisms such as TCP options for PDU alignment or a periodic marker in the byte stream) then I can see the value. But if the thought is to have a common encapsulation layer below TCP (i.e. iFCP headers) then this doesn't make any sense to me. The FCIP draft as it stands doesn't even require such a layer for tunneling FC frames. Such a layer would be pure overhead for FCIP. -----Original Message----- From: Ken Hirata [mailto:Ken.Hirata@Vixel.com] Sent: Thursday, January 04, 2001 6:47 PM To: Ips@Ece. Cmu. Edu Subject: Re: iFCP as an IP Storage Work Item David, Why do you want to standardize a common encapsulation protocol for FCIP and iFCP if their semantics and behavior are completely different? Would you want tunneling protocol implementations to also augment certain ELSs even though it isn't necessary for tunneling protocol operation? If a common encapsulation protocol was defined, I believe a negotiation protocol would be necessary to distinguish between usage as a gateway or tunneling protocol. After that behavior would be completely different; the tunneling protocol doesn't need to augment ELSs, resolve Fibre Channel Address Identifiers to IP/N_Port IDs, or do anything else the gateway protocol would do. Ken David Robinson wrote: > I am throwing in my hat to have the WG support both iFCP and > FCIP. From a business/customer perspective I find believable > markets for both approaches (both of which are still speculation). > >From a technical perspective they are similar enough that having > one standard mechanism and one different defacto mechanism will > cause more problems than it solves. > > It is clear that the semantic meaning of the two proposed protocols > cannot be merged as they do not operate in the same plane of > traditional stacks. However, from my reading of the two proposals > the encapsulation mechanisms are remarkably similar even though > their semantics are diverge significantly. What I have started > (before my two weeks away from work, ahhhh...) but haven't > yet finished, is an investigation on standardizing a common > encapsulation protocol for FC over TCP/IP. The the FCIP vs iFCP > becomes a higher level interpretation of the semantics of the > bits instead of also having two completely different stacks. > > -David -- Kenneth Hirata Vixel Corporation Irvine, CA 92618 Phone: (949) 450-6100 Email: khirata@vixel.com From owner-ips@ece.cmu.edu Fri Jan 5 17:23:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA24426 for ; Fri, 5 Jan 2001 17:23:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f05Jq4G27168 for ips-outgoing; Fri, 5 Jan 2001 14:52:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f05JpM027147 for ; Fri, 5 Jan 2001 14:51:22 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Fri, 5 Jan 2001 12:01:29 -0800 Message-ID: From: Charles Monia To: "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 5 Jan 2001 11:39:00 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: David Robinson [mailto:David.Robinson@EBay.Sun.COM] > Sent: Thursday, January 04, 2001 7:28 PM > To: Ips@Ece. Cmu. Edu > Subject: Re: iFCP as an IP Storage Work Item > > > Ken Hirata wrote: > > Why do you want to standardize a common encapsulation protocol > > for FCIP and iFCP if their semantics and behavior are completely > > different? Would you want tunneling protocol implementations to > > also augment certain ELSs even though it isn't necessary > for tunneling > > protocol operation? > > If I were to build hardware that either assisted or completely > processed both iFCP and FCIP, it sure would be easier to do > header parsing and other common processing if there was just > one format. > > > If a common encapsulation protocol was defined, I believe a > > negotiation protocol would be necessary to distinguish between > > usage as a gateway or tunneling protocol. > > Yes, either negotiation of a flag bit in the encapsulating > header used to choose which algorithm to use. > Hi David and Ken: I agree that a common encapsulation may make it marginally easier to build a multi-protocol box as well as having other benefits. However, from the above, I'm concerned that some might infer that multi-protocol support should be a requirement. Just to be clear on this point: While I'm all for doing things that encourage commonality (where it makes sense technically) I feel that a standard ought not to needlessly burden a product with supporting both architectures. With regard to 'negotiation', I believe such a handshake is unnecessary and undesirable. In a real system, I can't see a scenario where it buys anything to make this dynamic. As I see it, these choices are cast in concrete when the SAN is implemented and aren't going to change from day to day. Also, for hardwired boxes that only support one protocol, it simply adds complexity. If someone wants to build a multi-protocol box, I believe they'd be better off making this statically settable. Charles From owner-ips@ece.cmu.edu Fri Jan 5 18:20:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA25339 for ; Fri, 5 Jan 2001 18:20:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f05Kx6i29192 for ips-outgoing; Fri, 5 Jan 2001 15:59:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ausxc07.us.dell.com (ausxc07.us.dell.com [143.166.99.215]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f05KwA029166 for ; Fri, 5 Jan 2001 15:58:10 -0500 (EST) Received: by ausxc07.us.dell.com with Internet Mail Service (5.5.2650.21) id ; Fri, 5 Jan 2001 14:57:37 -0600 Message-ID: From: Dan_McConnell@Dell.com To: ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 5 Jan 2001 14:57:35 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Regardless of native end to end IP SCSI transports, it is clear that we need to address the existing "legacy" Fibre Channel devices (I still hesitate calling Fibre Channel legacy, but that's a different story). So, there is an obvious need for bridging the gap between current FC devices and the emerging IP storage networking realm. On one hand we have a standard tunneling protocol (FCIP) which would require the use of 1 FC switch port and 1 FCIP "link extender" port on each side for each remote tunnel desired. On the other hand the, iFCP equivalent would only require the use of 1 port on the iFCP gateway (not per tunnel) which has the ability to route sessions individually to the appropriate end iFCP gateway. Assuming that the E-port and L-port problems can be fixed, which supposedly should be no problem, iFCP definitely seems to be a viable solution. Whether iFCP or FCIP is the functionality winner or not, I would say that it would be a loss for the IPS working group to discard iFCP. Also, the iSNS protocol, compatible with both iSCSI and iFCP, has been accepted as a base for the iSCSI name service and it seems to me that using iFCP would provide a perfect match for future native iSCSI solutions to incorporate legacy FC devices over IP. While FCIP does address the issue of extending a FC link over IP, I believe that iFCP has added ip networking functionality and flexibility that is worth while for the IPS working group to incorporate. ...that's my 20 centavos..... Dan McConnell Storage Systems Group Storage Architecture and Technology Evaluation Dan_McConnell@Dell.com From owner-ips@ece.cmu.edu Fri Jan 5 19:11:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA25799 for ; Fri, 5 Jan 2001 19:11:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f05Ld7d00331 for ips-outgoing; Fri, 5 Jan 2001 16:39:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f05LcO000313 for ; Fri, 5 Jan 2001 16:38:24 -0500 (EST) Received: from sponge.cisco.com (sponge.cisco.com [171.71.61.25]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id NAA15455; Fri, 5 Jan 2001 13:38:25 -0800 (PST) Received: from DAPW2K (dhcp-161-44-68-178.cisco.com [161.44.68.178]) by sponge.cisco.com (Mirapoint) with SMTP id AES21743; Fri, 5 Jan 2001 15:38:17 -0600 (CST) From: "David Peterson" To: "Matt Wakeley" , "IPS Reflector" Subject: RE: new iSCSI draft - 02.txt Date: Fri, 5 Jan 2001 15:40:07 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: <3A53D012.D6A20748@agilent.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit The problem is: at that layer there is simply not a byte stream. The data is contained in socket buffers which are most likely mbuf clusters for most implementations. To "simply insert a few extra bytes" requires mbuf manipulation. Mbuf manipulation can processing intensive. Depending on how the mbuf data is packaged a small DMA operation may occur for these few extra bytes also. Dave -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Matt Wakeley Sent: Wednesday, January 03, 2001 7:21 PM To: IPS Reflector Subject: Re: new iSCSI draft - 02.txt David Peterson wrote: > > I believe there was agreement to remove the Urgent-Pointer framing mechanism > but don't recall any agreement to replace it with an in-stream marker. For a > software implementation it would be hard to support this type of framing > mechanism. I believe a TCP option indicating the message boundry or a > fixed-length PDU at a granularity to minimize overhead are much better > solutions and are workable for both software and hardware implementations. I > have not seen an agenda but I would hope this issue would be discussed at > the upcoming meeting in Orlando. > Dave What is so difficult for software to insert a few extra bytes in the byte stream? It's simply a layering problem: iSCSI layer <-> marker layer <-> TCP Normally, the marker layer simply transfers the bytes from the iSCSI layer to the TCP. Every X number of bytes, the marker layer inserts the marker into the byte stream. Since your software will probably not benefit from the receipt of the markers, it would negotiate not to receive the markers. It would only send markers *IF* the remote node requested them to be sent. -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Fri Jan 5 20:06:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA26250 for ; Fri, 5 Jan 2001 20:06:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f05N18d02625 for ips-outgoing; Fri, 5 Jan 2001 18:01:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f05N0c002613 for ; Fri, 5 Jan 2001 18:00:39 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f060Bw017499; Fri, 5 Jan 2001 16:11:59 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Charles Monia" , "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 5 Jan 2001 14:59:27 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Charles, I disagree with this assumption about rigidity of protocols. Using a common initialization field would allow node handling protocols to be confirmed. Initialization should also include a version number or option field for the encapsulation where this version or option be negotiated. If the node handling protocol, separate from the encapsulation specification, could be identified, then there would be fewer strange events if someone mis-configured IP ports. I am not a fan of using text strings to do this function and would prefer IANA defined values for rigid structures. In addition to indicating an encapsulation version or options designator, include the node type designator to signify how the node is handled to avoid strange failures. As a gateway, tunnel or perhaps as an end device in perhaps the distant future, the type of node protocols may vary to meet different needs. The lion's share of the work however could be covered within the encapsulation specifications and clever means of handling the node could be isolated from these details. It does not mean that in the end all node protocols will use identical encapsulation, but only that, as much as possible, encapsulation is kept common and the effort to encapsulate is seen as a separate task to that of handling the frame at the end point. I would hope that a transport could be developed that would not care how the node was handled and where this node handling function is defined within a separate process. It seems like a fairly clean separation of encapsulation and node handling that would help foster support for these markets especially if a few devices allow a wide range of possible node handling scenarios. I agree that IP fabric is more extensible to that of FC, however even with that assumption, the final goal of fully incorporating IP fabric without altering the FCP structures has not been achieved by either of these current proposals. With that said, I would be in favor of developing a universal FC encapsulation transport as a WG proposal that would support both iFCP and FCIP. Doug > > Ken Hirata wrote: > > > Why do you want to standardize a common encapsulation protocol > > > for FCIP and iFCP if their semantics and behavior are completely > > > different? Would you want tunneling protocol implementations to > > > also augment certain ELSs even though it isn't necessary > > for tunneling > > > protocol operation? > > > > If I were to build hardware that either assisted or completely > > processed both iFCP and FCIP, it sure would be easier to do > > header parsing and other common processing if there was just > > one format. > > > > > If a common encapsulation protocol was defined, I believe a > > > negotiation protocol would be necessary to distinguish between > > > usage as a gateway or tunneling protocol. > > > > Yes, either negotiation of a flag bit in the encapsulating > > header used to choose which algorithm to use. > > > Hi David and Ken: > > I agree that a common encapsulation may make it marginally easier > to build a > multi-protocol box as well as having other benefits. However, from the > above, I'm concerned that some might infer that multi-protocol support > should be a requirement. Just to be clear on this point: While > I'm all for > doing things that encourage commonality (where it makes sense > technically) I > feel that a standard ought not to needlessly burden a product with > supporting both architectures. > > With regard to 'negotiation', I believe such a handshake is > unnecessary and > undesirable. In a real system, I can't see a scenario where it buys > anything to make this dynamic. As I see it, these choices are cast in > concrete when the SAN is implemented and aren't going to change > from day to > day. Also, for hardwired boxes that only support one protocol, it simply > adds complexity. If someone wants to build a multi-protocol box, > I believe > they'd be better off making this statically settable. > > Charles > From owner-ips@ece.cmu.edu Fri Jan 5 20:07:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA26280 for ; Fri, 5 Jan 2001 20:07:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f05MV9F01799 for ips-outgoing; Fri, 5 Jan 2001 17:31:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f05MUx001787 for ; Fri, 5 Jan 2001 17:30:59 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Fri, 5 Jan 2001 14:41:09 -0800 Message-ID: From: Charles Monia To: "Ips (E-mail)" Cc: "Ken Hirata (E-mail)" Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 5 Jan 2001 14:30:33 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Ken Hirata [mailto:Ken.Hirata@Vixel.com] > Sent: Thursday, January 04, 2001 6:57 PM > To: Ips@Ece. Cmu. Edu > Subject: Re: iFCP as an IP Storage Work Item > > > Charles, > > There's one item with your reply that I have a question > about. With regards > to processing overhead, do you agree that FCIP as a tunneling protocol > would not have to look for ELSs in the Fibre Channel > datastream as iFCP > must do? > Hi Ken: First of all, ELS's are well off the performance path, so there's no effect on performance. More to the point, we process ELS's and other frame traffic because doing so gives us a big payoff in cost-performance. A payoff that comes from allowing iFCP implementations to intelligently leverage IP technology. For example, maintaining a seperate session for each N_PORT login gives the gateway a handle for controlling the flows between individual FC storage devices, thus fully exploiting IP-based routing and traffic management. Concealing the Fibre Channel transport infrastructure behind the FC side of the gateway eliminates the need to for a two-plane routing scheme. IP routing is then unconstrained by and fully decoupled from FC routing. More importantly, doing so makes it easier to integrate otherwise incompatible FC infrastructures. While I'm at it, and since this issue has come up several times, I should point out that the NAT-like address translation we do reflects a design choice made to exploit IP scalability. We could have made the tradeoffs differently without affecting iFCP in any fundamental way. Charles Charles Monia Senior Technology Consultant Nishan Systems email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ece.cmu.edu Fri Jan 5 20:10:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA26340 for ; Fri, 5 Jan 2001 20:10:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f05MQ6Y01620 for ips-outgoing; Fri, 5 Jan 2001 17:26:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f05MQ1001616 for ; Fri, 5 Jan 2001 17:26:01 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id BAAE51236 for ; Fri, 5 Jan 2001 14:26:00 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id OAA27510; Fri, 5 Jan 2001 14:25:46 -0800 (PST) Message-ID: <3A564A40.9E1E13ED@cup.hp.com> Date: Fri, 05 Jan 2001 14:27:12 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Response Length and response Data fields in SCSI Resonse PDU. Content-Type: multipart/mixed; boundary="------------A7B142C9809541D36667515B" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------A7B142C9809541D36667515B Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian/All, I do not see any changes in the new iSCSI draft that explain the semantics of the Response Length and Response Data fields described in Section 2.4 (SCSI Response PDU). Are these intended to describe something similar to the Response Length and Response Data sent in the FCP_RSP PDU of Fibre Channel ? (SCSI-FCP-2. rev 05. Section 9.4.10, 9.4.11). Some clarification on this subject would help. Thanks, Santosh Rao --------------A7B142C9809541D36667515B Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------A7B142C9809541D36667515B-- From owner-ips@ece.cmu.edu Fri Jan 5 21:48:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA28083 for ; Fri, 5 Jan 2001 21:48:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f060G9L04398 for ips-outgoing; Fri, 5 Jan 2001 19:16:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f060F9004369 for ; Fri, 5 Jan 2001 19:15:09 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 5 Jan 2001 19:15:03 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07041013A1@corpmx9.isus.emc.com> From: Black_David@emc.com To: dotis@sanlight.net, cmonia@NishanSystems.com, ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 5 Jan 2001 19:15:02 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Before this goes any further ... the two of you may be in violent agreement. Charles objected to a "handshake"; I don't think this objection would preclude one end of a connection announcing the protocol that it intends to use so that the other end can cleanly and quickly terminate the connection if it can't or won't use that protocol (ideally with an error code to the other end indicating protocol incompatibility, so the misconfiguration becomes obvious). Doug's suggestion of IANA-allocated numbers for identifying protocols for this purpose is reasonable. As to the virtues of being able to speak more than one protocol on a port, Fibre Channel provides examples of where this sort of thing has been added after the initial specifications were done, so I wouldn't dismiss this out of hand. --David > -----Original Message----- > From: Douglas Otis [SMTP:dotis@sanlight.net] > Sent: Friday, January 05, 2001 5:59 PM > To: Charles Monia; Ips@Ece. Cmu. Edu > Subject: RE: iFCP as an IP Storage Work Item > > Charles, > > I disagree with this assumption about rigidity of protocols. Using a > common > initialization field would allow node handling protocols to be confirmed. > Initialization should also include a version number or option field for > the > encapsulation where this version or option be negotiated. If the node > handling protocol, separate from the encapsulation specification, could be > identified, then there would be fewer strange events if someone > mis-configured IP ports. I am not a fan of using text strings to do this > function and would prefer IANA defined values for rigid structures. In > addition to indicating an encapsulation version or options designator, > include the node type designator to signify how the node is handled to > avoid > strange failures. > > As a gateway, tunnel or perhaps as an end device in perhaps the distant > future, the type of node protocols may vary to meet different needs. The > lion's share of the work however could be covered within the encapsulation > specifications and clever means of handling the node could be isolated > from > these details. It does not mean that in the end all node protocols will > use > identical encapsulation, but only that, as much as possible, encapsulation > is kept common and the effort to encapsulate is seen as a separate task to > that of handling the frame at the end point. I would hope that a > transport > could be developed that would not care how the node was handled and where > this node handling function is defined within a separate process. > > It seems like a fairly clean separation of encapsulation and node handling > that would help foster support for these markets especially if a few > devices > allow a wide range of possible node handling scenarios. I agree that IP > fabric is more extensible to that of FC, however even with that > assumption, > the final goal of fully incorporating IP fabric without altering the FCP > structures has not been achieved by either of these current proposals. > With > that said, I would be in favor of developing a universal FC encapsulation > transport as a WG proposal that would support both iFCP and FCIP. > > Doug > > > > Ken Hirata wrote: > > > > Why do you want to standardize a common encapsulation protocol > > > > for FCIP and iFCP if their semantics and behavior are completely > > > > different? Would you want tunneling protocol implementations to > > > > also augment certain ELSs even though it isn't necessary > > > for tunneling > > > > protocol operation? > > > > > > If I were to build hardware that either assisted or completely > > > processed both iFCP and FCIP, it sure would be easier to do > > > header parsing and other common processing if there was just > > > one format. > > > > > > > If a common encapsulation protocol was defined, I believe a > > > > negotiation protocol would be necessary to distinguish between > > > > usage as a gateway or tunneling protocol. > > > > > > Yes, either negotiation of a flag bit in the encapsulating > > > header used to choose which algorithm to use. > > > > > Hi David and Ken: > > > > I agree that a common encapsulation may make it marginally easier > > to build a > > multi-protocol box as well as having other benefits. However, from the > > above, I'm concerned that some might infer that multi-protocol support > > should be a requirement. Just to be clear on this point: While > > I'm all for > > doing things that encourage commonality (where it makes sense > > technically) I > > feel that a standard ought not to needlessly burden a product with > > supporting both architectures. > > > > With regard to 'negotiation', I believe such a handshake is > > unnecessary and > > undesirable. In a real system, I can't see a scenario where it buys > > anything to make this dynamic. As I see it, these choices are cast in > > concrete when the SAN is implemented and aren't going to change > > from day to > > day. Also, for hardwired boxes that only support one protocol, it simply > > adds complexity. If someone wants to build a multi-protocol box, > > I believe > > they'd be better off making this statically settable. > > > > Charles > > From owner-ips@ece.cmu.edu Fri Jan 5 21:52:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA28098 for ; Fri, 5 Jan 2001 21:52:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f05Nx9V04028 for ips-outgoing; Fri, 5 Jan 2001 18:59:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f05NwE004001 for ; Fri, 5 Jan 2001 18:58:14 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 5 Jan 2001 18:58:22 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07041013A0@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 5 Jan 2001 18:58:07 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > While I'm at it, and since this issue has come up several times, I should > point out that the NAT-like address translation we do reflects a design > choice made to exploit IP scalability. We could have made the tradeoffs > differently without affecting iFCP in any fundamental way. I'm not sure about that. An important consequence of the NAT-like address translation is that the allocation of the 24 bit addresses used for S_ID and D_ID need not be coordinated across the iFCP boxes. If two boxes happen to use the same address(es), the conflicts are resolved by: - Picking new addresses for the remote ports on the FC side of iFCP and translating appropriately. - Using the IP addresses on the IP side of iFCP to disambiguate which iFCP box is involved. If this address translation isn't done, the resulting need for coordination of address allocation strikes me as a "fundamental" change, especially given the fact that both FC fabrics and loops have their own ideas about how address allocation happens. --David From owner-ips@ece.cmu.edu Fri Jan 5 22:30:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA28594 for ; Fri, 5 Jan 2001 22:30:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f060tB705151 for ips-outgoing; Fri, 5 Jan 2001 19:55:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f060sp005131 for ; Fri, 5 Jan 2001 19:54:51 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Fri, 5 Jan 2001 17:05:02 -0800 Message-ID: From: Charles Monia To: Black_David@emc.com, dotis@sanlight.net, Charles Monia , ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 5 Jan 2001 16:54:26 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Friday, January 05, 2001 4:15 PM > To: dotis@sanlight.net; cmonia@nishansystems.com; ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > Before this goes any further ... the two of you may be > in violent agreement. Charles objected to a "handshake"; > I don't think this objection would preclude one end of > a connection announcing the protocol that it intends to > use so that the other end can cleanly and quickly > terminate the connection if it can't or won't use > that protocol (ideally with an error code to the > other end indicating protocol incompatibility, so > the misconfiguration becomes obvious). > Doug's > suggestion of IANA-allocated numbers for identifying > protocols for this purpose is reasonable. > > As to the virtues of being able to speak more than > one protocol on a port, Fibre Channel provides examples > of where this sort of thing has been added after the > initial specifications were done, so I wouldn't dismiss > this out of hand. > > --David Hi David: It seems to me that with or without a common encapsulation method, such boxes can be built 'now'. As I said earlier, a common encapsulation should make it easier, which is a good thing. So, what we're left with is the question of how such a box would shift gears. If an "in-band" method is easier, the simpler the handshake, the better. So an approach based on an IANA-assigned parameter seems reasonable to me. While this is a fruitful discussion, I'd be interested in hearing from the FCIP folks on this issue. Charles > > > > -----Original Message----- > > From: Douglas Otis [SMTP:dotis@sanlight.net] > > Sent: Friday, January 05, 2001 5:59 PM > > To: Charles Monia; Ips@Ece. Cmu. Edu > > Subject: RE: iFCP as an IP Storage Work Item > > > > Charles, > > > > I disagree with this assumption about rigidity of > protocols. Using a > > common > > initialization field would allow node handling protocols to > be confirmed. > > Initialization should also include a version number or > option field for > > the > > encapsulation where this version or option be negotiated. > If the node > > handling protocol, separate from the encapsulation > specification, could be > > identified, then there would be fewer strange events if someone > > mis-configured IP ports. I am not a fan of using text > strings to do this > > function and would prefer IANA defined values for rigid > structures. In > > addition to indicating an encapsulation version or options > designator, > > include the node type designator to signify how the node is > handled to > > avoid > > strange failures. > > > > As a gateway, tunnel or perhaps as an end device in perhaps > the distant > > future, the type of node protocols may vary to meet > different needs. The > > lion's share of the work however could be covered within > the encapsulation > > specifications and clever means of handling the node could > be isolated > > from > > these details. It does not mean that in the end all node > protocols will > > use > > identical encapsulation, but only that, as much as > possible, encapsulation > > is kept common and the effort to encapsulate is seen as a > separate task to > > that of handling the frame at the end point. I would hope that a > > transport > > could be developed that would not care how the node was > handled and where > > this node handling function is defined within a separate process. > > > > It seems like a fairly clean separation of encapsulation > and node handling > > that would help foster support for these markets especially if a few > > devices > > allow a wide range of possible node handling scenarios. I > agree that IP > > fabric is more extensible to that of FC, however even with that > > assumption, > > the final goal of fully incorporating IP fabric without > altering the FCP > > structures has not been achieved by either of these current > proposals. > > With > > that said, I would be in favor of developing a universal FC > encapsulation > > transport as a WG proposal that would support both iFCP and FCIP. > > > > Doug > > > > > > Ken Hirata wrote: > > > > > Why do you want to standardize a common encapsulation protocol > > > > > for FCIP and iFCP if their semantics and behavior are > completely > > > > > different? Would you want tunneling protocol > implementations to > > > > > also augment certain ELSs even though it isn't necessary > > > > for tunneling > > > > > protocol operation? > > > > > > > > If I were to build hardware that either assisted or completely > > > > processed both iFCP and FCIP, it sure would be easier to do > > > > header parsing and other common processing if there was just > > > > one format. > > > > > > > > > If a common encapsulation protocol was defined, I believe a > > > > > negotiation protocol would be necessary to distinguish between > > > > > usage as a gateway or tunneling protocol. > > > > > > > > Yes, either negotiation of a flag bit in the encapsulating > > > > header used to choose which algorithm to use. > > > > > > > Hi David and Ken: > > > > > > I agree that a common encapsulation may make it marginally easier > > > to build a > > > multi-protocol box as well as having other benefits. > However, from the > > > above, I'm concerned that some might infer that > multi-protocol support > > > should be a requirement. Just to be clear on this point: While > > > I'm all for > > > doing things that encourage commonality (where it makes sense > > > technically) I > > > feel that a standard ought not to needlessly burden a product with > > > supporting both architectures. > > > > > > With regard to 'negotiation', I believe such a handshake is > > > unnecessary and > > > undesirable. In a real system, I can't see a scenario > where it buys > > > anything to make this dynamic. As I see it, these > choices are cast in > > > concrete when the SAN is implemented and aren't going to change > > > from day to > > > day. Also, for hardwired boxes that only support one > protocol, it simply > > > adds complexity. If someone wants to build a multi-protocol box, > > > I believe > > > they'd be better off making this statically settable. > > > > > > Charles > > > > From owner-ips@ece.cmu.edu Sat Jan 6 00:07:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA00040 for ; Sat, 6 Jan 2001 00:07:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f062xDm07244 for ips-outgoing; Fri, 5 Jan 2001 21:59:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f062x7007238 for ; Fri, 5 Jan 2001 21:59:07 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id ZJ5LTQLD; Fri, 5 Jan 2001 18:59:06 -0800 From: "Y P Cheng" To: Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 5 Jan 2001 18:58:41 -0800 Message-ID: <002201c0778c$9326e360$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal In-Reply-To: <0F31E5C394DAD311B60C00E029101A07041013A1@corpmx9.isus.emc.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > Before this goes any further ... the two of you may be > in violent agreement. Charles objected to a "handshake"; > I don't think this objection would preclude one end of > a connection announcing the protocol that it intends to > use so that the other end can cleanly and quickly > terminate the connection if it can't or won't use > that protocol (ideally with an error code to the > other end indicating protocol incompatibility, so > the misconfiguration becomes obvious). Doug's > suggestion of IANA-allocated numbers for identifying > protocols for this purpose is reasonable. > > As to the virtues of being able to speak more than > one protocol on a port, Fibre Channel provides examples > of where this sort of thing has been added after the > initial specifications were done, so I wouldn't dismiss > this out of hand. > > --David I agree with David. While everyone thought that iFCP is just a gateway protocol, I do believe iFCP could be used by an HBA talking to a fibre channel device from an Ethernet connection. It puts together an FCP frame, adds a TCP/IP header, and out goes to the Ethernet wire. There is good reason that the same HBA may talk to another fibre channel device through an FCIP port. There would be different ways to create these connections. However, many fibre channel HBAs today support multiple protocols like FCP, IP, and VI at the same time. Therefore, it is possible for a future HBA to speak iFCP, FCIP, and even iSCSI on an Ethernet connection at the same time. While I don't know if this makes economical sense, but, technically, this is definitely possible. Y.P. Cheng, Connectcom Solutions. From owner-ips@ece.cmu.edu Sat Jan 6 00:11:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA00060 for ; Sat, 6 Jan 2001 00:11:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f063IEk07592 for ips-outgoing; Fri, 5 Jan 2001 22:18:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f063Hx007584 for ; Fri, 5 Jan 2001 22:17:59 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Fri, 5 Jan 2001 19:28:11 -0800 Message-ID: From: Charles Monia To: Black_David@emc.com Cc: ips@ece.cmu.edu, Charles Monia Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 5 Jan 2001 19:17:33 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Friday, January 05, 2001 3:58 PM > To: ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > > While I'm at it, and since this issue has come up several > times, I should > > point out that the NAT-like address translation we do > reflects a design > > choice made to exploit IP scalability. We could have made > the tradeoffs > > differently without affecting iFCP in any fundamental way. > > I'm not sure about that. An important consequence of the > NAT-like address > translation is that the allocation of the 24 bit addresses > used for S_ID and > D_ID > need not be coordinated across the iFCP boxes. If two boxes > happen to use > the same address(es), the conflicts are resolved by: > - Picking new addresses for the remote ports on the FC side > of iFCP and > translating appropriately. > - Using the IP addresses on the IP side of iFCP to disambiguate which > iFCP box is involved. > If this address translation isn't done, the resulting need > for coordination > of > address allocation strikes me as a "fundamental" change, > especially given > the fact that both FC fabrics and loops have their own ideas about how > address allocation happens. > Hi David: You are correct about the need to coordinate such address assigments across gateways. As you suggest, there are additional consequences stemming from the way N_PORT addresses are assigned on the FC side of the gateway. All of these can be handled by straighforward extensions to iFCP. Relative to the basic iFCP model for communications between N_PORTs, therefore, I don't consider these differences to be fundamental. Charles From owner-ips@ece.cmu.edu Sat Jan 6 03:58:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA14294 for ; Sat, 6 Jan 2001 03:58:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f066ji510882 for ips-outgoing; Sat, 6 Jan 2001 01:45:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f066jP010874 for ; Sat, 6 Jan 2001 01:45:25 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Fri, 5 Jan 2001 22:55:38 -0800 Message-ID: From: Joshua Tseng To: Y P Cheng , ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 5 Jan 2001 22:44:59 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > > I agree with David. While everyone thought that iFCP is just > a gateway > protocol, I do believe iFCP could be used by an HBA talking to a fibre > channel device from an Ethernet connection. It puts together > an FCP frame, > adds a TCP/IP header, and out goes to the Ethernet wire. > There is good > reason that the same HBA may talk to another fibre channel > device through an > FCIP port. There would be different ways to create these connections. <..rest deleted..> Sheesh...the Fibre Channel vendors are having a tough enough time getting one E_PORT to talk to another E_PORT. You are claiming that you can get an N_PORT to talk to an E_PORT. Let's get this straight here. iFCP encapsulates frames egressing N_PORTs. FCIP encapsulates frames egressing E_PORTs. Josh > > Y.P. Cheng, Connectcom Solutions. > From owner-ips@ece.cmu.edu Sat Jan 6 06:27:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA14911 for ; Sat, 6 Jan 2001 06:27:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f069ClW12884 for ips-outgoing; Sat, 6 Jan 2001 04:12:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f069CX012875 for ; Sat, 6 Jan 2001 04:12:33 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA15542 for ; Sat, 6 Jan 2001 10:12:26 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id KAA148064 for ; Sat, 6 Jan 2001 10:12:26 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569CC.00329302 ; Sat, 6 Jan 2001 10:12:24 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 6 Jan 2001 11:08:13 +0200 Subject: RE: new iSCSI draft - 02.txt Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dave, I think that a shim can be inserted between the socket calls and the TCP copy to mbuf or from mbuf. On many socket implementations this can be achieved by just changing the protocol "name" in the socket data structure and then have the shim revert it to the original. However we should not be very concerned about stacks that use a full-fledged mbuf structure and copy the data - their rates will be too low and they might as well reject the option. Julo "David Peterson" on 05/01/2001 23:40:07 Please respond to "David Peterson" To: "Matt Wakeley" , "IPS Reflector" cc: Subject: RE: new iSCSI draft - 02.txt The problem is: at that layer there is simply not a byte stream. The data is contained in socket buffers which are most likely mbuf clusters for most implementations. To "simply insert a few extra bytes" requires mbuf manipulation. Mbuf manipulation can processing intensive. Depending on how the mbuf data is packaged a small DMA operation may occur for these few extra bytes also. Dave -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Matt Wakeley Sent: Wednesday, January 03, 2001 7:21 PM To: IPS Reflector Subject: Re: new iSCSI draft - 02.txt David Peterson wrote: > > I believe there was agreement to remove the Urgent-Pointer framing mechanism > but don't recall any agreement to replace it with an in-stream marker. For a > software implementation it would be hard to support this type of framing > mechanism. I believe a TCP option indicating the message boundry or a > fixed-length PDU at a granularity to minimize overhead are much better > solutions and are workable for both software and hardware implementations. I > have not seen an agenda but I would hope this issue would be discussed at > the upcoming meeting in Orlando. > Dave What is so difficult for software to insert a few extra bytes in the byte stream? It's simply a layering problem: iSCSI layer <-> marker layer <-> TCP Normally, the marker layer simply transfers the bytes from the iSCSI layer to the TCP. Every X number of bytes, the marker layer inserts the marker into the byte stream. Since your software will probably not benefit from the receipt of the markers, it would negotiate not to receive the markers. It would only send markers *IF* the remote node requested them to be sent. -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Sat Jan 6 06:35:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA14956 for ; Sat, 6 Jan 2001 06:35:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f069Hm412953 for ips-outgoing; Sat, 6 Jan 2001 04:17:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f069HO012944 for ; Sat, 6 Jan 2001 04:17:24 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA195596 for ; Sat, 6 Jan 2001 10:17:18 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id KAA174134 for ; Sat, 6 Jan 2001 10:17:17 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569CC.003302CA ; Sat, 6 Jan 2001 10:17:10 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 6 Jan 2001 11:12:57 +0200 Subject: Re: iSCSI : Response Length and response Data fields in SCSI Resonse PDU. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, I am still working on it (and other items). The main reason I got 02 out was to get a discussion going on the "new" items - the marker, the recovery model for digest errors and for iSCSI format errors. The rest will be out soon. Julo Santosh Rao on 06/01/2001 00:27:12 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iSCSI : Response Length and response Data fields in SCSI Resonse PDU. Julian/All, I do not see any changes in the new iSCSI draft that explain the semantics of the Response Length and Response Data fields described in Section 2.4 (SCSI Response PDU). Are these intended to describe something similar to the Response Length and Response Data sent in the FCP_RSP PDU of Fibre Channel ? (SCSI-FCP-2. rev 05. Section 9.4.10, 9.4.11). Some clarification on this subject would help. Thanks, Santosh Rao - santoshr.vcf From owner-ips@ece.cmu.edu Sat Jan 6 13:05:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA16627 for ; Sat, 6 Jan 2001 13:05:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f06Fd4129224 for ips-outgoing; Sat, 6 Jan 2001 10:39:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from barry.mail.mindspring.net (barry.mail.mindspring.net [207.69.200.25]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f06FcG029207 for ; Sat, 6 Jan 2001 10:38:16 -0500 (EST) Received: from dad.ieee.org (user-vcaun3c.dsl.mindspring.com [216.175.92.108]) by barry.mail.mindspring.net (8.9.3/8.8.5) with ESMTP id KAA27089 for ; Sat, 6 Jan 2001 10:38:06 -0500 (EST) Message-Id: <4.3.2.7.2.20010106073511.00d037d0@pop.mindspring.com> X-Sender: ljlamers@pop.mindspring.com X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Date: Sat, 06 Jan 2001 07:38:18 -0800 To: ips@ece.cmu.edu From: "Lawrence J. Lamers" Subject: RE: iFCP as an IP Storage Work Item Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk The market analysis indicates that a fibre channel over internet protocol approach, such as FCIP, provides the solution needed to connect storage networks over Metropolitan and Wide Area Networks for the foreseeable future. The FCIP proposal still needs further work in order to provide the necessary features for it to have long term viability, but we see no reason that these features can not be added to this approach. The long term solution for native attachment of storage devices to an IP network is iSCSI, a solution that conceptually fits with the present operating system/software driver stacks, has broad industry support, and is in process of becoming a standard within IETF. ISCSI is a protocol to do SCSI commands and task management over Ethernet, much as FCP does for fibre channel, SBP does for 1394, and SIP does for SPI. Layering one protocol on top of another to achieve SCSI functionality seems to complicated an approach that will be difficult to debug and certify, a field day for Murphy's law. Certain market niches may be served by other solutions, but it is believed that the complementary solutions of FCIP and iSCSI are what should be standardized and promoted to the industry. This avoids confusion and divisiveness and gets the world to a storage networked future the fastest, with the lowest cost and the least un-productive effort. It is my belief based on the technical merits and market needs that the IETF should continue in its present course and support the FCoverIP work item and the iSCSI work item and not add additional work items that these solutions already address. Regards, Lawrence J. Lamers Principal Engineer Advanced Technology Group SAN Valley Systems, Inc. 408-234-0071 ljlamers@sanvalley.com ljlamers@ieee.org From owner-ips@ece.cmu.edu Sat Jan 6 14:20:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA17068 for ; Sat, 6 Jan 2001 14:20:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f06H1wR00637 for ips-outgoing; Sat, 6 Jan 2001 12:01:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f06H16000624 for ; Sat, 6 Jan 2001 12:01:06 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Sat, 6 Jan 2001 09:11:25 -0800 Message-ID: From: Charles Monia To: Y P Cheng Cc: ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Sat, 6 Jan 2001 09:00:37 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > I agree with David. While everyone thought that iFCP is just > a gateway > protocol, I do believe iFCP could be used by an HBA talking to a fibre > channel device from an Ethernet connection. It puts together > an FCP frame, > adds a TCP/IP header, and out goes to the Ethernet wire. Hi Y P: I think it's important to be clear on one thing. While a product can be built from a common encapsulation method, it is NOT iFCP. Such a product is basically leveraging a commmon envelope that encloses different messages. The sender and receiver reading the contents are different entities that speak different languages. In iFCP, the communicating objects are Fibre Channel storage devices and host adapters (N_PORTS) addressed through iFCP gateways. In FCIP, the communicating entities are FC fabric elements hauling frame traffic (backbone switches and FC switch elements). Anyhow, as I stated earlier, this is a useful but somewhat speculative conversation, since the tunneling folks have yet to be heard from on this topic. Charles > -----Original Message----- > From: Y P Cheng [mailto:ycheng@advansys.com] > Sent: Friday, January 05, 2001 6:59 PM > To: ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > > Before this goes any further ... the two of you may be > > in violent agreement. Charles objected to a "handshake"; > > I don't think this objection would preclude one end of > > a connection announcing the protocol that it intends to > > use so that the other end can cleanly and quickly > > terminate the connection if it can't or won't use > > that protocol (ideally with an error code to the > > other end indicating protocol incompatibility, so > > the misconfiguration becomes obvious). Doug's > > suggestion of IANA-allocated numbers for identifying > > protocols for this purpose is reasonable. > > > > As to the virtues of being able to speak more than > > one protocol on a port, Fibre Channel provides examples > > of where this sort of thing has been added after the > > initial specifications were done, so I wouldn't dismiss > > this out of hand. > > > > --David > > I agree with David. While everyone thought that iFCP is just > a gateway > protocol, I do believe iFCP could be used by an HBA talking to a fibre > channel device from an Ethernet connection. It puts together > an FCP frame, > adds a TCP/IP header, and out goes to the Ethernet wire. > There is good > reason that the same HBA may talk to another fibre channel > device through an > FCIP port. There would be different ways to create these connections. > However, many fibre channel HBAs today support multiple > protocols like FCP, > IP, and VI at the same time. Therefore, it is possible for a > future HBA to > speak iFCP, FCIP, and even iSCSI on an Ethernet connection at > the same time. > While I don't know if this makes economical sense, but, > technically, this is > definitely possible. > From owner-ips@ece.cmu.edu Sat Jan 6 14:29:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA17101 for ; Sat, 6 Jan 2001 14:29:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f06Hsxo01469 for ips-outgoing; Sat, 6 Jan 2001 12:54:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f06HsD001459 for ; Sat, 6 Jan 2001 12:54:13 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id ZJ5LTQVY; Sat, 6 Jan 2001 09:54:03 -0800 From: "Y P Cheng" To: "Joshua Tseng" , Subject: RE: iFCP as an IP Storage Work Item Date: Sat, 6 Jan 2001 09:53:36 -0800 Message-ID: <000001c07809$97ff3ea0$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > I think it's important to be clear on one thing. While a product can be > built from a common encapsulation method, it is NOT iFCP. Such a > product is basically leveraging a commmon envelope that > encloses different messages. The sender and receiver reading the contents > are different entities that speak different languages. > > Charles Totally agree. The HBA I am talking about does not care for E-Port or N-Port. > Sheesh...the Fibre Channel vendors are having a tough enough > time getting one E_PORT to talk to another E_PORT. You are claiming > that you can get an N_PORT to talk to an E_PORT. > > Josh Josh, the fact is that our HBA with 5 RISC engines and 4 MB of code can do the same thing Nishan and any FCIP vendors can do. However, having said that, my gut feeling is that it won't be that difficult to implement this HBA to speak iFCP and FCIP because it is only a "one-node SAN island" connecting to Ethernet capable of sending and receiving TCP/IP packets. It only needs to know the end-point addresses of the devices it wishes to connect. How do we get these addresses is what I expect from this IPS WG. The application software, not the HBA, will ensure a node with this HBA is visible to the world. As a one-node SAN island, all the handshakes with other iFCP and FCIP boxes are greatly simplified. It may only need minimum OSPF or FSPF exchanges. I am sure as an expert you'll have much better idea for the required exchanges. By the way, the HBA also speaks iSCSI so it can talk to SoIP devices which do not need the iFCP and FCIP boxes. Again, all the HBA cares is the end-point addresses. The HBA knows how to create and parse FCP frames and iSCSI PDUs in TCP/IP with right context and correct semantics. It does not worry about routing the packets or connecting to E-ports. > Let's get this straight here. iFCP encapsulates frames egressing > N_PORTs. FCIP encapsulates frames egressing E_PORTs. The above statement is mute to an HBA. Please remember the requirements for HBA are quite different from switch. Having made my argument, I do appreciate this IPS WG in willing to consider a common method for an HBA with a common TCP/IP encapsulation speaking to fibre channel or Ethernet storage devices. Y.P. Cheng, Connectcom Solutions. From owner-ips@ece.cmu.edu Sat Jan 6 16:26:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA17893 for ; Sat, 6 Jan 2001 16:26:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f06JH1S02776 for ips-outgoing; Sat, 6 Jan 2001 14:17:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([208.50.99.84]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f06JGv002769 for ; Sat, 6 Jan 2001 14:16:57 -0500 (EST) Received: from cx418298b (cx418298-b.orng1.occa.home.com [24.1.179.117]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id LAA10141 for ; Sat, 6 Jan 2001 11:16:50 -0800 (PST) Message-ID: <005101c07816$860593e0$75b30118@orng1.occa.home.com> From: "Murali Rajagopal" To: References: Subject: Agree or Disagree to Merge FCIP and iFCP Date: Sat, 6 Jan 2001 11:26:09 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit All: The FCIP and iFCP address the FC over IP network transport with entirely two different objectives. FCIP is a simple-minded tunnel that wants to see the basic FC connectivity extended over IP network. Period!!! iFCP's goal is much more than that .... But that does not mean that we should combine the two just because it appaers as though FCIP looks like a subset of iFCP. I am speaking for the authors of FCIP collectively, and in our opinion mixing the two in a single spec. or a common encapsulation method has little meaning.Yes we understand that everyone would like to see a single FC Over IP network solution and not two. But FCIP is really an extender for FC Switch Fabrics while iFCP is attempting to "replace" the FC Switching fabric. In this sense, iFCP is not in the best interest of either the FC Switch vendors or even the T11 FC community at large. So mixing them makes little business sense. In summary, the FCIP authors would like to keep the FCIP and the iFCP efforts seperate. I beleive that the iFCP folks are of the same opinion. Now with my TC hat on, I would like to propose that we proceed to solve this debate by getting a collective consensus from all folks on merging the two - protocol or otherwise. Please reply with a simple "Agree to merge" or "Disagree to merge" statement. No long explanations as to why or why not please! -Murali Rajagopal ----- Original Message ----- From: "Charles Monia" To: ; ; "Charles Monia" ; Sent: Friday, January 05, 2001 4:54 PM Subject: RE: iFCP as an IP Storage Work Item > > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Friday, January 05, 2001 4:15 PM > > To: dotis@sanlight.net; cmonia@nishansystems.com; ips@ece.cmu.edu > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > Before this goes any further ... the two of you may be > > in violent agreement. Charles objected to a "handshake"; > > I don't think this objection would preclude one end of > > a connection announcing the protocol that it intends to > > use so that the other end can cleanly and quickly > > terminate the connection if it can't or won't use > > that protocol (ideally with an error code to the > > other end indicating protocol incompatibility, so > > the misconfiguration becomes obvious). > > Doug's > > suggestion of IANA-allocated numbers for identifying > > protocols for this purpose is reasonable. > > > > As to the virtues of being able to speak more than > > one protocol on a port, Fibre Channel provides examples > > of where this sort of thing has been added after the > > initial specifications were done, so I wouldn't dismiss > > this out of hand. > > > > --David > > Hi David: > > It seems to me that with or without a common encapsulation method, such > boxes can be built 'now'. As I said earlier, a common encapsulation should > make it easier, which is a good thing. > > So, what we're left with is the question of how such a box would shift > gears. If an "in-band" method is easier, the simpler the handshake, the > better. So an approach based on an IANA-assigned parameter seems reasonable > to me. > > While this is a fruitful discussion, I'd be interested in hearing from the > FCIP folks on this issue. > > Charles > > > > > > > > -----Original Message----- > > > From: Douglas Otis [SMTP:dotis@sanlight.net] > > > Sent: Friday, January 05, 2001 5:59 PM > > > To: Charles Monia; Ips@Ece. Cmu. Edu > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > Charles, > > > > > > I disagree with this assumption about rigidity of > > protocols. Using a > > > common > > > initialization field would allow node handling protocols to > > be confirmed. > > > Initialization should also include a version number or > > option field for > > > the > > > encapsulation where this version or option be negotiated. > > If the node > > > handling protocol, separate from the encapsulation > > specification, could be > > > identified, then there would be fewer strange events if someone > > > mis-configured IP ports. I am not a fan of using text > > strings to do this > > > function and would prefer IANA defined values for rigid > > structures. In > > > addition to indicating an encapsulation version or options > > designator, > > > include the node type designator to signify how the node is > > handled to > > > avoid > > > strange failures. > > > > > > As a gateway, tunnel or perhaps as an end device in perhaps > > the distant > > > future, the type of node protocols may vary to meet > > different needs. The > > > lion's share of the work however could be covered within > > the encapsulation > > > specifications and clever means of handling the node could > > be isolated > > > from > > > these details. It does not mean that in the end all node > > protocols will > > > use > > > identical encapsulation, but only that, as much as > > possible, encapsulation > > > is kept common and the effort to encapsulate is seen as a > > separate task to > > > that of handling the frame at the end point. I would hope that a > > > transport > > > could be developed that would not care how the node was > > handled and where > > > this node handling function is defined within a separate process. > > > > > > It seems like a fairly clean separation of encapsulation > > and node handling > > > that would help foster support for these markets especially if a few > > > devices > > > allow a wide range of possible node handling scenarios. I > > agree that IP > > > fabric is more extensible to that of FC, however even with that > > > assumption, > > > the final goal of fully incorporating IP fabric without > > altering the FCP > > > structures has not been achieved by either of these current > > proposals. > > > With > > > that said, I would be in favor of developing a universal FC > > encapsulation > > > transport as a WG proposal that would support both iFCP and FCIP. > > > > > > Doug > > > > > > > > Ken Hirata wrote: > > > > > > Why do you want to standardize a common encapsulation protocol > > > > > > for FCIP and iFCP if their semantics and behavior are > > completely > > > > > > different? Would you want tunneling protocol > > implementations to > > > > > > also augment certain ELSs even though it isn't necessary > > > > > for tunneling > > > > > > protocol operation? > > > > > > > > > > If I were to build hardware that either assisted or completely > > > > > processed both iFCP and FCIP, it sure would be easier to do > > > > > header parsing and other common processing if there was just > > > > > one format. > > > > > > > > > > > If a common encapsulation protocol was defined, I believe a > > > > > > negotiation protocol would be necessary to distinguish between > > > > > > usage as a gateway or tunneling protocol. > > > > > > > > > > Yes, either negotiation of a flag bit in the encapsulating > > > > > header used to choose which algorithm to use. > > > > > > > > > Hi David and Ken: > > > > > > > > I agree that a common encapsulation may make it marginally easier > > > > to build a > > > > multi-protocol box as well as having other benefits. > > However, from the > > > > above, I'm concerned that some might infer that > > multi-protocol support > > > > should be a requirement. Just to be clear on this point: While > > > > I'm all for > > > > doing things that encourage commonality (where it makes sense > > > > technically) I > > > > feel that a standard ought not to needlessly burden a product with > > > > supporting both architectures. > > > > > > > > With regard to 'negotiation', I believe such a handshake is > > > > unnecessary and > > > > undesirable. In a real system, I can't see a scenario > > where it buys > > > > anything to make this dynamic. As I see it, these > > choices are cast in > > > > concrete when the SAN is implemented and aren't going to change > > > > from day to > > > > day. Also, for hardwired boxes that only support one > > protocol, it simply > > > > adds complexity. If someone wants to build a multi-protocol box, > > > > I believe > > > > they'd be better off making this statically settable. > > > > > > > > Charles > > > > > > > From owner-ips@ece.cmu.edu Sat Jan 6 16:33:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA17953 for ; Sat, 6 Jan 2001 16:33:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f06JX4103042 for ips-outgoing; Sat, 6 Jan 2001 14:33:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cleitus.hosting.pacbell.net (cleitus.hosting.pacbell.net [216.100.99.17]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f06JWh003028 for ; Sat, 6 Jan 2001 14:32:43 -0500 (EST) Received: from VENKAT1 (64-160-62-200.rhapsodynetworks.com [64.160.62.200] (may be forged)) by cleitus.hosting.pacbell.net id OAA28215; Sat, 6 Jan 2001 14:32:40 -0500 (EST) [ConcentricHost SMTP Relay 1.7] From: "Venkat Rangan" To: "Charles Monia" , "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Sat, 6 Jan 2001 11:34:40 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi Charles, Thanks for offering your perspective. I respect your opinions and I'm sure my responses would be exactly the same as yours if I were in your position. > That aside, the general thrust of this note seems > to reflect business rather than technical concerns. I am not sure if we can separate the overall market viability from technical. From our perspective, we do not want the IPS Working Group to endorse multiple technologies to address same or relatively close solutions. While it is easy to say that as a vendor it should be possible for them to support iSCSI, iFCP and FCIP protocols, the cost of developing a solution that addresses all three (which includes not just product development but testing, certifying and validating interoperable solutions) is something that we as a vendor would not like to be drawn into. If you think this is a business reason, and should be ignored, we disagree with that opinion. > Such confusion, if it exists, is best sorted out in the marketplace. In my > opinion, it's not the business of standards organizations to anoint any one > technology by picking winners and losers (as some would apparently wish). The standards organizations often pick something and others are forced to support that. We could let the marketplace decide, but that can be an expensive option for the market and can be detrimental to all vendors. As an example, while we are arguing about iFCP, FCIP and iSCSI, InfiniBand may enter the market and with a coherent proposal, may make this irrelevant. I assume that you agree that the combination of iSCSI, FCIP and iFCP has redundant technology elements. We believe the IP Storage Working Group needs to address the redundant parts which may mean picking something. When it does, we expect vendors to adopt and support what is picked. Do you not see a role in standards bodies to pick some technology? If you do not, why is iFCP being brought to the IP Storage Working Group as a work item? > Such assertions about processing overhead are totally without > foundation or substance. I do not think so. To perform cut-through routing, we only need to examine the first 8 bits of a frame (the domain id part of D_ID) to send it to an E_Port. If it is within the switch, you can do so by examining the first 24 bits of the frame (D_ID). Typically, most switches can do this by adding less than two to five microseconds to the latency. To perform a lookup to establish the From and To of both S_ID and D_ID and substitute the addresses and recompute a new CRC or digest would, in my opinion, take more cycles. Evidently, you seem to disagree, which is fine. Regards, Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Charles Monia Sent: Thursday, January 04, 2001 12:46 PM To: Ips@Ece. Cmu. Edu Cc: Venkat Rangan Subject: RE: iFCP as an IP Storage Work Item Hi Venkat: The issues you raise have been discussed at length before (but apparently not to your satisfaction). That aside, the general thrust of this note seems to reflect business rather than technical concerns. Within the parameters of the WG charter and IETF ground rules, it's my view that the ips wg needs to be a vehicle for bringing high-quality solutions to the marketplace, not obstructing technologies because they are inconvenient or conflict with someone's business model. See other responses below. Charles > -----Original Message----- > From: Venkat Rangan [mailto:venkat@rhapsodynetworks.com] > Sent: Thursday, January 04, 2001 9:05 AM > To: David Peterson; Ips@Ece. Cmu. Edu > Subject: RE: iFCP as an IP Storage Work Item > > > > We are also not in support of picking up iFCP as a IP Storage > Work Item. As > we indicated in earlier posts (FC-BB-2 exists thread and the > FCIP support > for N_Port etc.) we would not be in support of multiple ways > (i.e., FCIP and > iFCP) in which FC and IP networks/devices are made to coexist. > > In particular, our reasons for not supporting iFCP as a work > item are the > following. > > 1. Confusion in market place > > The overall goal of iFCP appears to be to leverage investment in IP > networking technology for networking storage. In this > respect, it is closer > in the goal of iSCSI. In a storage network based on IP networking > technology, we see iSCSI as a better alternative. It would be > confusing to > present both iSCSI and iFCP as two technologies for using the > IP network for > storage. Since the IP Storage Working Group and its members > have already > invested heavily in standardizing iSCSI, we do not see much value in > diluting the efforts on multiple technologies towards the > same end goal. > Such confusion, if it exists, is best sorted out in the marketplace. In my opinion, it's not the business of standards organizations to anoint any one technology by picking winners and losers (as some would apparently wish). > 2. Support for existing FC networks > > FCIP and its coordinated effort with FC-BB-2 supports > existing FC networks > better. The interoperability issues between FC and IP networks are > restricted to the border switches. Other infrastructure needs > such as FC > Name Server, Fabric Controller etc. are leveraged from standards work > already done in T11 working groups. The argument that iFCP is > useful in a > world of FC devices is not very strong, since FCIP is a > solution that not > only preserves investment in FC end nodes but also in FC switches. > To the contrary, the iFCP gateway model dovetails nicely with existing FC infrastuctures. iFCP's strong point is that the interface to existing FC networks becomes a gateway implementation issue. > 3. iFCP does not support FC loops and FC E_Port > > The drafts as submitted do not handle FC loops and FC > switches. There were > some email discussions where authors indicated that these are > trivial to add > to iFCP. We do not think it would be trivial to add E_Port > support and work > through the interoperability issues between iFCP gateways and > FC switches. > Adding FC loop support and E_Port support to an iFCP gateway > would make that > gateway device closer to an FC switch, so one could simply > use FC switches > with B_Port support and FCIP as the tunneling protocol. > Whether or not such support is trivial is, of course, a matter of opinion. Nonetheless, you seem to have missed the point. iFCP enables the gateway implementation to conceal the FC infrastructure. Given Fibre Channel's historic interoperability problems, some would consider that a benefit. Anyhow, the behavior on the FC side of the device is, of course, governed by the applicable FC standards (or their proprietary equivalents). It's not required for iFCP devices to interoperate with one another and hence doesn't belong in the spec. Since this issue has been mentioned a number of times, an informational appendix giving such implementation examples would apparently be useful. > 4. iFCP adds cost to deployment > > End nodes may be forced to support both FC N_Port and iSCSI. Forced by whom? Presumably, such concerns reflect market pressures and are irrelevant as such. > It is easier > for end nodes to just support iSCSI when storage network is > based on an IP > network. I assume that's a product development choice. >.............There is also an additional cost in testing a pure > iSCSI end node > with a bridged FC-iFCP end node. > Then, I'd suggest that you don't build such products. > 5. FCIP has broader coverage > > iFCP implies that it is only useful for translating FCP > traffic. FCIP is > able to tunnel all FC traffic through its border switches. It is also > possible to add N_Port connectivity using FCIP, although such > an attempt > would also compete with the goals of iSCSI. > > 6. iFCP addds processing overhead to frames > > In contrast to iSCSI, iFCP requires examination of every frame and > substitution of addresses at both iFCP Portals. This has the > potential to > add unwanted latencies as well as consume processing overhead > on gateways. Such assertions about processing overhead are totally without foundation or substance. > Given this, it would be hard to compete with cut-through routing based > products in a pure FC or IP based storage network. We also do > not believe > the NAT like function should really be the top priority for > the working > group to address. As an example, the first NAT RFC was RFC > 1631 (which was > an Informational RFC from the one vendor), well after several > other higher > priority items were addressed. We do not believe that FC > networks (when > deployed in the context of back-end storage) are running out > of address > space to warrant this to be addressed immediately. > > That's nothing more than unsubstantiated opinion and contrary to our experience. Apparently, earlier postings on this matter have gone unread. Charles Charles Monia Senior Technology Consultant Nishan Systems email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ece.cmu.edu Sat Jan 6 18:47:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA18549 for ; Sat, 6 Jan 2001 18:47:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f06LV4s04995 for ips-outgoing; Sat, 6 Jan 2001 16:31:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f06LU9004972 for ; Sat, 6 Jan 2001 16:30:09 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Sat, 6 Jan 2001 13:40:29 -0800 Message-ID: From: Charles Monia To: ips@ece.cmu.edu Cc: Murali Rajagopal Subject: RE: Agree or Disagree to Merge FCIP and iFCP Date: Sat, 6 Jan 2001 13:29:37 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Murali: Thanks for bringing this matter to a head. Here's my .02: 1. Merging the iFCP and FCIP specifications -- No, not feasible on technical grounds. Anyhow, I think this is one decision that can't be made by fiat. 2. Definition of a common encapsulation protocol -- Technically possible, practically not feasible. From my perspective, it's risky and difficult to manage as the client specs evolve over time. Besides, I assume the FCIP encapsulation is a done deal. Bottom line: I vote no (but would grudgingly try to accommodate the WG consensus on this matter). Charles Charles Monia Senior Technology Consultant Nishan Systems email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 > -----Original Message----- > From: Murali Rajagopal [mailto:muralir@lightsand.com] > Sent: Saturday, January 06, 2001 11:26 AM > To: ips@ece.cmu.edu > Subject: Agree or Disagree to Merge FCIP and iFCP > > > All: > > The FCIP and iFCP address the FC over IP network transport > with entirely two > different objectives. > FCIP is a simple-minded tunnel that wants to see the basic FC > connectivity > extended over IP network. Period!!! iFCP's goal is much more > than that .... > But that does not mean that we should combine the two just because it > appaers as though FCIP looks like a subset of iFCP. > > I am speaking for the authors of FCIP collectively, and in our opinion > mixing the two in a single spec. or a common encapsulation > method has little > meaning.Yes we understand that everyone would like to see a > single FC Over > IP network solution and not two. But FCIP is really an extender for FC > Switch Fabrics while iFCP is attempting to "replace" the FC Switching > fabric. In this sense, iFCP is not in the best interest of > either the FC > Switch vendors or even the T11 FC community at large. So > mixing them makes > little business sense. In summary, the FCIP authors would > like to keep the > FCIP and the iFCP efforts seperate. I beleive that the iFCP > folks are of the > same opinion. > > Now with my TC hat on, I would like to propose that we > proceed to solve this > debate by getting a collective consensus from all folks on > merging the two - > protocol or otherwise. > > Please reply with a simple "Agree to merge" or "Disagree to merge" > statement. No long explanations as to why or why not please! > > -Murali Rajagopal > > > ----- Original Message ----- > From: "Charles Monia" > To: ; ; "Charles Monia" > ; > Sent: Friday, January 05, 2001 4:54 PM > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > -----Original Message----- > > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > > Sent: Friday, January 05, 2001 4:15 PM > > > To: dotis@sanlight.net; cmonia@nishansystems.com; ips@ece.cmu.edu > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > Before this goes any further ... the two of you may be > > > in violent agreement. Charles objected to a "handshake"; > > > I don't think this objection would preclude one end of > > > a connection announcing the protocol that it intends to > > > use so that the other end can cleanly and quickly > > > terminate the connection if it can't or won't use > > > that protocol (ideally with an error code to the > > > other end indicating protocol incompatibility, so > > > the misconfiguration becomes obvious). > > > Doug's > > > suggestion of IANA-allocated numbers for identifying > > > protocols for this purpose is reasonable. > > > > > > As to the virtues of being able to speak more than > > > one protocol on a port, Fibre Channel provides examples > > > of where this sort of thing has been added after the > > > initial specifications were done, so I wouldn't dismiss > > > this out of hand. > > > > > > --David > > > > Hi David: > > > > It seems to me that with or without a common encapsulation > method, such > > boxes can be built 'now'. As I said earlier, a common encapsulation > should > > make it easier, which is a good thing. > > > > So, what we're left with is the question of how such a box > would shift > > gears. If an "in-band" method is easier, the simpler the > handshake, the > > better. So an approach based on an IANA-assigned parameter seems > reasonable > > to me. > > > > While this is a fruitful discussion, I'd be interested in > hearing from the > > FCIP folks on this issue. > > > > Charles > > > > > > > > > > > > -----Original Message----- > > > > From: Douglas Otis [SMTP:dotis@sanlight.net] > > > > Sent: Friday, January 05, 2001 5:59 PM > > > > To: Charles Monia; Ips@Ece. Cmu. Edu > > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > Charles, > > > > > > > > I disagree with this assumption about rigidity of > > > protocols. Using a > > > > common > > > > initialization field would allow node handling protocols to > > > be confirmed. > > > > Initialization should also include a version number or > > > option field for > > > > the > > > > encapsulation where this version or option be negotiated. > > > If the node > > > > handling protocol, separate from the encapsulation > > > specification, could be > > > > identified, then there would be fewer strange events if someone > > > > mis-configured IP ports. I am not a fan of using text > > > strings to do this > > > > function and would prefer IANA defined values for rigid > > > structures. In > > > > addition to indicating an encapsulation version or options > > > designator, > > > > include the node type designator to signify how the node is > > > handled to > > > > avoid > > > > strange failures. > > > > > > > > As a gateway, tunnel or perhaps as an end device in perhaps > > > the distant > > > > future, the type of node protocols may vary to meet > > > different needs. The > > > > lion's share of the work however could be covered within > > > the encapsulation > > > > specifications and clever means of handling the node could > > > be isolated > > > > from > > > > these details. It does not mean that in the end all node > > > protocols will > > > > use > > > > identical encapsulation, but only that, as much as > > > possible, encapsulation > > > > is kept common and the effort to encapsulate is seen as a > > > separate task to > > > > that of handling the frame at the end point. I would > hope that a > > > > transport > > > > could be developed that would not care how the node was > > > handled and where > > > > this node handling function is defined within a > separate process. > > > > > > > > It seems like a fairly clean separation of encapsulation > > > and node handling > > > > that would help foster support for these markets > especially if a few > > > > devices > > > > allow a wide range of possible node handling scenarios. I > > > agree that IP > > > > fabric is more extensible to that of FC, however even with that > > > > assumption, > > > > the final goal of fully incorporating IP fabric without > > > altering the FCP > > > > structures has not been achieved by either of these current > > > proposals. > > > > With > > > > that said, I would be in favor of developing a universal FC > > > encapsulation > > > > transport as a WG proposal that would support both iFCP > and FCIP. > > > > > > > > Doug > > > > > > > > > > Ken Hirata wrote: > > > > > > > Why do you want to standardize a common > encapsulation protocol > > > > > > > for FCIP and iFCP if their semantics and behavior are > > > completely > > > > > > > different? Would you want tunneling protocol > > > implementations to > > > > > > > also augment certain ELSs even though it isn't necessary > > > > > > for tunneling > > > > > > > protocol operation? > > > > > > > > > > > > If I were to build hardware that either assisted or > completely > > > > > > processed both iFCP and FCIP, it sure would be easier to do > > > > > > header parsing and other common processing if there was just > > > > > > one format. > > > > > > > > > > > > > If a common encapsulation protocol was defined, I > believe a > > > > > > > negotiation protocol would be necessary to > distinguish between > > > > > > > usage as a gateway or tunneling protocol. > > > > > > > > > > > > Yes, either negotiation of a flag bit in the encapsulating > > > > > > header used to choose which algorithm to use. > > > > > > > > > > > Hi David and Ken: > > > > > > > > > > I agree that a common encapsulation may make it > marginally easier > > > > > to build a > > > > > multi-protocol box as well as having other benefits. > > > However, from the > > > > > above, I'm concerned that some might infer that > > > multi-protocol support > > > > > should be a requirement. Just to be clear on this > point: While > > > > > I'm all for > > > > > doing things that encourage commonality (where it makes sense > > > > > technically) I > > > > > feel that a standard ought not to needlessly burden a > product with > > > > > supporting both architectures. > > > > > > > > > > With regard to 'negotiation', I believe such a handshake is > > > > > unnecessary and > > > > > undesirable. In a real system, I can't see a scenario > > > where it buys > > > > > anything to make this dynamic. As I see it, these > > > choices are cast in > > > > > concrete when the SAN is implemented and aren't going > to change > > > > > from day to > > > > > day. Also, for hardwired boxes that only support one > > > protocol, it simply > > > > > adds complexity. If someone wants to build a > multi-protocol box, > > > > > I believe > > > > > they'd be better off making this statically settable. > > > > > > > > > > Charles > > > > > > > > > > > From owner-ips@ece.cmu.edu Sat Jan 6 19:31:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA18765 for ; Sat, 6 Jan 2001 19:31:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f06MR5e05857 for ips-outgoing; Sat, 6 Jan 2001 17:27:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f06MQP005847 for ; Sat, 6 Jan 2001 17:26:25 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id ZJ5LTQ5B; Sat, 6 Jan 2001 14:26:24 -0800 From: "Y P Cheng" To: "'Ips@Ece. Cmu. Edu'" Subject: RE: iFCP as an IP Storage Work Item Date: Sat, 6 Jan 2001 14:25:55 -0800 Message-ID: <001d01c0782f$a2e84160$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > While it is easy to say that as a vendor it should be possible for them to > support iSCSI, iFCP and FCIP protocols, the cost of developing a solution > that addresses all three (which includes not just product development but > testing, certifying and validating interoperable solutions) is something > that we as a vendor would not like to be drawn into. If you think > this is a business reason, and should be ignored, we > disagree with that opinion. I have been trying to avoid taking side on this iFCP vs. FCIP debate on the technical merits. May be I can take my technical hat off and debate on the business merits. No, the world does not need two standards and the IPS WG should force the issue. While companies will always do their own, the mission of a standard committee is to find one and only one standard to make everyone's life easier. Failing to do so does not serve this community. If FCIP is good enough, why do I need iFCP? Do I really need the scalability of 4 billion fibre channel nodes visible to me? For Internet domain names I may need IPv6, but, for storage devices? 24-bits of D_ID with 16 million nodes are a lot of addresses. I do understand perfectly if one wishes to dominate the FC switch market. As a consumer, this is not my concern unless one can provide me alternatives with a much lower costs. As a customer, all I care is to have the ability to access storage devices on IP network at lower cost. I don't care the standard as long as there is one that gives me choices of low-cost vendors. Many people even believe among the networks of Ethernet, Fibre Channel, and InfiniBand, there will be only one winner. Therefore, among iFCP, FCIP, and iSCSI, please give me just one. Having said that, I do believe we need fibre channel before Ethernet folks taking over the world while InfiniBand lurks on the horizon. ISA was wonderful until EISA comes along. VESA was great until PCI appears. Now PCI-X and InfiniBand. We all suffer through technology transitions. Therefore, the last thing we need is to have another standard even before we start. Don't repeat VHS and Beta. I still have a lots of tapes in Beta format. From owner-ips@ece.cmu.edu Sat Jan 6 19:34:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA18783 for ; Sat, 6 Jan 2001 19:34:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f06Mx6g06345 for ips-outgoing; Sat, 6 Jan 2001 17:59:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f06Mwa006333 for ; Sat, 6 Jan 2001 17:58:36 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Sat, 6 Jan 2001 15:08:57 -0800 Message-ID: From: Charles Monia To: JP Raghavendra Rao Cc: ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Sat, 6 Jan 2001 14:58:05 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: JP Raghavendra Rao [mailto:jp.raghavendra@india.sun.com] > Sent: Thursday, January 04, 2001 8:19 AM > To: ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > I believe iFCP should be an IPS work item for the following > > technical reasons: > > > > 1) iFCP allows leverage of existing FCP-based driver stacks and > > preservation of the $$$ and @#$!!% that have been invested in them > > by vendor companies and their customers. > > > > I think the argument for preserving software doesn't make > strong case in the > face of a migration to a new mapping/tunneling protocol, new > software and new > administration challenges in spite of the fact that all of > this is likely to > mimic FC and contained in one or two edge router I don't know how you equate these. In my world, debugged and stable driver stacks are a precious commodity not to be discarded lightly. The other factors seem equally pertinent to all IP storage solutions. > - Today's > FC network is > difficult to administer and any bridging technology to a > different interconnect > is only going to compound it. > > It would be nice if somebody comes up with a stronger case > for connecting an > iSCSI host to a FC device - Is this attempted for the > survival of FC or for > a speedier deployment of iSCSI ? > How about the fact that users seem reluctant to trash their existing storage investments every time a new interconnect technology shows up. Is that strong enough? Also, storage interconnects have greater market longevity, so I wouldn't count on FC's demise any time soon. After all, people have been predicting the demise of parallel SCSI for a while, but after 10+ years, it's still thriving. Charles From owner-ips@ece.cmu.edu Sat Jan 6 22:57:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA21529 for ; Sat, 6 Jan 2001 22:57:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f071XAd08922 for ips-outgoing; Sat, 6 Jan 2001 20:33:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f071WF008911 for ; Sat, 6 Jan 2001 20:32:15 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Sat, 6 Jan 2001 17:41:37 -0800 Message-ID: From: Charles Monia To: "Venkat Rangan (E-mail)" Cc: "Ips (E-mail)" Subject: RE: iFCP as an IP Storage Work Item Date: Sat, 6 Jan 2001 17:31:48 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Venkat: I don't find much of technical substance to reply to here. Obviously, you're entitled to your views and a position based on your internal business and product development considerations is not something that can be productively debated in this or any other forum. I've interspersed some additional comments below. > -----Original Message----- > From: Venkat Rangan [mailto:venkat@rhapsodynetworks.com] > Sent: Saturday, January 06, 2001 11:35 AM > To: Charles Monia; Ips@Ece. Cmu. Edu > Subject: RE: iFCP as an IP Storage Work Item > > > Hi Charles, > > Thanks for offering your perspective. I respect your opinions > and I'm sure > my responses would be exactly the same as yours if I were in > your position. > > > That aside, the general thrust of this note seems > > to reflect business rather than technical concerns. > > I am not sure if we can separate the overall market viability from > technical. From our perspective, we do not want the IPS > Working Group to > endorse multiple technologies to address same or relatively > close solutions. > While it is easy to say that as a vendor it should be > possible for them to > support iSCSI, iFCP and FCIP protocols, the cost of > developing a solution > that addresses all three (which includes not just product > development but > testing, certifying and validating interoperable solutions) > is something > that we as a vendor would not like to be drawn into. > If you > think this is a > business reason, and should be ignored, we disagree with that opinion. > How can product development concerns be anything else? That being the case, I don't see how such issues can be productively addressed by the WG. That said, I'm having a hard time connecting the dots of your market viability argument. On the one hand you imply that iFCP is not viable from a marketing perspective, while asserting at the same time that you'll be drawn into building iFCP products. Why else would you have that problem except in response to the demands of the marketplace? If you truly believe it's not viable, then you can safely ignore the technology and go about your business. > > Such confusion, if it exists, is best sorted out in the > marketplace. In > my > > opinion, it's not the business of standards organizations > to anoint any > one > > technology by picking winners and losers (as some would > apparently wish). > > The standards organizations often pick something and others > are forced to > support that. > We could let the marketplace decide, but that can be an > expensive option for the market and can be detrimental to all > vendors. I don't understand where force enters the picture. Your position seems based on the arbitrary assumption that some solutions (namely yours) have a greater inherent claim to legitimacy than others. That's certainly a matter of opinion. I've always believed that market competition is a good thing. Obviously, you don't seem to share that view. >...............................................As an > example, while we are arguing about iFCP, FCIP and iSCSI, > InfiniBand may > enter the market and with a coherent proposal, may make this > irrelevant. > I don't think the time taken to settle this matter is going to make much difference in the scheme of things. Nonethless, it strikes me that iSCSI and FCIP are making good progress while this debate goes on. > I assume that you agree that the combination of iSCSI, FCIP > and iFCP has > redundant technology elements. Commonality -- yes to the extent that they all use the same set of wires and deliver storage solutions. Redundancy? Absolutely not. >...............................We believe the IP Storage > Working Group needs > to address the redundant parts which may mean picking > something. > When it > does, we expect vendors to adopt and support what is picked. Who decides what's 'redundant' and on what basis? > Do you not see > a role in standards bodies to pick some technology? Absolutely not -- even if it were my favorite solution (as opposed to yours). >..............................If you do > not, why is > iFCP being brought to the IP Storage Working Group as a work item? > We brought iFCP to the WG not because we want to legislate what the market should have but because we believe it can coexist with and offer benefits that are unique and distinct from the other technologies on the table. Will we also support iSCSI? You bet. We're in the business of supplying end-to-end IP-based storage solutions. Unlike some, we see these technologies as opportunities to be exploited, not threats to be feared or suppressed. > > Such assertions about processing overhead are totally without > > foundation or substance. > > I do not think so. To perform cut-through routing, we only > need to examine > the first 8 bits of a frame (the domain id part of D_ID) to > send it to an > E_Port. If it is within the switch, you can do so by > examining the first 24 > bits of the frame (D_ID). Typically, most switches can do > this by adding > less than two to five microseconds to the latency. To perform > a lookup to > establish the From and To of both S_ID and D_ID and > substitute the addresses > and recompute a new CRC or digest would, in my opinion, take > more cycles. > Evidently, you seem to disagree, which is fine. > While a discussion around clock cycles would be fun sometime, I'll simply point out that we also have a similar bag of tricks and we expect our performance to be on a par. Anyhow, in the context of a total system, I believe this is something that gets lost in the noise. Charles Charles Monia Senior Technology Consultant Nishan Systems email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ece.cmu.edu Sat Jan 6 23:15:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA21580 for ; Sat, 6 Jan 2001 23:15:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f072LEE09697 for ips-outgoing; Sat, 6 Jan 2001 21:21:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hotmail.com (law-f2.hotmail.com [209.185.131.65]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f070OQ007846 for ; Sat, 6 Jan 2001 19:24:27 -0500 (EST) Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Sat, 6 Jan 2001 16:24:21 -0800 Received: from 209.246.188.187 by law-test-www.hotmail.msn.com with HTTP; Sun, 07 Jan 2001 00:24:21 GMT X-Originating-IP: [209.246.188.187] From: "Eddy Quicksall" To: ips@ece.cmu.edu Subject: Markers Date: Sat, 06 Jan 2001 19:24:21 -0500 Mime-Version: 1.0 Content-Type: text/html Message-ID: X-OriginalArrivalTime: 07 Jan 2001 00:24:21.0541 (UTC) FILETIME=[2E3DB550:01C07840] Sender: owner-ips@ece.cmu.edu Precedence: bulk

Does anyone know why the proposal to use the urgent pointer was not approved?
 
If I understand the explanation of Markers, I see potential problems.
 
 
 

Get your FREE download of MSN Explorer at http://explorer.msn.com

From owner-ips@ece.cmu.edu Sun Jan 7 00:13:25 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA22170 for ; Sun, 7 Jan 2001 00:13:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f073IEu10577 for ips-outgoing; Sat, 6 Jan 2001 22:18:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f073IB010572 for ; Sat, 6 Jan 2001 22:18:11 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Sat, 6 Jan 2001 19:27:31 -0800 Message-ID: From: Charles Monia To: Y P Cheng Cc: "Ips (E-mail)" Subject: RE: iFCP as an IP Storage Work Item Date: Sat, 6 Jan 2001 19:17:40 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Y P: > I have been trying to avoid taking side on this iFCP vs. FCIP > debate on the > technical merits. May be I can take my technical hat off and > debate on the > business merits. According to whose criteria? Yours I guess. Others see the world differently. > No, the world does not need two standards I'm glad someone has the inside track on what the world needs. I'm perfectly content to let the world make that decision. >................................and the IPS WG > should force the > issue. While companies will always do their own, the mission > of a standard > committee is to find one and only one standard to make everyone's life > easier. Since when? I suggest you visit the T10 web site and look around (see http://www.t10.org/scsi-3.htm). At last count, there were six, count 'em, six SCSI encapsulations (not including iSCSI), all alive and well -- not to mention ATA. Incidentally, by a lot of measures, you'd probably be justified in concluding that ATA's what the world really needs. Other stuff below. > -----Original Message----- > From: Y P Cheng [mailto:ycheng@advansys.com] > Sent: Saturday, January 06, 2001 2:26 PM > To: 'Ips@Ece. Cmu. Edu' > Subject: RE: iFCP as an IP Storage Work Item > > > > While it is easy to say that as a vendor it should be > possible for them to > > support iSCSI, iFCP and FCIP protocols, the cost of > developing a solution > > that addresses all three (which includes not just product > development but > > testing, certifying and validating interoperable solutions) > is something > > that we as a vendor would not like to be drawn into. If you think > > this is a business reason, and should be ignored, we > > disagree with that opinion. > > I have been trying to avoid taking side on this iFCP vs. FCIP > debate on the > technical merits. May be I can take my technical hat off and > debate on the > business merits. > > No, the world does not need two standards and the IPS WG > should force the > issue. While companies will always do their own, the mission > of a standard > committee is to find one and only one standard to make everyone's life > easier. Failing to do so does not serve this community. > > If FCIP is good enough, why do I need iFCP? Good enough for whom? Without rehashing this issue yet again, there are valid constituencies for both solutions. >.......Do I really need the > scalability of 4 billion fibre channel nodes visible to me? The iSCSI folks and others planning to build directly attached IP storage devices are apt to find that argument strange. > For Internet > domain names I may need IPv6, but, for storage devices? > 24-bits of D_ID > with 16 million nodes are a lot of addresses. That address space disappears fast if you're trying to intergrate a lot of small FC sans. Each one consumes a 65 K block of FC addresses, most of which are unused. >..................................I do > understand perfectly if > one wishes to dominate the FC switch market. As a consumer, > this is not my > concern unless one can provide me alternatives with a much > lower costs. > > As a customer, all I care is to have the ability to access > storage devices > on IP network at lower cost. I don't care the standard as > long as there is > one that gives me choices of low-cost vendors. You also could care less about how many standards there are. >...........Many people > even believe > among the networks of Ethernet, Fibre Channel, and > InfiniBand, there will be > only one winner. And many people don't. I haven't seen the parallel SCSI and FC folks atart folding up their tents yet. >.......................Therefore, among iFCP, FCIP, and iSCSI, > please give me > just one. > Having said that, I do believe we need fibre channel before > Ethernet folks taking over the world while InfiniBand lurks > on the horizon. > ISA was wonderful until EISA comes along. VESA was great > until PCI appears. > Now PCI-X and InfiniBand. We all suffer through technology > transitions. > Therefore, the last thing we need is to have another standard > even before we > start. Don't repeat VHS and Beta. I still have a lots of > tapes in Beta > format. > > And I have a lot of DVDs, VHS, 8MM, etc, etc, etc. Charles From owner-ips@ece.cmu.edu Sun Jan 7 01:23:07 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA24637 for ; Sun, 7 Jan 2001 01:23:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f074JF211391 for ips-outgoing; Sat, 6 Jan 2001 23:19:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f074IG011380 for ; Sat, 6 Jan 2001 23:18:16 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 034272BF for ; Sat, 6 Jan 2001 20:18:10 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id UAA06093; Sat, 6 Jan 2001 20:18:02 -0800 (PST) Message-ID: <3A57EE53.97898A4@cup.hp.com> Date: Sat, 06 Jan 2001 20:19:31 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : NOP-OUT & NOP-IN issues. Content-Type: multipart/mixed; boundary="------------8EF5842A246230839B3304C1" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------8EF5842A246230839B3304C1 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, I have several concerns regarding the current semantics for the NOP operation, as defined in the lastest iSCSI draft. (iSCSI-02.txt) : Section 2.12 NOP-OUT PDU ========================= 1) Quoting from the draft : "The NOP-Out can be sent by an initiator because of a NOP-In with the poll bit set, in which case the Target Tag will copy the NOP-In value." There is no poll bit in the NOP-IN PDU. What is the intent of the above ? 2) Quoting from the draft : "The NOP-Out MUST have the Initiator Task Tag set only if the P bit is one or the DataRN field is set." There is no DataRN field in the NOP-OUT PDU. Is the intent of this to state that : "The NOP-OUT must have the Initiator Task Tag set if the NOP-OUT is being used to acknowledge DataRN[s] from received READ Data PDU[s]" ? Some clarification would help here. 3) Quoting from the draft : "The NOP-Out MUST have the Target Tag set only if it issued in response to a NOP-In with the P bit one, in which case it copies the Target Tag from the NOP-In PDU." Some comments on the above : i) There is no P bit in a NOP-IN PDU. ii) If a target wanted to originate a NOP for some reason, it should use NOP-OUT. The NOP-OUT should be intended for the originator of the NOP and the responder of the NOP should reply with NOP-IN. This keeps the NOP semantics simple and intuitive. ii) Why is there a need for a target to originate a NOP-OUT ? In the interests of simplicity, targets should be prohibited from originating the NOP operation. iii) With the above semantics, how is an initator to know if a received NOP-IN is being originated by a target [to which it needs to respond with a NOP-OUT] or the NOP-IN is being received in response to a transmitted NOP-OUT ? The semantics of NOP-OUT used by the originator of the NOP operation and NOP-IN used by the responder of the NOP operation seem more intuitive. Section 2.13. NOP-IN PDU ========================= 1) References to a P bit in the NOP-IN PDU which does not exist. 2) This section specifies the use of 0 for Initiator Task Tag when the target originates a NOP operation. 0 is a valid value for Initiator Task Tag and its usage can confuse the initiator b/n a target originated NOP-IN and a target response to NOP-OUT sent on an initiator task tag of 0. The reserved value for Initiator Task Tag should be 0xFFFFFFFF, as is the case with the Target Task Tag. Thanks, Santosh --------------8EF5842A246230839B3304C1 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------8EF5842A246230839B3304C1-- From owner-ips@ece.cmu.edu Sun Jan 7 01:25:22 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA24913 for ; Sun, 7 Jan 2001 01:25:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f074hGb11798 for ips-outgoing; Sat, 6 Jan 2001 23:43:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f074gv011791 for ; Sat, 6 Jan 2001 23:42:58 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 450945C8 for ; Sat, 6 Jan 2001 20:42:57 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id UAA07169; Sat, 6 Jan 2001 20:42:51 -0800 (PST) Message-ID: <3A57F425.740CCC51@cup.hp.com> Date: Sat, 06 Jan 2001 20:44:21 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Unsolicited Inbound Commands for initiators. Content-Type: multipart/mixed; boundary="------------6E904D08677CB56DA85F14A5" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------6E904D08677CB56DA85F14A5 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, Can a Logout command PDU be originated by a target ? If so, this would eliminate operations like the target sending AEN with an iSCSI Event Indicator of "Target Requests Logout on this connection". In general, what is the policy of the iSCSI draft towards un-solicited inbound command PDUs received by an initiator. Which of the following commands can an initiator receive : 1) Login Command (Could be received during device discovery process when another initiator on the SAN is discovering its devices.) 2) Logout Command (Could be originated by targets as a form of error recovery.) 3) Text Command 4) NOP-OUT Command. (Could be originated by a target to test a connection or to request DataRN acknowledgement, when the max un-acknowledged DataRN limit has been reached.) Specifc elaboration in the draft of the commands that targets are allowed to originate would be desirable. Thanks, Santosh --------------6E904D08677CB56DA85F14A5 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------6E904D08677CB56DA85F14A5-- From owner-ips@ece.cmu.edu Sun Jan 7 01:26:31 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA25071 for ; Sun, 7 Jan 2001 01:26:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f074kFM11859 for ips-outgoing; Sat, 6 Jan 2001 23:46:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f074jf011848 for ; Sat, 6 Jan 2001 23:45:41 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Sat, 6 Jan 2001 20:55:04 -0800 Message-ID: From: Charles Monia To: "Lawrence J. Lamers" Cc: ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Sat, 6 Jan 2001 20:45:09 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi: It appears to me that this thread has pretty much deteriorated into a billboard for marketing messages and positioning statements. Apparently, a lot of folks have decided that repeating the same mantras over and over is a good substitute for an in-depth technical discourse based on the merits. Solemn pronouncements about the viability of this or that "long term solution" also appear to serve the same purpose. I'm a also little fed up with the bogus notion that it ought to be the job of this organization to bless one technology under the pretext that others are "unproductive", would "cause confusion and divisiveness", or would be "difficult to debug", etc, etc, ad nauseum. Those who think they have a better way ought to waste less time trying to stifle innovation and take their case to the marketplace. Yes, iFCP, like any other proposal, needs to come up to the bar by uniquely adding technical value. In that regard, no one gets a pass. As far as that criteria for acceptance is concerned, we believe we meet the standard. Next.... Charles Charles Monia Senior Technology Consultant Nishan Systems email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 > -----Original Message----- > From: Lawrence J. Lamers [mailto:ljlamers@ieee.org] > Sent: Saturday, January 06, 2001 7:38 AM > To: ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > The market analysis indicates that a fibre channel over > internet protocol approach, such as FCIP, provides the solution > needed to connect storage networks over Metropolitan and Wide > Area Networks for the foreseeable future. The FCIP proposal > still needs further work in order to provide the necessary > features for it to have long term viability, but we see no > reason that these features can not be added to this approach. > > The long term solution for native attachment of storage > devices to an IP network is iSCSI, a solution that conceptually > fits with the present operating system/software driver stacks, > has broad industry support, and is in process of becoming a > standard within IETF. ISCSI is a protocol to do SCSI commands > and task management over Ethernet, much as FCP does for fibre > channel, SBP does for 1394, and SIP does for SPI. Layering one > protocol on top of another to achieve SCSI functionality seems > to complicated an approach that will be difficult to debug and > certify, a field day for Murphy's law. > > Certain market niches may be served by other solutions, but it > is believed that the complementary solutions of FCIP and iSCSI > are what should be standardized and promoted to the industry. > This avoids confusion and divisiveness and gets the world to a > storage networked future the fastest, with the lowest cost and > the least un-productive effort. > > It is my belief based on the technical merits and market needs > that the IETF should continue in its present course and support > the FCoverIP work item and the iSCSI work item and not add > additional work items that these solutions already address. > > Regards, > > Lawrence J. Lamers > Principal Engineer Advanced Technology Group > SAN Valley Systems, Inc. > 408-234-0071 > ljlamers@sanvalley.com > ljlamers@ieee.org > From owner-ips@ece.cmu.edu Sun Jan 7 04:18:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA07159 for ; Sun, 7 Jan 2001 04:18:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f077AKQ13984 for ips-outgoing; Sun, 7 Jan 2001 02:10:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0779o013957 for ; Sun, 7 Jan 2001 02:09:50 -0500 (EST) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id XAA29823 for ; Sat, 6 Jan 2001 23:09:46 -0800 (PST) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.151.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id XAA18990 for ; Sat, 6 Jan 2001 23:09:45 -0800 (PST) Received: from ebay.sun.com by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id XAA06469; Sat, 6 Jan 2001 23:09:45 -0800 (PST) Message-ID: <3A581654.B50E398@ebay.sun.com> Date: Sat, 06 Jan 2001 23:10:12 -0800 From: David Robinson Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.75 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Agree or Disagree to Merge FCIP and iFCP References: <005101c07816$860593e0$75b30118@orng1.occa.home.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Murali Rajagopal wrote: > Now with my TC hat on, I would like to propose that we proceed to solve this > debate by getting a collective consensus from all folks on merging the two - > protocol or otherwise. > > Please reply with a simple "Agree to merge" or "Disagree to merge" > statement. No long explanations as to why or why not please! Even with a common encapsulation encoding, I don't believe the two can be merged in any meaningful sense of the word. -David From owner-ips@ece.cmu.edu Sun Jan 7 15:04:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA09830 for ; Sun, 7 Jan 2001 15:04:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f07HgeH01260 for ips-outgoing; Sun, 7 Jan 2001 12:42:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f07Hg0001253 for ; Sun, 7 Jan 2001 12:42:00 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA112504 for ; Sun, 7 Jan 2001 18:41:54 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA50578 for ; Sun, 7 Jan 2001 18:41:44 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569CD.00613113 ; Sun, 7 Jan 2001 18:41:35 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 7 Jan 2001 19:37:23 +0200 Subject: Re: iSCSI : NOP-OUT & NOP-IN issues. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk My reply in the text - thanks, Julo Santosh Rao on 07/01/2001 06:19:31 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iSCSI : NOP-OUT & NOP-IN issues. Julian, I have several concerns regarding the current semantics for the NOP operation, as defined in the lastest iSCSI draft. (iSCSI-02.txt) : Section 2.12 NOP-OUT PDU ========================= 1) Quoting from the draft : "The NOP-Out can be sent by an initiator because of a NOP-In with the poll bit set, in which case the Target Tag will copy the NOP-In value." There is no poll bit in the NOP-IN PDU. What is the intent of the above ? Bit 7 of the second byte (it was a typo) should read P - it is fixed in 03.txt 2) Quoting from the draft : "The NOP-Out MUST have the Initiator Task Tag set only if the P bit is one or the DataRN field is set." There is no DataRN field in the NOP-OUT PDU. Is the intent of this to state that : "The NOP-OUT must have the Initiator Task Tag set if the NOP-OUT is being used to acknowledge DataRN[s] from received READ Data PDU[s]" ? Some clarification would help here. read ExpDataRN. Correction and an explanation for the field have been added 3) Quoting from the draft : "The NOP-Out MUST have the Target Tag set only if it issued in response to a NOP-In with the P bit one, in which case it copies the Target Tag from the NOP-In PDU." Some comments on the above : i) There is no P bit in a NOP-IN PDU. ii) If a target wanted to originate a NOP for some reason, it should use NOP-OUT. The NOP-OUT should be intended for the originator of the NOP and the responder of the NOP should reply with NOP-IN. This keeps the NOP semantics simple and intuitive. ii) Why is there a need for a target to originate a NOP-OUT ? In the interests of simplicity, targets should be prohibited from originating the NOP operation. iii) With the above semantics, how is an initator to know if a received NOP-IN is being originated by a target [to which it needs to respond with a NOP-OUT] or the NOP-IN is being received in response to a transmitted NOP-OUT ? In and Out are always relative to the initiator. A receiver of a NOP knows if the NOP is originated by the target through the target task tag. The code differentiation is meant for a stateless protocol analyzer The semantics of NOP-OUT used by the originator of the NOP operation and NOP-IN used by the responder of the NOP operation seem more intuitive. Section 2.13. NOP-IN PDU ========================= 1) References to a P bit in the NOP-IN PDU which does not exist. 2) This section specifies the use of 0 for Initiator Task Tag when the target originates a NOP operation. 0 is a valid value for Initiator Task Tag and its usage can confuse the initiator b/n a target originated NOP-IN and a target response to NOP-OUT sent on an initiator task tag of 0. The reserved value for Initiator Task Tag should be 0xFFFFFFFF, as is the case with the Target Task Tag. Thanks, Santosh corections where made - this typo was reported by several people From owner-ips@ece.cmu.edu Sun Jan 7 15:08:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA09863 for ; Sun, 7 Jan 2001 15:08:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f07HjdM01321 for ips-outgoing; Sun, 7 Jan 2001 12:45:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f07HjJ001313 for ; Sun, 7 Jan 2001 12:45:19 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA145238 for ; Sun, 7 Jan 2001 18:45:13 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA67956 for ; Sun, 7 Jan 2001 18:45:13 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569CD.0061816E ; Sun, 7 Jan 2001 18:45:01 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 7 Jan 2001 19:40:50 +0200 Subject: Re: iSCSI : NOP-OUT & NOP-IN issues. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, I will consider renaming. However the codes have now overloaded meaning (for that we have earlier two different NOPs - and I don't feel easy having a NOP response to no command. Regards, Julo Santosh Rao on 07/01/2001 06:29:13 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: santoshr@cup.hp.com Subject: Re: iSCSI : NOP-OUT & NOP-IN issues. Julian, If NOP-OUT and NOP-IN were to be re-named to NOP Command and NOP Response PDUs, this would further enhance ease of understanding of the NOP operation semantics, removing any issues about target originating the NOP operation through the use of NOP-IN. Thanks & Regards, Santosh sip - Santosh Rao wrote: > The NOP-OUT should be intended for the originator of the NOP > and the responder of the NOP should reply with NOP-IN. > This keeps the NOP semantics simple and intuitive. > From owner-ips@ece.cmu.edu Sun Jan 7 23:54:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA14138 for ; Sun, 7 Jan 2001 23:54:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f082X1T10184 for ips-outgoing; Sun, 7 Jan 2001 21:33:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f082WKU10172 for ; Sun, 7 Jan 2001 21:32:20 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 28A4A7F0 for ; Sun, 7 Jan 2001 18:32:19 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA01004; Sun, 7 Jan 2001 18:32:09 -0800 (PST) Message-ID: <3A592705.B12F6A15@cup.hp.com> Date: Sun, 07 Jan 2001 18:33:42 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Login Response PDU Issues. Content-Type: multipart/mixed; boundary="------------5EE9DF34A44C87BA968AC215" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------5EE9DF34A44C87BA968AC215 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, Some concerns regarding the Login Response PDU section of the latest iSCSI draft : Section 2.11.3. Login Response Status ====================================== 1) Should targets use a REJECT PDU or Login Response PDU with a status of "reject login" to indicate Login Reject ? Section 2.11.3 and 4.4 contradict each other on this subject. A standard REJECT PDU used to reject ALL non-scsi PDUs [and ONLY non-scsi PDUs] would be desirable and would provide the following benefits : o Allows more specific error information to be conveyed within SCSI Task Mgmt response PDUs and SCSI Response PDUs, on a reject of SCSI Task Mgmt Cmd or SCSI Command. (thru the Response Data for Scsi Response PDU and Response field for SCSI Task Mgmt response PDU). o Consistent with Fibre Channel semantics for the standard LS_RJT for all ELS'. This also allows for easy mapping from iSCSI Response PDUs to FC ELS responses and vice versa in iSCSI-FC bridges. (since an iSCSI-FC bridge would be mapping certain types of operations and their responses from one protocol to another like Login. Logout, NOP-OUT, etc.) o REJECT field in the Login Response PDU can be made reserved. 2) Whatever the mechanism that will be finally used for login rejection [be it the REJECT PDU or "reject login" in login response PDU], it should make provisions for detailed reason codes and reason explanations that allow the initiator to determine the cause of the rejection. This does not currently exist in the draft. 3) Quoting from the draft : "In the case that the Status is reject login the initiator should immediately close down its end of the TCP connection, thus freeing up the target's port for some other connection." On a REJECT, initiators may wish to retry Login. [perhaps, with a modification to their Login Command PDU or TEXT parameters . Perhaps, the word "may" might be a better fit instead of "should" here. Thanks, Santosh --------------5EE9DF34A44C87BA968AC215 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------5EE9DF34A44C87BA968AC215-- From owner-ips@ece.cmu.edu Mon Jan 8 00:42:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA14474 for ; Mon, 8 Jan 2001 00:42:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f083Y2Q11307 for ips-outgoing; Sun, 7 Jan 2001 22:34:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f083XPU11294 for ; Sun, 7 Jan 2001 22:33:25 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 3AC50367 for ; Sun, 7 Jan 2001 19:33:24 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id TAA03649; Sun, 7 Jan 2001 19:33:19 -0800 (PST) Message-ID: <3A59355C.7F06B3FD@cup.hp.com> Date: Sun, 07 Jan 2001 19:34:52 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : On the subject of R2T and Task Tags. Content-Type: multipart/mixed; boundary="------------785321837367D3D40FDF9568" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------785321837367D3D40FDF9568 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit On the subject of R2T and task tags ..... Section 2.16. R2T ============= 1) Differing nomenclatures "Target Task Tag" and "Target Transfer Tag" for the same field. Only 1 of the 2 should be used. The current semanticsof this field seem to call for a "Target R2T Tag" name and not a "Target Task Tag", since this is not per task. 2) The READ I/O PDUs do not use a "Target Task Tag". The WRITE I/O PDUs use a "Target Task Tag" per R2T and not per task. It seems like the only unique per I/O identifier is the Initiator Task Tag. (?) Is it the intention of iSCSI that targets should be using the Initiator Task Tag as the per I/O lookup tag ? If this is true, then, the name "Initiator Task Tag" is a misnomer and it should just be "Task Tag", since it is used by both initiators and targets. Explicitly stating the semantics of "Target Task Tag" in Section 2.2 would help, particularly since, it seems like iSCSI's concept of "Target Task Tag" differs from the semantics of the equivalent(?) RX_ID in the Fibre Channel header. iSCSI seems to be more closely aligned with Parallel SCSI's single Queue Tag model than Fibre Channel's OX_ID/RX_ID tag model. 3) Quoting from the draft : "The present document does not limit the number of outstanding datatransfers." As we had discussed earlier, this needs to be removed, since MaxOutstandingR2T bounds the number of outstanding R2Ts. Thanks, Santosh --------------785321837367D3D40FDF9568 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------785321837367D3D40FDF9568-- From owner-ips@ece.cmu.edu Mon Jan 8 04:12:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA28630 for ; Mon, 8 Jan 2001 04:12:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f086kA914302 for ips-outgoing; Mon, 8 Jan 2001 01:46:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f086jBU14277 for ; Mon, 8 Jan 2001 01:45:11 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Sun, 7 Jan 2001 22:54:38 -0800 Message-ID: From: Joshua Tseng To: Y P Cheng , "'Ips@Ece. Cmu. Edu'" Subject: RE: iFCP as an IP Storage Work Item Date: Sun, 7 Jan 2001 22:44:44 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk See below > No, the world does not need two standards and the IPS WG > should force the > issue. While companies will always do their own, the mission > of a standard > committee is to find one and only one standard to make everyone's life > easier. Failing to do so does not serve this community. > > If FCIP is good enough, why do I need iFCP? Do I really need the > scalability of 4 billion fibre channel nodes visible to me? > For Internet > domain names I may need IPv6, but, for storage devices? > 24-bits of D_ID > with 16 million nodes are a lot of addresses. I do > understand perfectly if > one wishes to dominate the FC switch market. As a consumer, > this is not my > concern unless one can provide me alternatives with a much > lower costs. Where have we heard the above line of reasoning before? Does it sound familiar to...."why do we need OSPF if we have RIP?" ..."why do we need IP/MPLS if we already have ATM?"...."why do we need DiffServ when we already have a standard for IP precedence?" .... "why do we need Qualcomm's CDMA when we have GSM?".... "why do we need iSCSI when we have Fibre Channel?" If the technology is superior and/or provides additional capabilities, it should be admitted to IETF. Admitting only the first standard to come along is a recipe for mediocrity. The Internet would not be what it is today if the IETF were to operate according to this standard. It hasn't in the past, and it should not start today with the IPS WG. iFCP provides truly superior capabilities to the alternative technologies, and may prove to be disruptive to the business plans of some companies. I believe therein lies the real reason why some object to its admission to the IPS. Josh From owner-ips@ece.cmu.edu Mon Jan 8 04:52:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA28830 for ; Mon, 8 Jan 2001 04:52:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0887Bp15484 for ips-outgoing; Mon, 8 Jan 2001 03:07:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0886rU15477 for ; Mon, 8 Jan 2001 03:06:53 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id JAA152668 for ; Mon, 8 Jan 2001 09:06:46 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id JAA199888 for ; Mon, 8 Jan 2001 09:06:45 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569CE.002C8723 ; Mon, 8 Jan 2001 09:06:21 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 8 Jan 2001 09:58:16 +0200 Subject: Re: iSCSI : On the subject of R2T and Task Tags. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Answers in text - Julo Santosh Rao on 08/01/2001 05:34:52 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iSCSI : On the subject of R2T and Task Tags. On the subject of R2T and task tags ..... Section 2.16. R2T ============= 1) Differing nomenclatures "Target Task Tag" and "Target Transfer Tag" for the same field. Only 1 of the 2 should be used. The current semanticsof this field seem to call for a "Target R2T Tag" name and not a "Target Task Tag", since this is not per task. will fix 2) The READ I/O PDUs do not use a "Target Task Tag". The WRITE I/O PDUs use a "Target Task Tag" per R2T and not per task. It seems like the only unique per I/O identifier is the Initiator Task Tag. (?) Is it the intention of iSCSI that targets should be using the Initiator Task Tag as the per I/O lookup tag ? If this is true, then, the name "Initiator Task Tag" is a misnomer and it should just be "Task Tag", since it is used by both initiators and targets. We use initiator and target for the task to state who is "generating" the tag Explicitly stating the semantics of "Target Task Tag" in Section 2.2 would help, particularly since, it seems like iSCSI's concept of "Target Task Tag" differs from the semantics of the equivalent(?) RX_ID in the Fibre Channel header. iSCSI seems to be more closely aligned with Parallel SCSI's single Queue Tag model than Fibre Channel's OX_ID/RX_ID tag model. iSCSI is completely neutral to the targets use of the target transfer tag it can follow both models. The iSCSI protocol treats the as an opaque handle to give back to the target to do the association. 3) Quoting from the draft : "The present document does not limit the number of outstanding datatransfers." As we had discussed earlier, this needs to be removed, since MaxOutstandingR2T bounds the number of outstanding R2Ts. will remove the offending statement Thanks, Santosh - santoshr.vcf From owner-ips@ece.cmu.edu Mon Jan 8 04:52:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA28841 for ; Mon, 8 Jan 2001 04:52:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f087oBQ15224 for ips-outgoing; Mon, 8 Jan 2001 02:50:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f087nBU15200 for ; Mon, 8 Jan 2001 02:49:11 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id IAA171058 for ; Mon, 8 Jan 2001 08:49:06 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA193360 for ; Mon, 8 Jan 2001 08:49:06 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569CE.002AF145 ; Mon, 8 Jan 2001 08:49:02 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 8 Jan 2001 09:44:49 +0200 Subject: Re: iSCSI : Unsolicited Inbound Commands for initiators. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, We did attempt to keep-up the SAM model in which initiators issue commands and targets receive them and issue responses. A command is bound to an initiator-task-id and there are no unsolicited commands. NOP is a "collapse" of a message (something not bound by a task-id) and a command/response. Logout was meant to enable clean termination. The 2 alternative to kick it off are AEN and NOP and we choose AEN. Julo Santosh Rao on 07/01/2001 06:44:21 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iSCSI : Unsolicited Inbound Commands for initiators. Julian, Can a Logout command PDU be originated by a target ? If so, this would eliminate operations like the target sending AEN with an iSCSI Event Indicator of "Target Requests Logout on this connection". In general, what is the policy of the iSCSI draft towards un-solicited inbound command PDUs received by an initiator. Which of the following commands can an initiator receive : 1) Login Command (Could be received during device discovery process when another initiator on the SAN is discovering its devices.) 2) Logout Command (Could be originated by targets as a form of error recovery.) 3) Text Command 4) NOP-OUT Command. (Could be originated by a target to test a connection or to request DataRN acknowledgement, when the max un-acknowledged DataRN limit has been reached.) Specifc elaboration in the draft of the commands that targets are allowed to originate would be desirable. Thanks, Santosh From owner-ips@ece.cmu.edu Mon Jan 8 14:49:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA15056 for ; Mon, 8 Jan 2001 14:49:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08HE1X01420 for ips-outgoing; Mon, 8 Jan 2001 12:14:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-1.cisco.com (sj-msg-core-1.cisco.com [171.71.163.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08HDKU01389 for ; Mon, 8 Jan 2001 12:13:20 -0500 (EST) Received: from sponge.cisco.com (sponge.cisco.com [171.71.61.25]) by sj-msg-core-1.cisco.com (8.9.3/8.9.1) with ESMTP id JAA15777; Mon, 8 Jan 2001 09:12:26 -0800 (PST) Received: from DAPW2K (dhcp-161-44-68-178.cisco.com [161.44.68.178]) by sponge.cisco.com (Mirapoint) with SMTP id AEZ02392; Mon, 8 Jan 2001 11:12:13 -0600 (CST) From: "David Peterson" To: , Subject: RE: new iSCSI draft - 02.txt Date: Mon, 8 Jan 2001 11:14:02 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit The point is (whether you use an mbuf/mblk/zbuf/socket buffer or whatever) some type of manipulation will be required. Bottom line is a periodic marker does not work well for a software based implementation. As I've indicated before I do not believe a framing mechanism is required at this point in time. We should start looking at what it will take to use SCTP (or something similar). When the 10G pipes become available in the future we will then be ready/positioned and have a solution that should work for all. -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Saturday, January 06, 2001 3:08 AM To: ips@ece.cmu.edu Subject: RE: new iSCSI draft - 02.txt Dave, I think that a shim can be inserted between the socket calls and the TCP copy to mbuf or from mbuf. On many socket implementations this can be achieved by just changing the protocol "name" in the socket data structure and then have the shim revert it to the original. However we should not be very concerned about stacks that use a full-fledged mbuf structure and copy the data - their rates will be too low and they might as well reject the option. Julo "David Peterson" on 05/01/2001 23:40:07 Please respond to "David Peterson" To: "Matt Wakeley" , "IPS Reflector" cc: Subject: RE: new iSCSI draft - 02.txt The problem is: at that layer there is simply not a byte stream. The data is contained in socket buffers which are most likely mbuf clusters for most implementations. To "simply insert a few extra bytes" requires mbuf manipulation. Mbuf manipulation can processing intensive. Depending on how the mbuf data is packaged a small DMA operation may occur for these few extra bytes also. Dave -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Matt Wakeley Sent: Wednesday, January 03, 2001 7:21 PM To: IPS Reflector Subject: Re: new iSCSI draft - 02.txt David Peterson wrote: > > I believe there was agreement to remove the Urgent-Pointer framing mechanism > but don't recall any agreement to replace it with an in-stream marker. For a > software implementation it would be hard to support this type of framing > mechanism. I believe a TCP option indicating the message boundry or a > fixed-length PDU at a granularity to minimize overhead are much better > solutions and are workable for both software and hardware implementations. I > have not seen an agenda but I would hope this issue would be discussed at > the upcoming meeting in Orlando. > Dave What is so difficult for software to insert a few extra bytes in the byte stream? It's simply a layering problem: iSCSI layer <-> marker layer <-> TCP Normally, the marker layer simply transfers the bytes from the iSCSI layer to the TCP. Every X number of bytes, the marker layer inserts the marker into the byte stream. Since your software will probably not benefit from the receipt of the markers, it would negotiate not to receive the markers. It would only send markers *IF* the remote node requested them to be sent. -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Mon Jan 8 15:39:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA16227 for ; Mon, 8 Jan 2001 15:39:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08IiYT04837 for ips-outgoing; Mon, 8 Jan 2001 13:44:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08IhYU04797 for ; Mon, 8 Jan 2001 13:43:35 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f08JrW023770; Mon, 8 Jan 2001 11:53:35 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Charles Monia" , "Y P Cheng" Cc: "Ips \(E-mail\)" Subject: RE: iFCP as an IP Storage Work Item Date: Mon, 8 Jan 2001 10:41:08 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Charles, With respect to merging FCIP and iFCP encapsulation, there are many technical merits for doing so without looking at the marketing issues. You have noted in your view of FCIP and iFCP as being in two separate markets and thus not likely to cooperate at the encapsulation level. It would seem you use marketing concerns in your positions. I would hope however that this group would have the ability to bring these two segments of the SAN market a bit closer together. I also see merit in the iFCP effort in that iSCSI is divergent with respect to existing markets. There will be many areas where FCIP and iFCP will find common solutions with many common problems. In the spirit of furthering common goals, iFCP and FCIP should use a common encapsulation where possible. I would not wish to bet if iFCP or iSCSI becomes a larger player in the marketplace. Looking at complexity, I would not place too many chips on iSCSI. I do not think this group needs to decide such winners and losers. If there were two iSCSI solutions or two iFCP solutions then there would a reason to merge these proposals. If there are two FC encapsulations proposals, this two should be merged. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Charles Monia > Sent: Saturday, January 06, 2001 7:18 PM > To: Y P Cheng > Cc: Ips (E-mail) > Subject: RE: iFCP as an IP Storage Work Item > > > Hi Y P: > > > I have been trying to avoid taking side on this iFCP vs. FCIP > > debate on the > > technical merits. May be I can take my technical hat off and > > debate on the > > business merits. > > According to whose criteria? Yours I guess. Others see the world > differently. > > > No, the world does not need two standards > > I'm glad someone has the inside track on what the world needs. > I'm perfectly > content to let the world make that decision. > > >................................and the IPS WG > > should force the > > issue. While companies will always do their own, the mission > > of a standard > > committee is to find one and only one standard to make everyone's life > > easier. > > Since when? I suggest you visit the T10 web site and look around (see > http://www.t10.org/scsi-3.htm). At last count, there were six, count 'em, > six SCSI encapsulations (not including iSCSI), all alive and well > -- not to > mention ATA. Incidentally, by a lot of measures, you'd probably be > justified in concluding that ATA's what the world really needs. > > Other stuff below. > > > -----Original Message----- > > From: Y P Cheng [mailto:ycheng@advansys.com] > > Sent: Saturday, January 06, 2001 2:26 PM > > To: 'Ips@Ece. Cmu. Edu' > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > While it is easy to say that as a vendor it should be > > possible for them to > > > support iSCSI, iFCP and FCIP protocols, the cost of > > developing a solution > > > that addresses all three (which includes not just product > > development but > > > testing, certifying and validating interoperable solutions) > > is something > > > that we as a vendor would not like to be drawn into. If you think > > > this is a business reason, and should be ignored, we > > > disagree with that opinion. > > > > I have been trying to avoid taking side on this iFCP vs. FCIP > > debate on the > > technical merits. May be I can take my technical hat off and > > debate on the > > business merits. > > > > No, the world does not need two standards and the IPS WG > > should force the > > issue. While companies will always do their own, the mission > > of a standard > > committee is to find one and only one standard to make everyone's life > > easier. Failing to do so does not serve this community. > > > > If FCIP is good enough, why do I need iFCP? > > Good enough for whom? Without rehashing this issue yet again, there are > valid constituencies for both solutions. > > >.......Do I really need the > > scalability of 4 billion fibre channel nodes visible to me? > > The iSCSI folks and others planning to build directly attached IP storage > devices are apt to find that argument strange. > > > For Internet > > domain names I may need IPv6, but, for storage devices? > > 24-bits of D_ID > > with 16 million nodes are a lot of addresses. > > That address space disappears fast if you're trying to intergrate a lot of > small FC sans. Each one consumes a 65 K block of FC addresses, > most of which > are unused. > > >..................................I do > > understand perfectly if > > one wishes to dominate the FC switch market. As a consumer, > > this is not my > > concern unless one can provide me alternatives with a much > > lower costs. > > > > As a customer, all I care is to have the ability to access > > storage devices > > on IP network at lower cost. I don't care the standard as > > long as there is > > one that gives me choices of low-cost vendors. > > You also could care less about how many standards there are. > > >...........Many people > > even believe > > among the networks of Ethernet, Fibre Channel, and > > InfiniBand, there will be > > only one winner. > > And many people don't. I haven't seen the parallel SCSI and FC > folks atart > folding up their tents yet. > > >.......................Therefore, among iFCP, FCIP, and iSCSI, > > please give me > > just one. > > Having said that, I do believe we need fibre channel before > > Ethernet folks taking over the world while InfiniBand lurks > > on the horizon. > > ISA was wonderful until EISA comes along. VESA was great > > until PCI appears. > > Now PCI-X and InfiniBand. We all suffer through technology > > transitions. > > Therefore, the last thing we need is to have another standard > > even before we > > start. Don't repeat VHS and Beta. I still have a lots of > > tapes in Beta > > format. > > > > > > And I have a lot of DVDs, VHS, 8MM, etc, etc, etc. > > Charles > From owner-ips@ece.cmu.edu Mon Jan 8 15:40:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA16249 for ; Mon, 8 Jan 2001 15:40:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08Iva805375 for ips-outgoing; Mon, 8 Jan 2001 13:57:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08IuuU05341 for ; Mon, 8 Jan 2001 13:56:56 -0500 (EST) Received: from sponge.cisco.com (sponge.cisco.com [171.71.61.25]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id KAA18067; Mon, 8 Jan 2001 10:56:52 -0800 (PST) Received: from DAPW2K (dhcp-161-44-68-178.cisco.com [161.44.68.178]) by sponge.cisco.com (Mirapoint) with SMTP id AEZ03819; Mon, 8 Jan 2001 12:56:41 -0600 (CST) From: "David Peterson" To: , "Ips@Ece. Cmu. Edu" Cc: Subject: RE: iFCP as an IP Storage Work Item Date: Mon, 8 Jan 2001 12:58:30 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <3A53A43F.DF33335E@sun.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Regarding #1: I believe the existing FC host drivers and HBA's will work quite well using FCIP or an iSCSI gateway. A gateway protocol is not required to allow an iSCSI HBA to function with an existing FC device. Regarding #2: under IETF rules there will be no support from multiple companies, only individual contributors:) -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Mark A. Carlson Sent: Wednesday, January 03, 2001 4:14 PM To: Ips@Ece. Cmu. Edu Cc: 'David Peterson'; 'Black_David@emc.com' Subject: Re: iFCP as an IP Storage Work Item In my mind, the issue of whether this should be a work item should only revolve around two issues: 1) Is there a customer need for this kind of protocol that is not being addressed by this or other working groups? 2) Is there enough interest in the working group (from multiple companies) to support the development of the protocol? On 1), I suspect that we will need to address some sort of "gateway" protocol for customers to use in transitioning from FC Fabrics to IP Networks. Today's reality is that we have FC host OS drivers and HBAs as well as FC devices that will be with us for some time during the transition. If you assume that you need a gateway protocol that does not require the upgrade of either end, I think that 1) is satisfied. There may be other gateway protocols needed as well for when you have an iSCSI HBA and OS Driver, but existing FC devices. Look at the market for the FC/Parallel SCSI routers/bridges to see a similar transitioning market. Whether we standardize it or not, these type of devices will exist due to the market need. On 2), I'd like to see more diversity in the support of iFCP among individuals from more than just the handful of companies that have spoken so far, but the real proof is in the interest in helping to author the related documents. One possibility is to task a subgroup with FC to IP storage transition protocols, and leave the FCIP subgroup to focus just on the bridging and tunneling issues. My thoughts. -- mark From owner-ips@ece.cmu.edu Mon Jan 8 15:40:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA16268 for ; Mon, 8 Jan 2001 15:40:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08IKcl03946 for ips-outgoing; Mon, 8 Jan 2001 13:20:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from tenornetworks.com (rtu.tenornetworks.com [63.77.213.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08IJxU03893 for ; Mon, 8 Jan 2001 13:19:59 -0500 (EST) Received: from tenornet.com (newman [192.168.0.185]) by tenornetworks.com (Switch-2.1.0/Switch-2.1.0) with SMTP id f08IJvF24557 for ; Mon, 8 Jan 2001 13:19:58 -0500 (EST) Received: from 192.168.0.185 by tenornet.com; MON, 8 Jan 2001 13:07:51 -0700 Received: by newman.tenornet.com with Internet Mail Service (5.5.2650.21) id ; Mon, 8 Jan 2001 13:07:50 -0500 Message-ID: <6B190B34070BD411ACA000B0D0214E563D3484@newman.tenornet.com> From: "Kavi, Prabhu" To: "'ips@ece.cmu.edu'" Subject: Questions about FCIP Date: Mon, 8 Jan 2001 13:07:41 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I have some questions about the Fibre Channel over TCP/IP I-D. I understand IP networks well, but am a Fibre Channel newbie and am asking these questions to learn more about Fibre Channel's requirements rather than to challenge the document. Here are my questions: 1. Why Fibre Channel over TCP, rather than over UDP? Is it because of an expectation of lossy IP networks? Or is it equally valid to put Fibre Channel over UDP (and therefore can be the subject of another I-D). 2. What are the typical values of the ED_TOV and R_A_TOV timers? Do I understand this right to say that these timers effectively limit the latency across the IP network? 3. What is an acceptable loss ratio of Fibre Channel datagrams that still allows the SAN devices to deliver appropriate end user performance (order of magnitude estimate is sufficient). The reasons I ask are: * I believe it is invalid to assume that IP networks are necessarily more lossy than Fibre Channel networks. A well-engineered IP network, using a combination of both Diff-Serv and MPLS, can achieve (close to) zero loss. * Stacking one reliable delivery protocol on top of another can possibly lead to interesting performance problems after a loss occurs, depending upon the timer values for TCP and Fibre Channel. Thanks in advance for your answers. Prabhu Kavi ---------------------------------------------------------------------- Prabhu Kavi Phone: 1-978-264-4900 x125 Director, Adv. Prod. Planning Fax: 1-978-264-0671 Tenor Networks Email: prabhu_kavi@tenornetworks.com 100 Nagog Park WWW: www.tenornetworks.com Acton, MA 01720 From owner-ips@ece.cmu.edu Mon Jan 8 15:42:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA16322 for ; Mon, 8 Jan 2001 15:42:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08I2Ud03272 for ips-outgoing; Mon, 8 Jan 2001 13:02:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.com (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08Hv8U03076 for ; Mon, 8 Jan 2001 12:57:08 -0500 (EST) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Mon, 8 Jan 2001 09:57:03 -0800 Message-ID: From: Robert Snively To: "'Murali Rajagopal'" , ips@ece.cmu.edu Subject: RE: Agree or Disagree to Merge FCIP and iFCP Date: Mon, 8 Jan 2001 09:57:01 -0800 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk I disagree with the merger of FCIP and iFCP. Bob Snively -----Original Message----- From: Murali Rajagopal [mailto:muralir@lightsand.com] Sent: Saturday, January 06, 2001 11:26 AM To: ips@ece.cmu.edu Subject: Agree or Disagree to Merge FCIP and iFCP --- snip--- Now with my TC hat on, I would like to propose that we proceed to solve this debate by getting a collective consensus from all folks on merging the two - protocol or otherwise. Please reply with a simple "Agree to merge" or "Disagree to merge" statement. From owner-ips@ece.cmu.edu Mon Jan 8 15:44:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA16390 for ; Mon, 8 Jan 2001 15:44:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08J5iq05751 for ips-outgoing; Mon, 8 Jan 2001 14:05:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08J5MU05729 for ; Mon, 8 Jan 2001 14:05:22 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Mon, 8 Jan 2001 11:14:45 -0800 Message-ID: From: Charles Monia To: Douglas Otis Cc: "Ips (E-mail)" Subject: RE: iFCP as an IP Storage Work Item Date: Mon, 8 Jan 2001 11:04:45 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Doug: Maybe we're closer to agreement than you think. Here's my response to Murali's call for consensus: >>>>>>>>>>>> "Thanks for bringing this matter to a head. Here's my .02: 1. Merging the iFCP and FCIP specifications -- No, not feasible on technical grounds. Anyhow, I think this is one decision that can't be made by fiat. 2. Definition of a common encapsulation protocol -- Technically possible, practically not feasible. From my perspective, it's risky and difficult to manage as the client specs evolve over time. Besides, I assume the FCIP encapsulation is a done deal. Bottom line: I vote no (but would grudgingly try to accommodate the WG consensus on this matter)." >>>>>>>>>>>>> The gist is that we're willing work with the FCIP community to achieve a common encapsulation if that is the consensus of the WG. Since the FCIP folks also have a stake in this, I suggest addressing your concerns to them as well. Charles > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Monday, January 08, 2001 10:41 AM > To: Charles Monia; Y P Cheng > Cc: Ips (E-mail) > Subject: RE: iFCP as an IP Storage Work Item > > > Charles, > > With respect to merging FCIP and iFCP encapsulation, there are many > technical merits for doing so without looking at the > marketing issues. You > have noted in your view of FCIP and iFCP as being in two > separate markets > and thus not likely to cooperate at the encapsulation level. > It would seem > you use marketing concerns in your positions. I would hope > however that > this group would have the ability to bring these two segments > of the SAN > market a bit closer together. I also see merit in the iFCP > effort in that > iSCSI is divergent with respect to existing markets. There > will be many > areas where FCIP and iFCP will find common solutions with many common > problems. > > In the spirit of furthering common goals, iFCP and FCIP > should use a common > encapsulation where possible. I would not wish to bet if > iFCP or iSCSI > becomes a larger player in the marketplace. Looking at > complexity, I would > not place too many chips on iSCSI. I do not think this group needs to > decide such winners and losers. If there were two iSCSI > solutions or two > iFCP solutions then there would a reason to merge these > proposals. If there > are two FC encapsulations proposals, this two should be merged. > > Doug > From owner-ips@ece.cmu.edu Mon Jan 8 15:45:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA16402 for ; Mon, 8 Jan 2001 15:45:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08IrbT05219 for ips-outgoing; Mon, 8 Jan 2001 13:53:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08Ir0U05194 for ; Mon, 8 Jan 2001 13:53:00 -0500 (EST) Received: from sponge.cisco.com (sponge.cisco.com [171.71.61.25]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id KAA15810; Mon, 8 Jan 2001 10:53:02 -0800 (PST) Received: from DAPW2K (dhcp-161-44-68-178.cisco.com [161.44.68.178]) by sponge.cisco.com (Mirapoint) with SMTP id AEZ03759; Mon, 8 Jan 2001 12:52:52 -0600 (CST) From: "David Peterson" To: "Joshua Tseng" , Subject: RE: iFCP as an IP Storage Work Item Date: Mon, 8 Jan 2001 12:54:41 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit See comments below [dap]: -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Joshua Tseng Sent: Wednesday, January 03, 2001 6:22 PM To: ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item I believe iFCP should be an IPS work item for the following technical reasons: 1) iFCP allows leverage of existing FCP-based driver stacks and preservation of the $$$ and @#$!!% that have been invested in them by vendor companies and their customers. [dap]: Maybe I'm mistaken but I believe connecting an existing FCP-based driver stack to an iSCSI gateway or FCIP implementation would preserve the investment. 2) IP switching and routing technology is far more mature, stable, and scalable than the Fibre Channel equivalent. iFCP leverages this aspect of IP technology, while FCIP does not. [dap]: The IP switching and routing technology may also be used in a FCIP implementation that is FC-BB-2 capable. With regard to your reasons why iFCP should not be permitted: > > The iFCP proposal for encapsulating Fibre Channel frames > should not be a > work item for the IP Storage working group. > This view is taken by Cisco Systems and Brocade Communications for the > following technical reasons. > > 1. FCIP is aligned with the existing FC-BB-2 work in progress. We support independent development of the FCIP separate from iFCP, but this point has nothing to do with iFCP. > 2. FCIP will function with any existing/future FC-4 protocol such as > FCP/FCP-2, VI, SRP. Again, this point says nothing about the technical feasibility of iFCP. Regardless, iFCP can encapsulate any protocol that FCIP does. > 3. A native iSCSI device will provide the same functionality > as a device > connected to an iFCP gateway. Just because iSCSI is an official IPS WG item doesn't mean that FC devices will disappear from the face of the earth. There is still a market need to internetwork FC devices over IP networks. [dap]: Correct, this is what FCIP and FC-BB-2 are all about. > 4. Combining iFCP and FCIP does not make any sense since they are > implemented using two different routing planes. In addition, FCIP is a > tunneling protocol and requires mimimal processing of the > Fibre Channel > frame. In contrast, iFCP is a gateway protocol and requires much more > processing such as the reading and modification of the Fibre > Channel header, > augmentation data for specific Fibre Channel Extended Link > Services, and > special handling of specific Fibre Channel Extended Link Services. I disagree that the amount of processing is prohibitive in any way. I would like to understand why you feel this way. As iFCP is a gateway protocol, a gateway device typically has more resources available to perform these types of operations than an end device. > 5. Existing Fibre Channel device addressing capability in > conjunction with > FCIP is viable at this point in time. Again, this says nothing about iFCP. [dap]: You are missing the point. Collectively these are the technical reasons we believe yet another protocol is not required to achieve the same functionality that iSCSI and FCIP already provide. > 6. Manipulation of N_Port ID's as specified in iFCP will make > debugging/analysis extremely difficult. I am confused as to why you think debugging/analysis will be difficult. Please explain. I don't think this is true, but even if it is, the same can be said of many technologies that are described in RFC's. Take NAT for example. I and my former network buddies have had a #@!&% of a time working with NAT. Does that mean NAT should be outlawed in the IETF? No, of course not! It is needed to internetwork subnets numbered under the "old" IPv4 regime. Until we reach the promised land of IPv6 everywhere, we will continue to see a lot of NAT. Even after IPv6, I still think there will be a lot NAT around, probably until the year 2654. Same thing for iSCSI. I believe iFCP and iSCSI will coexist into the forseeable future. [dap]: An always-current translation table will be required to debug and analyze the iFCP protocol. > > In summary, FCIP exists today and is well-defined. FCIP > provides all of the > features > necessary to tunnel Fibre Channel over IP networks. Similarly, iSCSI Of course FCIP allows tunneling--it IS a tunneling protocol. iFCP is NOT a tunneling protocol. iFCP does NOT tunnel Fibre Channel frames. Rather, it allows FC devices to be natively internetworked over an IP fabric. You can't do that with FCIP. You can't do that with iSCSI. iFCP is NOT redundant. Regards, Josh > provides the > capability for executing the SCSI command set over IP > networks. Creating > yet another > protocol, such as iFCP, is redundant. > > David Peterson > Cisco Systems, Inc (SRBU) > Lead Architect - Standards Development > Office: 763-398-1007 > Cell: 612-802-3299 > e-mail: dap@cisco.com > > Bob Snively > Brocade Communications Systems > Principal Engineer > Office: 408 487 8135 > e-mail: rsnively@brocade.com > > Mark Bakke > Cisco Systems, Inc (SRBU) > Chief Architect > From owner-ips@ece.cmu.edu Mon Jan 8 16:52:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA17343 for ; Mon, 8 Jan 2001 16:52:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08JbZT06911 for ips-outgoing; Mon, 8 Jan 2001 14:37:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08JahU06885 for ; Mon, 8 Jan 2001 14:36:44 -0500 (EST) Received: from sponge.cisco.com (sponge.cisco.com [171.71.61.25]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id LAA14039 for ; Mon, 8 Jan 2001 11:36:40 -0800 (PST) Received: from DAPW2K (dhcp-161-44-68-178.cisco.com [161.44.68.178]) by sponge.cisco.com (Mirapoint) with SMTP id AEZ04445; Mon, 8 Jan 2001 13:36:31 -0600 (CST) From: "David Peterson" To: Subject: RE: Agree or Disagree to Merge FCIP and iFCP Date: Mon, 8 Jan 2001 13:38:20 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <005101c07816$860593e0$75b30118@orng1.occa.home.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Disagree to merge. Dave > > Now with my TC hat on, I would like to propose that we proceed to > solve this > debate by getting a collective consensus from all folks on > merging the two - > protocol or otherwise. > > Please reply with a simple "Agree to merge" or "Disagree to merge" > statement. No long explanations as to why or why not please! > > -Murali Rajagopal > From owner-ips@ece.cmu.edu Mon Jan 8 17:32:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA17809 for ; Mon, 8 Jan 2001 17:32:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08KOa408562 for ips-outgoing; Mon, 8 Jan 2001 15:24:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08KOHU08532 for ; Mon, 8 Jan 2001 15:24:17 -0500 (EST) Received: from centralmail2.Central.Sun.COM ([129.147.62.11]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id NAA11837; Mon, 8 Jan 2001 13:24:15 -0700 (MST) Received: from thor.Central.Sun.COM (thor.Central.Sun.COM [129.147.248.69]) by centralmail2.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id NAA22821; Mon, 8 Jan 2001 13:24:13 -0700 (MST) Received: from Sun.COM (sunray11.Central.Sun.COM [129.147.71.13]) by thor.Central.Sun.COM (8.9.1b+Sun/8.9.1) with ESMTP id NAA14854; Mon, 8 Jan 2001 13:24:07 -0700 (MST) Message-ID: <3A5A21EC.9E179900@Sun.COM> Date: Mon, 08 Jan 2001 13:24:12 -0700 From: mark carlson Organization: Sun Microsystems X-Mailer: Mozilla 4.75 [en] (X11; U; SunOS 5.7 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: David Peterson CC: "Ips@Ece. Cmu. Edu" , Black_David@emc.com Subject: Re: iFCP as an IP Storage Work Item References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David Peterson wrote: > > Regarding #1: I believe the existing FC host drivers and HBA's will > work quite well using FCIP or an iSCSI gateway. A gateway protocol > is not required to allow an iSCSI HBA to function with an existing FC > device. In order to use FCIP, I have to purchase at least 2 FC switches and 2 FCIP devices (unless it's in the switch) to connect my FC HBA and my FC Disk even if they are local to each other. While you could do it this way, my preference would be a simple, cheap conversion box at each end. This same conversion box, if it supported the iSCSI protocol as well, would transition me to iSCSI when I have an iSCSI HBA. > > Regarding #2: under IETF rules there will be no support from multiple > companies, only individual contributors:) You misunderstand. While there is no requirement that the individual contributors come from separate companies, I would not like to see all the authors of the document coming from the same company ;-) -- mark From owner-ips@ece.cmu.edu Mon Jan 8 17:32:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA17823 for ; Mon, 8 Jan 2001 17:32:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08L0kO10087 for ips-outgoing; Mon, 8 Jan 2001 16:00:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08KxgU10050 for ; Mon, 8 Jan 2001 15:59:42 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Mon, 8 Jan 2001 13:09:16 -0800 Message-ID: From: Joshua Tseng To: David Peterson , ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Mon, 8 Jan 2001 12:59:14 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk David, See below: > > See comments below [dap]: > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Joshua Tseng > Sent: Wednesday, January 03, 2001 6:22 PM > To: ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > I believe iFCP should be an IPS work item for the following > technical reasons: > > 1) iFCP allows leverage of existing FCP-based driver stacks and > preservation of the $$$ and @#$!!% that have been invested in them > by vendor companies and their customers. > > [dap]: Maybe I'm mistaken but I believe connecting an > existing FCP-based > driver stack to an iSCSI gateway or FCIP implementation would preserve > the investment. I agree iSCSI and FCIP are viable solutions. But there are also end users who would like to migrate to a 100% IP network, and to internetwork all of their existing storage assets through a 100% IP network. FCIP cannot provide this, and iFCP is the most efficient and straightforward way to accomplish this. > > 2) IP switching and routing technology is far more mature, > stable, and scalable than the Fibre Channel equivalent. iFCP > leverages this aspect of IP technology, while FCIP does not. > > [dap]: The IP switching and routing technology may also be used > in a FCIP implementation that is FC-BB-2 capable. Yes, agreed, but FCIP connects E_PORTs, and that requires that a Fibre Channel fabric be in place. Once again, some customers would rather not have this, in order to internetwork their existing FC storage assets. > > With regard to your reasons why iFCP should not be permitted: > > > > The iFCP proposal for encapsulating Fibre Channel frames > > should not be a > > work item for the IP Storage working group. > > This view is taken by Cisco Systems and Brocade > Communications for the > > following technical reasons. > > > > 1. FCIP is aligned with the existing FC-BB-2 work in progress. > > We support independent development of the FCIP separate from iFCP, but > this point has nothing to do with iFCP. > > > 2. FCIP will function with any existing/future FC-4 protocol such as > > FCP/FCP-2, VI, SRP. > > Again, this point says nothing about the technical > feasibility of iFCP. > Regardless, iFCP can encapsulate any protocol that FCIP does. > > > 3. A native iSCSI device will provide the same functionality > > as a device > > connected to an iFCP gateway. > > Just because iSCSI is an official IPS WG item doesn't mean that > FC devices will disappear from the face of the earth. There is still > a market need to internetwork FC devices over IP networks. > > [dap]: Correct, this is what FCIP and FC-BB-2 are all about. > > > 4. Combining iFCP and FCIP does not make any sense since they are > > implemented using two different routing planes. In > addition, FCIP is a > > tunneling protocol and requires mimimal processing of the > > Fibre Channel > > frame. In contrast, iFCP is a gateway protocol and requires > much more > > processing such as the reading and modification of the Fibre > > Channel header, > > augmentation data for specific Fibre Channel Extended Link > > Services, and > > special handling of specific Fibre Channel Extended Link Services. > > I disagree that the amount of processing is prohibitive in any > way. I would like to understand why you feel this way. As iFCP is > a gateway protocol, a gateway device typically has more resources > available to perform these types of operations than an end device. > > > 5. Existing Fibre Channel device addressing capability in > > conjunction with > > FCIP is viable at this point in time. > > Again, this says nothing about iFCP. > > [dap]: You are missing the point. Collectively these are the technical > reasons > we believe yet another protocol is not required to achieve the same > functionality > that iSCSI and FCIP already provide. We believe iFCP provides additional capabilities that FCIP cannot provide. With regard to iSCSI, iFCP is a gateway technology and is designed to work with existing FC devices. iSCSI has no such objectives in mind. > > > 6. Manipulation of N_Port ID's as specified in iFCP will make > > debugging/analysis extremely difficult. > > I am confused as to why you think debugging/analysis will be > difficult. Please explain. I don't think this is true, but even > if it is, the same can be said of many technologies that are described > in RFC's. Take NAT for example. I and my former network buddies have > had a #@!&% of a time working with NAT. Does that mean NAT should be > outlawed in the IETF? No, of course not! It is needed to > internetwork > subnets numbered under the "old" IPv4 regime. Until we reach the > promised land of IPv6 everywhere, we will continue to see a > lot of NAT. > Even after IPv6, I still think there will be a lot NAT > around, probably > until the year 2654. Same thing for iSCSI. I believe iFCP and iSCSI > will coexist into the forseeable future. > > [dap]: An always-current translation table will be required > to debug and > analyze the iFCP protocol. Yes, and this is the norm in IP networking itself. ARP tables and caches exist in every network switch and router, to translate IP addresses to ethernet MAC addresses. This hasn't proven to be a serious problem in debugging IP networks. > > > > > In summary, FCIP exists today and is well-defined. FCIP > > provides all of the > > features > > necessary to tunnel Fibre Channel over IP networks. > Similarly, iSCSI > > Of course FCIP allows tunneling--it IS a tunneling protocol. iFCP > is NOT a tunneling protocol. iFCP does NOT tunnel Fibre > Channel frames. > Rather, it allows FC devices to be natively internetworked over an IP > fabric. You can't do that with FCIP. You can't do that with > iSCSI. iFCP > is NOT redundant. > > Regards, > Josh > > > provides the > > capability for executing the SCSI command set over IP > > networks. Creating > > yet another > > protocol, such as iFCP, is redundant. > > > > David Peterson > > Cisco Systems, Inc (SRBU) > > Lead Architect - Standards Development > > Office: 763-398-1007 > > Cell: 612-802-3299 > > e-mail: dap@cisco.com > > > > Bob Snively > > Brocade Communications Systems > > Principal Engineer > > Office: 408 487 8135 > > e-mail: rsnively@brocade.com > > > > Mark Bakke > > Cisco Systems, Inc (SRBU) > > Chief Architect > > > From owner-ips@ece.cmu.edu Mon Jan 8 17:32:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA17835 for ; Mon, 8 Jan 2001 17:32:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08KGcD08222 for ips-outgoing; Mon, 8 Jan 2001 15:16:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08KFnU08191 for ; Mon, 8 Jan 2001 15:15:50 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id 2A2B4973 for ; Mon, 8 Jan 2001 12:15:49 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id MAA24127; Mon, 8 Jan 2001 12:15:47 -0800 (PST) Message-ID: <3A5A1FA5.DD5A309F@hp.com> Date: Mon, 08 Jan 2001 12:14:29 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI: data numbering Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Two points on the data numbering. 1) It seems to me that the negociation of the Data numbering is useless and must be removed from the spec. Because : - there is the P bit in the SCSI data (READ) pdu header and the initiator must ack if P is set. Doing this way gives all flexibility to the target. - if the target wants the data numbering, the initiator MUST do it. Based on that, negociating the data numbering doesn't add any value. We have to remove it. The "P" bit is perfect. 2) The SCSI data READ pdu header must contains a target task tag. Doing that, avoids the target to do a lookup from the initiator task tag each time it receives a data ack (NOP out). The lookups (using hashing table for ex) must be kept to a minimum because they slow down operations. We have to avoid them on the performance path. In iSCSI the only place where a lookup takes place is for Abort task (lookup done on the target side) for "retry" a command (lookup done on the target side) and for read data ack (lookup done on the target side too). In the two first operations a lookup on the initiator task tag can not be avoided because the target may not have time to give a tag to the initiator before things go wrong. However it is not a problem because it is not on the performance path. But for read data acknowledgement, it is on the performance path. This lookup can be easily avoided if the target provides a tag in the data PDU, the NOP out passes back this tag to the target. The tag is the same all along the [READ] command. Regards, Pierre From owner-ips@ece.cmu.edu Mon Jan 8 17:34:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA17893 for ; Mon, 8 Jan 2001 17:34:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08L1bg10111 for ips-outgoing; Mon, 8 Jan 2001 16:01:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08L1WU10105 for ; Mon, 8 Jan 2001 16:01:33 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f08MCe023876; Mon, 8 Jan 2001 14:12:40 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Charles Monia" Cc: "Ips \(E-mail\)" Subject: RE: iFCP as an IP Storage Work Item Date: Mon, 8 Jan 2001 13:00:16 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Charles, Standards consolidation can not be placed one individual. Those wishing to create any FC encapsulating transport whether it is eventually used as a tunnel, gateway, or target would need to justify lack of cooperation in encapsulation consolidation. Not interfering with a potential iSCSI market is little justification for making an exception. In my view, neither has a strong case for not consolidating their encapsulation schemes. Your concerns about eventual extensions can be handled through provisions within the encapsulating structures. In the long run, finding a means for independent maturation will not degrade a final product, but improve it. Success of any solution using a common encapsulation will improve the standings of all solutions. In that case, you may have a smaller piece of the pie, but the pie gets bigger. Doug > Hi Doug: > > Maybe we're closer to agreement than you think. > > Here's my response to Murali's call for consensus: > > >>>>>>>>>>>> > "Thanks for bringing this matter to a head. > > Here's my .02: > > 1. Merging the iFCP and FCIP specifications -- No, not feasible on > technical grounds. Anyhow, I think this is one decision that can't be > made by fiat. > > 2. Definition of a common encapsulation protocol -- Technically possible, > practically not feasible. From my perspective, it's risky and difficult to > manage as the client specs evolve over time. Besides, I assume the FCIP > encapsulation is a done deal. Bottom line: I vote no (but would grudgingly > try to accommodate the WG consensus on this matter)." > >>>>>>>>>>>>> > > The gist is that we're willing work with the FCIP community to achieve a > common encapsulation if that is the consensus of the WG. Since the FCIP > folks also have a stake in this, I suggest addressing your > concerns to them > as well. > > Charles > > > -----Original Message----- > > From: Douglas Otis [mailto:dotis@sanlight.net] > > Sent: Monday, January 08, 2001 10:41 AM > > To: Charles Monia; Y P Cheng > > Cc: Ips (E-mail) > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > Charles, > > > > With respect to merging FCIP and iFCP encapsulation, there are many > > technical merits for doing so without looking at the > > marketing issues. You > > have noted in your view of FCIP and iFCP as being in two > > separate markets > > and thus not likely to cooperate at the encapsulation level. > > It would seem > > you use marketing concerns in your positions. I would hope > > however that > > this group would have the ability to bring these two segments > > of the SAN > > market a bit closer together. I also see merit in the iFCP > > effort in that > > iSCSI is divergent with respect to existing markets. There > > will be many > > areas where FCIP and iFCP will find common solutions with many common > > problems. > > > > In the spirit of furthering common goals, iFCP and FCIP > > should use a common > > encapsulation where possible. I would not wish to bet if > > iFCP or iSCSI > > becomes a larger player in the marketplace. Looking at > > complexity, I would > > not place too many chips on iSCSI. I do not think this group needs to > > decide such winners and losers. If there were two iSCSI > > solutions or two > > iFCP solutions then there would a reason to merge these > > proposals. If there > > are two FC encapsulations proposals, this two should be merged. > > > > Doug > > > > > > From owner-ips@ece.cmu.edu Mon Jan 8 17:48:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA18110 for ; Mon, 8 Jan 2001 17:48:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08L2bK10140 for ips-outgoing; Mon, 8 Jan 2001 16:02:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08L2NU10128 for ; Mon, 8 Jan 2001 16:02:23 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Mon, 8 Jan 2001 13:11:57 -0800 Message-ID: From: Charles Monia To: "David Peterson (E-mail)" Cc: "Ips (E-mail)" Subject: RE: iFCP as an IP Storage Work Item Date: Mon, 8 Jan 2001 13:01:56 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi David: Thanks for the feedback. To avoid losing sight of the forest for the trees, I'd like to step back and look at the broader picture. Our basic position is that the combination of features provided by iFCP, taken in totality, provide unique and useful capabilities. Yes, there is overlap to some degree with other proposals. iSCSI is a new paradigm for storage and FCIP will provide users with the ability to use IP in a limited role to integrate existing FC infrastructures. As we've stated before in some detail, to the extent that iFCP fully leverages IP technology as the basis for a FC storage solution, we believe it offers unique intrinsic value. Obviously, others may not see it that way. At this point, I can't think of what more to say that might change their perception. So be it. As to the technical difficulty of debugging such systems, we've had a lot of hands on experience here and haven't found it to be a problem. What's more, the experts tell me there are network black boxes that do similar kinds of address mapping, so I assume the problem (if there is one) is manageable. In any event, I believe reasonable people can disagree on this point. Charles > -----Original Message----- > From: David Peterson [mailto:dap@cisco.com] > Sent: Monday, January 08, 2001 10:55 AM > To: Joshua Tseng; ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > See comments below [dap]: > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Joshua Tseng > Sent: Wednesday, January 03, 2001 6:22 PM > To: ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > I believe iFCP should be an IPS work item for the following > technical reasons: > > 1) iFCP allows leverage of existing FCP-based driver stacks and > preservation of the $$$ and @#$!!% that have been invested in them > by vendor companies and their customers. > > [dap]: Maybe I'm mistaken but I believe connecting an > existing FCP-based > driver stack to an iSCSI gateway or FCIP implementation would preserve > the investment. > > 2) IP switching and routing technology is far more mature, > stable, and scalable than the Fibre Channel equivalent. iFCP > leverages this aspect of IP technology, while FCIP does not. > > [dap]: The IP switching and routing technology may also be used > in a FCIP implementation that is FC-BB-2 capable. > > With regard to your reasons why iFCP should not be permitted: > > > > The iFCP proposal for encapsulating Fibre Channel frames > > should not be a > > work item for the IP Storage working group. > > This view is taken by Cisco Systems and Brocade > Communications for the > > following technical reasons. > > > > 1. FCIP is aligned with the existing FC-BB-2 work in progress. > > We support independent development of the FCIP separate from iFCP, but > this point has nothing to do with iFCP. > > > 2. FCIP will function with any existing/future FC-4 protocol such as > > FCP/FCP-2, VI, SRP. > > Again, this point says nothing about the technical > feasibility of iFCP. > Regardless, iFCP can encapsulate any protocol that FCIP does. > > > 3. A native iSCSI device will provide the same functionality > > as a device > > connected to an iFCP gateway. > > Just because iSCSI is an official IPS WG item doesn't mean that > FC devices will disappear from the face of the earth. There is still > a market need to internetwork FC devices over IP networks. > > [dap]: Correct, this is what FCIP and FC-BB-2 are all about. > > > 4. Combining iFCP and FCIP does not make any sense since they are > > implemented using two different routing planes. In > addition, FCIP is a > > tunneling protocol and requires mimimal processing of the > > Fibre Channel > > frame. In contrast, iFCP is a gateway protocol and requires > much more > > processing such as the reading and modification of the Fibre > > Channel header, > > augmentation data for specific Fibre Channel Extended Link > > Services, and > > special handling of specific Fibre Channel Extended Link Services. > > I disagree that the amount of processing is prohibitive in any > way. I would like to understand why you feel this way. As iFCP is > a gateway protocol, a gateway device typically has more resources > available to perform these types of operations than an end device. > > > 5. Existing Fibre Channel device addressing capability in > > conjunction with > > FCIP is viable at this point in time. > > Again, this says nothing about iFCP. > > [dap]: You are missing the point. Collectively these are the technical > reasons > we believe yet another protocol is not required to achieve the same > functionality > that iSCSI and FCIP already provide. > > > 6. Manipulation of N_Port ID's as specified in iFCP will make > > debugging/analysis extremely difficult. > > I am confused as to why you think debugging/analysis will be > difficult. Please explain. I don't think this is true, but even > if it is, the same can be said of many technologies that are described > in RFC's. Take NAT for example. I and my former network buddies have > had a #@!&% of a time working with NAT. Does that mean NAT should be > outlawed in the IETF? No, of course not! It is needed to > internetwork > subnets numbered under the "old" IPv4 regime. Until we reach the > promised land of IPv6 everywhere, we will continue to see a > lot of NAT. > Even after IPv6, I still think there will be a lot NAT > around, probably > until the year 2654. Same thing for iSCSI. I believe iFCP and iSCSI > will coexist into the forseeable future. > > [dap]: An always-current translation table will be required > to debug and > analyze the iFCP protocol. > > > > > In summary, FCIP exists today and is well-defined. FCIP > > provides all of the > > features > > necessary to tunnel Fibre Channel over IP networks. > Similarly, iSCSI > > Of course FCIP allows tunneling--it IS a tunneling protocol. iFCP > is NOT a tunneling protocol. iFCP does NOT tunnel Fibre > Channel frames. > Rather, it allows FC devices to be natively internetworked over an IP > fabric. You can't do that with FCIP. You can't do that with > iSCSI. iFCP > is NOT redundant. > > Regards, > Josh > > > provides the > > capability for executing the SCSI command set over IP > > networks. Creating > > yet another > > protocol, such as iFCP, is redundant. > > > > David Peterson > > Cisco Systems, Inc (SRBU) > > Lead Architect - Standards Development > > Office: 763-398-1007 > > Cell: 612-802-3299 > > e-mail: dap@cisco.com > > > > Bob Snively > > Brocade Communications Systems > > Principal Engineer > > Office: 408 487 8135 > > e-mail: rsnively@brocade.com > > > > Mark Bakke > > Cisco Systems, Inc (SRBU) > > Chief Architect > > > From owner-ips@ece.cmu.edu Mon Jan 8 18:15:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA18465 for ; Mon, 8 Jan 2001 18:15:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08LKcf10762 for ips-outgoing; Mon, 8 Jan 2001 16:20:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mcmail.mcdata.com ([144.49.1.5]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08LJoU10736 for ; Mon, 8 Jan 2001 16:19:50 -0500 (EST) Received: from exchange5.mcdata.com (exchange5.mcdata.com [144.49.33.200]) by mcmail.mcdata.com (8.8.6 (PHNE_14041)/8.8.6) with ESMTP id FAA04285; Mon, 8 Jan 2001 05:16:30 -0700 (MST) Received: by exchange5.mcdata.com with Internet Mail Service (5.5.2650.21) id ; Mon, 8 Jan 2001 14:19:33 -0700 Message-ID: From: Henry Yang To: "'Murali Rajagopal'" , ips@ece.cmu.edu Subject: RE: Agree or Disagree to Merge FCIP and iFCP Date: Mon, 8 Jan 2001 14:19:32 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I disagree with merging FCIP and iFCP. Henry Yang -----Original Message----- From: Murali Rajagopal [mailto:muralir@lightsand.com] Sent: Saturday, January 06, 2001 12:26 PM To: ips@ece.cmu.edu Subject: Agree or Disagree to Merge FCIP and iFCP All: The FCIP and iFCP address the FC over IP network transport with entirely two different objectives. FCIP is a simple-minded tunnel that wants to see the basic FC connectivity extended over IP network. Period!!! iFCP's goal is much more than that .... But that does not mean that we should combine the two just because it appaers as though FCIP looks like a subset of iFCP. I am speaking for the authors of FCIP collectively, and in our opinion mixing the two in a single spec. or a common encapsulation method has little meaning.Yes we understand that everyone would like to see a single FC Over IP network solution and not two. But FCIP is really an extender for FC Switch Fabrics while iFCP is attempting to "replace" the FC Switching fabric. In this sense, iFCP is not in the best interest of either the FC Switch vendors or even the T11 FC community at large. So mixing them makes little business sense. In summary, the FCIP authors would like to keep the FCIP and the iFCP efforts seperate. I beleive that the iFCP folks are of the same opinion. Now with my TC hat on, I would like to propose that we proceed to solve this debate by getting a collective consensus from all folks on merging the two - protocol or otherwise. Please reply with a simple "Agree to merge" or "Disagree to merge" statement. No long explanations as to why or why not please! -Murali Rajagopal ----- Original Message ----- From: "Charles Monia" To: ; ; "Charles Monia" ; Sent: Friday, January 05, 2001 4:54 PM Subject: RE: iFCP as an IP Storage Work Item > > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Friday, January 05, 2001 4:15 PM > > To: dotis@sanlight.net; cmonia@nishansystems.com; ips@ece.cmu.edu > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > Before this goes any further ... the two of you may be > > in violent agreement. Charles objected to a "handshake"; > > I don't think this objection would preclude one end of > > a connection announcing the protocol that it intends to > > use so that the other end can cleanly and quickly > > terminate the connection if it can't or won't use > > that protocol (ideally with an error code to the > > other end indicating protocol incompatibility, so > > the misconfiguration becomes obvious). > > Doug's > > suggestion of IANA-allocated numbers for identifying > > protocols for this purpose is reasonable. > > > > As to the virtues of being able to speak more than > > one protocol on a port, Fibre Channel provides examples > > of where this sort of thing has been added after the > > initial specifications were done, so I wouldn't dismiss > > this out of hand. > > > > --David > > Hi David: > > It seems to me that with or without a common encapsulation method, such > boxes can be built 'now'. As I said earlier, a common encapsulation should > make it easier, which is a good thing. > > So, what we're left with is the question of how such a box would shift > gears. If an "in-band" method is easier, the simpler the handshake, the > better. So an approach based on an IANA-assigned parameter seems reasonable > to me. > > While this is a fruitful discussion, I'd be interested in hearing from the > FCIP folks on this issue. > > Charles > > > > > > > > -----Original Message----- > > > From: Douglas Otis [SMTP:dotis@sanlight.net] > > > Sent: Friday, January 05, 2001 5:59 PM > > > To: Charles Monia; Ips@Ece. Cmu. Edu > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > Charles, > > > > > > I disagree with this assumption about rigidity of > > protocols. Using a > > > common > > > initialization field would allow node handling protocols to > > be confirmed. > > > Initialization should also include a version number or > > option field for > > > the > > > encapsulation where this version or option be negotiated. > > If the node > > > handling protocol, separate from the encapsulation > > specification, could be > > > identified, then there would be fewer strange events if someone > > > mis-configured IP ports. I am not a fan of using text > > strings to do this > > > function and would prefer IANA defined values for rigid > > structures. In > > > addition to indicating an encapsulation version or options > > designator, > > > include the node type designator to signify how the node is > > handled to > > > avoid > > > strange failures. > > > > > > As a gateway, tunnel or perhaps as an end device in perhaps > > the distant > > > future, the type of node protocols may vary to meet > > different needs. The > > > lion's share of the work however could be covered within > > the encapsulation > > > specifications and clever means of handling the node could > > be isolated > > > from > > > these details. It does not mean that in the end all node > > protocols will > > > use > > > identical encapsulation, but only that, as much as > > possible, encapsulation > > > is kept common and the effort to encapsulate is seen as a > > separate task to > > > that of handling the frame at the end point. I would hope that a > > > transport > > > could be developed that would not care how the node was > > handled and where > > > this node handling function is defined within a separate process. > > > > > > It seems like a fairly clean separation of encapsulation > > and node handling > > > that would help foster support for these markets especially if a few > > > devices > > > allow a wide range of possible node handling scenarios. I > > agree that IP > > > fabric is more extensible to that of FC, however even with that > > > assumption, > > > the final goal of fully incorporating IP fabric without > > altering the FCP > > > structures has not been achieved by either of these current > > proposals. > > > With > > > that said, I would be in favor of developing a universal FC > > encapsulation > > > transport as a WG proposal that would support both iFCP and FCIP. > > > > > > Doug > > > > > > > > Ken Hirata wrote: > > > > > > Why do you want to standardize a common encapsulation protocol > > > > > > for FCIP and iFCP if their semantics and behavior are > > completely > > > > > > different? Would you want tunneling protocol > > implementations to > > > > > > also augment certain ELSs even though it isn't necessary > > > > > for tunneling > > > > > > protocol operation? > > > > > > > > > > If I were to build hardware that either assisted or completely > > > > > processed both iFCP and FCIP, it sure would be easier to do > > > > > header parsing and other common processing if there was just > > > > > one format. > > > > > > > > > > > If a common encapsulation protocol was defined, I believe a > > > > > > negotiation protocol would be necessary to distinguish between > > > > > > usage as a gateway or tunneling protocol. > > > > > > > > > > Yes, either negotiation of a flag bit in the encapsulating > > > > > header used to choose which algorithm to use. > > > > > > > > > Hi David and Ken: > > > > > > > > I agree that a common encapsulation may make it marginally easier > > > > to build a > > > > multi-protocol box as well as having other benefits. > > However, from the > > > > above, I'm concerned that some might infer that > > multi-protocol support > > > > should be a requirement. Just to be clear on this point: While > > > > I'm all for > > > > doing things that encourage commonality (where it makes sense > > > > technically) I > > > > feel that a standard ought not to needlessly burden a product with > > > > supporting both architectures. > > > > > > > > With regard to 'negotiation', I believe such a handshake is > > > > unnecessary and > > > > undesirable. In a real system, I can't see a scenario > > where it buys > > > > anything to make this dynamic. As I see it, these > > choices are cast in > > > > concrete when the SAN is implemented and aren't going to change > > > > from day to > > > > day. Also, for hardwired boxes that only support one > > protocol, it simply > > > > adds complexity. If someone wants to build a multi-protocol box, > > > > I believe > > > > they'd be better off making this statically settable. > > > > > > > > Charles > > > > > > > From owner-ips@ece.cmu.edu Mon Jan 8 19:28:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA18995 for ; Mon, 8 Jan 2001 19:28:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08M7dA12068 for ips-outgoing; Mon, 8 Jan 2001 17:07:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.com (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08M76U12053 for ; Mon, 8 Jan 2001 17:07:06 -0500 (EST) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Mon, 8 Jan 2001 14:07:01 -0800 Message-ID: <176B85B0D4E0D411BA5200508B692E0A35D1D0@hq-ex-1.brocade.com> From: Camden Ford To: "'mark carlson'" , David Peterson Cc: "Ips@Ece. Cmu. Edu" , Black_David@emc.com Subject: RE: iFCP as an IP Storage Work Item Date: Mon, 8 Jan 2001 14:07:00 -0800 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk Interesting comment..... If your FC disks and FC HBAs are local why would you use FCIP? -----Original Message----- From: mark carlson [mailto:mark.carlson@sun.com] Sent: Monday, January 08, 2001 12:24 PM To: David Peterson Cc: Ips@Ece. Cmu. Edu; Black_David@emc.com Subject: Re: iFCP as an IP Storage Work Item David Peterson wrote: > > Regarding #1: I believe the existing FC host drivers and HBA's will > work quite well using FCIP or an iSCSI gateway. A gateway protocol > is not required to allow an iSCSI HBA to function with an existing FC > device. In order to use FCIP, I have to purchase at least 2 FC switches and 2 FCIP devices (unless it's in the switch) to connect my FC HBA and my FC Disk even if they are local to each other. While you could do it this way, my preference would be a simple, cheap conversion box at each end. This same conversion box, if it supported the iSCSI protocol as well, would transition me to iSCSI when I have an iSCSI HBA. > > Regarding #2: under IETF rules there will be no support from multiple > companies, only individual contributors:) You misunderstand. While there is no requirement that the individual contributors come from separate companies, I would not like to see all the authors of the document coming from the same company ;-) -- mark From owner-ips@ece.cmu.edu Mon Jan 8 20:25:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA19383 for ; Mon, 8 Jan 2001 20:25:36 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08NPft14296 for ips-outgoing; Mon, 8 Jan 2001 18:25:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08NOtU14282 for ; Mon, 8 Jan 2001 18:24:55 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Mon, 8 Jan 2001 15:34:30 -0800 Message-ID: From: Joshua Tseng To: "Ips@Ece. Cmu. Edu" Subject: New iSNS Draft Document Available Date: Mon, 8 Jan 2001 15:24:27 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Friends, My colleagues and I are pleased to announce the availability of the new iSNS document as an official IP Storage Working Group draft. Until this document becomes available at the IETF web site, it can be retrieved at the following URL: http://www.nishansystems.com/ietf/draft-ietf-ips-iSNS-00.txt Regards, Josh Tseng From owner-ips@ece.cmu.edu Mon Jan 8 20:26:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA19405 for ; Mon, 8 Jan 2001 20:26:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08NOgi14276 for ips-outgoing; Mon, 8 Jan 2001 18:24:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08NOUU14269 for ; Mon, 8 Jan 2001 18:24:30 -0500 (EST) Received: from centralmail1.Central.Sun.COM ([129.147.62.10]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id QAA26738; Mon, 8 Jan 2001 16:24:28 -0700 (MST) Received: from thor.Central.Sun.COM (thor.Central.Sun.COM [129.147.248.69]) by centralmail1.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id QAA20640; Mon, 8 Jan 2001 16:24:26 -0700 (MST) Received: from sun.com (lab-dhcp71-6.Central.Sun.COM [172.20.71.216]) by thor.Central.Sun.COM (8.9.1b+Sun/8.9.1) with ESMTP id QAA19233; Mon, 8 Jan 2001 16:24:19 -0700 (MST) Message-ID: <3A5A4DF6.6178CA30@sun.com> Date: Mon, 08 Jan 2001 16:32:06 -0700 From: "Mark A. Carlson" Reply-To: mark.carlson@sun.com Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.75 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Camden Ford CC: "Ips@Ece. Cmu. Edu" Subject: Re: iFCP as an IP Storage Work Item References: <176B85B0D4E0D411BA5200508B692E0A35D1D0@hq-ex-1.brocade.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Camden Ford wrote: > > Interesting comment..... > > If your FC disks and FC HBAs are local why would you use FCIP? I wouldn't ;-) I would want a lightweight protocol for my conversion boxes so that I would not need a FC Fabric anywhere and could do pure IP with existing devices and HBAs. -- mark From owner-ips@ece.cmu.edu Mon Jan 8 20:27:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA19418 for ; Mon, 8 Jan 2001 20:27:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08NohJ14806 for ips-outgoing; Mon, 8 Jan 2001 18:50:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08NoNU14790 for ; Mon, 8 Jan 2001 18:50:23 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 8 Jan 2001 18:50:32 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07041013B8@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: iFCP: Please calm down ... Date: Mon, 8 Jan 2001 18:50:15 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, Some of the discussion about iFCP has drifted into business and marketing concerns. If you really feel that you must share such items, please send them directly to Steve Bellovin (smb@research.att.com) and myself - the list is for technical discussions only. The input on whether iFCP should be a work item is appreciated, but some of it is getting repetitive - I would ask people posting on this topic to make sure they have something new to add, as opposed to a "me too" or a desire to have the last word in some discussion. "me too"s can be sent directly to Steve and me rather than consuming list bandwidth, and "who has the last word on what" is not among the important decision criteria for whether to undertake work on iFCP. I suspect the time from here to a decision is still measured in weeks for a variety of reasons. I would hope that the enthusiasm equal to that shown for commenting on iFCP could be applied to working out the details of both iSCSI and FCIP on the list and at the interim meeting - progress on these protocols is necessary for the continued health of the WG. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Jan 8 20:33:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA19477 for ; Mon, 8 Jan 2001 20:33:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f08Nihn14677 for ips-outgoing; Mon, 8 Jan 2001 18:44:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f08NiOU14667 for ; Mon, 8 Jan 2001 18:44:25 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f090tI023990; Mon, 8 Jan 2001 16:55:19 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Henry Yang" , "'Murali Rajagopal'" , Subject: RE: Agree or Disagree to Merge FCIP and iFCP Encapsulation Date: Mon, 8 Jan 2001 15:42:54 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Henry, A question can greatly influence an answer. There should not be a concern about protecting iSCSI with respect to iFCP. Julian was the first to suggest efforts to merge aspects of iFCP and FCIP. I would not expect these efforts however will merge into a single document and so a question of merger should look at a larger picture. In general, you will find a more stable environment if specifications isolate independent functions combined by higher uses. I can agree with your comment that FCIP and iFCP not merge without agreeing this lack of merger should also influence a decision to merge encapsulation. Such consideration will improve the likelihood of common support ASICs as well as leveraging common practices. This transforms both proposals into a simple encapsulation transport combined with various frame handling conventions. Keeping node handling aspects separate from encapsulation also allows encapsulation specifications to stabilize sooner and yet allow changes with respect to the way frames are handled at the end nodes. You have not answered a meaningful question. Merge FCIP with iFCP, NO. Merge FCIP and iFCP encapsulation, YES. (I would also hope for a bit more justification that a simple yes or no.) Doug > I disagree with merging FCIP and iFCP. > > Henry Yang > > > -----Original Message----- > From: Murali Rajagopal [mailto:muralir@lightsand.com] > Sent: Saturday, January 06, 2001 12:26 PM > To: ips@ece.cmu.edu > Subject: Agree or Disagree to Merge FCIP and iFCP > > > All: > > The FCIP and iFCP address the FC over IP network transport with > entirely two > different objectives. > FCIP is a simple-minded tunnel that wants to see the basic FC connectivity > extended over IP network. Period!!! iFCP's goal is much more than > that .... > But that does not mean that we should combine the two just because it > appaers as though FCIP looks like a subset of iFCP. > > I am speaking for the authors of FCIP collectively, and in our opinion > mixing the two in a single spec. or a common encapsulation method > has little > meaning.Yes we understand that everyone would like to see a single FC Over > IP network solution and not two. But FCIP is really an extender for FC > Switch Fabrics while iFCP is attempting to "replace" the FC Switching > fabric. In this sense, iFCP is not in the best interest of either the FC > Switch vendors or even the T11 FC community at large. So mixing them makes > little business sense. In summary, the FCIP authors would like to keep the > FCIP and the iFCP efforts seperate. I beleive that the iFCP folks > are of the > same opinion. > > Now with my TC hat on, I would like to propose that we proceed to > solve this > debate by getting a collective consensus from all folks on > merging the two - > protocol or otherwise. > > Please reply with a simple "Agree to merge" or "Disagree to merge" > statement. No long explanations as to why or why not please! > > -Murali Rajagopal > > > ----- Original Message ----- > From: "Charles Monia" > To: ; ; "Charles Monia" > ; > Sent: Friday, January 05, 2001 4:54 PM > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > -----Original Message----- > > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > > Sent: Friday, January 05, 2001 4:15 PM > > > To: dotis@sanlight.net; cmonia@nishansystems.com; ips@ece.cmu.edu > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > Before this goes any further ... the two of you may be > > > in violent agreement. Charles objected to a "handshake"; > > > I don't think this objection would preclude one end of > > > a connection announcing the protocol that it intends to > > > use so that the other end can cleanly and quickly > > > terminate the connection if it can't or won't use > > > that protocol (ideally with an error code to the > > > other end indicating protocol incompatibility, so > > > the misconfiguration becomes obvious). > > > Doug's > > > suggestion of IANA-allocated numbers for identifying > > > protocols for this purpose is reasonable. > > > > > > As to the virtues of being able to speak more than > > > one protocol on a port, Fibre Channel provides examples > > > of where this sort of thing has been added after the > > > initial specifications were done, so I wouldn't dismiss > > > this out of hand. > > > > > > --David > > > > Hi David: > > > > It seems to me that with or without a common encapsulation method, such > > boxes can be built 'now'. As I said earlier, a common encapsulation > should > > make it easier, which is a good thing. > > > > So, what we're left with is the question of how such a box would shift > > gears. If an "in-band" method is easier, the simpler the handshake, the > > better. So an approach based on an IANA-assigned parameter seems > reasonable > > to me. > > > > While this is a fruitful discussion, I'd be interested in > hearing from the > > FCIP folks on this issue. > > > > Charles > > > > > > > > > > > > -----Original Message----- > > > > From: Douglas Otis [SMTP:dotis@sanlight.net] > > > > Sent: Friday, January 05, 2001 5:59 PM > > > > To: Charles Monia; Ips@Ece. Cmu. Edu > > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > Charles, > > > > > > > > I disagree with this assumption about rigidity of > > > protocols. Using a > > > > common > > > > initialization field would allow node handling protocols to > > > be confirmed. > > > > Initialization should also include a version number or > > > option field for > > > > the > > > > encapsulation where this version or option be negotiated. > > > If the node > > > > handling protocol, separate from the encapsulation > > > specification, could be > > > > identified, then there would be fewer strange events if someone > > > > mis-configured IP ports. I am not a fan of using text > > > strings to do this > > > > function and would prefer IANA defined values for rigid > > > structures. In > > > > addition to indicating an encapsulation version or options > > > designator, > > > > include the node type designator to signify how the node is > > > handled to > > > > avoid > > > > strange failures. > > > > > > > > As a gateway, tunnel or perhaps as an end device in perhaps > > > the distant > > > > future, the type of node protocols may vary to meet > > > different needs. The > > > > lion's share of the work however could be covered within > > > the encapsulation > > > > specifications and clever means of handling the node could > > > be isolated > > > > from > > > > these details. It does not mean that in the end all node > > > protocols will > > > > use > > > > identical encapsulation, but only that, as much as > > > possible, encapsulation > > > > is kept common and the effort to encapsulate is seen as a > > > separate task to > > > > that of handling the frame at the end point. I would hope that a > > > > transport > > > > could be developed that would not care how the node was > > > handled and where > > > > this node handling function is defined within a separate process. > > > > > > > > It seems like a fairly clean separation of encapsulation > > > and node handling > > > > that would help foster support for these markets especially if a few > > > > devices > > > > allow a wide range of possible node handling scenarios. I > > > agree that IP > > > > fabric is more extensible to that of FC, however even with that > > > > assumption, > > > > the final goal of fully incorporating IP fabric without > > > altering the FCP > > > > structures has not been achieved by either of these current > > > proposals. > > > > With > > > > that said, I would be in favor of developing a universal FC > > > encapsulation > > > > transport as a WG proposal that would support both iFCP and FCIP. > > > > > > > > Doug > > > > > > > > > > Ken Hirata wrote: > > > > > > > Why do you want to standardize a common encapsulation protocol > > > > > > > for FCIP and iFCP if their semantics and behavior are > > > completely > > > > > > > different? Would you want tunneling protocol > > > implementations to > > > > > > > also augment certain ELSs even though it isn't necessary > > > > > > for tunneling > > > > > > > protocol operation? > > > > > > > > > > > > If I were to build hardware that either assisted or completely > > > > > > processed both iFCP and FCIP, it sure would be easier to do > > > > > > header parsing and other common processing if there was just > > > > > > one format. > > > > > > > > > > > > > If a common encapsulation protocol was defined, I believe a > > > > > > > negotiation protocol would be necessary to distinguish between > > > > > > > usage as a gateway or tunneling protocol. > > > > > > > > > > > > Yes, either negotiation of a flag bit in the encapsulating > > > > > > header used to choose which algorithm to use. > > > > > > > > > > > Hi David and Ken: > > > > > > > > > > I agree that a common encapsulation may make it marginally easier > > > > > to build a > > > > > multi-protocol box as well as having other benefits. > > > However, from the > > > > > above, I'm concerned that some might infer that > > > multi-protocol support > > > > > should be a requirement. Just to be clear on this point: While > > > > > I'm all for > > > > > doing things that encourage commonality (where it makes sense > > > > > technically) I > > > > > feel that a standard ought not to needlessly burden a product with > > > > > supporting both architectures. > > > > > > > > > > With regard to 'negotiation', I believe such a handshake is > > > > > unnecessary and > > > > > undesirable. In a real system, I can't see a scenario > > > where it buys > > > > > anything to make this dynamic. As I see it, these > > > choices are cast in > > > > > concrete when the SAN is implemented and aren't going to change > > > > > from day to > > > > > day. Also, for hardwired boxes that only support one > > > protocol, it simply > > > > > adds complexity. If someone wants to build a multi-protocol box, > > > > > I believe > > > > > they'd be better off making this statically settable. > > > > > > > > > > Charles > > > > > > > > > > > From owner-ips@ece.cmu.edu Mon Jan 8 23:09:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA22725 for ; Mon, 8 Jan 2001 23:09:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f091vwd16952 for ips-outgoing; Mon, 8 Jan 2001 20:57:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f091v3U16931 for ; Mon, 8 Jan 2001 20:57:03 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Mon, 8 Jan 2001 18:06:39 -0800 Message-ID: From: Kevin Gibbons To: "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Mon, 8 Jan 2001 17:56:36 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk With respect to merging FCIP and iFCP: I do not feel they should be merged. I do not think the capabilities of iFCP and FCIP could be easily merged because their connection models are significantly different. Regards, Kevin ----------------------------------------- Kevin Gibbons Nishan Systems, Inc. (408) 519-3756 kgibbons@NishanSystems.com ----------------------------------------- From owner-ips@ece.cmu.edu Mon Jan 8 23:10:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA22740 for ; Mon, 8 Jan 2001 23:10:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09201u16988 for ips-outgoing; Mon, 8 Jan 2001 21:00:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f091xBU16976 for ; Mon, 8 Jan 2001 20:59:11 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Mon, 8 Jan 2001 18:08:47 -0800 Message-ID: From: Kevin Gibbons To: "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Mon, 8 Jan 2001 17:58:43 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk With respect to iFCP as an IP Storage work item: I believe there are enough technical contributions to IP storage for iFCP to be considered as a work item. My reasoning is based on the ability of iFCP to support traffic management for individual storage entities. According to the iFCP specification, individual iFCP entities behind a gateway have separate addresses and registrations in the iSNS. This allows MPLS and other traffic management standards to be specified and used for individual sessions between initiators and targets. This provides the ability to control individual session storage traffic, ensuring storage data can receive priority service. Additionally, each connection between an initiator and target has a separate connection built. If problems occur, the connection can be torn down without impact to other storage entities. Regards, Kevin ----------------------------------------- Kevin Gibbons Nishan Systems, Inc. (408) 519-3756 kgibbons@NishanSystems.com ----------------------------------------- From owner-ips@ece.cmu.edu Tue Jan 9 00:06:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA23541 for ; Tue, 9 Jan 2001 00:06:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f093J1B18239 for ips-outgoing; Mon, 8 Jan 2001 22:19:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f093I4U18211 for ; Mon, 8 Jan 2001 22:18:04 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 24B5DFF7 for ; Mon, 8 Jan 2001 19:18:03 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id TAA05756; Mon, 8 Jan 2001 19:17:51 -0800 (PST) Message-ID: <3A5A833E.363C5770@cup.hp.com> Date: Mon, 08 Jan 2001 19:19:26 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Further review feedback on latest iSCSI-02.txt draft. Content-Type: multipart/mixed; boundary="------------EE9CF31DC2CEE0EE50782B9A" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------EE9CF31DC2CEE0EE50782B9A Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian/All, More feedback on the latest iSCSI draft (02.txt. dec 30, '00) : Section 1.2.2.3. Data PDU Numbering ============================ What is the scope of DataRNs ? Is this maintained per task or per connection or per session ? This should be explicitly stated in this section as is done in earlier sections for CmdRN and StatRN. Section 1.2.8.3. Max iSCSI PDU Size ============================ The login parameter for the Max iSCSI PDU Size as described in this section is missing in the Login/Text keys in Appendix C. Section 2.5. SCSI Task Management Command PDU ======================================= The "Logical Unit Number" field in the figure should state "Logical Unit Number or reserved (0)" since the LUN is a don't care value in case of the Target Reset task mgmt command. Section 2.6. SCSI Task Management Response ================================== The referenced Task Tag field can be removed since the "No Task Found" response error is not valid. (A Target should respond with an accept for an Abort Task request specifying an invalid Initator Task Tag). Section 2.7 READ SCSI Data PDU ========================== 1) The READ SCSI Data PDU does not contain a Target Task Tag. Hence, when initators acknowledge DataRNs using NOP-OUT, targets will have to lookup the I/O based on its Initator Task Tag. It would be simpler and more optimal for targets to have a single lookup mechanism based on their Target Task Tags. This would also be in sync with Fibre Channel semantics where target lookups are based on only RX_ID. This is possible if Target Task Tags are used for READ SCSI Data PDUs. Section 2.17 Asynchronous Event ========================= 1) "Target requests Logout on this connection" iSCSI event indicator. Why not just allow the target to originate a Logout as is done in Fibre Channel ? Besides, this AEN does not meet the requirement. How does the target convey the CID of the connection to be logged out ? I do not think SAM-2 lays any guidelines for how the SCSI interconnect protocol handle its native PDUs. (i.e. non-scsi operations). Fibre Channel allows unsolicited inbound traffic for ELS and BLS commands and allowing Logout and NOP-OUT (perhaps better thought of as a NOP command) commands to be originated by targets would be desirable and would get rid of this particular AEN iSCSI event indicator. Section 2.19. Reject PDU =================== What is the value-add from copying the received iSCSI header into the Reject PDU ? Returning the Initiator Task Tag alone in the Reject PDU allows the initiator to look up its per-task data structure to determine what iSCSI PDU was transmitted on the Initiator Task Tag. 2) Is the reject PDU intended to be used for both SCSI and non-SCSI PDUs ? The REJECT PDU should be used to reject ALL non-scsi PDUs. For scsi PDUs, sending a response back thru the standard response PDUs (SCSI response or SCSI Task Mgmt response) would be preferrable since this allows the communication of specific information related to the failure of the SCSI PDU in the respnse Data or Response Code fields of the SCSI Response or SCSI Task Mgmt Response. 3) More detailed Reason Code/Reason Explanation fields in the Reject PDU is highly desirable. This will greatly help enhance the interop abilities of initiators and targets by being able to isolate quickly the reasons for a PDU Reject. I will send in further comments to the reflector on desired reason code and reason explanations for the Reject PDU. Section 4.4. Format Errors ==================== Quoting from the draft : "When a session is active whenever an initiator receives an iSCSI PDU with a format error, for which it has an outstanding task, it MUST abort the target task and report the error as a SCSI check condition status with a sense key of 4h (hardware error)." Is the above referring to an initiator's action or a target's ? What is the reasoning behind this ? It seems to complicate Format Error Handling with no clear benefit. Is it the expectation that iSCSI initiators will fake a format error as a CHECK CONDITION back to the SCSI Layer ? If so, why ? And why the choice of "Hardware Error" ? Last, but not the least, the above statement is referring to all iSCSI PDUs and advocates that iSCSI initiators must fake a CHECK CONDITION back to the SCSI layer on any format error of any iSCSI PDU. This is not applicable for non-scsi iSCSI PDUs which have no SCSI context. I believe the REJECT PDU should only be used for non-scsi PDUs since Rejects to SCSI PDUs will need a different type of reason code/explanation information best fitted into the existing SCSI Response PDUs. Thanks, Santosh Rao --------------EE9CF31DC2CEE0EE50782B9A Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------EE9CF31DC2CEE0EE50782B9A-- From owner-ips@ece.cmu.edu Tue Jan 9 00:09:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA23574 for ; Tue, 9 Jan 2001 00:09:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09390618076 for ips-outgoing; Mon, 8 Jan 2001 22:09:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cleitus.hosting.pacbell.net (cleitus.hosting.pacbell.net [216.100.99.17]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09387U18055 for ; Mon, 8 Jan 2001 22:08:07 -0500 (EST) Received: from VENKAT1 (64-160-62-200.rhapsodynetworks.com [64.160.62.200] (may be forged)) by cleitus.hosting.pacbell.net id WAA09514; Mon, 8 Jan 2001 22:07:35 -0500 (EST) [ConcentricHost SMTP Relay 1.7] From: "Venkat Rangan" To: "Joshua Tseng" , "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Mon, 8 Jan 2001 19:09:35 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Josh, Thanks for the clarification that iFCP is only presented as a gateway protocol. The one comment we would make is that we have FC to SCSI gateways already in place, without the need for any standards body standardizing a new protocol. The function of the gateway is defined by the standards for the two protocols being "connected", and gateway details are left as implementation details. On another note, it should be feasible to build a gateway that receives FCP frames from an N_Port or NL_Port of a SCSI Initiator and map the FCP frames into iSCSI frames. The frames are sent on an IP interface and routed by a normal IP network and another gateway reconverts the iSCSI PDUs back to FCP frames and sends them to the target. You will notice that this does not require any routing in the FC Plane and accomplishes the same end goals as iFCP. Also, this does not require any further standards work, besides the usual FCP, iSCSI and related naming protocols. This also provides the same scalability of number of nodes on the network, because the conversion from locally significant S_ID and D_ID to iSCSI IP addresses can be done, with help from a standardized naming effort such as iSNS. Based on these, we believe the need for IP Storage working group to pick up iFCP as a work item is reduced. Regards, Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Joshua Tseng Sent: Thursday, January 04, 2001 10:54 AM To: Ips@Ece. Cmu. Edu Subject: RE: iFCP as an IP Storage Work Item I don't want to stifle any creative technical discussion here, but I feel the need to remind everybody that iFCP is positioned as a gateway technology only. While the thought of "native" iFCP HBA's might be interesting, this discussion is completely irrelevant with regard to whether iFCP should or should not become an IPS work item. iFCP is being proposed as an IPS work item purely on its merits as a gateway technology. Regards, Josh > -----Original Message----- > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > Sent: Thursday, January 04, 2001 5:47 AM > To: 'ips@ece.cmu.edu' > Subject: FW: iFCP as an IP Storage Work Item > > > > > -----Original Message----- > From: Stephen Byan > Sent: Thursday, January 04, 2001 8:40 AM > To: 'Bill Terrell' > Subject: RE: iFCP as an IP Storage Work Item > > > It's all the FC stuff that lets iFCP work over an unreliable > data transport > like UDP. It's redundant when running over TCP/IP. > > Regards, > -Steve > > > -----Original Message----- > > From: Bill Terrell [mailto:terrell@troikanetworks.com] > > Sent: Wednesday, January 03, 2001 6:10 PM > > To: 'Stephen Byan' > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > >The downside of this advantage is that native iFCP devices would be > > burdened > > >with greater complexity and cost. I therefor think iFCP > > should not be an IP > > >Storage work item. > > > > > >Regards, > > >-Steve > > > > How is a native iFCP endpoint (initiator or target) more > > complex or costly > > than an iSCSI native endpoint? What are the specific > > difficulties inherent > > to native iFCP devices versus native iSCSI devices? > > > > Bill > > > From owner-ips@ece.cmu.edu Tue Jan 9 01:09:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA24388 for ; Tue, 9 Jan 2001 01:09:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09432E18959 for ips-outgoing; Mon, 8 Jan 2001 23:03:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0942FU18943 for ; Mon, 8 Jan 2001 23:02:16 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id EE0BB3A0 for ; Mon, 8 Jan 2001 20:02:14 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id UAA07769; Mon, 8 Jan 2001 20:02:10 -0800 (PST) Message-ID: <3A5A8D77.B8F4CBEB@cup.hp.com> Date: Mon, 08 Jan 2001 20:03:03 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Concerns regarding DataRN in READ Data PDU. Content-Type: multipart/mixed; boundary="------------4B98345FCDAC632891497AB2" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------4B98345FCDAC632891497AB2 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian/All, Can anybody comment on the true benefit that the DataRN feature provides that justifies the complexity and performance issues it introduces to the protocol ? I see the following issues with DataRN : 1) It adds a performance penalty in that on every READ Data PDU targets will have to initialize one additional 32-bit word in the header. 2) Extra traffic on the outbound context from initiators sending NOP-OUTs to acknowledge the DataRNs. 3) Performance path checks for the "P" bit on every READ PDU. (Alternatively, checks for the DataNumber field from the Login Text Keys). 4) Implementing DataRNs per task is too short-lived to provide any real benefit. Implementing them per connection or session can result in quick use up of the 32-bit DataRN space (since each frame consumes one DataRN), adding additional complexity to the target code to use the "P" bit appropriately. 5) Further, "DataNumber", which is the size of the un-acknowledged DataRN window is negotiated at login time, whereas this is really desired to be a dynamic variable based on the target's current I/O load and the number of initiators speaking to it. Thus, DataRN can end up being under-utilized resulting in too much NOP-OUT traffic, or it can be over-utilized resulting in too much usage of the "P" bit in the READ PDUs, which, again, can cause too much NOP-OUT traffic. 6) Considering that the DataRN generation and acknowledgement are functions that would typically be in SCSI hardware assist engines and there is a need to keep these as simple as possible, adding such features in the spec means that these SCSI assist engines MUST implement DataRN generation and acknowledgement [since initiators have no way to turn it off if a target decides to use it] What is the true benefit of this feature ? Is it intended to be used as : a) Will allow targets to do partial freeing of their memory buffers as and when DataRNs are received ? b) Will allow targets to send back only the un-acknowledged data (as seen from the ExpDataRN received on the last NOP-OUT from the initiator) when the initiators retry commands with the "Retry" bit set ? If the benefit intended is (a), then, in order to derive this benefit, targets will have to advertise their DataNumber login key to be < the avg. number of frames transmitted per READ I/O. Having a DataNumber > the avg. number of frames per READ I/O implies the life of the task ends [and buffers for the task get released] before the initiator sends NOP-OUT, thereby, defeating benefit (a). If the benefit intended is (b) [and i do not see this explicitly stated in the draft], then, this is a dangerous assumption which can lead to data corruption issues. When commands are retried with the "Retry" bit set, are targets allowed to send back partial data based on the last ExpDataRN they received ? If so, this can have multiple issues like : o complexity in handling I/O underrun cases for the retried command which only saw partial data transfer from the target. o If the target only sent back partial data based on its last ExpDataRN received, then, it can open up possible data corruption windows. All in all, I believe that the DataRN is adding too much complexity and potential performance impact to iSCSI for the benefits it may provide. Moreover and most importantly, the draft does not give initiators any control over the usage of this feature and if a target decides to use this feature, initiators will have to support it. This exposes the initiators to all of the above issues without being able to turn off this feature. I would like to therefore ask that : either, a) DataRN support be removed from the iSCSI draft. or b) The support for DataRN be negotiated at login time giving initators control over this feature and allowing them to disable this feature. This will allow SCSI hardware assist engines to skip implementation of this functionality and their associated software can merely turn off DataNumber in the login negotiation. Thanks, Santosh --------------4B98345FCDAC632891497AB2 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------4B98345FCDAC632891497AB2-- From owner-ips@ece.cmu.edu Tue Jan 9 03:05:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA07576 for ; Tue, 9 Jan 2001 03:05:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f095eiP20344 for ips-outgoing; Tue, 9 Jan 2001 00:40:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f095dkU20326 for ; Tue, 9 Jan 2001 00:39:46 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Mon, 8 Jan 2001 21:49:24 -0800 Message-ID: From: Joshua Tseng To: Venkat Rangan Cc: "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Mon, 8 Jan 2001 21:39:18 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Venkat, > > Josh, > > Thanks for the clarification that iFCP is only presented as a gateway > protocol. The one comment we would make is that we have FC to > SCSI gateways > already in place, without the need for any standards body > standardizing a > new protocol. The function of the gateway is defined by the > standards for > the two protocols being "connected", and gateway details are left as > implementation details. Actually, what I meant by gateway protocol is that it is a protocol spoken by gateways. It doesn't specify anything about how to implement the protocol, which is internal to the gateway device as you point out. iFCP is the protocol that shows up on the IP network when at least two iFCP gateways transport storage data between them. > > On another note, it should be feasible to build a gateway > that receives FCP > frames from an N_Port or NL_Port of a SCSI Initiator and map > the FCP frames > into iSCSI frames. The frames are sent on an IP interface and > routed by a > normal IP network and another gateway reconverts the iSCSI > PDUs back to FCP > frames and sends them to the target. You will notice that > this does not > require any routing in the FC Plane and accomplishes the same > end goals as > iFCP. Also, this does not require any further standards work, > besides the > usual FCP, iSCSI and related naming protocols. This also > provides the same > scalability of number of nodes on the network, because the > conversion from > locally significant S_ID and D_ID to iSCSI IP addresses can > be done, with > help from a standardized naming effort such as iSNS. Yes, I agree. But this is a different gateway which has nothing to do with an iFCP gateway. Definitely these types of gateways will be needed as we transition to iSCSI. The new iSNS draft has a diagram showing both iSCSI-FC gateways and iFCP gateways. They have different roles. > > Based on these, we believe the need for IP Storage working > group to pick up > iFCP as a work item is reduced. hmmm....you lost me here. An iFCP gateway has nothing to do with iSCSI. They are completely separate. In order to use iSCSI, you need one of the following: 1) Two iSCSI devices 2) One iSCSI device, one FC device, and one iSCSI-FC gateway (which you described above) The situation of two FC devices and two iSCSI-FC gateways is clearly not a design objective of iSCSI. Of course it can be done, but we believe iFCP is clearly the best solution here. Fibre Channel has its issues, but one thing is certain: the FCP driver stacks are very stable and can provide excellent performance for storage applications. But the drawback is FC's networking capabilities leave much to be desired. On the other hand, IP networking capabilities are quite stable and work exceedingly well, but the iSCSI driver stack does not exist yet. So, iFCP's objective is to marry the best of both worlds--to take the existing stable, high-performance driver stacks of Fibre Channel and directly internetwork their individual N_PORTs using TCP/IP. Therefore, iFCP is an opportunity to leverage the existing and proven Fibre Channel driver stacks and combine them with the capabilities that IP networking can provide. Until the day we have stable iSCSI driver stacks everywhere, this may prove to be an attractive alternative for many end users. Another comparison I liken it to is the need for NAT until we have IPv6 everywhere. Josh > > Regards, > > Venkat Rangan > Rhapsody Networks Inc. > http://www.rhapsodynetworks.com > > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Joshua Tseng > Sent: Thursday, January 04, 2001 10:54 AM > To: Ips@Ece. Cmu. Edu > Subject: RE: iFCP as an IP Storage Work Item > > > I don't want to stifle any creative technical discussion here, > but I feel the need to remind everybody that iFCP is positioned > as a gateway technology only. While the thought of "native" > iFCP HBA's might be interesting, this discussion is > completely irrelevant with regard to whether iFCP should > or should not become an IPS work item. iFCP is being proposed > as an IPS work item purely on its merits as a gateway technology. > > Regards, > Josh > > > -----Original Message----- > > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > > Sent: Thursday, January 04, 2001 5:47 AM > > To: 'ips@ece.cmu.edu' > > Subject: FW: iFCP as an IP Storage Work Item > > > > > > > > > > -----Original Message----- > > From: Stephen Byan > > Sent: Thursday, January 04, 2001 8:40 AM > > To: 'Bill Terrell' > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > It's all the FC stuff that lets iFCP work over an unreliable > > data transport > > like UDP. It's redundant when running over TCP/IP. > > > > Regards, > > -Steve > > > > > -----Original Message----- > > > From: Bill Terrell [mailto:terrell@troikanetworks.com] > > > Sent: Wednesday, January 03, 2001 6:10 PM > > > To: 'Stephen Byan' > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > >The downside of this advantage is that native iFCP > devices would be > > > burdened > > > >with greater complexity and cost. I therefor think iFCP > > > should not be an IP > > > >Storage work item. > > > > > > > >Regards, > > > >-Steve > > > > > > How is a native iFCP endpoint (initiator or target) more > > > complex or costly > > > than an iSCSI native endpoint? What are the specific > > > difficulties inherent > > > to native iFCP devices versus native iSCSI devices? > > > > > > Bill > > > > > > From owner-ips@ece.cmu.edu Tue Jan 9 03:13:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA07616 for ; Tue, 9 Jan 2001 03:13:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f095Vi120200 for ips-outgoing; Tue, 9 Jan 2001 00:31:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mcmail.mcdata.com ([144.49.1.5]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f095VWU20191 for ; Tue, 9 Jan 2001 00:31:32 -0500 (EST) Received: from exchange5.mcdata.com (exchange5.mcdata.com [144.49.33.200]) by mcmail.mcdata.com (8.8.6 (PHNE_14041)/8.8.6) with ESMTP id NAA11794; Mon, 8 Jan 2001 13:28:12 -0700 (MST) Received: by exchange5.mcdata.com with Internet Mail Service (5.5.2650.21) id ; Mon, 8 Jan 2001 22:31:16 -0700 Message-ID: From: "Mike O'Donnell" To: "'Murali Rajagopal '" , "'ips@ece.cmu.edu '" Subject: RE: Agree or Disagree to Merge FCIP and iFCP Date: Mon, 8 Jan 2001 22:31:11 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Murali, As a co-author of the FCIP proposal, I see no value in diluting the committee's resources (not only at IETF but, ANSI as well in terms of FC-BB-2) in merging the two techniques. I disagree to merge the techniques. Regards, Michael E. O'Donnell ===================== McDATA Corporation modonnell@mcdata.com 303.460.4142 ===================== -----Original Message----- From: Murali Rajagopal To: ips@ece.cmu.edu Sent: 1/6/01 12:26 PM Subject: Agree or Disagree to Merge FCIP and iFCP All: The FCIP and iFCP address the FC over IP network transport with entirely two different objectives. FCIP is a simple-minded tunnel that wants to see the basic FC connectivity extended over IP network. Period!!! iFCP's goal is much more than that .... But that does not mean that we should combine the two just because it appaers as though FCIP looks like a subset of iFCP. I am speaking for the authors of FCIP collectively, and in our opinion mixing the two in a single spec. or a common encapsulation method has little meaning.Yes we understand that everyone would like to see a single FC Over IP network solution and not two. But FCIP is really an extender for FC Switch Fabrics while iFCP is attempting to "replace" the FC Switching fabric. In this sense, iFCP is not in the best interest of either the FC Switch vendors or even the T11 FC community at large. So mixing them makes little business sense. In summary, the FCIP authors would like to keep the FCIP and the iFCP efforts seperate. I beleive that the iFCP folks are of the same opinion. Now with my TC hat on, I would like to propose that we proceed to solve this debate by getting a collective consensus from all folks on merging the two - protocol or otherwise. Please reply with a simple "Agree to merge" or "Disagree to merge" statement. No long explanations as to why or why not please! -Murali Rajagopal ----- Original Message ----- From: "Charles Monia" To: ; ; "Charles Monia" ; Sent: Friday, January 05, 2001 4:54 PM Subject: RE: iFCP as an IP Storage Work Item > > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Friday, January 05, 2001 4:15 PM > > To: dotis@sanlight.net; cmonia@nishansystems.com; ips@ece.cmu.edu > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > Before this goes any further ... the two of you may be > > in violent agreement. Charles objected to a "handshake"; > > I don't think this objection would preclude one end of > > a connection announcing the protocol that it intends to > > use so that the other end can cleanly and quickly > > terminate the connection if it can't or won't use > > that protocol (ideally with an error code to the > > other end indicating protocol incompatibility, so > > the misconfiguration becomes obvious). > > Doug's > > suggestion of IANA-allocated numbers for identifying > > protocols for this purpose is reasonable. > > > > As to the virtues of being able to speak more than > > one protocol on a port, Fibre Channel provides examples > > of where this sort of thing has been added after the > > initial specifications were done, so I wouldn't dismiss > > this out of hand. > > > > --David > > Hi David: > > It seems to me that with or without a common encapsulation method, such > boxes can be built 'now'. As I said earlier, a common encapsulation should > make it easier, which is a good thing. > > So, what we're left with is the question of how such a box would shift > gears. If an "in-band" method is easier, the simpler the handshake, the > better. So an approach based on an IANA-assigned parameter seems reasonable > to me. > > While this is a fruitful discussion, I'd be interested in hearing from the > FCIP folks on this issue. > > Charles > > > > > > > > -----Original Message----- > > > From: Douglas Otis [SMTP:dotis@sanlight.net] > > > Sent: Friday, January 05, 2001 5:59 PM > > > To: Charles Monia; Ips@Ece. Cmu. Edu > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > Charles, > > > > > > I disagree with this assumption about rigidity of > > protocols. Using a > > > common > > > initialization field would allow node handling protocols to > > be confirmed. > > > Initialization should also include a version number or > > option field for > > > the > > > encapsulation where this version or option be negotiated. > > If the node > > > handling protocol, separate from the encapsulation > > specification, could be > > > identified, then there would be fewer strange events if someone > > > mis-configured IP ports. I am not a fan of using text > > strings to do this > > > function and would prefer IANA defined values for rigid > > structures. In > > > addition to indicating an encapsulation version or options > > designator, > > > include the node type designator to signify how the node is > > handled to > > > avoid > > > strange failures. > > > > > > As a gateway, tunnel or perhaps as an end device in perhaps > > the distant > > > future, the type of node protocols may vary to meet > > different needs. The > > > lion's share of the work however could be covered within > > the encapsulation > > > specifications and clever means of handling the node could > > be isolated > > > from > > > these details. It does not mean that in the end all node > > protocols will > > > use > > > identical encapsulation, but only that, as much as > > possible, encapsulation > > > is kept common and the effort to encapsulate is seen as a > > separate task to > > > that of handling the frame at the end point. I would hope that a > > > transport > > > could be developed that would not care how the node was > > handled and where > > > this node handling function is defined within a separate process. > > > > > > It seems like a fairly clean separation of encapsulation > > and node handling > > > that would help foster support for these markets especially if a few > > > devices > > > allow a wide range of possible node handling scenarios. I > > agree that IP > > > fabric is more extensible to that of FC, however even with that > > > assumption, > > > the final goal of fully incorporating IP fabric without > > altering the FCP > > > structures has not been achieved by either of these current > > proposals. > > > With > > > that said, I would be in favor of developing a universal FC > > encapsulation > > > transport as a WG proposal that would support both iFCP and FCIP. > > > > > > Doug > > > > > > > > Ken Hirata wrote: > > > > > > Why do you want to standardize a common encapsulation protocol > > > > > > for FCIP and iFCP if their semantics and behavior are > > completely > > > > > > different? Would you want tunneling protocol > > implementations to > > > > > > also augment certain ELSs even though it isn't necessary > > > > > for tunneling > > > > > > protocol operation? > > > > > > > > > > If I were to build hardware that either assisted or completely > > > > > processed both iFCP and FCIP, it sure would be easier to do > > > > > header parsing and other common processing if there was just > > > > > one format. > > > > > > > > > > > If a common encapsulation protocol was defined, I believe a > > > > > > negotiation protocol would be necessary to distinguish between > > > > > > usage as a gateway or tunneling protocol. > > > > > > > > > > Yes, either negotiation of a flag bit in the encapsulating > > > > > header used to choose which algorithm to use. > > > > > > > > > Hi David and Ken: > > > > > > > > I agree that a common encapsulation may make it marginally easier > > > > to build a > > > > multi-protocol box as well as having other benefits. > > However, from the > > > > above, I'm concerned that some might infer that > > multi-protocol support > > > > should be a requirement. Just to be clear on this point: While > > > > I'm all for > > > > doing things that encourage commonality (where it makes sense > > > > technically) I > > > > feel that a standard ought not to needlessly burden a product with > > > > supporting both architectures. > > > > > > > > With regard to 'negotiation', I believe such a handshake is > > > > unnecessary and > > > > undesirable. In a real system, I can't see a scenario > > where it buys > > > > anything to make this dynamic. As I see it, these > > choices are cast in > > > > concrete when the SAN is implemented and aren't going to change > > > > from day to > > > > day. Also, for hardwired boxes that only support one > > protocol, it simply > > > > adds complexity. If someone wants to build a multi-protocol box, > > > > I believe > > > > they'd be better off making this statically settable. > > > > > > > > Charles > > > > > > > From owner-ips@ece.cmu.edu Tue Jan 9 05:08:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA08258 for ; Tue, 9 Jan 2001 05:08:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f097Vm621870 for ips-outgoing; Tue, 9 Jan 2001 02:31:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f097VfU21865 for ; Tue, 9 Jan 2001 02:31:41 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 97A8B5D8; Mon, 8 Jan 2001 23:31:40 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id XAA18438; Mon, 8 Jan 2001 23:30:35 -0800 (PST) From: Santosh Rao Message-Id: <200101090730.XAA18438@hpcuhe.cup.hp.com> Subject: Re: iSCSI : On the subject of R2T and Task Tags. To: julian_satran@il.ibm.com Date: Mon, 8 Jan 2001 23:30:35 -0800 (PST) Cc: ips@ece.cmu.edu, santoshr@hpcuhe.cup.hp.com (Santosh Rao) In-Reply-To: from "julian_satran@il.ibm.com" at Jan 08, 2001 09:58:16 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Thanks for the clarifications. One further question below (that I've asked in several different ways a couple of times now ;) : > 2) The READ I/O PDUs do not use a "Target Task Tag". > The WRITE I/O PDUs use a "Target Task Tag" per R2T > and not per task. > > It seems like the only unique per I/O identifier is the Initiator > Task Tag. (?) Is it the intention of iSCSI that targets should be > using the Initiator Task Tag as the per I/O lookup tag ? > > If this is true, then, the name "Initiator Task Tag" is a misnomer > and it should just be "Task Tag", since it is used by both initiators > and > targets. > We use initiator and target for the task to state who is "generating" > the tag Since a READ Data PDU has no Target Task Tag, is it the expectation that iSCSI targets should perform a lookup on the Initiator Task Tag to obtain their per-task data structure ? This is a performance penalty, if so. Again, with WRITE Data PDUs there is no target assigned per-task tag. While one could argue about vertically encoding the Target Task Tag and the Target Transfer Tag into the same "Target Transfer Tag", this would seem like an ugly hack to work around a shortcoming in the PDU header. It would be desirable to have seperate Target Task Tag fields for both READ and WRITE Data PDUs. Thanks, Santosh -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Tue Jan 9 07:00:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA09040 for ; Tue, 9 Jan 2001 07:00:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09AHtL02499 for ips-outgoing; Tue, 9 Jan 2001 05:17:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09AHLU02489 for ; Tue, 9 Jan 2001 05:17:21 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id 3C179FEC for ; Tue, 9 Jan 2001 02:17:19 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id CAA01404; Tue, 9 Jan 2001 02:17:17 -0800 (PST) Message-ID: <3A5AE4DF.A617ECB5@hp.com> Date: Tue, 09 Jan 2001 02:15:59 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI: target resource "leak"/logout command Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, There is a corner case where - a [group of] command completion[s] will never be acked (ExpStatRN) - the corresponding task[s] are not "retried" because they are perfectly completed from the initiator point of view and - the session continues to work fine. Thus the resource for the task[s] may never be released on the target (the target waits for ExpStatRN acking the completion before releasing task resources) because on the target side the ack of the completion is not received and on the initiator will not retry the command. A resource (memory) "leak" happens on the target. First i described the scenario, then the solution. Scenario ======= Session with 2 TCP connections 1 and 2 1) A command completion (StatRN=10) is received by the initiator (over cx 1) for the command 5. 2) The initiator sends back ExpStatRN=11 with a command over TCP cx 1 3) The TCP cx 1 drops and the command 5 (so ExpStatRN=11) will never make it to the target. 4) The command 5 is not retried on the connection 2 because for the initiator it is perfectly completed. 5) The target never gets the completion acked and doesn't know when to release resources. Solution ======= - Add the field ExpStatRN in the logout command. - The initiator puts in this field the last value of ExpStatRN of the failed cx. (Not the ExpStatRN value of the connection used to transport the logout) - The target when receiving the logout, reads ExpStatRN and is able to free up the resources. Regards, Pierre From owner-ips@ece.cmu.edu Tue Jan 9 07:02:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA09075 for ; Tue, 9 Jan 2001 07:02:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f099fsL23360 for ips-outgoing; Tue, 9 Jan 2001 04:41:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f099fDU23350 for ; Tue, 9 Jan 2001 04:41:13 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id B05A9186E for ; Tue, 9 Jan 2001 01:41:12 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id BAA01205; Tue, 9 Jan 2001 01:41:11 -0800 (PST) Message-ID: <3A5ADC69.78416558@hp.com> Date: Tue, 09 Jan 2001 01:39:53 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI: Abort task Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, It seems that we need to specify more the "Abort task" to avoid some problems. Below are 3 points we should add in the spec: 1) An "Abort task" must be sent on the TCP connection that was used to send the command. This does not apply if the connection used for the command has been "logged out". In this case of course, the "Abort task" can be send on another connection and it will not hurt. It is to avoid that a command that was delayed in the network, finally make it after the "Abort task" as been processed. 2) When the initiator receives the "Abort task" response, if the ExpCmdRN returned in the "Abort task" response is <= CmdRN of the aborted task, the initiator must re-use this CmdRN for another task (any). 3) The target when receiving an "Abort task" for a task whose the CmdRN (retreived from the initiator task tag) is >= ExpCmdRN must not bridge this CmdRN. I mean by bridge: allow ExpCmdRN to pass the CmdRN value of the aborted task. It will be up to the initiator to send another command with the same CmdRN in order to allow the target to move ExpCmdRN over the CmdRN value of the aborted task. "3)" Is to prevent to following scenario: Two TCP connections 1 and 2 in the session ExpCmdRN = 2 a) the command with CmdRN = 3 is sent over connection1 b) the command with CmdRN = 4 is sent over connection 2 c) the command CmdRN = 4 reaches the target d) the initiator aborts the command 4 and bridges CmdRN=4 e) the target receives the command 3, and as CmdRN=4 is bridged, advances ExpCmdRN to 5 f) the initiator sends another command (not related with the aborted one) with CmdRN=4 g) the target drops this command because CmdRN is less than ExpCmdRN. "2)" needs to exist because if the initiator doesn't re-use the CmdRN the target will deadlock because ExpCmdRN will be stuck behind the CmdRN of the aborted command. An alternative to the points 2) and 3) would be the initiator not re-using the CmdRN of the aborted commands. But this requires the target to bridge all the CmdRNs of the Aborted tasks (need extra states) and anyway there is a scenario where it doesn't work. Hence this is a bad alternative. Scenario where it doesn't work: Session with 2 connections 1 and 2. ExpCmdRN=1 a) Command with CmdRN=4 is sent over connection1 b) Connection 1 drops and the command 4 will never make it to the target. c) A logout for the connection 1 is sent over connection 2 d) The initiator sends an "Abort task" (for the task CmdRN=4) over the connection 2 e) The target, from the referenced initiator task tag cannot find the CmdRN of the aborted task because the initiator task tag corresponds to nothing because the command has never been received. Hence the target is not able to bridge CmdRN=4 f) As the initiator does not re-use CmdRN=4 the target will deadlock when ExpCmdRN will be 4. Regards, Pierre From owner-ips@ece.cmu.edu Tue Jan 9 11:15:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA15094 for ; Tue, 9 Jan 2001 11:15:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09Dc3H05138 for ips-outgoing; Tue, 9 Jan 2001 08:38:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f098lAU22709 for ; Tue, 9 Jan 2001 03:47:10 -0500 (EST) Received: from divyaroot.India.Sun.COM ([129.158.225.35]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id AAA25702 for ; Tue, 9 Jan 2001 00:47:05 -0800 (PST) Received: from helix (helix [129.158.226.51]) by divyaroot.India.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.0) with SMTP id OAA29278 for ; Tue, 9 Jan 2001 14:16:59 +0530 (IST) Message-Id: <200101090846.OAA29278@divyaroot.India.Sun.COM> Date: Tue, 9 Jan 2001 14:18:49 -0500 (GMT) From: Raghavendra Rao Reply-To: Raghavendra Rao Subject: Re: iSCSI : On the subject of R2T and Task Tags. To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: iW8iaxGDhC8S6mp08SX/9w== Sender: owner-ips@ece.cmu.edu Precedence: bulk > Since a READ Data PDU has no Target Task Tag, is it the expectation that > iSCSI targets should perform a lookup on the Initiator Task Tag to obtain > their per-task data structure ? This is a performance penalty, if so. > For a SCSI READ command, I don't think it'll be really useful to have a target task tag. > Again, with WRITE Data PDUs there is no target assigned > per-task tag. While one could argue about vertically encoding the Target There is - this is done during R2T time and every solicited data PDU from the initiator goes out with this tag. BTW, SAM-2 rev 14 has 64 bit task tags; How are we handling situations like these ? -JP From owner-ips@ece.cmu.edu Tue Jan 9 11:18:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA15135 for ; Tue, 9 Jan 2001 11:18:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09E7Bc05876 for ips-outgoing; Tue, 9 Jan 2001 09:07:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09E6gU05861 for ; Tue, 9 Jan 2001 09:06:42 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id PAA107320 for ; Tue, 9 Jan 2001 15:06:35 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id PAA69032 for ; Tue, 9 Jan 2001 15:06:35 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569CF.004D7D8A ; Tue, 9 Jan 2001 15:06:24 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 9 Jan 2001 16:01:47 +0200 Subject: Re: iSCSI: data numbering Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Pierre, 1 - is fixed already in 03 (the text was ambiguous) 2 -will fix in 03 Julo Pierre Labat on 08/01/2001 22:14:29 Please respond to Pierre Labat To: ips@ece.cmu.edu cc: Subject: iSCSI: data numbering Julian, Two points on the data numbering. 1) It seems to me that the negociation of the Data numbering is useless and must be removed from the spec. Because : - there is the P bit in the SCSI data (READ) pdu header and the initiator must ack if P is set. Doing this way gives all flexibility to the target. - if the target wants the data numbering, the initiator MUST do it. Based on that, negociating the data numbering doesn't add any value. We have to remove it. The "P" bit is perfect. 2) The SCSI data READ pdu header must contains a target task tag. Doing that, avoids the target to do a lookup from the initiator task tag each time it receives a data ack (NOP out). The lookups (using hashing table for ex) must be kept to a minimum because they slow down operations. We have to avoid them on the performance path. In iSCSI the only place where a lookup takes place is for Abort task (lookup done on the target side) for "retry" a command (lookup done on the target side) and for read data ack (lookup done on the target side too). In the two first operations a lookup on the initiator task tag can not be avoided because the target may not have time to give a tag to the initiator before things go wrong. However it is not a problem because it is not on the performance path. But for read data acknowledgement, it is on the performance path. This lookup can be easily avoided if the target provides a tag in the data PDU, the NOP out passes back this tag to the target. The tag is the same all along the [READ] command. Regards, Pierre From owner-ips@ece.cmu.edu Tue Jan 9 12:02:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA16732 for ; Tue, 9 Jan 2001 12:02:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09Eo4N07144 for ips-outgoing; Tue, 9 Jan 2001 09:50:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ludwig.troikanetworks.com (host03.troikanetworks.com [12.31.172.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09EnqU07137 for ; Tue, 9 Jan 2001 09:49:52 -0500 (EST) Received: by host03.troikanetworks.com with Internet Mail Service (5.5.2653.19) id ; Tue, 9 Jan 2001 06:49:55 -0800 Message-ID: From: Wayland Jeong To: ips@ece.cmu.edu, julian_satran@il.ibm.com Subject: iSCSI : Negotiation of PDU Sizes Date: Tue, 9 Jan 2001 06:49:54 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Section 1.2.8.3 of the iSCSI draft specifies: "Each end of the iSCSI session specifies during login the maximum size of an iSCSI PDU it will accept." I did not see any mention of a Login/Text key that allows iSCSI end points to negotiate the maximum PDU size. Is there some other mechanism for doing this negotiation? -Wayland From owner-ips@ece.cmu.edu Tue Jan 9 12:03:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA16808 for ; Tue, 9 Jan 2001 12:03:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09Ee5506838 for ips-outgoing; Tue, 9 Jan 2001 09:40:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.lss.emc.com (mercury.eng.emc.com [168.159.40.77]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09Ed2U06811 for ; Tue, 9 Jan 2001 09:39:02 -0500 (EST) Received: by mercury.eng.emc.com with Internet Mail Service (5.5.2653.19) id ; Tue, 9 Jan 2001 09:36:06 -0500 Message-ID: <221EA3AEAB55D411A2DA00D0B774E6F70153FF44@marino.lss.emc.com> From: "harwood, jack" To: "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Tue, 9 Jan 2001 09:44:25 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk While in concept FCIP and iFCP both provide gateway functionality I do not believe they can be merged into one specification. Both however do provide useful functions and should be supported as work group items for IPS, I understand FCIP is already a work item today. As mentioned in previous emails FCIP connects E_Ports, which supports exisiting switch SAN technology. As such, all of the traffic is sent across one connection without regard to priority with many FC ports funneled through single E_Ports. The benefit to this is connecting existing SAN technology with all of the control provided by the FC domain. iFCP provides the ability to map the FC port connections to the IP domain and allow for distinguishing FC connections. With this the FC traffic can be prioritized according to the connection, traffic, application, etc... This perspective for a gateway allows for building SANs while utilizing network traffic management tools. As neither covers all cases for connecting Fibre Channel domains both should be pursued. Thanks jack ***************************************** jack harwood EMC Corporation 228 South St. Hopkinton, MA 01748 Phone: 508-435-1000 x22094 FAX: 508-544-2164 email: harwood_jack@emc.com From owner-ips@ece.cmu.edu Tue Jan 9 12:05:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA16944 for ; Tue, 9 Jan 2001 12:05:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09ET3906465 for ips-outgoing; Tue, 9 Jan 2001 09:29:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ludwig.troikanetworks.com (host03.troikanetworks.com [12.31.172.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09ESAU06445 for ; Tue, 9 Jan 2001 09:28:12 -0500 (EST) Received: by host03.troikanetworks.com with Internet Mail Service (5.5.2653.19) id ; Tue, 9 Jan 2001 06:28:16 -0800 Message-ID: From: Wayland Jeong To: "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Date: Tue, 9 Jan 2001 06:28:16 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I am in support of having iFCP as a IP Storage Work Item for the following two technical reasons. 1. FCIP is currently not fully specified. As it stands today, FCIP relies on the concepts of Autonomous Regions (AR's) and Border Switches (BSW's). The role of a BSW and the structure of AR's in a FC fabric are not currently well defined in FC-SW-2 (btw, I am willing to be educated here). A new effort within T11, FC-BB-2, has the charter for encompassing connectivity to IP networks, but the current BB specification only addresses ATM and SONET. I have no doubt that T11 can solve these problems, but without seeing a proposal, it is difficult for me to support FCIP in its current form. 2. iFCP is a more robust way of connecting SAN islands. As I understand it, an FCIP connected wide area SAN is virtually a contiguous fabric (maybe not from a routing perspective, but at least from a naming perspective). I believe that the Achilles heel of FC fabrics is its use of a single coherent name space and its reliance on distributed services (i.e. fabric SNS). When two remote SAN's are connected together, re-configurations in one SAN will impose a re-configuration in the other SAN. SSP's for example, have to jump through hoops to avoid these issues (they don't like the idea of having their customer networks have the ability to re-configure their data center). Furthermore, when AR's with conflicting domains are connected, those domains become isolated until the global domain assignments can be made. I don't see this as a very scalable solution. The iFCP gateway definition will more effectively isolate FC SAN island's allowing for connectivity over legacy IP networks, without tightly coupling the remote storage devices. I would like to put one caveat on my position. I want to see iFCP extended, fully embracing connectivity to existing FC fabrics (i.e. E-ports) and fully supporting multi-protocol SAN's (IP, VI, etc.). -Wayland From owner-ips@ece.cmu.edu Tue Jan 9 14:05:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA19719 for ; Tue, 9 Jan 2001 14:05:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09GW7I10174 for ips-outgoing; Tue, 9 Jan 2001 11:32:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09GVoU10167 for ; Tue, 9 Jan 2001 11:31:50 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id RAA243388 for ; Tue, 9 Jan 2001 17:31:43 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA28100 for ; Tue, 9 Jan 2001 17:31:43 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569CF.005AC96B ; Tue, 9 Jan 2001 17:31:38 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 9 Jan 2001 18:27:27 +0200 Subject: Re: iSCSI: data numbering Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk It is already in 03. I discovered it also on rereading. Thanks, Julo Pierre Labat on 08/01/2001 22:14:29 Please respond to Pierre Labat To: ips@ece.cmu.edu cc: Subject: iSCSI: data numbering Julian, Two points on the data numbering. 1) It seems to me that the negociation of the Data numbering is useless and must be removed from the spec. Because : - there is the P bit in the SCSI data (READ) pdu header and the initiator must ack if P is set. Doing this way gives all flexibility to the target. - if the target wants the data numbering, the initiator MUST do it. Based on that, negociating the data numbering doesn't add any value. We have to remove it. The "P" bit is perfect. 2) The SCSI data READ pdu header must contains a target task tag. Doing that, avoids the target to do a lookup from the initiator task tag each time it receives a data ack (NOP out). The lookups (using hashing table for ex) must be kept to a minimum because they slow down operations. We have to avoid them on the performance path. In iSCSI the only place where a lookup takes place is for Abort task (lookup done on the target side) for "retry" a command (lookup done on the target side) and for read data ack (lookup done on the target side too). In the two first operations a lookup on the initiator task tag can not be avoided because the target may not have time to give a tag to the initiator before things go wrong. However it is not a problem because it is not on the performance path. But for read data acknowledgement, it is on the performance path. This lookup can be easily avoided if the target provides a tag in the data PDU, the NOP out passes back this tag to the target. The tag is the same all along the [READ] command. Regards, Pierre From owner-ips@ece.cmu.edu Tue Jan 9 14:06:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA19735 for ; Tue, 9 Jan 2001 14:06:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09Gu8810922 for ips-outgoing; Tue, 9 Jan 2001 11:56:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09GttU10915 for ; Tue, 9 Jan 2001 11:55:55 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA143398 for ; Tue, 9 Jan 2001 17:54:57 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA67852 for ; Tue, 9 Jan 2001 17:54:58 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569CF.005CE9A1 ; Tue, 9 Jan 2001 17:54:51 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 9 Jan 2001 18:50:38 +0200 Subject: Re: iSCSI : Further review feedback on latest iSCSI-02.txt draft. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk replys in text - thanks Julo Santosh Rao on 09/01/2001 05:19:26 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iSCSI : Further review feedback on latest iSCSI-02.txt draft. Julian/All, More feedback on the latest iSCSI draft (02.txt. dec 30, '00) : Section 1.2.2.3. Data PDU Numbering ============================ What is the scope of DataRNs ? Is this maintained per task or per connection or per session ? This should be explicitly stated in this section as is done in earlier sections for CmdRN and StatRN. [js] per command and it says so [/js] Section 1.2.8.3. Max iSCSI PDU Size ============================ The login parameter for the Max iSCSI PDU Size as described in this section is missing in the Login/Text keys in Appendix C. [js] added in 03 already Section 2.5. SCSI Task Management Command PDU ======================================= The "Logical Unit Number" field in the figure should state "Logical Unit Number or reserved (0)" since the LUN is a don't care value in case of the Target Reset task mgmt command. [js] will fix [/js] Section 2.6. SCSI Task Management Response ================================== The referenced Task Tag field can be removed since the "No Task Found" response error is not valid. (A Target should respond with an accept for an Abort Task request specifying an invalid Initator Task Tag). [js]Different people have different opinions on this - even with HP - talk to Pierre! [/js] Section 2.7 READ SCSI Data PDU ========================== 1) The READ SCSI Data PDU does not contain a Target Task Tag. Hence, when initators acknowledge DataRNs using NOP-OUT, targets will have to lookup the I/O based on its Initator Task Tag. It would be simpler and more optimal for targets to have a single lookup mechanism based on their Target Task Tags. This would also be in sync with Fibre Channel semantics where target lookups are based on only RX_ID. This is possible if Target Task Tags are used for READ SCSI Data PDUs. [js] fixed already in 03 [js] Section 2.17 Asynchronous Event ========================= 1) "Target requests Logout on this connection" iSCSI event indicator. Why not just allow the target to originate a Logout as is done in Fibre Channel ? Besides, this AEN does not meet the requirement. How does the target convey the CID of the connection to be logged out ? [js] CID an timeout appear now in parameters [/js] I do not think SAM-2 lays any guidelines for how the SCSI interconnect protocol handle its native PDUs. (i.e. non-scsi operations). Fibre Channel allows unsolicited inbound traffic for ELS and BLS commands and allowing Logout and NOP-OUT (perhaps better thought of as a NOP command) commands to be originated by targets would be desirable and would get rid of this particular AEN iSCSI event indicator. Section 2.19. Reject PDU =================== What is the value-add from copying the received iSCSI header into the Reject PDU ? Returning the Initiator Task Tag alone in the Reject PDU allows the initiator to look up its per-task data structure to determine what iSCSI PDU was transmitted on the Initiator Task Tag. 2) Is the reject PDU intended to be used for both SCSI and non-SCSI PDUs ? The REJECT PDU should be used to reject ALL non-scsi PDUs. For scsi PDUs, sending a response back thru the standard response PDUs (SCSI response or SCSI Task Mgmt response) would be preferrable since this allows the communication of specific information related to the failure of the SCSI PDU in the respnse Data or Response Code fields of the SCSI Response or SCSI Task Mgmt Response. 3) More detailed Reason Code/Reason Explanation fields in the Reject PDU is highly desirable. This will greatly help enhance the interop abilities of initiators and targets by being able to isolate quickly the reasons for a PDU Reject. I will send in further comments to the reflector on desired reason code and reason explanations for the Reject PDU. [js] reject PDU indicates an iSCSI error, format digest etc. Handling it is implementation dependent[/js] Section 4.4. Format Errors ==================== Quoting from the draft : "When a session is active whenever an initiator receives an iSCSI PDU with a format error, for which it has an outstanding task, it MUST abort the target task and report the error as a SCSI check condition status with a sense key of 4h (hardware error)." Is the above referring to an initiator's action or a target's ? [js] the text says initiator [/js] What is the reasoning behind this ? It seems to complicate Format Error Handling with no clear benefit. Is it the expectation that iSCSI initiators will fake a format error as a CHECK CONDITION back to the SCSI Layer ? If so, why ? And why the choice of "Hardware Error" ? [js} the error was generated by a faulty controller and I did not find any other SCSI sense fit for it[/js] Last, but not the least, the above statement is referring to all iSCSI PDUs and advocates that iSCSI initiators must fake a CHECK CONDITION back to the SCSI layer on any format error of any iSCSI PDU. [js] only if a task is running [/js] This is not applicable for non-scsi iSCSI PDUs which have no SCSI context. I believe the REJECT PDU should only be used for non-scsi PDUs since Rejects to SCSI PDUs will need a different type of reason code/explanation information best fitted into the existing SCSI Response PDUs. Thanks, Santosh Rao - santoshr.vcf From owner-ips@ece.cmu.edu Tue Jan 9 14:09:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA19794 for ; Tue, 9 Jan 2001 14:09:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09Ga7V10300 for ips-outgoing; Tue, 9 Jan 2001 11:36:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09GZxU10289 for ; Tue, 9 Jan 2001 11:35:59 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 9 Jan 2001 11:36:09 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07041013C1@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: Merge FCIP and iFCP? Date: Tue, 9 Jan 2001 11:35:47 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk In a further attempt to get efforts refocused on the important work before us ... I believe that it is the rough consensus of the WG that IF work on iFCP is undertaken, THEN the iFCP and FCIP protocols (and their protocol specifications) should not be merged. Anyone who disagrees should send me email directly rather than posting to the list. Also, please note the "IF" in the above consensus call. OTOH, the issue of whether a common encapsulation is to be used is open. Note that a common encapsulation could consist almost entirely of using the reserved bits in the FCIP header as an (IANA-allocated) protocol number field and agreeing to use common encodings of the SOF and EOF frame delimiters. With my WG co-chair hat off, I have a technical comment to add. One of the arguments made against a common encapsulation has been to observe the wide variety of IP in IP tunnels that exist in IETF specs. In 20/20 hindsight, and based on having worked on aspects of tunneling protocols (e.g., RFC 2983) my opinion is that this is more of an argument in favor of a common encapsulation because there are too many sorts of IP in IP tunnels (i.e., if things could be done over again from a blank slate, there would be fewer). --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Tue Jan 9 15:01:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA20444 for ; Tue, 9 Jan 2001 15:01:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09HX8h12035 for ips-outgoing; Tue, 9 Jan 2001 12:33:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gate.internaut.com ([64.38.134.108]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09HX2U12028 for ; Tue, 9 Jan 2001 12:33:03 -0500 (EST) Received: from e1kj2 (kidneybean [64.38.134.109]) by gate.internaut.com (8.10.2/8.10.2) with SMTP id f09HU0C25090 for ; Tue, 9 Jan 2001 09:30:01 -0800 From: "Bernard Aboba" To: Subject: iSCSI and the IPSEC replay window Date: Tue, 9 Jan 2001 09:33:06 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: <0F31E5C394DAD311B60C00E029101A07041013B8@corpmx9.isus.emc.com> X-Mimeole: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit At IETF 49, we had a presentation on use of IPSEC in iSCSI. While I'm generally positive on the concept of re-using IPSEC in this way, there are some things to think about. One of these is the effect of the IPSEC replay window on TCP behavior. At the 1+ Gbps speeds of iSCSI, it strikes me that even a small variation in delay between two alternate paths will result in falling outside the IPSEC replay window if it is set to a small, fixed value (say 64 packets). So the size of the IPSEC replay window should probably scale with transmission speed. Do we understand how this ought to work and is there a potential for some unforseen effects? Inquiring minds want to know ;) From owner-ips@ece.cmu.edu Tue Jan 9 15:03:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA20456 for ; Tue, 9 Jan 2001 15:03:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09HvH912696 for ips-outgoing; Tue, 9 Jan 2001 12:57:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09HumU12679 for ; Tue, 9 Jan 2001 12:56:49 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id EAFF06DE for ; Tue, 9 Jan 2001 09:56:47 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id JAA03927; Tue, 9 Jan 2001 09:56:46 -0800 (PST) Message-ID: <3A5B5090.14444240@hp.com> Date: Tue, 09 Jan 2001 09:55:28 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI: header digest error at initiator Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, In the section "4.5 Digest errors" you say that if the initiator detects an header digest error in an incoming iSCSI PDU, the TCP connection must be restarted. Is it to address the following case or for something else? 1) the initiator issues a READ 2) one of the inbound data PDU has a header digest error 3) the initiator (as specified in "1.2.5 iSCSI Full Feature Phase": "Initiators MUST NOT perform any score boarding for data and the residual count calculation is to be performed by the targets".) doesn't check the total length of the data received when the completion comes. Because it trusts the target and TCP. 4) the initiator thinks the READ is ok and in fact it is not. To solve that, you propose the drop of the connection in 2) ? Regards, Pierre From owner-ips@ece.cmu.edu Tue Jan 9 16:12:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA21473 for ; Tue, 9 Jan 2001 16:12:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09Id9413976 for ips-outgoing; Tue, 9 Jan 2001 13:39:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hotmail.com (f242.law11.hotmail.com [64.4.17.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09IcpU13961 for ; Tue, 9 Jan 2001 13:38:51 -0500 (EST) Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Tue, 9 Jan 2001 10:38:45 -0800 Received: from 168.191.255.104 by lw11fd.law11.hotmail.msn.com with HTTP; Tue, 09 Jan 2001 18:38:45 GMT X-Originating-IP: [168.191.255.104] From: "Eddy Quicksall" To: ips@ece.cmu.edu Subject: markers Date: Tue, 09 Jan 2001 13:38:45 -0500 Mime-Version: 1.0 Content-Type: text/plain; format=flowed Message-ID: X-OriginalArrivalTime: 09 Jan 2001 18:38:45.0803 (UTC) FILETIME=[66046FB0:01C07A6B] Sender: owner-ips@ece.cmu.edu Precedence: bulk Why was the proposal to use the urgent pointer dropped? I see some problems with the markers. ESQuicksall@hotmail.com _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com From owner-ips@ece.cmu.edu Tue Jan 9 19:17:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA23502 for ; Tue, 9 Jan 2001 19:17:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09MlNn21877 for ips-outgoing; Tue, 9 Jan 2001 17:47:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09MkIU21844 for ; Tue, 9 Jan 2001 17:46:18 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 940A728B; Tue, 9 Jan 2001 15:46:17 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 745AB116; Tue, 9 Jan 2001 17:46:16 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id OAA04599; Tue, 9 Jan 2001 14:46:15 -0800 (PST) Message-ID: <3A5B94B0.FE534C8A@agilent.com> Date: Tue, 09 Jan 2001 14:46:08 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Wayland Jeong Cc: IPS Reflector Subject: Re: iFCP as an IP Storage Work Item References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Sorry, Fibre Channel, does not guarantee delivery. The main reason is that FC does not have transport layer as robust as TCP. A FC ACK simply informs that the frame was delivered. FC does not have any reliable mechanism to handle retransmission when an ACK is not delivered. -Matt Wayland Jeong wrote: > > I think there are some misconceptions here. As you probably know, the ANSI > FC standard is a collection of specifications that are thick enough to choke > an elephant. Of that large body of work, only a portion of it has been > implemented in the commercial world. I would guess from your comments that > you are referring to one of the ways that FC attempts to guarantee delivery. > The two most notable mechanisms in FC are buffer-to-buffer credit and > Class-2. > > Now, BB credit is a link-layer concept which ensures that when a FC frame is > sent, it will not be dropped on the floor due to congestion. You can achieve > the same functionality in Ethernet using 802.3x flow control. That said, > BB-credit is a layer below what is encapsulated in iFCP. iFCP encapsulates > FC at frame level and largely does not concern itself with an FC primitives > (frame delimiters are an exception to this). > > Fibre Channel has a class of service called Class-2 which provides an ACK > mechanism to guarantee delivery to the end station. It uses frame-level > acknowledgements of received data as well as end-to-end buffer credit. This > level of service ensures that data sent is delivered at the FC peer level. > But, Class-2 was not intended to be a congestion management solution. > Class-2 is an error recovery concept and it will not work well in a lossy > network (i.e. IP networks with congestion management through WRED and such). > Moreover, Class-2 is not generally deployed and the FC industry has largely > settled on Class-3. > > Thus, there is nothing within a FC encapsulated in iFCP environment that > makes this protocol inherently reliable. iFCP specifies TCP as the transport > layer and mFCP uses UDP in conjunction with a well-behaved network. > > -----Original Message----- > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > Sent: Thursday, January 04, 2001 5:47 AM > To: 'ips@ece.cmu.edu' > Subject: FW: iFCP as an IP Storage Work Item > > -----Original Message----- > From: Stephen Byan > Sent: Thursday, January 04, 2001 8:40 AM > To: 'Bill Terrell' > Subject: RE: iFCP as an IP Storage Work Item > > It's all the FC stuff that lets iFCP work over an unreliable data transport > like UDP. It's redundant when running over TCP/IP. > > Regards, > -Steve > > > -----Original Message----- > > From: Bill Terrell [mailto:terrell@troikanetworks.com] > > Sent: Wednesday, January 03, 2001 6:10 PM > > To: 'Stephen Byan' > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > >The downside of this advantage is that native iFCP devices would be > > burdened > > >with greater complexity and cost. I therefor think iFCP > > should not be an IP > > >Storage work item. > > > > > >Regards, > > >-Steve > > > > How is a native iFCP endpoint (initiator or target) more > > complex or costly > > than an iSCSI native endpoint? What are the specific > > difficulties inherent > > to native iFCP devices versus native iSCSI devices? > > > > Bill > > From owner-ips@ece.cmu.edu Tue Jan 9 19:17:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA23514 for ; Tue, 9 Jan 2001 19:17:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09MnG721930 for ips-outgoing; Tue, 9 Jan 2001 17:49:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09Mn1U21921 for ; Tue, 9 Jan 2001 17:49:01 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id CC527E63; Tue, 9 Jan 2001 14:49:00 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id OAA16478; Tue, 9 Jan 2001 14:48:52 -0800 (PST) Message-ID: <3A5B95B6.29771A0B@cup.hp.com> Date: Tue, 09 Jan 2001 14:50:30 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Raghavendra Rao Cc: ips@ece.cmu.edu Subject: Re: iSCSI : On the subject of R2T and Task Tags. References: <200101090846.OAA29278@divyaroot.India.Sun.COM> Content-Type: multipart/mixed; boundary="------------E570FC9D13C6307F123FFA58" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------E570FC9D13C6307F123FFA58 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Raghavendra Rao wrote: > > Again, with WRITE Data PDUs there is no target assigned > > per-task tag. While one could argue about vertically encoding the Target > > There is - this is done during R2T time and every solicited data PDU > from the initiator goes out with this tag. The "Target Transfer Tag" that you are referring to is per R2T and not per task. There is no per-task identifier that can be used by the targets to track the task. They will either have to : a) use the Initiator Task Tag or b) resort to some form of vertical encoding of the Target Task Tag (a per-task target id) into the Target Transfer Tag (the PDU defined per-R2T target id). (a) has perofrmance impacts. (b) is an ugly implementation-specific kludge and also reduces the Target Task Tag space due to the vertical encoding of 2 fields into the 32-bit space. Thanks, Santosh --------------E570FC9D13C6307F123FFA58 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------E570FC9D13C6307F123FFA58-- From owner-ips@ece.cmu.edu Tue Jan 9 19:18:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA23528 for ; Tue, 9 Jan 2001 19:18:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09MbFN21544 for ips-outgoing; Tue, 9 Jan 2001 17:37:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09MagU21524 for ; Tue, 9 Jan 2001 17:36:42 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 9 Jan 2001 17:36:52 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07041013C6@corpmx9.isus.emc.com> From: Black_David@emc.com To: esquicksall@hotmail.com, ips@ece.cmu.edu Subject: RE: markers Date: Tue, 9 Jan 2001 17:36:31 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk >Why was the proposal to use the urgent pointer dropped? Here's its tombstone. --David From: Allison Mankin [mankin@isi.edu] Sent: Monday, December 04, 2000 5:26 PM To: ips@ece.cmu.edu Cc: iesg@ietf.org Subject: IESG review of IPS use of URG Pointer The IESG discussed the IPS Urgent Pointer proposal Thursday during its bi-weekly teleconference. The unanimous view was that the use of the TCP Urgent Pointer for this proposed function is not consistent with well-established TCP semantics. Since the IESG specifically ruled out modifications to Internet transport protocols when approving the IPS charter, the IESG will not approve an IPS protocol document that specifies use of the TCP Urgent Pointer in this way. From owner-ips@ece.cmu.edu Tue Jan 9 19:20:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA23557 for ; Tue, 9 Jan 2001 19:20:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09NQGh23057 for ips-outgoing; Tue, 9 Jan 2001 18:26:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09NPSU23037 for ; Tue, 9 Jan 2001 18:25:28 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id E3768643 for ; Tue, 9 Jan 2001 16:25:27 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id E0C67156 for ; Tue, 9 Jan 2001 18:25:26 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id PAA08949 for ; Tue, 9 Jan 2001 15:25:25 -0800 (PST) Message-ID: <3A5B9DDE.1378F557@agilent.com> Date: Tue, 09 Jan 2001 15:25:18 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Login Response PDU Issues. References: <3A592705.B12F6A15@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > 3) Quoting from the draft : > > "In the case that the Status is reject login the initiator should > immediately > close down its end of the TCP connection, thus freeing up the target's > port for some other connection." > > On a REJECT, initiators may wish to retry Login. [perhaps, with a > modification to their Login Command PDU or TEXT parameters . > > Perhaps, the word "may" might be a better fit instead of "should" here. What is so bad with always closing a TCP connection when a login is rejected and opening a new TCP connection when issuing a login? Makes it simple. Only one way to do things. Only one path to test. -Matt Wakeley Agilent Technologies > > Thanks, > Santosh From owner-ips@ece.cmu.edu Tue Jan 9 19:20:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA23579 for ; Tue, 9 Jan 2001 19:20:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09N9GV22540 for ips-outgoing; Tue, 9 Jan 2001 18:09:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09N8VU22514 for ; Tue, 9 Jan 2001 18:08:31 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 9 Jan 2001 18:08:06 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07041013C7@corpmx9.isus.emc.com> From: Black_David@emc.com To: aboba@internaut.com, ips@ece.cmu.edu Subject: RE: iSCSI and the IPSEC replay window Date: Tue, 9 Jan 2001 18:08:05 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Hmm ... I would have thought that the separate TCP connections for the alternate paths would use separate IPsec SAs and hence would not share a replay window, making this a non- issue. Steve Kent had a number of things to say in the ipsec WG about running IPsec at gigabit speeds, all of which are probably applicable to iSCSI, but best left to the ipsec WG. --David > -----Original Message----- > From: Bernard Aboba [SMTP:aboba@internaut.com] > Sent: Tuesday, January 09, 2001 12:33 PM > To: ips@ece.cmu.edu > Subject: iSCSI and the IPSEC replay window > > At IETF 49, we had a presentation on use of IPSEC in > iSCSI. While I'm generally positive on the concept > of re-using IPSEC in this way, there are some things > to think about. > > One of these is the effect of the IPSEC replay window > on TCP behavior. At the 1+ Gbps speeds of iSCSI, it > strikes me that even a small variation in delay > between two alternate paths will result in falling > outside the IPSEC replay window if it is set to a > small, fixed value (say 64 packets). > > So the size of the IPSEC replay window should probably > scale with transmission speed. > > Do we understand how this ought to work and is there > a potential for some unforseen effects? > > Inquiring minds want to know ;) From owner-ips@ece.cmu.edu Tue Jan 9 19:21:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA23590 for ; Tue, 9 Jan 2001 19:21:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09M9G820729 for ips-outgoing; Tue, 9 Jan 2001 17:09:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09M8wU20722 for ; Tue, 9 Jan 2001 17:08:58 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 7AF6C464 for ; Tue, 9 Jan 2001 15:08:57 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 0E912139 for ; Tue, 9 Jan 2001 17:08:54 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id OAA29947 for ; Tue, 9 Jan 2001 14:08:52 -0800 (PST) Message-ID: <3A5B8BEE.8896E9EA@agilent.com> Date: Tue, 09 Jan 2001 14:08:46 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iFCP as an IP Storage Work Item References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Joshua Tseng wrote: > > I don't want to stifle any creative technical discussion here, > but I feel the need to remind everybody that iFCP is positioned > as a gateway technology only. If iFCP is a "gateway", that to me means that it will convert FCP to iSCSI. In that case, FCP is defined as a standard, and iSCSI is a work in progress by this group. Those building such gateways then only have to figure out how to make one side of the gateway look native to the other. What is there to standardize? -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Tue Jan 9 19:23:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA23607 for ; Tue, 9 Jan 2001 19:23:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09NEGv22694 for ips-outgoing; Tue, 9 Jan 2001 18:14:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09NDxU22684 for ; Tue, 9 Jan 2001 18:13:59 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 3FED04C4 for ; Tue, 9 Jan 2001 16:13:59 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 0AA1DDB for ; Tue, 9 Jan 2001 18:13:31 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id PAA08167 for ; Tue, 9 Jan 2001 15:13:17 -0800 (PST) Message-ID: <3A5B9B07.A3076F78@agilent.com> Date: Tue, 09 Jan 2001 15:13:11 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Agree or Disagree to Merge FCIP and iFCP References: <005101c07816$860593e0$75b30118@orng1.occa.home.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I disagree to merge FCIP and iFCP. When do we get to vote on booting iFCP out of this WG? -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Tue Jan 9 19:23:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA23618 for ; Tue, 9 Jan 2001 19:23:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09N2Lo22315 for ips-outgoing; Tue, 9 Jan 2001 18:02:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09N1MU22291 for ; Tue, 9 Jan 2001 18:01:22 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id 2B5301C9B for ; Tue, 9 Jan 2001 15:01:20 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id PAA07222; Tue, 9 Jan 2001 15:01:18 -0800 (PST) Message-ID: <3A5B97F1.AFB92689@hp.com> Date: Tue, 09 Jan 2001 15:00:01 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI : Further review feedback on latest iSCSI-02.txt draft. Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian >>>>>>>>> Section 2.6. SCSI Task Management Response ================================== The referenced Task Tag field can be removed since the "No Task Found" response error is not valid. (A Target should respond with an accept for an Abort Task request specifying an invalid Initator Task Tag). [js]Different people have different opinions on this - even with HP - talk to Pierre! [/js] <<<<<<<<< We are in agreement, it is just a point of terminology. Some month ago, we had a discussion on the IPS about what to do in case the target doesn't find the task to abort. I requested that the Abort response returns "OK" (something different than Function rejected). You added the value "Task not found". But "Function complete" can works too. Regards, Pierre From owner-ips@ece.cmu.edu Tue Jan 9 19:24:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA23631 for ; Tue, 9 Jan 2001 19:24:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09LuFu20334 for ips-outgoing; Tue, 9 Jan 2001 16:56:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09Lu5U20324 for ; Tue, 9 Jan 2001 16:56:05 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id AD5AC29 for ; Tue, 9 Jan 2001 14:56:04 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id A70B913B for ; Tue, 9 Jan 2001 16:56:03 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id NAA28396 for ; Tue, 9 Jan 2001 13:56:01 -0800 (PST) Message-ID: <3A5B88EB.C539DE34@agilent.com> Date: Tue, 09 Jan 2001 13:55:55 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 Cc: IPS Reflector Subject: Re: new iSCSI draft - 02.txt References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David Peterson wrote: > > The point is (whether you use an mbuf/mblk/zbuf/socket buffer or whatever) > some type of manipulation will be required. Bottom line is a periodic marker > does not work well for a software based implementation. As I've indicated > before > I do not believe a framing mechanism is required at this point in time. We > should start looking at what it will take to use SCTP (or something > similar). Perhaps it's not needed now by the first generation software implementations, but it must be standardized now so that when it is needed by tomorrow's hardware implementations, it will be there. If you relegate the future to SCTP, then that means that today's products (using TCP) will not be able to interoperate with tomorrow's products (using SCTP). As I have stated, the marker is negotiated, and only supplied if the receiver requests it. -Matt Wakeley Agilent Technologies > When the 10G pipes become available in the future we will then be > ready/positioned > and have a solution that should work for all. From owner-ips@ece.cmu.edu Tue Jan 9 19:29:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA23665 for ; Tue, 9 Jan 2001 19:29:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09N7GD22460 for ips-outgoing; Tue, 9 Jan 2001 18:07:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09N71U22453 for ; Tue, 9 Jan 2001 18:07:01 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id E6CE1277 for ; Tue, 9 Jan 2001 16:07:00 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 8686312D for ; Tue, 9 Jan 2001 18:06:59 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id PAA07835 for ; Tue, 9 Jan 2001 15:06:57 -0800 (PST) Message-ID: <3A5B998B.E8D09DC2@agilent.com> Date: Tue, 09 Jan 2001 15:06:51 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI: Byte padding requirement question References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > > Wan-Hui, > > Yes it is a side effect of the marker and we could have made it optional > but it ends up being messy if you admit > non-symmetric situations in which you send markers but do not receive them > or viceversa. What makes it messy in non-symmetric situations? > As padding does not practically add too much I thought it would be simpler > for the implementers to have it always (and I've resisted previous requests > to include padding!). Besides keeping markers out of iscsi headers, the main reason for padding is that drivers want integers in the iscsi headers to always align on integer boundaries. Otherwise, most risc based processors will have to do a byte copy to align the headers on integer boundaries. -Matt Wakeley Agilent Technologies > > Regards, > Julo > > "Lee, WanHuiHendra" on 05/01/2001 04:49:00 > > Please respond to "Lee, WanHuiHendra" > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: iSCSI: Byte padding requirement question > > Julian, > > In the length field description (section 2.2.3 Length): > "... The length field accounts for proper iSCSI PDU content; whatever > padding is required to reach a 4 byte boundary in the TCP stream is implied > by the protocol but not accounted for in the length field." > > Is this a side effect requirement due to the added "framing using marker at > fixed interval" mechanism ? > > If yes, since framing is a negotiated feature, would it make sense to make > the byte padding only a requirement if framing is enabled ? > > If this is not due to the framing feature, could you let us know why it is > a > requirement (I did not see it in previous drafts) ? > > Thanks, > Wan-Hui From owner-ips@ece.cmu.edu Tue Jan 9 20:27:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA24180 for ; Tue, 9 Jan 2001 20:27:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09NhIg23640 for ips-outgoing; Tue, 9 Jan 2001 18:43:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09Nh0U23630 for ; Tue, 9 Jan 2001 18:43:00 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 8FA733D8 for ; Tue, 9 Jan 2001 16:42:59 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 4329F156 for ; Tue, 9 Jan 2001 18:42:58 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id PAA09880 for ; Tue, 9 Jan 2001 15:42:56 -0800 (PST) Message-ID: <3A5BA1FA.652B86CD@agilent.com> Date: Tue, 09 Jan 2001 15:42:50 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : On the subject of R2T and Task Tags. References: <200101090846.OAA29278@divyaroot.India.Sun.COM> <3A5B95B6.29771A0B@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit The task tags in iSCSI should work the same as they do in FCP: The initiator supplies a initiator task tag per task that it uses to associate any received messages from the target to that task. [a "undefined" initiator task tag should have a value of 0xffffffff since a value of zero is a valid number for a table look up ]. The target supplies a target task tag per task that it sends to the initiator where a response from the initiator is expected. The target uses the target task tag to associate any solicited received messages from the initiator (other than the initial command) to that task. For SCSI READ commands, there is no need to specifiy a target task tag. Typically in FCP today, the targets to NOT assign target task tags (RX_IDs in FC terms) for READs. For SCSI WRITE commands, the target MAY assign a target task tag to the I/O that it uses when sending R2Ts to the initiator. The initiator must then echo that target task tag in replies it sends to the target. I strongly object to the words in section 2.16 "All outstanding R2T should have different Target Transfer Tags". This specification should not dictate to the target how it chooses or uses the Target Task Tags. -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Tue Jan 9 20:28:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA24191 for ; Tue, 9 Jan 2001 20:27:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A08IY24284 for ips-outgoing; Tue, 9 Jan 2001 19:08:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A07lU24270 for ; Tue, 9 Jan 2001 19:07:48 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Tue, 9 Jan 2001 16:17:34 -0800 Message-ID: From: Charles Monia To: Matt Wakeley Cc: IPS Reflector Subject: RE: iFCP as an IP Storage Work Item Date: Tue, 9 Jan 2001 16:07:19 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Matt: To clarify matters. The response was in reference to a thread discussing a "native" iFCP device. I assume the authors' envisioned that this would be implemented in a NIC of some sort. Hence, the reminder. In any event, iFCP gateways convert iFCP to native FC. Charles > -----Original Message----- > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > Sent: Tuesday, January 09, 2001 2:09 PM > To: IPS Reflector > Subject: Re: iFCP as an IP Storage Work Item > > > Joshua Tseng wrote: > > > > I don't want to stifle any creative technical discussion here, > > but I feel the need to remind everybody that iFCP is positioned > > as a gateway technology only. > > If iFCP is a "gateway", that to me means that it will convert > FCP to iSCSI. > In that case, FCP is defined as a standard, and iSCSI is a > work in progress by > this group. Those building such gateways then only have to > figure out how to > make one side of the gateway look native to the other. What > is there to > standardize? > > -Matt Wakeley > Agilent Technologies > From owner-ips@ece.cmu.edu Tue Jan 9 20:28:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA24202 for ; Tue, 9 Jan 2001 20:28:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09NpIP23850 for ips-outgoing; Tue, 9 Jan 2001 18:51:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ludwig.troikanetworks.com (host03.troikanetworks.com [12.31.172.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09NolU23839 for ; Tue, 9 Jan 2001 18:50:47 -0500 (EST) Received: by host03.troikanetworks.com with Internet Mail Service (5.5.2653.19) id ; Tue, 9 Jan 2001 15:50:54 -0800 Message-ID: From: Wayland Jeong To: "'Matt Wakeley'" , Wayland Jeong Cc: IPS Reflector Subject: RE: iFCP as an IP Storage Work Item Date: Tue, 9 Jan 2001 15:50:53 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Yes. I did not mean to imply that Class-2 had a built-in retry mechanism. Class-2 is simply an error handling mechanism which provides a more timely notification of un-delivered data. Any reliable delivery mechanism similar to TCP would need to be implemented as a session layer above FC Class-2. Thanks for the clarification. -----Original Message----- From: Matt Wakeley [mailto:matt_wakeley@agilent.com] Sent: Tuesday, January 09, 2001 2:46 PM To: Wayland Jeong Cc: IPS Reflector Subject: Re: iFCP as an IP Storage Work Item Sorry, Fibre Channel, does not guarantee delivery. The main reason is that FC does not have transport layer as robust as TCP. A FC ACK simply informs that the frame was delivered. FC does not have any reliable mechanism to handle retransmission when an ACK is not delivered. -Matt Wayland Jeong wrote: > > I think there are some misconceptions here. As you probably know, the ANSI > FC standard is a collection of specifications that are thick enough to choke > an elephant. Of that large body of work, only a portion of it has been > implemented in the commercial world. I would guess from your comments that > you are referring to one of the ways that FC attempts to guarantee delivery. > The two most notable mechanisms in FC are buffer-to-buffer credit and > Class-2. > > Now, BB credit is a link-layer concept which ensures that when a FC frame is > sent, it will not be dropped on the floor due to congestion. You can achieve > the same functionality in Ethernet using 802.3x flow control. That said, > BB-credit is a layer below what is encapsulated in iFCP. iFCP encapsulates > FC at frame level and largely does not concern itself with an FC primitives > (frame delimiters are an exception to this). > > Fibre Channel has a class of service called Class-2 which provides an ACK > mechanism to guarantee delivery to the end station. It uses frame-level > acknowledgements of received data as well as end-to-end buffer credit. This > level of service ensures that data sent is delivered at the FC peer level. > But, Class-2 was not intended to be a congestion management solution. > Class-2 is an error recovery concept and it will not work well in a lossy > network (i.e. IP networks with congestion management through WRED and such). > Moreover, Class-2 is not generally deployed and the FC industry has largely > settled on Class-3. > > Thus, there is nothing within a FC encapsulated in iFCP environment that > makes this protocol inherently reliable. iFCP specifies TCP as the transport > layer and mFCP uses UDP in conjunction with a well-behaved network. > > -----Original Message----- > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > Sent: Thursday, January 04, 2001 5:47 AM > To: 'ips@ece.cmu.edu' > Subject: FW: iFCP as an IP Storage Work Item > > -----Original Message----- > From: Stephen Byan > Sent: Thursday, January 04, 2001 8:40 AM > To: 'Bill Terrell' > Subject: RE: iFCP as an IP Storage Work Item > > It's all the FC stuff that lets iFCP work over an unreliable data transport > like UDP. It's redundant when running over TCP/IP. > > Regards, > -Steve > > > -----Original Message----- > > From: Bill Terrell [mailto:terrell@troikanetworks.com] > > Sent: Wednesday, January 03, 2001 6:10 PM > > To: 'Stephen Byan' > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > >The downside of this advantage is that native iFCP devices would be > > burdened > > >with greater complexity and cost. I therefor think iFCP > > should not be an IP > > >Storage work item. > > > > > >Regards, > > >-Steve > > > > How is a native iFCP endpoint (initiator or target) more > > complex or costly > > than an iSCSI native endpoint? What are the specific > > difficulties inherent > > to native iFCP devices versus native iSCSI devices? > > > > Bill > > From owner-ips@ece.cmu.edu Tue Jan 9 20:29:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA24213 for ; Tue, 9 Jan 2001 20:29:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09NWK923218 for ips-outgoing; Tue, 9 Jan 2001 18:32:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09NVZU23197 for ; Tue, 9 Jan 2001 18:31:35 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 912A772E for ; Tue, 9 Jan 2001 16:31:34 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 963AA126 for ; Tue, 9 Jan 2001 18:29:37 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id PAA09154 for ; Tue, 9 Jan 2001 15:29:35 -0800 (PST) Message-ID: <3A5B9ED8.55CB9786@agilent.com> Date: Tue, 09 Jan 2001 15:29:28 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Login Response PDU Issues. References: <3A592705.B12F6A15@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > > Julian, > > Some concerns regarding the Login Response PDU section of the latest > iSCSI draft : > > Section 2.11.3. Login Response Status > ====================================== > 1) Should targets use a REJECT PDU or Login Response PDU with a status > of "reject login" to indicate Login Reject ? Section 2.11.3 and 4.4 > contradict each other on this subject. > > A standard REJECT PDU used to reject ALL non-scsi PDUs > [and ONLY non-scsi PDUs] would be desirable and would provide > the following benefits : There already is one. Check out section 2.19 "Reject". > o Allows more specific error information to be conveyed > within SCSI Task Mgmt response PDUs and SCSI Response > PDUs, on a reject of SCSI Task Mgmt Cmd or SCSI Command. > (thru the Response Data for Scsi Response PDU and > Response field for SCSI Task Mgmt response PDU). > > o Consistent with Fibre Channel semantics for the standard > LS_RJT for all ELS'. This also allows for easy mapping > from iSCSI Response PDUs to FC ELS responses and vice versa > in iSCSI-FC bridges. > (since an iSCSI-FC bridge would be mapping certain types of > operations and their responses from one protocol to another > like > Login. Logout, NOP-OUT, etc.) > > o REJECT field in the Login Response PDU can be made reserved. > > 2) Whatever the mechanism that will be finally used for login rejection > [be it the REJECT PDU or "reject login" in login response PDU], it > should > make provisions for detailed reason codes and reason explanations that > allow > the initiator to determine the cause of the rejection. This does not > currently exist in the draft. Perhaps you should provide a detailed list of reason codes and reason explanations you would like to see. -Matt Wakeley Agilent Technologies > Thanks, > Santosh From owner-ips@ece.cmu.edu Tue Jan 9 20:31:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA24262 for ; Tue, 9 Jan 2001 20:31:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A01I424111 for ips-outgoing; Tue, 9 Jan 2001 19:01:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A00gU24091 for ; Tue, 9 Jan 2001 19:00:43 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Tue, 9 Jan 2001 16:10:29 -0800 Message-ID: From: Charles Monia To: "Ips (E-mail)" Cc: Joshua Tseng Subject: New iFCP Draft available Date: Tue, 9 Jan 2001 16:00:14 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Folks: We have a submitted new version of iFCP to the IETS archive. In order to make it more readily available for review we have also posted copy at the following location. http://www.nishansystems.com/ietf/draft-monia-ips-iFCP-01.txt The revised version contains the following major changes: 1. Annex A -- A proposed extension defining an FC-4 address-transparent mode for iFCP. This mode eliminates the address translation process at the cost in scalability. The annex was prepared to articulate the design issues and to serve as a basis for further technical discussions. 2. Added explanatory text describing the manner in which a gateway implementation might incorporate support for FC loops and E_PORTs. This does not involve substantive changes to the iFCP protocol but does suggest ways in which the gateway architecture can be leveraged. This will probably be expanded and included in an annex at some point. 3. Expanded and modified the N_PORT login session model. 4. Modified the encapsulation to add provisions for additional end-to-end error detection. Many thanks to the co-authors for their contributions. Charles Charles Monia Senior Technology Consultant Nishan Systems email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ece.cmu.edu Tue Jan 9 20:31:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA24273 for ; Tue, 9 Jan 2001 20:31:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09NnHS23785 for ips-outgoing; Tue, 9 Jan 2001 18:49:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09NmbU23768 for ; Tue, 9 Jan 2001 18:48:37 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 8B7DF3EE for ; Tue, 9 Jan 2001 16:48:36 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id C14C881 for ; Tue, 9 Jan 2001 18:48:35 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id PAA10128 for ; Tue, 9 Jan 2001 15:48:34 -0800 (PST) Message-ID: <3A5BA34C.BBB76866@agilent.com> Date: Tue, 09 Jan 2001 15:48:28 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: Questions about FCIP References: <6B190B34070BD411ACA000B0D0214E563D3484@newman.tenornet.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit "Kavi, Prabhu" wrote: > > I have some questions about the Fibre Channel over TCP/IP I-D. > I understand IP networks well, but am a Fibre Channel newbie and > am asking these questions to learn more about Fibre Channel's > requirements rather than to challenge the document. Here are > my questions: > > 1. Why Fibre Channel over TCP, rather than over UDP? Is it > because of an expectation of lossy IP networks? Or is it > equally valid to put Fibre Channel over UDP (and therefore > can be the subject of another I-D). Because UDP does not implement congestion management. > 2. What are the typical values of the ED_TOV and R_A_TOV timers? > Do I understand this right to say that these timers effectively > limit the latency across the IP network? In Fibre Channel, the ED_TOV and R_A_TOV timer values are advertised from the switches to the N_Ports during fabric login. If a switch knows it has a FCIP port, it could increase theses values as needed (but they are already unreasonably huge to begin with) > 3. What is an acceptable loss ratio of Fibre Channel datagrams > that still allows the SAN devices to deliver appropriate > end user performance (order of magnitude estimate is > sufficient). > > The reasons I ask are: > > * I believe it is invalid to assume that IP networks are > necessarily more lossy than Fibre Channel networks. A > well-engineered IP network, using a combination of both > Diff-Serv and MPLS, can achieve (close to) zero loss. > * Stacking one reliable delivery protocol on top of another can > possibly lead to interesting performance problems after a loss > occurs, depending upon the timer values for TCP and Fibre Channel. You talk about stacking one reliable protocol on another. TCP is one, what is the other? Don't kid yourself into thinking fibre channel is reliable! -Matt Wakeley Agilent Technologies > > Thanks in advance for your answers. > > Prabhu Kavi > ---------------------------------------------------------------------- > Prabhu Kavi Phone: 1-978-264-4900 x125 > Director, Adv. Prod. Planning Fax: 1-978-264-0671 > Tenor Networks Email: prabhu_kavi@tenornetworks.com > 100 Nagog Park WWW: www.tenornetworks.com > Acton, MA 01720 From owner-ips@ece.cmu.edu Tue Jan 9 20:34:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA24284 for ; Tue, 9 Jan 2001 20:34:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f09NUGl23145 for ips-outgoing; Tue, 9 Jan 2001 18:30:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f09NTfU23128 for ; Tue, 9 Jan 2001 18:29:41 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 9 Jan 2001 18:29:03 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07041013C9@corpmx9.isus.emc.com> From: Black_David@emc.com To: matt_wakeley@agilent.com, ips@ece.cmu.edu Subject: RE: Agree or Disagree to Merge FCIP and iFCP Date: Tue, 9 Jan 2001 18:29:01 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > I disagree to merge FCIP and iFCP. Noted. > When do we get to vote on booting iFCP out of this WG? That's not the way things work. As noted previously, whether to work on iFCP is a rechartering decision that gets made by WG co-chairs (Steve and myself) in consultation with the ADs for approval by the IESG. Cogent technical input to that decision is still welcome either on the list or directly to Steve and me. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Tue Jan 9 22:01:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA25726 for ; Tue, 9 Jan 2001 22:01:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A0pTk25269 for ips-outgoing; Tue, 9 Jan 2001 19:51:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcars04e.ca.nortel.com (h56s242a129n47.user.nortelnetworks.com [47.129.242.56]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A0oKU25235 for ; Tue, 9 Jan 2001 19:50:20 -0500 (EST) Received: from zbl6c012.corpeast.baynetworks.com by zcars04e.ca.nortel.com; Tue, 9 Jan 2001 19:49:32 -0500 Received: from zbl6c002.corpeast.baynetworks.com ([132.245.205.52]) by zbl6c012.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id ZLK0BDD3; Tue, 9 Jan 2001 19:49:31 -0500 Received: from nortelnetworks.com (dhcp223-140.engeast.baynetworks.com [192.32.223.140]) by zbl6c002.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id CN99DAF1; Tue, 9 Jan 2001 19:49:31 -0500 Message-ID: <3A5BB1BF.27CEED03@nortelnetworks.com> Date: Tue, 09 Jan 2001 19:50:07 -0500 X-Sybari-Space: 00000000 00000000 00000000 From: "Victor Firoiu" Organization: Nortel Networks X-Mailer: Mozilla 4.76 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector CC: "Franco Travostino" Subject: Performance of iSCSI, FCIP and iFCP Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Orig: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi, we would like to add to the recent discussion on IP storage protocols a performance point of view showing significant differences between the protocols discussed. Any comment would be highly appreciated. Briefly, transporting a set of storage connections between two storage enclaves using multiple TCP sessions (iSCSI and iFCP) provides significantly higher aggregate average throughput than transporting the same set of storage connections over a single TCP session (FCIP), the difference being proportional to the square of the number of TCP sessions. This is due to the way TCP congestion control reacts to packet losses. Here we have two questions: why are we talking about packet losses and why is this difference in throughput. There are several reasons for packet losses: network congestion, link errors and network errors. Network congestion is pervasive in current IP networks, where the only way to control congestion is through dropping packets. Traffic engineering, admission control and bandwidth reservation are currently in early stages of definition. DiffServ supporting QoS infrastructure will not be widely available in the near future (especially when we realize that it is not a simple matter of asserting the EF PHB bit as if it were the old IP ToS; it rather needs network services and supporting SLA negotiation). The DiffServ EF PHB RFC is currently under redefinition. On the other hand, if such supporting QoS infrastructure were indeed available and pervasive, today, why would we need TCP to begin with? Even in a perfectly engineered network, link errors occur. If we take the Fibre Channel objective of 10^-12 Bit Error Rate, for a 10Gb/s link, this is one error every 100 seconds. Network errors also occur with significant frequency in IP networks. Jonathan Stone and Craig Partridge recently reported in Sigcomm 2000 that network errors caught by TCP checksum occur with significant frequency (between one packet in 1100 and 1 in 32000) and without link CRC catching it. For the second question, TCP throughput is impacted by each packet loss. Following TCP's congestion control algorithm existent in all major implementations (Tahoe, Reno, New-Reno, SACK) each packet loss results in the TCP sender's congestion window being reduced to half of its current value, and therefore (assuming constant Round Trip Time), TCP's throughput is halved. After that, the window increases by roughly one packet every two Round Trip Times (assuming the popular Delayed-Acknowledgement algorithm). The temporary decrease in TCP's rate translates into an amount of data missing transmission opportunity. As we show later, for N storage connections sharing an IP "pipe" of rate E, the amount of data missing the opportunity to be transmitted due to a packet loss is D(N) = E^2/(N^2)*RTT^2/(256*M) in the case of iSCSI and iFCP, and D(1) = E^2*RTT^2/(256*M) = D(N)*N^2 in the case of FCIP where RTT = Round Trip Time M = packet size For example, for a set of N=100 connections totaling E=10Gb/s, RTT=10ms, M=1500B, the data not transmitted in time due to a packet loss for iSCSI or iFCP is D(N)=2.6MB. For the same set transported over one TCP session as in FCIP, the data not sent in time is D(1)= 26GB, a 10,000 fold increase. The time interval for TCP to recover its sending rate to its initial value after a packet loss is I(N)= 0.833seconds in the case of iSCSI and iFCP, and I(1)=83.3seconds in the case of FCIP. Observe that in the case of FCIP, the time to recover its rate, I(1)=83.3s, is of the same order of magnitude as the time between two packet losses due exclusively to the link Bit Error Rate. In other words, a packet losse occurs almost immediately after TCP has recovered its rate. This means that FCIP delivers on average about 3/4 of the required 10Gb/s rate, since 1/4 of rate is lost during the time TCP rate increases linearly from 1/2 to full rate. (More precisely, the effective rate is 8.27Gb/s because 1/4 of rate is lost during 83.3s, and the time between two errors is now 120.825s due to decreased sending rate). By comparison, iSCSI or iFCP deliver approximately 9.99979Gb/s (i.e., lost 1/4 of one TCP full rate of 100Mb/s during 0.833s out of a 100s interval). In conclusion, from a performance point of view, transporting storage connections (SCSI or Fibre Channel) on multiple TCP sessions is much more effective than tunneling through a single TCP session, and the difference is proportional to the square of the number of TCP sessions. The math. For a TCP session to sustain a rate of C bits/second, the TCP's congestion window W (measured in number packets) has to be W=RTT*C/(8*M) where RTT = Round Trip Time in seconds M = packet size in Bytes The time needed by the TCP sender to recover from a single packet loss and have its sending rate reach the previous C value is I = 2*RTT*W/2 = RTT*W = RTT^2*C/(8*M) The total amount of data (in Bytes) missing the opportunity to be transmitted in this time interval I is D = C/8*I/4 = C^2*RTT^2/(256*M) If we consider a set of N storage connections sharing an IP "pipe" of rate E, they can be transported in N TCP sessions, as in iSCSI or iFCP. Assuming all connections equal, each TCP session sends at a rate of E/N. One packet loss impacts only one TCP session, and thus, the total amount of data missing the opportunity to be transmitted due to a packet loss is D(N) = E^2/(N^2)*RTT^2/(256*M) On the other hand, if the same set of N storage connections is transported in one TCP session, as in FCIP, the total amount of data losing the opportunity to be transmitted due to a packet loss is D(1) = E^2*RTT^2/(256*M) = D(N)*N^2. For more details on TCP performance see for example: "Modeling TCP Reno Performance: A Simple Model and its Empirical Validation." J. Padhye, V. Firoiu, D. Towsley and J. Kurose, IEEE/ACM Transactions on Networking, April 2000. Franco Travostino, Victor Firoiu ----- Content Internetworking Lab, Technology Center Nortel Networks, Inc. 600 Technology Park Billerica, MA 01821 USA From owner-ips@ece.cmu.edu Tue Jan 9 23:21:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA27089 for ; Tue, 9 Jan 2001 23:21:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A34Nc27999 for ips-outgoing; Tue, 9 Jan 2001 22:04:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A34HU27993 for ; Tue, 9 Jan 2001 22:04:18 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 182F21960 for ; Tue, 9 Jan 2001 19:04:10 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id TAA07965; Tue, 9 Jan 2001 19:04:05 -0800 (PST) Message-ID: <3A5BD187.7520B642@cup.hp.com> Date: Tue, 09 Jan 2001 19:05:43 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Initiators expected to fake CHECK CONDITIONS. Content-Type: multipart/mixed; boundary="------------981462BFADCD0BBD099F400A" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------981462BFADCD0BBD099F400A Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, My comments embedded below. Thanks, Santosh > Section 4.4. Format Errors > ==================== > Quoting from the draft : > "When a session is active whenever an initiator receives an iSCSI PDU > with a format error, for which it has an outstanding task, it MUST > abort the target task and report the error as a SCSI check condition > status with a sense key of 4h (hardware error)." > What is the reasoning behind this ? It seems to complicate Format Error > Handling with no clear benefit. Is it the expectation that iSCSI > initiators > will fake a format error as a CHECK CONDITION back to the SCSI >Layer ? If so, why ? And why the choice of "Hardware Error" ? [js} the error was generated by a faulty controller and I did not find any other SCSI sense fit for it[/js] A clean layering should be maintained between iSCSI layer errors and SCSI layer errors. A format error on a iSCSI PDU header does not constitute a SCSI error. The specific error returned back to the SCSI upper layer driver in this case is really O.S. specific since each O.S. has its own error codes to deal with interconnect transport errors. Expecting initiators to fabricate sense data on behalf of a target and faking a CHECK CONDITION back to the upper layers is : a) violating layering b/n iSCSI and SCSI layers. b) adds un-necessary complexity to the initiator which now needs to build sense data internally on behalf of a target. c) will cause different error recovery paths to be taken in the upper layer SCSI driver which is expecting to see some std O.S. defined error codes to be returned on interconnect transport errors. > Last, but not the least, the above statement is referring to all iSCSI > PDUs and advocates that iSCSI initiators must fake a CHECK CONDITION > back to the SCSI layer on any format error of any iSCSI PDU. [js] only if a task is running [/js] I'm somewhat confused here. Let us say a format error occurs on a Logout. What is the error recovery to be done in this case if there are other outstanding tasks ? Is this section advocating erroring ALL the outstanding tasks [that did not encounter format error] ? > This is not applicable for non-scsi iSCSI PDUs which have no SCSI > context. I believe the REJECT PDU should only be used for non-scsi > PDUs since Rejects to SCSI PDUs will need a different type of reason > code/explanation information best fitted into the existing SCSI Response --------------981462BFADCD0BBD099F400A Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------981462BFADCD0BBD099F400A-- From owner-ips@ece.cmu.edu Tue Jan 9 23:22:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA27100 for ; Tue, 9 Jan 2001 23:22:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A3COs28168 for ips-outgoing; Tue, 9 Jan 2001 22:12:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A3BPU28148 for ; Tue, 9 Jan 2001 22:11:25 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 884F616B2 for ; Tue, 9 Jan 2001 19:11:21 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id TAA08418; Tue, 9 Jan 2001 19:11:16 -0800 (PST) Message-ID: <3A5BD336.F983F9F8@cup.hp.com> Date: Tue, 09 Jan 2001 19:12:55 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: header digest error at initiator References: <3A5B5090.14444240@hp.com> Content-Type: multipart/mixed; boundary="------------E478BC8B10BEBC534631EFC9" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------E478BC8B10BEBC534631EFC9 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Pierre Labat wrote: > Julian, > > In the section "4.5 Digest errors" > you say that if the initiator detects an header digest error > in an incoming iSCSI PDU, the TCP connection must be > restarted. > > Is it to address the following case or for something else? > > 1) the initiator issues a READ > > 2) one of the inbound data PDU has a header digest error > > 3) the initiator (as specified in "1.2.5 iSCSI Full Feature Phase": > "Initiators MUST NOT perform any score boarding for data > and the residual count calculation is to be performed by the > targets".) > doesn't check the total length of the data received when the > completion > comes. Because it trusts the target and TCP. > > 4) the initiator thinks the READ is ok and in fact it is not. This sounds like extreme behaviour to me. Why is iSCSI attempting to prevent initiators from score-boarding data ? This is a standard practice most parallel scsi and FC initiators would follow to detect cases of I/O underrun. Not doing this will cause the initiator to trust the target's resid value and if there was an underrun, reporting success back to upper layers can result in data corruption. What is the justification for NOT allowing initiators to score-board ? > > To solve that, you propose the drop of the connection in 2) ? > If this is the intention of the recommended error recovery, it is the result of not allowing score-boarding. By score-boarding an initiator would detect an underrun and would just error the affected I/O back. In this case, all outstanding tasks on that connection are being affected. Both the Format Error and Digest Error handling seem too extreme. A format error or digest error recovery should only involve the affected task and none others. Thanks, Santosh --------------E478BC8B10BEBC534631EFC9 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------E478BC8B10BEBC534631EFC9-- From owner-ips@ece.cmu.edu Tue Jan 9 23:23:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA27111 for ; Tue, 9 Jan 2001 23:23:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A2lNC27664 for ips-outgoing; Tue, 9 Jan 2001 21:47:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A2l2U27658 for ; Tue, 9 Jan 2001 21:47:02 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 9EDF2131C; Tue, 9 Jan 2001 18:47:01 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA06730; Tue, 9 Jan 2001 18:46:56 -0800 (PST) Message-ID: <3A5BCD83.17F74969@cup.hp.com> Date: Tue, 09 Jan 2001 18:48:35 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI : DataRN issues. References: Content-Type: multipart/mixed; boundary="------------8C49C8F656D7D6F2B611B004" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------8C49C8F656D7D6F2B611B004 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Hi Julian, Some further comments on this. Thanks, Santosh julian_satran@il.ibm.com wrote: > Section 1.2.2.3. Data PDU Numbering > ============================ > What is the scope of DataRNs ? Is this maintained per task or per > connection or per session ? This should be explicitly stated in this > section as is done in earlier sections for CmdRN and StatRN. > [js] per command and it says so [/js] I believe that the DataRN feature and all its related overheads add a level of complexity that is going to discourage both targets and initiators from using this. I am yet to see anyone state sufficient benefits of this feature that can justify this complexity. Moreover, the benefit it may provide [if any, and if somebody does use it] is in the error path context and its overheads hurt the performance path. Therefore, in the interests of keeping iSCSI simple, I would like to request removal of all references to DataRN and DataNumber. Thanks, Santosh --------------8C49C8F656D7D6F2B611B004 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------8C49C8F656D7D6F2B611B004-- From owner-ips@ece.cmu.edu Tue Jan 9 23:24:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA27122 for ; Tue, 9 Jan 2001 23:24:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A3IQv28295 for ips-outgoing; Tue, 9 Jan 2001 22:18:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A3HoU28287 for ; Tue, 9 Jan 2001 22:17:51 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id EB55A4DD for ; Tue, 9 Jan 2001 19:17:49 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id TAA08680; Tue, 9 Jan 2001 19:17:45 -0800 (PST) Message-ID: <3A5BD4BB.6CE409A4@cup.hp.com> Date: Tue, 09 Jan 2001 19:19:23 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Login Response PDU Issues. References: <3A592705.B12F6A15@cup.hp.com> <3A5B9ED8.55CB9786@agilent.com> Content-Type: multipart/mixed; boundary="------------B6B6FCA8418F2479642671E4" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------B6B6FCA8418F2479642671E4 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Matt Wakeley wrote: > Santosh Rao wrote: > > > > Julian, > > > > Some concerns regarding the Login Response PDU section of the latest > > iSCSI draft : > > > > Section 2.11.3. Login Response Status > > ====================================== > > 1) Should targets use a REJECT PDU or Login Response PDU with a status > > of "reject login" to indicate Login Reject ? Section 2.11.3 and 4.4 > > contradict each other on this subject. > > > > A standard REJECT PDU used to reject ALL non-scsi PDUs > > [and ONLY non-scsi PDUs] would be desirable and would provide > > the following benefits : > > There already is one. Check out section 2.19 "Reject". The above comments are questioning whether to use REJECT PDU (as defined in section 2.19) or to use the "reject login" Login response PDU. It seems like there's redundant solutions being provided here. The "Status" field in the Login Response should be removed since a login reject should be conveyed through the REJECT PDU. Additionally, the above also questions the use of REJECT PDU for scsi PDUs since their reject could do with more specific error information conveying the reason for the failure. REJECT PDU would be desirable for use with non-scsi PDUs along the lines of a standard LS_RJT for all ELS in fibre channel. Thanks, Santosh --------------B6B6FCA8418F2479642671E4 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------B6B6FCA8418F2479642671E4-- From owner-ips@ece.cmu.edu Tue Jan 9 23:28:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA27147 for ; Tue, 9 Jan 2001 23:28:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A2JMh27072 for ips-outgoing; Tue, 9 Jan 2001 21:19:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A2J2U27066 for ; Tue, 9 Jan 2001 21:19:02 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id DA5101FC for ; Tue, 9 Jan 2001 18:19:01 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA05129; Tue, 9 Jan 2001 18:18:52 -0800 (PST) Message-ID: <3A5BC6EE.26DCFF5E@cup.hp.com> Date: Tue, 09 Jan 2001 18:20:30 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Task Mgmt Response - "No Task Found". Content-Type: multipart/mixed; boundary="------------1DC6E4FD29CB3764B397D219" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------1DC6E4FD29CB3764B397D219 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, Some further comments on this issue below. > Section 2.6. SCSI Task Management Response > ================================== > The referenced Task Tag field can be removed since the > "No Task Found" response error is not valid. > (A Target should respond with an accept for an Abort Task > request specifying an invalid Initator Task Tag). > [js]Different people have different opinions on this - even with HP - > talk > to Pierre! [/js] I did talk with Pierre and others as well. We are in consensus on this issue and the issues on this section are : a) The "No Task Found" SHOULD be removed. When a Target receives an Abort Task for a task not in its task set, the target SHOULD return "Function Complete". Not doing so is violation of SAM-2 Rev 15 Section 6.1 which reads : "If the LUN supports this function, a resonse of FUNCTION COMPLETE shall indicate that the task was aborted or was not in the task set." (not in task set to me implies the No Task Found case that is being listed seperately.). Also note that SAM-2 specifies only one of the following return values for the task mgmt response : i) Function Complete ii) Function Rejected iii) Service Delivery or target Failure. The (iii) value is missing in Section 2.6 and the "No Task Found" is not required. b) Since the "No Task Found" can be removed, the corresponding "Referenced Tag Field" can also be removed, since its description states "I.T.T. of the task not found" which is no longer a valid return code. Thanks, Santosh --------------1DC6E4FD29CB3764B397D219 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------1DC6E4FD29CB3764B397D219-- From owner-ips@ece.cmu.edu Tue Jan 9 23:28:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA27158 for ; Tue, 9 Jan 2001 23:28:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A28MQ26853 for ips-outgoing; Tue, 9 Jan 2001 21:08:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gordan.pl.gadzoox.com (i248.gadzoox.com [216.52.31.248]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A27iU26842 for ; Tue, 9 Jan 2001 21:07:44 -0500 (EST) Received: by gordan.pl.gadzoox.com with Internet Mail Service (5.5.2650.21) id ; Tue, 9 Jan 2001 18:08:24 -0800 Message-ID: <312419998E3CD211A52900A0C991A47A263156@gordan.pl.gadzoox.com> From: Vi Chau To: "'Murali Rajagopal'" , ips@ece.cmu.edu Subject: RE: Agree or Disagree to Merge FCIP and iFCP Date: Tue, 9 Jan 2001 18:08:21 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I diagree to merge FCIP and iFCP protocols. Vi Chau Gadzoox Networks, Inc. 16241 Laguna Canyon Road, Suite 100 Irvine, CA 92618-3611 949-789-4639 From owner-ips@ece.cmu.edu Tue Jan 9 23:28:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA27169 for ; Tue, 9 Jan 2001 23:28:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A2IRh27053 for ips-outgoing; Tue, 9 Jan 2001 21:18:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gordan.pl.gadzoox.com (i248.gadzoox.com [216.52.31.248]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A2I4U27046 for ; Tue, 9 Jan 2001 21:18:04 -0500 (EST) Received: by gordan.pl.gadzoox.com with Internet Mail Service (5.5.2650.21) id ; Tue, 9 Jan 2001 18:18:44 -0800 Message-ID: <312419998E3CD211A52900A0C991A47A263157@gordan.pl.gadzoox.com> From: Vi Chau To: "'Victor Firoiu'" , IPS Reflector Cc: Franco Travostino Subject: RE: Performance of iSCSI, FCIP and iFCP Date: Tue, 9 Jan 2001 18:18:42 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi, There is nothing in the FCIP I-D that limits the tunnel between the SAN islands to one TCP connection. The FCIP gateways can open as many TCP connections as it can support to reach any number of destination gateways. Vi Chau Gadzoox Networks, Inc. 16241 Laguna Canyon Road, Suite 100 Irvine, CA 92618-3611 949-789-4639 > -----Original Message----- > From: Victor Firoiu [mailto:vfiroiu@nortelnetworks.com] > Sent: Tuesday, January 09, 2001 4:50 PM > To: IPS Reflector > Cc: Franco Travostino > Subject: Performance of iSCSI, FCIP and iFCP > > > > Hi, > > we would like to add to the recent discussion on IP storage > protocols a > performance point of view showing significant differences between the > protocols discussed. Any comment would be highly appreciated. > > Briefly, transporting a set of storage connections between two storage > enclaves using multiple TCP sessions (iSCSI and iFCP) provides > significantly higher aggregate average throughput than transporting > the same set of storage connections over a single TCP session (FCIP), > the difference being proportional to the square of the number of TCP > sessions. This is due to the way TCP congestion control reacts to > packet losses. > > Here we have two questions: why are we talking about packet losses and > why is this difference in throughput. > > There are several reasons for packet losses: network congestion, > link errors and network errors. > > Network congestion is pervasive in current IP networks, where the only > way to control congestion is through dropping packets. Traffic > engineering, admission control and bandwidth reservation are currently > in early stages of definition. DiffServ supporting QoS infrastructure > will not be widely available in the near future (especially when we > realize that it is not a simple matter of asserting the EF PHB bit as > if it were the old IP ToS; it rather needs network services and > supporting SLA negotiation). The DiffServ EF PHB RFC is currently > under redefinition. On the other hand, if such supporting QoS > infrastructure were indeed available and pervasive, today, why would > we need TCP to begin with? > > Even in a perfectly engineered network, link errors occur. If we take > the Fibre Channel objective of 10^-12 Bit Error Rate, for a 10Gb/s > link, this is one error every 100 seconds. > > Network errors also occur with significant frequency in IP networks. > Jonathan Stone and Craig Partridge recently reported in Sigcomm 2000 > that network errors caught by TCP checksum occur with significant > frequency (between one packet in 1100 and 1 in 32000) and without link > CRC catching it. > > For the second question, TCP throughput is impacted by each packet > loss. Following TCP's congestion control algorithm existent in all > major implementations (Tahoe, Reno, New-Reno, SACK) each packet loss > results in the TCP sender's congestion window being reduced to half of > its current value, and therefore (assuming constant Round Trip Time), > TCP's throughput is halved. After that, the window increases by > roughly one packet every two Round Trip Times (assuming the popular > Delayed-Acknowledgement algorithm). > > The temporary decrease in TCP's rate translates into an amount of data > missing transmission opportunity. As we show later, for N storage > connections sharing an IP "pipe" of rate E, the amount of data missing > the opportunity to be transmitted due to a packet loss is > D(N) = E^2/(N^2)*RTT^2/(256*M) > in the case of iSCSI and iFCP, and > D(1) = E^2*RTT^2/(256*M) = D(N)*N^2 > in the case of FCIP > where > RTT = Round Trip Time > M = packet size > > For example, for a set of N=100 connections totaling E=10Gb/s, > RTT=10ms, M=1500B, the data not transmitted in time due to a packet > loss for iSCSI or iFCP is D(N)=2.6MB. For the same set transported > over one TCP session as in FCIP, the data not sent in time is > D(1)= 26GB, a 10,000 fold increase. > > The time interval for TCP to recover its sending rate to its initial > value after a packet loss is I(N)= 0.833seconds in the case of iSCSI > and iFCP, and I(1)=83.3seconds in the case of FCIP. > > Observe that in the case of FCIP, the time to recover its rate, > I(1)=83.3s, is of the same order of magnitude as the time between two > packet losses due exclusively to the link Bit Error Rate. In other > words, a packet losse occurs almost immediately after TCP has > recovered its rate. This means that FCIP delivers on average about > 3/4 of the required 10Gb/s rate, since 1/4 of rate is lost during the > time TCP rate increases linearly from 1/2 to full rate. (More > precisely, the effective rate is 8.27Gb/s because 1/4 of rate is lost > during 83.3s, and the time between two errors is now 120.825s due to > decreased sending rate). By comparison, iSCSI or iFCP deliver > approximately 9.99979Gb/s (i.e., lost 1/4 of one TCP full rate of > 100Mb/s during 0.833s out of a 100s interval). > > In conclusion, from a performance point of view, transporting > storage connections (SCSI or Fibre Channel) on multiple TCP sessions > is much more effective than tunneling through a single TCP > session, and the difference is proportional to the square of the > number of TCP sessions. > > > The math. > > For a TCP session to sustain a rate of C bits/second, the TCP's > congestion window W (measured in number packets) has to be > W=RTT*C/(8*M) > where > RTT = Round Trip Time in seconds > M = packet size in Bytes > > The time needed by the TCP sender to recover from a single packet > loss and have its sending rate reach the previous C value is > I = 2*RTT*W/2 = RTT*W = RTT^2*C/(8*M) > > The total amount of data (in Bytes) missing the opportunity to be > transmitted in this time interval I is > D = C/8*I/4 = C^2*RTT^2/(256*M) > > If we consider a set of N storage connections sharing an IP "pipe" of > rate E, they can be transported in N TCP sessions, as in iSCSI or > iFCP. Assuming all connections equal, each TCP session sends at a > rate of E/N. One packet loss impacts only one TCP session, and thus, > the total amount of data missing the opportunity to be transmitted due > to a packet loss is D(N) = E^2/(N^2)*RTT^2/(256*M) > > On the other hand, if the same set of N storage connections is > transported in one TCP session, as in FCIP, the total amount of data > losing the opportunity to be transmitted due to a packet loss is > D(1) = E^2*RTT^2/(256*M) = D(N)*N^2. > > For more details on TCP performance see for example: > "Modeling TCP Reno Performance: A Simple Model and its Empirical > Validation." J. Padhye, V. Firoiu, D. Towsley and J. Kurose, IEEE/ACM > Transactions on Networking, April 2000. > > > Franco Travostino, Victor Firoiu > ----- > Content Internetworking Lab, Technology Center > Nortel Networks, Inc. > 600 Technology Park > Billerica, MA 01821 USA > From owner-ips@ece.cmu.edu Tue Jan 9 23:28:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA27180 for ; Tue, 9 Jan 2001 23:28:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A2rNB27767 for ips-outgoing; Tue, 9 Jan 2001 21:53:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A2qbU27754 for ; Tue, 9 Jan 2001 21:52:38 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 178C9AF4 for ; Tue, 9 Jan 2001 18:52:37 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA06955; Tue, 9 Jan 2001 18:52:32 -0800 (PST) Message-ID: <3A5BCED2.5CEF70D1@cup.hp.com> Date: Tue, 09 Jan 2001 18:54:10 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : LOGO originated by target. Content-Type: multipart/mixed; boundary="------------5B5DDAD4ADAC10FD6E6B78A3" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------5B5DDAD4ADAC10FD6E6B78A3 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit > Section 2.17 Asynchronous Event > ========================= > 1) "Target requests Logout on this connection" iSCSI event > indicator. Why not just allow the target to originate a Logout > as is done in Fibre Channel ? > Besides, this AEN does not meet the requirement. > How does the target convey the CID of the connection to be > logged out ? > [js] CID an timeout appear now in parameters [/js] The simplest approach and the most expeditious error recovery is to allow the target to originate a Logout. To send an AEN and then wait for the initiator to log out is opening up the target to dependencies on the initiator honoring and acting on that AEN, which is un-necessary. This should not be a violation of SAM since SAM does not restrict unsolicited traffic for the interconnect specific commands. (Fibre Chanel does this.) Thanks, Santosh > I do not think SAM-2 lays any guidelines for how the SCSI > interconnect protocol handle its native PDUs. (i.e. non-scsi > operations). > Fibre Channel allows unsolicited inbound traffic for ELS and > LS commands and allowing Logout and NOP-OUT > erhaps better thought of as a NOP command) commands to > be originated by targets would be desirable and would get rid of this > particular AEN iSCSI event indicator. --------------5B5DDAD4ADAC10FD6E6B78A3 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------5B5DDAD4ADAC10FD6E6B78A3-- From owner-ips@ece.cmu.edu Wed Jan 10 00:34:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA28070 for ; Wed, 10 Jan 2001 00:34:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A3QQd28474 for ips-outgoing; Tue, 9 Jan 2001 22:26:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A3Q6U28467 for ; Tue, 9 Jan 2001 22:26:06 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id E05D435F for ; Tue, 9 Jan 2001 19:26:05 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id TAA09099; Tue, 9 Jan 2001 19:26:01 -0800 (PST) Message-ID: <3A5BD6AB.FB7F5590@cup.hp.com> Date: Tue, 09 Jan 2001 19:27:39 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : On the subject of R2T and Task Tags. References: <200101090846.OAA29278@divyaroot.India.Sun.COM> <3A5B95B6.29771A0B@cup.hp.com> <3A5BA1FA.652B86CD@agilent.com> Content-Type: multipart/mixed; boundary="------------F37041FECA55FD7B25379839" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------F37041FECA55FD7B25379839 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Matt Wakeley wrote: > The task tags in iSCSI should work the same as they do in FCP: > > The target supplies a target task tag per task that it sends to the initiator > where a response from the initiator is expected. The target uses the target > task tag to associate any solicited received messages from the initiator > (other than the initial command) to that task. > > For SCSI WRITE commands, the target MAY assign a target task tag to the I/O > that it uses when sending R2Ts to the initiator. The initiator must then echo > that target task tag in replies it sends to the target. I strongly object to > the words in section 2.16 "All outstanding R2T should have different Target > Transfer Tags". This specification should not dictate to the target how it > chooses or uses the Target Task Tags. > I agree with Matt on the semantics of "Target Task Tag" for WRITE PDUs and would also like to request a "Target Task Tag" per task for WRITE PDUs. In addition, since multiple R2Ts can be outstanding, an additional qualifier is required to track the R2T (Target R2T Tag) along the lines of a (RX_ID + SEQ_ID) that Fibre Channel uses. Thanks, Santosh --------------F37041FECA55FD7B25379839 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------F37041FECA55FD7B25379839-- From owner-ips@ece.cmu.edu Wed Jan 10 03:56:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA12322 for ; Wed, 10 Jan 2001 03:56:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A6hVK02279 for ips-outgoing; Wed, 10 Jan 2001 01:43:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A6hCU02269 for ; Wed, 10 Jan 2001 01:43:13 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id D6B541B41 for ; Tue, 9 Jan 2001 22:43:11 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id WAA11403; Tue, 9 Jan 2001 22:43:10 -0800 (PST) Message-ID: <3A5C0430.593CACD4@hp.com> Date: Tue, 09 Jan 2001 22:41:52 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI : On the subject of R2T and Task Tags. References: <200101090846.OAA29278@divyaroot.India.Sun.COM> <3A5B95B6.29771A0B@cup.hp.com> <3A5BA1FA.652B86CD@agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt Wakeley wrote: > The task tags in iSCSI should work the same as they do in FCP: > > The initiator supplies a initiator task tag per task that it uses to associate > any received messages from the target to that task. [a "undefined" initiator > task tag should have a value of 0xffffffff since a value of zero is a valid > number for a table look up ]. > > The target supplies a target task tag per task that it sends to the initiator > where a response from the initiator is expected. The target uses the target > task tag to associate any solicited received messages from the initiator > (other than the initial command) to that task. > > For SCSI READ commands, there is no need to specifiy a target task tag. > Typically in FCP today, the targets to NOT assign target task tags (RX_IDs in > FC terms) for READs. > > For SCSI WRITE commands, the target MAY assign a target task tag to the I/O > that it uses when sending R2Ts to the initiator. The initiator must then echo > that target task tag in replies it sends to the target. I strongly object to > the words in section 2.16 "All outstanding R2T should have different Target > Transfer Tags". This specification should not dictate to the target how it > chooses or uses the Target Task Tags. Matt, I agree that the draft must not impose to the target to have a different tag for each R2T (it is up to the target implementation to decide). I disagree to impose a unique target tag for the whole IO. It imposes the target to do score boarding to know when all the data have been received. Letting the possiblility to the target to have one tag per R2T (or add a subtag identifying the R2T as proposed Santosh) allows the target to dispense with the score boarding. The target knows that all the data has been received when a data PDU with the "F" bit set has been received for each R2T. Regards, Pierre > > > -Matt Wakeley > Agilent Technologies From owner-ips@ece.cmu.edu Wed Jan 10 03:58:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA12338 for ; Wed, 10 Jan 2001 03:58:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A7xWt03650 for ips-outgoing; Wed, 10 Jan 2001 02:59:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A7xTU03645 for ; Wed, 10 Jan 2001 02:59:29 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id 91F8DF17 for ; Tue, 9 Jan 2001 23:59:28 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id XAA11976; Tue, 9 Jan 2001 23:59:27 -0800 (PST) Message-ID: <3A5C1611.2BAD0302@hp.com> Date: Tue, 09 Jan 2001 23:58:09 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: target resource "leak"/logout command Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit sip - Santosh Rao wrote: > +----------------------------------------------------------+ > | This is a broadcast message. Any reply to this | > | message will broadcast to the entire distribution. | > | To reply only to the individual submitter, send mail | > | directly to < mailto:santoshr@cup.hp.com > > +----------------------------------------------------------+ > > This is a multi-part message in MIME format. > > ------------------------------------------------------------------------ > Pierre, > > Some comments embedded below. Perhaps, Doug or some of the other target > folks can add more comments here. > > Thanks, > Santosh > > Pierre Labat wrote: > > > Scenario > > ======= > > Session with 2 TCP connections 1 and 2 > > > > 1) A command completion (StatRN=10) is received by > > the initiator (over cx 1) for the command 5. > > > > 2) The initiator sends back ExpStatRN=11 with a command over TCP cx 1 > > > > 3) The TCP cx 1 drops and the command 5 (so ExpStatRN=11) will never > > make it to the target. > > A connection drop is followed by the initiator migrating its outstanding > commands to a new connection and issuing a Logout on the CID of the failed > connection. The logout will cause the target to clean up all its resources. The logout doesn't cause the target to cleanup all its resources. The target MAY keep the resources corresponding to the task pending on the failed connection. If we don't assume that we can throw away the "retry" strategy. The logout indicates to the target that it has to throw away all that is incoming on the failed connection and that the command pending will be retried (or aborted) from another connection. > > > In any cases, no self respecting target will [and can afford to] keep its > I/O buffers and SEST around forever after sending the Status. They will run > out of SEST type of resources at the target end. The typical target may > start a timer once it sends the SCSI Status PDU for a SCSI command and on > timeout would clean up the SEST entry and release the I/O buffers for that > timed out I/O which did not get an updated ExpStatRN. I know they are such timeouts, but it is better to free up the resources sooner on the target by designing a robust protocol. Timers have to be the "last resort". > > > Further, on such a timer pop, the target may choose to use a NOP-OUT with > the "Ping" bit set to test the connection. > > However, this starts a new issue here. Once again, the philosophy of StatRN > here seems to be to optimize the recovery paths and affect the performance > paths !! The only goal of StatRN is to fasten a recovery using "retry". > > > How does the draft cover the situation when an initator has no further > outbound traffic on whch it can send an update to ExpStatRN ? If there was > a burst of traffic from one initiator followed by a period of quiescence, > it could end up not sending ExpStatRN update to the target and this would > eat up resources on the target, affecting other initiators that may still > be talking to it. The target sends a NOP-in with the P bit set. Regards, Pierre From owner-ips@ece.cmu.edu Wed Jan 10 03:59:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA12366 for ; Wed, 10 Jan 2001 03:59:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A7Mdh02971 for ips-outgoing; Wed, 10 Jan 2001 02:22:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A7LWU02960 for ; Wed, 10 Jan 2001 02:21:32 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id 688921363 for ; Tue, 9 Jan 2001 23:21:30 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id XAA11666; Tue, 9 Jan 2001 23:21:28 -0800 (PST) Message-ID: <3A5C0D2B.68B20523@hp.com> Date: Tue, 09 Jan 2001 23:20:11 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI : Login Response PDU Issues. References: <3A592705.B12F6A15@cup.hp.com> <3A5B9DDE.1378F557@agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt Wakeley wrote: > Santosh Rao wrote: > > > 3) Quoting from the draft : > > > > "In the case that the Status is reject login the initiator should > > immediately > > close down its end of the TCP connection, thus freeing up the target's > > port for some other connection." > > > > On a REJECT, initiators may wish to retry Login. [perhaps, with a > > modification to their Login Command PDU or TEXT parameters . > > > > Perhaps, the word "may" might be a better fit instead of "should" here. > > What is so bad with always closing a TCP connection when a login is rejected > and opening a new TCP connection when issuing a login? Makes it simple. Only > one way to do things. Only one path to test. As it is not on the performance path i prefer to keep it simple too. If the target doesn't close the connection after rejecting the login when will it do it? It must add some code to manage such kind of things, better to avoid. Regards, Pierre > > > -Matt Wakeley > Agilent Technologies > > > > > Thanks, > > Santosh From owner-ips@ece.cmu.edu Wed Jan 10 06:55:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA13666 for ; Wed, 10 Jan 2001 06:55:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0A9VZq14980 for ips-outgoing; Wed, 10 Jan 2001 04:31:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gate.internaut.com ([64.38.134.108]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A9UjU14968 for ; Wed, 10 Jan 2001 04:30:45 -0500 (EST) Received: from e1kj2 (kidneybean [64.38.134.109]) by gate.internaut.com (8.10.2/8.10.2) with SMTP id f0A9RWR12931; Wed, 10 Jan 2001 01:27:33 -0800 From: "Bernard Aboba" To: , Subject: RE: iSCSI and the IPSEC replay window Date: Wed, 10 Jan 2001 01:30:43 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <0F31E5C394DAD311B60C00E029101A07041013C7@corpmx9.isus.emc.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit >Hmm ... I would have thought that the separate >TCP connections for the alternate paths >would use separate IPsec SAs and hence would >not share a replay window I was talking about a single TCP connection over a single IPSEC SA. By "alternate paths" I meant a routing change that might result in re-ordering. From owner-ips@ece.cmu.edu Wed Jan 10 13:25:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA23413 for ; Wed, 10 Jan 2001 13:24:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0AFXZJ24472 for ips-outgoing; Wed, 10 Jan 2001 10:33:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A4DFU29333 for ; Tue, 9 Jan 2001 23:13:15 -0500 (EST) Received: from divyaroot.India.Sun.COM ([129.158.226.35]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id UAA12313 for ; Tue, 9 Jan 2001 20:13:13 -0800 (PST) Received: from helix (helix [129.158.226.51]) by divyaroot.India.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.0) with SMTP id JAA13095 for ; Wed, 10 Jan 2001 09:43:11 +0530 (IST) Message-Id: <200101100413.JAA13095@divyaroot.India.Sun.COM> Date: Wed, 10 Jan 2001 09:45:02 -0500 (GMT) From: Raghavendra Rao Reply-To: Raghavendra Rao Subject: Re: iSCSI : On the subject of R2T and Task Tags. To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: 0Kn6hoOZEtA3hrQs8D4rMw== Sender: owner-ips@ece.cmu.edu Precedence: bulk > The "Target Transfer Tag" that you are referring to is per R2T and not per task. > There is no per-task identifier that can be used by the targets to track the > task. They will either have to : I don't think per R2T target tag makes sense. It would simplify if target maintains one-to-one mapping of target and initiator task tags. So, until the task is terminated, the target should simply post the same target task identifier in every R2T it transmits corresponding to an initiator task tag. -JP From owner-ips@ece.cmu.edu Wed Jan 10 13:28:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA23511 for ; Wed, 10 Jan 2001 13:28:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0AFXXA24469 for ips-outgoing; Wed, 10 Jan 2001 10:33:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0A97aU04741 for ; Wed, 10 Jan 2001 04:07:37 -0500 (EST) Received: from divyaroot.India.Sun.COM ([129.158.225.35]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id BAA17808 for ; Wed, 10 Jan 2001 01:07:34 -0800 (PST) Received: from helix (helix [129.158.226.51]) by divyaroot.India.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.0) with SMTP id OAA04633 for ; Wed, 10 Jan 2001 14:37:32 +0530 (IST) Message-Id: <200101100907.OAA04633@divyaroot.India.Sun.COM> Date: Wed, 10 Jan 2001 14:39:23 -0500 (GMT) From: Raghavendra Rao Reply-To: Raghavendra Rao Subject: Re: iSCSI : On the subject of R2T and Task Tags. To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: 0hMb33AMEX5UP4679WxBQg== Sender: owner-ips@ece.cmu.edu Precedence: bulk > > It imposes the target to do score boarding to know when all the data have > been received. Letting the possiblility to the > I'm missing something here - When the 'F' bit is per R2T, why would one need to spend additional effort to know whether the all the data has arrived for the R2T ? The 'F' bit helps in finding out if all the data for the R2T arrived or not. The length + buffer offset help in finding out if all the data for the task has arrived or not. -JP From owner-ips@ece.cmu.edu Wed Jan 10 13:28:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA23523 for ; Wed, 10 Jan 2001 13:28:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0AFZ6f24513 for ips-outgoing; Wed, 10 Jan 2001 10:35:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0ADJrU18587 for ; Wed, 10 Jan 2001 08:19:53 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA15719; Wed, 10 Jan 2001 08:19:51 -0500 (EST) Message-Id: <200101101319.IAA15719@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-isns-00.txt Date: Wed, 10 Jan 2001 08:19:51 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iSNS Internet Storage Name Service Author(s) : K. Gibbons et al. Filename : draft-ietf-ips-isns-00.txt Pages : 43 Date : 09-Jan-01 The Internet Storage Name Service (iSNS) provides a generic framework for configuring and managing various storage entities and their usage attributes in an IP-based storage network. This framework includes the iSNS Protocol (iSNSP), which defines how entities communicate with the iSNS server to discover and utilize available networked storage resources. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-isns-00.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-isns-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-isns-00.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20010109124241.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-isns-00.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-isns-00.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20010109124241.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Wed Jan 10 14:24:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA25041 for ; Wed, 10 Jan 2001 14:24:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0AGIab25873 for ips-outgoing; Wed, 10 Jan 2001 11:18:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0AG3Vk25370 for ; Wed, 10 Jan 2001 11:03:31 -0500 (EST) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 4BA1F22B5; Wed, 10 Jan 2001 08:03:30 -0800 (PST) Received: from mk731913.cup.hp.com (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id IAA04860; Wed, 10 Jan 2001 08:05:40 -0800 (PST) Message-Id: <5.0.0.25.2.20010110075802.024dae70@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.0 Date: Wed, 10 Jan 2001 08:03:02 -0800 To: "Victor Firoiu" From: Michael Krause Subject: Re: Performance of iSCSI, FCIP and iFCP Cc: IPS Reflector , "Franco Travostino" In-Reply-To: <3A5BB1BF.27CEED03@nortelnetworks.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 07:50 PM 1/9/2001 -0500, Victor Firoiu wrote: >In conclusion, from a performance point of view, transporting >storage connections (SCSI or Fibre Channel) on multiple TCP sessions >is much more effective than tunneling through a single TCP >session, and the difference is proportional to the square of the >number of TCP sessions. Hence why many of us want to support multiple TCP connections per iSCSI session (even if this over a single physical port). Some of us don't believe a single connection is the right solution to achieve maximum throughput and focusing in on a strictly single connection objective especially if the fabric diameter is large (as some have suggested) is the wrong paradigm to pursue. Note: For hardware or software implementations, the cost of multiple connections is rather trivial to support in terms of resources and there are advantages w.r.t. QoS and other policies (e.g. multi-pathing through a fabric) that one can take advantage of with multiple connections. Nice write up. Thanks, Mike From owner-ips@ece.cmu.edu Wed Jan 10 18:02:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA02379 for ; Wed, 10 Jan 2001 18:02:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0AKmsv03844 for ips-outgoing; Wed, 10 Jan 2001 15:48:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0AKiYk03714 for ; Wed, 10 Jan 2001 15:44:35 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 207B0747 for ; Wed, 10 Jan 2001 13:44:29 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id D6EC2216 for ; Wed, 10 Jan 2001 15:41:45 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id MAA22394 for ; Wed, 10 Jan 2001 12:41:43 -0800 (PST) Message-ID: <3A5CC8FB.A0E954B9@agilent.com> Date: Wed, 10 Jan 2001 12:41:31 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 Cc: IPS Reflector Subject: Re: iSCSI : DataRN issues. References: <3A5BCD83.17F74969@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I originally made this request on 11/7/00 and therefore agree that DataRN should be removed. I especially take exception to the statement on page 87, refering to "18 DataNumber" that "An initiator MUST support data numbering if requested". An initiator should not be required to support a feature that might help recover an error faster. -Matt Wakeley Agilent Technologies Santosh Rao wrote: > > Hi Julian, > > Some further comments on this. > > Thanks, > Santosh > > julian_satran@il.ibm.com wrote: > > > Section 1.2.2.3. Data PDU Numbering > > ============================ > > What is the scope of DataRNs ? Is this maintained per task or per > > connection or per session ? This should be explicitly stated in this > > section as is done in earlier sections for CmdRN and StatRN. > > [js] per command and it says so [/js] > > I believe that the DataRN feature and all its related overheads add a level of > complexity that is going to discourage both targets and initiators from using > this. I am yet to see anyone state sufficient benefits of this feature that > can justify this complexity. Moreover, the benefit it may provide [if any, and > if somebody does use it] is in the error path context and its overheads hurt > the performance path. > > Therefore, in the interests of keeping iSCSI simple, I would like to request > removal of all references to DataRN and DataNumber. > > Thanks, > Santosh From owner-ips@ece.cmu.edu Wed Jan 10 18:02:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA02408 for ; Wed, 10 Jan 2001 18:02:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0AKMFj03053 for ips-outgoing; Wed, 10 Jan 2001 15:22:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcars04f.ca.nortel.com (h57s242a129n47.user.nortelnetworks.com [47.129.242.57]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0AJtLk02306 for ; Wed, 10 Jan 2001 14:55:21 -0500 (EST) Received: from zcard00m.ca.nortel.com by zcars04f.ca.nortel.com; Wed, 10 Jan 2001 14:50:40 -0500 Received: from zbl6c002.corpeast.baynetworks.com ([132.245.205.52]) by zcard00m.ca.nortel.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id CNPSYYBH; Wed, 10 Jan 2001 14:47:01 -0500 Received: from nortelnetworks.com (dhcp223-140.engeast.baynetworks.com [192.32.223.140]) by zbl6c002.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id CN99DBPJ; Wed, 10 Jan 2001 14:47:01 -0500 Message-ID: <3A5CBC58.756953D2@nortelnetworks.com> Date: Wed, 10 Jan 2001 14:47:36 -0500 X-Sybari-Space: 00000000 00000000 00000000 From: "Victor Firoiu" Organization: Nortel Networks X-Mailer: Mozilla 4.76 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Vi Chau CC: IPS Reflector Subject: Re: Performance of iSCSI, FCIP and iFCP References: <312419998E3CD211A52900A0C991A47A263157@gordan.pl.gadzoox.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Orig: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Vi, thanks for your comments. Indeed, there is no such limitation in the FCIP ID. But there is no consideration for multiple TCP connections either. What specifically governs the use of TCP flow A as opposed to TCP flow B when two FCIP gateways are communicating? As we've seen in our examples, the performance is significantly (quadratically) impacted by the choice of number of TCP connections. Fine-grain control over multiple TCP connections thus becomes mandatory between any two FC islands of non-trivial size. The most natural way to multiplex storage connections (FC or SCSI) on multiple TCP connections is to have one or a few TCP connections per storage connection, that is, to use storage connection awareness for assigning TCP connections to storage connections (iSCSI, iFCP). Surely, it is possible to multiplex storage connections into TCP connections without any knowledge of the identity of storage connections. But this brings up issues including load balancing, interdependency between storage connections sharing TCP connections, managing storage connections that are striped across TCP connections (as thoroughly discussed within the iSCSI community). It seems that the iSCSI and iFCP solution of assigning one (or a few) TCP connection(s) for each storage connections has many practical and performance merits that needs a special attention in FCIP also. Again, any comments are very welcome. Victor Firoiu ----- Content Internetworking Lab, Technology Center Nortel Networks, Inc. 600 Technology ParkBillerica, MA 01821 USA Vi Chau wrote: > > Hi, > > There is nothing in the FCIP I-D that limits the > tunnel between the SAN islands to one TCP connection. The > FCIP gateways can open as many TCP connections as it can > support to reach any number of destination gateways. > > Vi Chau > Gadzoox Networks, Inc. > 16241 Laguna Canyon Road, Suite 100 > Irvine, CA 92618-3611 > 949-789-4639 > From owner-ips@ece.cmu.edu Wed Jan 10 18:06:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA02744 for ; Wed, 10 Jan 2001 18:06:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0AL3qG04337 for ips-outgoing; Wed, 10 Jan 2001 16:03:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0AKodk03913 for ; Wed, 10 Jan 2001 15:50:39 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 4CBD9B5 for ; Wed, 10 Jan 2001 13:50:31 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 29746143 for ; Wed, 10 Jan 2001 15:50:29 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id MAA22924 for ; Wed, 10 Jan 2001 12:50:27 -0800 (PST) Message-ID: <3A5CCB08.74D95ED9@agilent.com> Date: Wed, 10 Jan 2001 12:50:16 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : On the subject of R2T and Task Tags. References: <200101090846.OAA29278@divyaroot.India.Sun.COM> <3A5B95B6.29771A0B@cup.hp.com> <3A5BA1FA.652B86CD@agilent.com> <3A5BD6AB.FB7F5590@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > I agree with Matt on the semantics of "Target Task Tag" for WRITE PDUs and would > also like to request a "Target Task Tag" per task for WRITE PDUs. > > In addition, since multiple R2Ts can be outstanding, an additional qualifier is > required to track the R2T (Target R2T Tag) along the lines of a (RX_ID + SEQ_ID) > that Fibre Channel uses. I don't think an additional qualifier is needed. For what purpose? The target just sends out all the R2Ts, and counts all the bytes coming back (remember - TCP guarantees to deliver each byte only once!). Also remember, the iSCSI spec mandates that the initiator send only what was requested in the R2T... nothing more. -Matt Wakeley Agilent Technologies > > Thanks, > Santosh From owner-ips@ece.cmu.edu Wed Jan 10 18:56:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA04890 for ; Wed, 10 Jan 2001 18:56:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0AL3cU04319 for ips-outgoing; Wed, 10 Jan 2001 16:03:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0AKxOk04179 for ; Wed, 10 Jan 2001 15:59:24 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id B7F59389 for ; Wed, 10 Jan 2001 13:59:20 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id A333A135 for ; Wed, 10 Jan 2001 15:59:19 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id MAA23851 for ; Wed, 10 Jan 2001 12:59:18 -0800 (PST) Message-ID: <3A5CCD1A.647D096B@agilent.com> Date: Wed, 10 Jan 2001 12:59:06 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI: Logout for a Session Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Throughout the iSCSI document, it talks about the logout command closing a TCP connection. Especially for the purposes of error recover to terminate a TCP connection and "clean up" so that another TCP connection can be established to continue the session. There seems to be no way to indicate that a session is being terminated. There should be a flag in the logout command that indicates that the *session* is being terminated (as well as the TCP connection). -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Wed Jan 10 18:57:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA04913 for ; Wed, 10 Jan 2001 18:57:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0AM3NW06055 for ips-outgoing; Wed, 10 Jan 2001 17:03:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0ALoNk05677 for ; Wed, 10 Jan 2001 16:50:23 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id A6E48ED3; Wed, 10 Jan 2001 13:50:22 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id NAA15698; Wed, 10 Jan 2001 13:50:17 -0800 (PST) Message-ID: <3A5CD97E.62977962@cup.hp.com> Date: Wed, 10 Jan 2001 13:51:58 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Matt Wakeley Cc: IPS Reflector Subject: Re: iSCSI: Logout for a Session References: <3A5CCD1A.647D096B@agilent.com> Content-Type: multipart/mixed; boundary="------------1906CE993DD7ABAFFDCE6277" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------1906CE993DD7ABAFFDCE6277 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Matt Wakeley wrote: > Throughout the iSCSI document, it talks about the logout command closing a TCP > connection. Especially for the purposes of error recover to terminate a TCP > connection and "clean up" so that another TCP connection can be established to > continue the session. There seems to be no way to indicate that a session is > being terminated. > > There should be a flag in the logout command that indicates that the *session* > is being terminated (as well as the TCP connection). > There is. Look at the "Reason Code" field in the Logout Command PDU. (Section 2.14 of the draft dated dec 30, 2000). - Santosh Rao --------------1906CE993DD7ABAFFDCE6277 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------1906CE993DD7ABAFFDCE6277-- From owner-ips@ece.cmu.edu Wed Jan 10 18:58:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA04925 for ; Wed, 10 Jan 2001 18:58:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0AL3uN04343 for ips-outgoing; Wed, 10 Jan 2001 16:03:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0AKqdk03979 for ; Wed, 10 Jan 2001 15:52:40 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id EF464296 for ; Wed, 10 Jan 2001 13:52:35 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id CF7DC143 for ; Wed, 10 Jan 2001 15:52:34 -0500 (EST) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id MAA22988 for ; Wed, 10 Jan 2001 12:52:33 -0800 (PST) Message-ID: <3A5CCB85.5FDE7AF8@agilent.com> Date: Wed, 10 Jan 2001 12:52:21 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : On the subject of R2T and Task Tags. References: <200101090846.OAA29278@divyaroot.India.Sun.COM> <3A5B95B6.29771A0B@cup.hp.com> <3A5BA1FA.652B86CD@agilent.com> <3A5C0430.593CACD4@hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Pierre Labat wrote: > I agree that the draft must not impose to the target to have a different tag > for each R2T (it is up to the target implementation to decide). I disagree > to impose a unique target tag for the whole IO. It imposes the target to do > score boarding > to know when all the data have been received. Letting the possiblility to the > target > to have one tag per R2T (or add a subtag identifying the R2T as proposed Santosh) > > allows the target to dispense with the score boarding. The target knows that all > the > data has been received when a data PDU with the "F" bit set has been received > for each R2T. There is no need to do scoreboarding. Just count the number of bytes received. If they all add up, then you've received all the data. I for one would not rely on a PDU with the "F" bit set. I would want to cross check it with the correct number of bytes. > > Regards, > > Pierre > > > > > -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Wed Jan 10 20:33:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA08488 for ; Wed, 10 Jan 2001 20:33:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0AMiXX07219 for ips-outgoing; Wed, 10 Jan 2001 17:44:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0AMMck06605 for ; Wed, 10 Jan 2001 17:22:38 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Wed, 10 Jan 2001 14:22:32 -0800 Message-ID: From: Joshua Tseng To: ips@ece.cmu.edu Subject: RE: I-D ACTION:draft-ietf-ips-isns-00.txt Date: Wed, 10 Jan 2001 14:22:09 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Can we be provided with 20 minutes on the agenda for the Orlando meeting to discuss how iSNS benefits iSCSI and FCIP? Thanks! Josh Tseng > -----Original Message----- > From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org] > Sent: Wednesday, January 10, 2001 5:20 AM > Cc: ips@ece.cmu.edu > Subject: I-D ACTION:draft-ietf-ips-isns-00.txt > > > A New Internet-Draft is available from the on-line > Internet-Drafts directories. > This draft is a work item of the IP Storage Working Group of the IETF. > > Title : iSNS Internet Storage Name Service > Author(s) : K. Gibbons et al. > Filename : draft-ietf-ips-isns-00.txt > Pages : 43 > Date : 09-Jan-01 > > The Internet Storage Name Service (iSNS) provides a generic > framework for configuring and managing various storage entities and > their usage attributes in an IP-based storage network. This > framework includes the iSNS Protocol (iSNSP), which defines how > entities communicate with the iSNS server to discover and utilize > available networked storage resources. > > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-ietf-ips-isns-00.txt > > Internet-Drafts are also available by anonymous FTP. Login > with the username > "anonymous" and a password of your e-mail address. After logging in, > type "cd internet-drafts" and then > "get draft-ietf-ips-isns-00.txt". > > A list of Internet-Drafts directories can be found in > http://www.ietf.org/shadow.html > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > > > Internet-Drafts can also be obtained by e-mail. > > Send a message to: > mailserv@ietf.org. > In the body type: > "FILE /internet-drafts/draft-ietf-ips-isns-00.txt". > > NOTE: The mail server at ietf.org can return the document in > MIME-encoded form by using the "mpack" utility. To use this > feature, insert the command "ENCODING mime" before the "FILE" > command. To decode the response(s), you will need "munpack" or > a MIME-compliant mail reader. Different MIME-compliant > mail readers > exhibit different behavior, especially when dealing with > "multipart" MIME messages (i.e. documents which have been split > up into multiple messages), so check your local documentation on > how to manipulate these messages. > > > Below is the data which will enable a MIME compliant mail reader > implementation to automatically retrieve the ASCII version of the > Internet-Draft. > From owner-ips@ece.cmu.edu Wed Jan 10 20:36:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA08538 for ; Wed, 10 Jan 2001 20:36:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0AMME206575 for ips-outgoing; Wed, 10 Jan 2001 17:22:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thalia.fm.intel.com (thalia.fm.intel.com [132.233.247.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0AMGtk06429 for ; Wed, 10 Jan 2001 17:16:55 -0500 (EST) Received: from SMTP (fmsmsxvs02-1.fm.intel.com [132.233.42.202]) by thalia.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.33 2000/11/21 19:27:27 smothers Exp $) with SMTP id WAA11522; Wed, 10 Jan 2001 22:18:16 GMT Received: from fmsmsx18.intel.com ([132.233.48.18]) by 132.233.48.202 (Norton AntiVirus for Internet Email Gateways 1.0) ; Wed, 10 Jan 2001 22:16:53 0000 (GMT) Received: by fmsmsx18.fm.intel.com with Internet Mail Service (5.5.2650.21) id ; Wed, 10 Jan 2001 14:16:50 -0800 Message-ID: <31E3E2A9D8DDD311AC43009027C68052C8017B@orsmsx59.jf.intel.com> From: "Grun, Paul" To: "'Matt Wakeley'" , IPS Reflector Subject: RE: iSCSI: Logout for a Session Date: Wed, 10 Jan 2001 14:16:46 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="ISO-8859-1" X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id f0AMI2m06458 Sender: owner-ips@ece.cmu.edu Precedence: bulk X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id f0AMME306575 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ietf.org id UAA08538 Isn't a session terminated when the last TCP connection is closed? -paul Paul Grun                                              Intel Corporation Enterprise Platform Group Fabric Components Division (503) 677-6768 paul.grun@intel.com > -----Original Message----- > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > Sent: Wednesday, January 10, 2001 12:59 PM > To: IPS Reflector > Subject: iSCSI: Logout for a Session > > > Throughout the iSCSI document, it talks about the logout > command closing a TCP > connection. Especially for the purposes of error recover to > terminate a TCP > connection and "clean up" so that another TCP connection can > be established to > continue the session. There seems to be no way to indicate > that a session is > being terminated. > > There should be a flag in the logout command that indicates > that the *session* > is being terminated (as well as the TCP connection). > > -Matt Wakeley > Agilent Technologies > From owner-ips@ece.cmu.edu Wed Jan 10 20:36:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA08549 for ; Wed, 10 Jan 2001 20:36:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0AN3Sq07697 for ips-outgoing; Wed, 10 Jan 2001 18:03:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0AMp2k07403 for ; Wed, 10 Jan 2001 17:51:02 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 290BF11A0 for ; Wed, 10 Jan 2001 14:51:01 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id OAA22499; Wed, 10 Jan 2001 14:50:48 -0800 (PST) Message-ID: <3A5CE7AD.143B073A@cup.hp.com> Date: Wed, 10 Jan 2001 14:52:29 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Areas Of Concern. Content-Type: multipart/mixed; boundary="------------396F7479EF1A42723FF433D5" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------396F7479EF1A42723FF433D5 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian and all authors of the iSCSI draft, The following broad concerns have been expressed in several rounds of review feedback on the latest version of the iSCSI draft (dated dec 30, 2000) : 1) DataRNissues. (requests for its removal.) 2) Task Tag Management issues. (requests for a unique Target Task Tag per task for WRITE I/Os.) 3) Header Format Error handling issues. (concerns regarding the requirement that initiators to be SCSI aware and fake CHECK CONDITIONs to their upper layers on behalf of targets.) 4) Allowing LOGO to be originated by targets. 5) Reject PDU issues. (to be used for only non-scsi PDUs and all non-scsi PDUs. removal of redundant mechanisms to reject in login response PDU. ) Your position on these concerns and whether/how they are going to be addressed will be much appreciated. Thanks, Santosh Rao --------------396F7479EF1A42723FF433D5 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------396F7479EF1A42723FF433D5-- From owner-ips@ece.cmu.edu Wed Jan 10 20:39:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA08572 for ; Wed, 10 Jan 2001 20:39:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0AMMLU06586 for ips-outgoing; Wed, 10 Jan 2001 17:22:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0AMAYk06270 for ; Wed, 10 Jan 2001 17:10:34 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 60CAC1814; Wed, 10 Jan 2001 14:10:31 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id OAA17639; Wed, 10 Jan 2001 14:10:27 -0800 (PST) Message-ID: <3A5CDE37.C3A97810@cup.hp.com> Date: Wed, 10 Jan 2001 14:12:08 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Raghavendra Rao Cc: ips@ece.cmu.edu Subject: Re: iSCSI : On the subject of R2T and Task Tags. References: <200101100413.JAA13095@divyaroot.India.Sun.COM> Content-Type: multipart/mixed; boundary="------------BC796B4938A3D88858997728" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------BC796B4938A3D88858997728 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit When multiple R2Ts are outstanding, the targets need a tag per R2T to track their SCSI State Table Entries for each R2T. While its usage depends on how each individual programming model implements this feature, I believe the header should provide support to track each R2T using an id that the target can lookup for its R2T data buffer pointers. Not having this id will cause a more expensive lookup that needs to be based on the Buffer Offset that the WRITE Data PDUs will carry. Implementations can choose to not use (0xFFFF) or use a single value for the Target R2T Tag if they do not need this feature or specify MaxOutstandingR2T as 1. - Santosh Rao Raghavendra Rao wrote: > > The "Target Transfer Tag" that you are referring to is per R2T and not per > task. > > There is no per-task identifier that can be used by the targets to track the > > task. They will either have to : > > I don't think per R2T target tag makes sense. > > It would simplify if target maintains one-to-one mapping of target and initiator > task tags. So, until the task is terminated, the target should simply post the > same target task identifier in every R2T it transmits corresponding to an > initiator task tag. > > -JP --------------BC796B4938A3D88858997728 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------BC796B4938A3D88858997728-- From owner-ips@ece.cmu.edu Wed Jan 10 21:17:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA09361 for ; Wed, 10 Jan 2001 21:17:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0B1TvY10969 for ips-outgoing; Wed, 10 Jan 2001 20:29:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0B1T4f10951 for ; Wed, 10 Jan 2001 20:29:07 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0B2dk026960; Wed, 10 Jan 2001 18:39:51 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Matt Wakeley" , "IPS Reflector" Subject: RE: iSCSI : On the subject of R2T and Task Tags. Date: Wed, 10 Jan 2001 17:27:27 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <3A5CCB85.5FDE7AF8@agilent.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt, With independent error checking requiring an additional level of error handling requesting repeated reads, there could be old responses combined with the retry from failed data digests. (If requesting overlapped reads are prohibited, retry is an exception.) Without some means of identifying which retry phase data is associated, a complete set would be difficult to confirm. As order of each data portion is not mandated and these transfers themselves may overlap, byte counts can not reveal completion. Should does not mean 'must.' Doug > Pierre Labat wrote: > > > I agree that the draft must not impose to the target to have a > different tag > > for each R2T (it is up to the target implementation to decide). > I disagree > > to impose a unique target tag for the whole IO. It imposes the > target to do > > score boarding > > to know when all the data have been received. Letting the > possiblility to the > > target > > to have one tag per R2T (or add a subtag identifying the R2T as > proposed Santosh) > > > > allows the target to dispense with the score boarding. The > target knows that all > > the > > data has been received when a data PDU with the "F" bit set has > been received > > for each R2T. > > There is no need to do scoreboarding. Just count the number of bytes > received. If they all add up, then you've received all the data. > I for one > would not rely on a PDU with the "F" bit set. I would want to > cross check it > with the correct number of bytes. > > > > > > Regards, > > > > Pierre > > > > > > > > > -Matt Wakeley > Agilent Technologies > From owner-ips@ece.cmu.edu Wed Jan 10 21:19:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA09373 for ; Wed, 10 Jan 2001 21:19:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0ANmGw08784 for ips-outgoing; Wed, 10 Jan 2001 18:48:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0ANeSk08632 for ; Wed, 10 Jan 2001 18:40:28 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id 8FAC1765 for ; Wed, 10 Jan 2001 15:40:27 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id PAA19476; Wed, 10 Jan 2001 15:40:26 -0800 (PST) Message-ID: <3A5D1968.9227D23A@hp.com> Date: Wed, 10 Jan 2001 18:24:40 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI : On the subject of R2T and Task Tags. References: <200101090846.OAA29278@divyaroot.India.Sun.COM> <3A5B95B6.29771A0B@cup.hp.com> <3A5BA1FA.652B86CD@agilent.com> <3A5BD6AB.FB7F5590@cup.hp.com> <3A5CCB08.74D95ED9@agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt Wakeley wrote: > Santosh Rao wrote: > > > I agree with Matt on the semantics of "Target Task Tag" for WRITE PDUs and would > > also like to request a "Target Task Tag" per task for WRITE PDUs. > > > > In addition, since multiple R2Ts can be outstanding, an additional qualifier is > > required to track the R2T (Target R2T Tag) along the lines of a (RX_ID + SEQ_ID) > > that Fibre Channel uses. > > I don't think an additional qualifier is needed. For what purpose? > > The target just sends out all the R2Ts, and counts all the bytes coming back > (remember - TCP guarantees to deliver each byte only once!). Also remember, > the iSCSI spec mandates that the initiator send only what was requested in the > R2T... nothing more. Matt, To have differents tag or (tag + subtag) one for each R2T allows the target to find directly the buffer address. For example using the tag as an index in a table. If not, the target has to go through a list of R2T control structures and for each one compare the offset with with the interval recorded in the R2T structure, in order to find the right buffer. As it doesn't hurt on the initiator side, why not to do it? In fact an R2T and the data related are as an exchange. Hence it is natural to have one identifier/exchange. Of course if the target want to use the same it could. Regards, Pierre > > > -Matt Wakeley > Agilent Technologies > > > > > Thanks, > > Santosh From owner-ips@ece.cmu.edu Wed Jan 10 21:22:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA09411 for ; Wed, 10 Jan 2001 21:22:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0B03Gh09181 for ips-outgoing; Wed, 10 Jan 2001 19:03:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0ANxmk09075 for ; Wed, 10 Jan 2001 18:59:48 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id F17355D8 for ; Wed, 10 Jan 2001 15:59:47 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id PAA19676; Wed, 10 Jan 2001 15:59:46 -0800 (PST) Message-ID: <3A5D1DEC.737F90C7@hp.com> Date: Wed, 10 Jan 2001 18:43:56 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI: retry commands not mandatory Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Where ===== Section 4.1 Connection failure ----------------------- Beginning of page 62 "- the initiator will reissue all outstanding commands with their original Initiator task tag..." Problem: ======= Reading the chapter it is not clear that the initiator has possibility to abort the task and restart it. For some applications that want a total control on the commands, they don't want the "low level" iSCSI to "retry" the command. They want to be able to restart the command exactly when and how they want. For them they must have the possibility to NOT have the command "retried" by iSCSI. Solution ======= The phrase must be change with "- the initiator will abort or reissue the command....". Regards, Pierre From owner-ips@ece.cmu.edu Wed Jan 10 21:23:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA09422 for ; Wed, 10 Jan 2001 21:23:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0B1Bqn10569 for ips-outgoing; Wed, 10 Jan 2001 20:11:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0B1Axf10553 for ; Wed, 10 Jan 2001 20:10:59 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id 7E4A714B for ; Wed, 10 Jan 2001 17:10:58 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA20559; Wed, 10 Jan 2001 17:10:57 -0800 (PST) Message-ID: <3A5D082F.6037AA26@hp.com> Date: Wed, 10 Jan 2001 17:11:11 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI: Ordered delivery capability notification] Content-Type: multipart/mixed; boundary="------------BA2F059E6BBC901F2631925D" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------BA2F059E6BBC901F2631925D Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Hoops! I forgot to add the "iSCSI" prefix --------------BA2F059E6BBC901F2631925D Content-Type: message/rfc822 Content-Disposition: inline Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA20415; Wed, 10 Jan 2001 17:01:00 -0800 (PST) Sender: plabat@cup.hp.com Message-ID: <3A5D2C4A.EB8CC024@hp.com> Date: Wed, 10 Jan 2001 19:45:14 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Ordered delivery capability notification Content-Type: text/plain; charset=us-ascii X-Mozilla-Status2: 00000000 Content-Transfer-Encoding: 7bit Julian, Where: ====== 1.2.2.1 Command numbering "iSCSI supports ordered command delivery within a session" "The target iSCSI layer SHOULD deliver the commands to the target SCSI layer in the order specified by CmdRN". Problem: ======= An initiator is unable to know if the iSCSI target layer support ordered command delivery. Hence it cannot trust iSCSI to have ordered delivery. Hence all the iSCSI ordering mechanism becomes useless because it can't be trusted. The problem is not that iSCSI target layers can't support ordered delivery, the problem is that the initiator/application doesn't know anything about it. Solution: ====== a) Add a Login/Text key Ordered: b) Put the explanation in for the key: "If the target answered "yes" and is unable to maintain the order for whatever reason, it must notify the initiator and drop the session" If the target answered "yes", when it receives a "retried" command (retry bit), it must be able to retry the command in such a way that the effect on the initiator and target device when the command completes is the same as if the command would have not needed to be retried. This doesn't mean always restart from where left, a READ can have all the data re-transmitted because it is transparent a completion time. --------------BA2F059E6BBC901F2631925D-- From owner-ips@ece.cmu.edu Wed Jan 10 22:04:26 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA10993 for ; Wed, 10 Jan 2001 22:04:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0B11pa10368 for ips-outgoing; Wed, 10 Jan 2001 20:01:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0B112f10359 for ; Wed, 10 Jan 2001 20:01:03 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id 5D08E1092 for ; Wed, 10 Jan 2001 17:01:01 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA20415; Wed, 10 Jan 2001 17:01:00 -0800 (PST) Message-ID: <3A5D2C4A.EB8CC024@hp.com> Date: Wed, 10 Jan 2001 19:45:14 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Ordered delivery capability notification Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Where: ====== 1.2.2.1 Command numbering "iSCSI supports ordered command delivery within a session" "The target iSCSI layer SHOULD deliver the commands to the target SCSI layer in the order specified by CmdRN". Problem: ======= An initiator is unable to know if the iSCSI target layer support ordered command delivery. Hence it cannot trust iSCSI to have ordered delivery. Hence all the iSCSI ordering mechanism becomes useless because it can't be trusted. The problem is not that iSCSI target layers can't support ordered delivery, the problem is that the initiator/application doesn't know anything about it. Solution: ====== a) Add a Login/Text key Ordered: b) Put the explanation in for the key: "If the target answered "yes" and is unable to maintain the order for whatever reason, it must notify the initiator and drop the session" If the target answered "yes", when it receives a "retried" command (retry bit), it must be able to retry the command in such a way that the effect on the initiator and target device when the command completes is the same as if the command would have not needed to be retried. This doesn't mean always restart from where left, a READ can have all the data re-transmitted because it is transparent a completion time. From owner-ips@ece.cmu.edu Wed Jan 10 22:08:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA11054 for ; Wed, 10 Jan 2001 22:08:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0B1Rq110928 for ips-outgoing; Wed, 10 Jan 2001 20:27:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0B1Raf10920 for ; Wed, 10 Jan 2001 20:27:36 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id D07EB4C6 for ; Wed, 10 Jan 2001 17:27:35 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA20683; Wed, 10 Jan 2001 17:27:34 -0800 (PST) Message-ID: <3A5D0C13.3380D31E@hp.com> Date: Wed, 10 Jan 2001 17:27:47 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI: CmdRN mandatory Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Where: ====== 1.2.2.1 Command numbering "Command numbering for sessions that will only be made up of one connection is optional." Problem ====== The targets have to be built to handle the case where a session doesn't use the command numbering and the case where a session use it. It is : - two code paths to test - a code bloat - add complexity hence potential errors Solution ====== Make CmdRN mandatory even with one connection per session. Regards, Pierre From owner-ips@ece.cmu.edu Wed Jan 10 22:50:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA12383 for ; Wed, 10 Jan 2001 22:50:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0B1CrJ10585 for ips-outgoing; Wed, 10 Jan 2001 20:12:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cosrel1.hp.com (cosrel1.hp.com [156.153.255.170]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0B1Ckf10581 for ; Wed, 10 Jan 2001 20:12:46 -0500 (EST) Received: from omgw5.rsvl.itc.hp.com (omgw5.rsvl.itc.hp.com [15.34.240.65]) by cosrel1.hp.com (Postfix) with ESMTP id 679C2940 for ; Wed, 10 Jan 2001 18:12:49 -0700 (MST) Received: from xpabh1.boi.hp.com (xpabh1.boi.hp.com [15.56.8.33]) by omgw5.rsvl.itc.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with ESMTP id SAA20116 for ; Wed, 10 Jan 2001 18:12:43 -0700 (MST) Received: by xpabh1.boi.hp.com with Internet Mail Service (5.5.2653.19) id ; Wed, 10 Jan 2001 17:12:43 -0800 Message-ID: From: "VOIGT,DOUG (HP-Boise,ex1)" To: IPS Reflector Subject: RE: iSCSI : Concerns regarding DataRN in READ Data PDU. Date: Wed, 10 Jan 2001 17:11:41 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk It seems to me that this feature is redundant with TCP's own delivery guarantees, in an effort to minimize work loss in the event of TCP connection failure. I too believe this level of complexity and potential normal performance impact compared with accelerated recovery from improbable error events is not likely to yield a net gain. Is recoverable TCP connection loss more common than disk array controller failure? Since different controllers in the same array are (quite reasonably) viewed as different targets (pending SAM change), iSCSI level failover won't compensate for controller failure. This example leads me to suggest that the benefit we are shooting for has a high likelihood of being overshadowed by other failure modes whose recovery is less graceful. Doug Voigt > -----Original Message----- > From: Santosh Rao [mailto:santoshr@cup.hp.com] > Sent: Monday, January 08, 2001 9:03 PM > To: IPS Reflector > Subject: iSCSI : Concerns regarding DataRN in READ Data PDU. > > > Julian/All, > > Can anybody comment on the true benefit that the DataRN > feature provides > that justifies the complexity and performance issues it introduces to > the protocol ? > > I see the following issues with DataRN : > > 1) It adds a performance penalty in that on every READ Data PDU > targets will have to initialize one additional 32-bit word in the > header. > > 2) Extra traffic on the outbound context from initiators sending > NOP-OUTs to acknowledge the DataRNs. > > 3) Performance path checks for the "P" bit on every READ PDU. > (Alternatively, checks for the DataNumber field from the Login > Text Keys). > > 4) Implementing DataRNs per task is too short-lived to provide any > real benefit. Implementing them per connection or session can result > in quick use up of the 32-bit DataRN space (since each frame consumes > one DataRN), adding additional complexity to the target code > to use the > "P" > bit appropriately. > > 5) Further, "DataNumber", which is the size of the un-acknowledged > DataRN window is negotiated at login time, whereas this is really > desired to be a dynamic variable based on the target's > current I/O load > and the number of initiators speaking to it. Thus, DataRN can end up > being > under-utilized resulting in too much NOP-OUT traffic, or it can be > over-utilized resulting in too much usage of the "P" bit in the READ > PDUs, which, again, can cause too much NOP-OUT traffic. > > 6) Considering that the DataRN generation and acknowledgement are > functions > that would typically be in SCSI hardware assist engines and there is a > need to > keep these as simple as possible, adding such features in the > spec means > > that these SCSI assist engines MUST implement DataRN generation > and acknowledgement [since initiators have no way to turn it off if a > target decides to use it] > > What is the true benefit of this feature ? Is it intended to > be used as > : > > a) Will allow targets to do partial freeing of their memory buffers as > and when DataRNs are received ? > > b) Will allow targets to send back only the un-acknowledged data (as > seen from the ExpDataRN received on the last NOP-OUT from the > initiator) > when the initiators retry commands with the "Retry" bit set ? > > If the benefit intended is (a), then, in order to derive this benefit, > targets will have to advertise their DataNumber login key to be < the > avg. number of frames transmitted per READ I/O. Having a DataNumber > > the avg. number of frames per READ I/O implies the life of > the task ends > [and buffers for the task get released] before the initiator sends > NOP-OUT, thereby, defeating benefit (a). > > If the benefit intended is (b) [and i do not see this > explicitly stated > in the draft], then, this is a dangerous assumption which can lead to > data corruption issues. > > When commands are retried with the "Retry" bit set, are > targets allowed > to send back partial data based on the last ExpDataRN they > received ? If > so, this can have multiple issues like : > > o complexity in handling I/O underrun cases for the retried command > which only saw partial data transfer from the target. > > o If the target only sent back partial data based > on its last ExpDataRN received, then, it can open up > possible data > > corruption windows. > > All in all, I believe that the DataRN is adding too much > complexity and > potential performance impact to iSCSI for the benefits it may provide. > > Moreover and most importantly, the draft does not give initiators any > control over the usage of this feature and if a target decides to use > this feature, initiators will have to support it. This exposes the > initiators to all of the above issues without being able to turn off > this feature. > > I would like to therefore ask that : > > either, > a) DataRN support be removed from the iSCSI draft. > > or > b) The support for DataRN be negotiated at login time giving initators > control over this feature and allowing them to disable this feature. > This will allow SCSI hardware assist engines to skip implementation of > this functionality and their associated software can merely turn off > DataNumber in the login negotiation. > > Thanks, > Santosh > > > > From owner-ips@ece.cmu.edu Wed Jan 10 23:31:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA13115 for ; Wed, 10 Jan 2001 23:31:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0B2NsY12074 for ips-outgoing; Wed, 10 Jan 2001 21:23:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0B2N9f12060 for ; Wed, 10 Jan 2001 21:23:09 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id 36A351B82 for ; Wed, 10 Jan 2001 18:23:08 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA21203; Wed, 10 Jan 2001 18:23:06 -0800 (PST) Message-ID: <3A5D1918.9459950B@hp.com> Date: Wed, 10 Jan 2001 18:23:20 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI: Connection replacement Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Where ===== 2.10 Login command Problem ====== In the case where - the maximum number of connections/session is reached - the initiator is faster than the target to detect a failed TCP connection, the establishement of a new connection to replace the bad one will fail. The Login on the new connection will be rejected by the target because the maximum number of connections is reached and the target has not yet detected that there were a failed connection. In the event of a server adapter failure this problem will almost always happen. As soon as the harware fails, the server open a new connection using a new adapter. The target will not be fast enough to realize the connection is bad. Solution ====== a) in the Login message add a field (RecoverID) to inform the target that this connection is to replace a failed one. The value 0xffff means it is not a connection used to replace an old one. b) After the login phase, the first message sent to the target on this new connection must be a Logout for the failed connection. If not the target close the connection. Regards, Pierre From owner-ips@ece.cmu.edu Thu Jan 11 00:18:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA13603 for ; Thu, 11 Jan 2001 00:18:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0B3FuI13064 for ips-outgoing; Wed, 10 Jan 2001 22:15:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0B3Faf13054 for ; Wed, 10 Jan 2001 22:15:36 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Wed, 10 Jan 2001 19:15:33 -0800 Message-ID: From: Charles Monia To: "Julian Satran (E-mail)" Cc: "Ips (E-mail)" Subject: iSCSI: Question on MaxCmdRN update rule Date: Wed, 10 Jan 2001 19:15:08 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Julo: I don't understand the MaxCmdRN update policy as defined below: "2.3.10 MaxCmdRN - maximum CmdRN acceptable from this initiator MaxCmdRN is a reference number that the target iSCSI returns to the initiator to indicate the maximum CmdRN the initiator can send. The initiator must ignore values not between the current value of the ExpCmdRN and MaxCmdRN; this may be required when updates arrive out of order (they travel on different TCP connections)." As I interpret the above, values not within the upper and lower limits given above must be discarded. I would think this parameter should be updated when a new value greater than the current MaxCmdRN is received. Charles From owner-ips@ece.cmu.edu Thu Jan 11 00:19:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA13615 for ; Thu, 11 Jan 2001 00:19:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0B2wtY12698 for ips-outgoing; Wed, 10 Jan 2001 21:58:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0B2wGf12687 for ; Wed, 10 Jan 2001 21:58:17 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id DB4E512C9 for ; Wed, 10 Jan 2001 18:58:15 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA12942; Wed, 10 Jan 2001 18:58:09 -0800 (PST) Message-ID: <3A5D21A7.BDDD10F4@cup.hp.com> Date: Wed, 10 Jan 2001 18:59:51 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Pierre Labat Cc: ips@ece.cmu.edu Subject: Re: iSCSI: CmdRN mandatory References: <3A5D0C13.3380D31E@hp.com> Content-Type: multipart/mixed; boundary="------------970838426EC6BBB0C1808A39" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------970838426EC6BBB0C1808A39 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit The additional advantage this provides is to allow ExpCmdRN based dynamic flow control at the iSCSI Command layer to augment the current SCSI Queue Full based flow control even with single connection sessions. CmdRN should be made compulsory given the dis-advantages of not having it as stated by pierre and the advantage of having it. - Santosh Rao Pierre Labat wrote: > Solution > ====== > Make CmdRN mandatory even with one connection per session. > > Regards, > > Pierre --------------970838426EC6BBB0C1808A39 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------970838426EC6BBB0C1808A39-- From owner-ips@ece.cmu.edu Thu Jan 11 01:38:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA18283 for ; Thu, 11 Jan 2001 01:38:36 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0B4G2x14130 for ips-outgoing; Wed, 10 Jan 2001 23:16:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cleitus.hosting.pacbell.net (cleitus.hosting.pacbell.net [216.100.99.17]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0B4F5f14096 for ; Wed, 10 Jan 2001 23:15:05 -0500 (EST) Received: from VENKAT1 (64-160-62-200.rhapsodynetworks.com [64.160.62.200] (may be forged)) by cleitus.hosting.pacbell.net id XAA14578; Wed, 10 Jan 2001 23:14:48 -0500 (EST) [ConcentricHost SMTP Relay 1.7] From: "Venkat Rangan" To: "Douglas Otis" , "Matt Wakeley" , "IPS Reflector" Subject: RE: iSCSI : On the subject of R2T and Task Tags. Date: Wed, 10 Jan 2001 20:17:02 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Doug, The draft make should not make any statements about how a target chooses to assign Target Task Tags. Some may choose to assign it per R2T, others may choose to assign it per task, based on how they choose to implement the target. Since the initiator is required to reflect it, I do not believe there are any interoperability issues. If an implementation wishes to do a Task-SubTask breakdown, they can choose to partition the 32 bits into portions that can be used for identifying resources within the target. Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Douglas Otis Sent: Wednesday, January 10, 2001 5:27 PM To: Matt Wakeley; IPS Reflector Subject: RE: iSCSI : On the subject of R2T and Task Tags. Matt, With independent error checking requiring an additional level of error handling requesting repeated reads, there could be old responses combined with the retry from failed data digests. (If requesting overlapped reads are prohibited, retry is an exception.) Without some means of identifying which retry phase data is associated, a complete set would be difficult to confirm. As order of each data portion is not mandated and these transfers themselves may overlap, byte counts can not reveal completion. Should does not mean 'must.' Doug > Pierre Labat wrote: > > > I agree that the draft must not impose to the target to have a > different tag > > for each R2T (it is up to the target implementation to decide). > I disagree > > to impose a unique target tag for the whole IO. It imposes the > target to do > > score boarding > > to know when all the data have been received. Letting the > possiblility to the > > target > > to have one tag per R2T (or add a subtag identifying the R2T as > proposed Santosh) > > > > allows the target to dispense with the score boarding. The > target knows that all > > the > > data has been received when a data PDU with the "F" bit set has > been received > > for each R2T. > > There is no need to do scoreboarding. Just count the number of bytes > received. If they all add up, then you've received all the data. > I for one > would not rely on a PDU with the "F" bit set. I would want to > cross check it > with the correct number of bytes. > > > > > > Regards, > > > > Pierre > > > > > > > > > -Matt Wakeley > Agilent Technologies > From owner-ips@ece.cmu.edu Thu Jan 11 04:37:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA27985 for ; Thu, 11 Jan 2001 04:37:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0B7L5k17646 for ips-outgoing; Thu, 11 Jan 2001 02:21:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0B7KQf17636 for ; Thu, 11 Jan 2001 02:20:27 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 341C8E39; Wed, 10 Jan 2001 23:20:26 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id XAA26173; Wed, 10 Jan 2001 23:20:20 -0800 (PST) From: Santosh Rao Message-Id: <200101110720.XAA26173@hpcuhe.cup.hp.com> Subject: Re: iSCSI : On the subject of R2T and Task Tags. To: venkat@rhapsodynetworks.com (Venkat Rangan) Date: Wed, 10 Jan 2001 23:20:20 -0800 (PST) Cc: dotis@sanlight.net (Douglas Otis), matt_wakeley@agilent.com (Matt Wakeley), ips@ece.cmu.edu (IPS Reflector) In-Reply-To: from "Venkat Rangan" at Jan 10, 2001 08:17:02 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit The draft must provide for seperate fields to track the task and the R2T along the lines of RX_ID and SEQ_ID in Fibre Channel. Implementations are free to use them as they please and must use 0xFFFFFFFF as their un-used values in these fields. The advantage of defining the fields explicitly in the header as opposed to allowing targets to vertically encode a portion of the tag field for tracking the R2T is that these standard fields are interpret-able by iSCSI Analyzers and will make debugging and tracking the sequence of the I/O easier than implementation specific vertical encoding into a single field. Merging the 2 into a single field is akin to attempting to merge the OX_ID, SEQ_ID and SEQ_CNT fields of FC into a single field and allowing implementation specific vertical encoding. Standard defined fields make for easier introp verification and debugging. - Santosh > > Doug, > > The draft make should not make any statements about how a target chooses to > assign Target Task Tags. Some may choose to assign it per R2T, others may > choose to assign it per task, based on how they choose to implement the > target. Since the initiator is required to reflect it, I do not believe > there are any interoperability issues. If an implementation wishes to do a > Task-SubTask breakdown, they can choose to partition the 32 bits into > portions that can be used for identifying resources within the target. > > Venkat Rangan > Rhapsody Networks Inc. > http://www.rhapsodynetworks.com > > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Douglas Otis > Sent: Wednesday, January 10, 2001 5:27 PM > To: Matt Wakeley; IPS Reflector > Subject: RE: iSCSI : On the subject of R2T and Task Tags. > > > Matt, > > With independent error checking requiring an additional level of error > handling requesting repeated reads, there could be old responses combined > with the retry from failed data digests. (If requesting overlapped reads are > prohibited, retry is an exception.) Without some means of identifying which > retry phase data is associated, a complete set would be difficult to > confirm. As order of each data portion is not mandated and these transfers > themselves may overlap, byte counts can not reveal completion. Should does > not mean 'must.' > > Doug > > > Pierre Labat wrote: > > > > > I agree that the draft must not impose to the target to have a > > different tag > > > for each R2T (it is up to the target implementation to decide). > > I disagree > > > to impose a unique target tag for the whole IO. It imposes the > > target to do > > > score boarding > > > to know when all the data have been received. Letting the > > possiblility to the > > > target > > > to have one tag per R2T (or add a subtag identifying the R2T as > > proposed Santosh) > > > > > > allows the target to dispense with the score boarding. The > > target knows that all > > > the > > > data has been received when a data PDU with the "F" bit set has > > been received > > > for each R2T. > > > > There is no need to do scoreboarding. Just count the number of bytes > > received. If they all add up, then you've received all the data. > > I for one > > would not rely on a PDU with the "F" bit set. I would want to > > cross check it > > with the correct number of bytes. > > > > > > > > > > Regards, > > > > > > Pierre > > > > > > > > > > > > > -Matt Wakeley > > Agilent Technologies > > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Thu Jan 11 05:15:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA28211 for ; Thu, 11 Jan 2001 05:15:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0B8D8U18534 for ips-outgoing; Thu, 11 Jan 2001 03:13:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0B8Cuf18528 for ; Thu, 11 Jan 2001 03:12:56 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 76DD6C5C; Thu, 11 Jan 2001 00:12:55 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id AAA29904; Thu, 11 Jan 2001 00:12:49 -0800 (PST) From: Santosh Rao Message-Id: <200101110812.AAA29904@hpcuhe.cup.hp.com> Subject: Re: iSCSI: Logout for a Session To: paul.grun@intel.com (Grun, Paul) Date: Thu, 11 Jan 2001 00:12:49 -0800 (PST) Cc: matt_wakeley@agilent.com ('Matt Wakeley'), ips@ece.cmu.edu (IPS Reflector) In-Reply-To: <31E3E2A9D8DDD311AC43009027C68052C8017B@orsmsx59.jf.intel.com> from "Grun, Paul" at Jan 10, 2001 02:16:46 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Sender: owner-ips@ece.cmu.edu Precedence: bulk X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id f0B8D8V18534 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ietf.org id FAA28211 The use of a Logout to close the connection [and the session if this is the last connection being closed] is preferred to just closing the TCP connection since the logout command PDU provides the other end a "Reason Code" that explains the reason for the close. 2 comments regarding logout : i) The draft should change : "An initiator MAY use a logout command to remove a connection from a session." to : "An initiator MUST use a logout command to remove a connection from a session." This ensures only one way of doing things, communicates a reason code for the close of connection and does its bit for the cause of successful interopability. ii) The reason code : "2 - Remove the connection at targets requests (requested through an AEN)" along with the corresponding AEN iSCSI event indicator : "2 Target requests Logout on this connection " MUST be removed and targets allowed to originate a logout. This allows targets the ability to drive the logout and is a useful feature in cases like targets wishing to commence an online firmware upgrade on one of their channels. In such situations, it is un-desirable for the targets to send AENs and then wait [and hope] that the initiator will log the target out. For all one knows, several O.S. drivers may decide to ignore AENs ! Targets SHOULD be allowed to originate logouts. - Santosh > > Isn't a session terminated when the last TCP connection is closed? > -paul > > Paul Grun                                              > Intel Corporation > Enterprise Platform Group > Fabric Components Division > (503) 677-6768 > paul.grun@intel.com > > > > -----Original Message----- > > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > > Sent: Wednesday, January 10, 2001 12:59 PM > > To: IPS Reflector > > Subject: iSCSI: Logout for a Session > > > > > > Throughout the iSCSI document, it talks about the logout > > command closing a TCP > > connection. Especially for the purposes of error recover to > > terminate a TCP > > connection and "clean up" so that another TCP connection can > > be established to > > continue the session. There seems to be no way to indicate > > that a session is > > being terminated. > > > > There should be a flag in the logout command that indicates > > that the *session* > > is being terminated (as well as the TCP connection). > > > > -Matt Wakeley > > Agilent Technologies > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Thu Jan 11 07:28:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA29115 for ; Thu, 11 Jan 2001 07:28:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0BABC020440 for ips-outgoing; Thu, 11 Jan 2001 05:11:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0BAAof20430 for ; Thu, 11 Jan 2001 05:10:50 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 1D6402F5 for ; Thu, 11 Jan 2001 03:10:50 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 633C4145 for ; Thu, 11 Jan 2001 05:10:49 -0500 (EST) Received: from agilent.com (cos1nai132083.cos.agilent.com [141.184.132.83]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id CAA21133 for ; Thu, 11 Jan 2001 02:10:47 -0800 (PST) Message-ID: <3A5D86A5.83A173CA@agilent.com> Date: Thu, 11 Jan 2001 02:10:45 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI: Logout for a Session References: <3A5CCD1A.647D096B@agilent.com> <3A5CD97E.62977962@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > > Matt Wakeley wrote: > > > There should be a flag in the logout command that indicates that the *session* > > is being terminated (as well as the TCP connection). > > > > There is. Look at the "Reason Code" field in the Logout Command PDU. (Section > 2.14 of the draft dated dec 30, 2000). Ok - guess I missed that... -Matt > > - Santosh Rao From owner-ips@ece.cmu.edu Thu Jan 11 14:07:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA12570 for ; Thu, 11 Jan 2001 14:07:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0BGj5r10165 for ips-outgoing; Thu, 11 Jan 2001 11:45:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0BGica10146 for ; Thu, 11 Jan 2001 11:44:38 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA150708 for ; Thu, 11 Jan 2001 17:44:31 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA240544 for ; Thu, 11 Jan 2001 17:44:31 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D1.005BF5E1 ; Thu, 11 Jan 2001 17:44:27 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 11 Jan 2001 18:40:16 +0200 Subject: new iSCSI draft - 03.txt Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, I've just submitted to the Internet-Drafts repository and to our list archive (at CMU) a new version of the draft. There are many changes from the previous versions. A major change - security is revamped (hopefully better but not yet final). As we are past major ticket items (hopefully) we could dedicate more time to details and many long overdue fixes are finally in. If you had a question or a point to make after January 7 take a renewed look at the document and restate your point. You can find the new document at: http://www.haifa.il.ibm.com/satran/ips/draft-ietf-ips-iSCSI-03.txt Regards, Julo From owner-ips@ece.cmu.edu Thu Jan 11 14:08:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA12608 for ; Thu, 11 Jan 2001 14:08:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0BHG6011182 for ips-outgoing; Thu, 11 Jan 2001 12:16:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0BHFEa11131 for ; Thu, 11 Jan 2001 12:15:14 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA12046 for ; Thu, 11 Jan 2001 18:15:09 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA267318 for ; Thu, 11 Jan 2001 18:15:08 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D1.005EC257 ; Thu, 11 Jan 2001 18:15:01 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 11 Jan 2001 19:10:47 +0200 Subject: Re: iSCSI : Concerns regarding DataRN in READ Data PDU. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, DataRN was meant for what you call case B to be used by targets that perform long inbound transfers (like mirorring or tape copy) . Initiators have very few things to do to suport it and targets will use as they see fit. Setting a counter in the header is not exactly an large overhead - it is a local counter in the command block (unprotected). As for your statements about the data corruption window - could you explain please? Julo Santosh Rao on 09/01/2001 06:03:03 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iSCSI : Concerns regarding DataRN in READ Data PDU. Julian/All, Can anybody comment on the true benefit that the DataRN feature provides that justifies the complexity and performance issues it introduces to the protocol ? I see the following issues with DataRN : 1) It adds a performance penalty in that on every READ Data PDU targets will have to initialize one additional 32-bit word in the header. 2) Extra traffic on the outbound context from initiators sending NOP-OUTs to acknowledge the DataRNs. 3) Performance path checks for the "P" bit on every READ PDU. (Alternatively, checks for the DataNumber field from the Login Text Keys). 4) Implementing DataRNs per task is too short-lived to provide any real benefit. Implementing them per connection or session can result in quick use up of the 32-bit DataRN space (since each frame consumes one DataRN), adding additional complexity to the target code to use the "P" bit appropriately. 5) Further, "DataNumber", which is the size of the un-acknowledged DataRN window is negotiated at login time, whereas this is really desired to be a dynamic variable based on the target's current I/O load and the number of initiators speaking to it. Thus, DataRN can end up being under-utilized resulting in too much NOP-OUT traffic, or it can be over-utilized resulting in too much usage of the "P" bit in the READ PDUs, which, again, can cause too much NOP-OUT traffic. 6) Considering that the DataRN generation and acknowledgement are functions that would typically be in SCSI hardware assist engines and there is a need to keep these as simple as possible, adding such features in the spec means that these SCSI assist engines MUST implement DataRN generation and acknowledgement [since initiators have no way to turn it off if a target decides to use it] What is the true benefit of this feature ? Is it intended to be used as : a) Will allow targets to do partial freeing of their memory buffers as and when DataRNs are received ? b) Will allow targets to send back only the un-acknowledged data (as seen from the ExpDataRN received on the last NOP-OUT from the initiator) when the initiators retry commands with the "Retry" bit set ? If the benefit intended is (a), then, in order to derive this benefit, targets will have to advertise their DataNumber login key to be < the avg. number of frames transmitted per READ I/O. Having a DataNumber > the avg. number of frames per READ I/O implies the life of the task ends [and buffers for the task get released] before the initiator sends NOP-OUT, thereby, defeating benefit (a). If the benefit intended is (b) [and i do not see this explicitly stated in the draft], then, this is a dangerous assumption which can lead to data corruption issues. When commands are retried with the "Retry" bit set, are targets allowed to send back partial data based on the last ExpDataRN they received ? If so, this can have multiple issues like : o complexity in handling I/O underrun cases for the retried command which only saw partial data transfer from the target. o If the target only sent back partial data based on its last ExpDataRN received, then, it can open up possible data corruption windows. All in all, I believe that the DataRN is adding too much complexity and potential performance impact to iSCSI for the benefits it may provide. Moreover and most importantly, the draft does not give initiators any control over the usage of this feature and if a target decides to use this feature, initiators will have to support it. This exposes the initiators to all of the above issues without being able to turn off this feature. I would like to therefore ask that : either, a) DataRN support be removed from the iSCSI draft. or b) The support for DataRN be negotiated at login time giving initators control over this feature and allowing them to disable this feature. This will allow SCSI hardware assist engines to skip implementation of this functionality and their associated software can merely turn off DataNumber in the login negotiation. Thanks, Santosh - santoshr.vcf From owner-ips@ece.cmu.edu Thu Jan 11 14:08:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA12643 for ; Thu, 11 Jan 2001 14:08:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0BGw4n10564 for ips-outgoing; Thu, 11 Jan 2001 11:58:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cleitus.hosting.pacbell.net (cleitus.hosting.pacbell.net [216.100.99.17]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0BGvVa10554 for ; Thu, 11 Jan 2001 11:57:31 -0500 (EST) Received: from VENKAT1 (64-160-62-200.rhapsodynetworks.com [64.160.62.200] (may be forged)) by cleitus.hosting.pacbell.net id LAA11177; Thu, 11 Jan 2001 11:53:47 -0500 (EST) [ConcentricHost SMTP Relay 1.7] From: "Venkat Rangan" To: "Pierre Labat" , Subject: RE: iSCSI: Connection replacement Date: Thu, 11 Jan 2001 08:56:01 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <3A5D1918.9459950B@hp.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Pierre, There is another aspect to connection replacement - any in progress data/status transfers on the original connection may now be sent on the new connection, thus "breaking" the connection allegiance requirementg. If we allowed retaining the same CID as the failed connection, and sent that along in the Login of the replacing connection (along with the RecoverId as you suggest), we could have the target maintain connection allegiance with respect to the CID (although not with respect to TCP connection). Do you have a suggestion on how the in-progress I/Os can be handled? Regards, Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Pierre Labat Sent: Wednesday, January 10, 2001 6:23 PM To: ips@ece.cmu.edu Subject: iSCSI: Connection replacement Julian, Where ===== 2.10 Login command Problem ====== In the case where - the maximum number of connections/session is reached - the initiator is faster than the target to detect a failed TCP connection, the establishement of a new connection to replace the bad one will fail. The Login on the new connection will be rejected by the target because the maximum number of connections is reached and the target has not yet detected that there were a failed connection. In the event of a server adapter failure this problem will almost always happen. As soon as the harware fails, the server open a new connection using a new adapter. The target will not be fast enough to realize the connection is bad. Solution ====== a) in the Login message add a field (RecoverID) to inform the target that this connection is to replace a failed one. The value 0xffff means it is not a connection used to replace an old one. b) After the login phase, the first message sent to the target on this new connection must be a Logout for the failed connection. If not the target close the connection. Regards, Pierre From owner-ips@ece.cmu.edu Thu Jan 11 14:10:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA12676 for ; Thu, 11 Jan 2001 14:10:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0BHIOa11248 for ips-outgoing; Thu, 11 Jan 2001 12:18:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0BHHXa11226 for ; Thu, 11 Jan 2001 12:17:33 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA47772 for ; Thu, 11 Jan 2001 18:17:24 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA86574 for ; Thu, 11 Jan 2001 18:17:24 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D1.005EF5C5 ; Thu, 11 Jan 2001 18:17:13 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 11 Jan 2001 19:13:02 +0200 Subject: Re: iSCSI : Concerns regarding DataRN in READ Data PDU. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk And BTW the same coments where made with regard to ftp! Was it a good thing that it it was unrecoverable? Should we repeat this story for the SCSI Extended Copy? Julo Santosh Rao on 09/01/2001 06:03:03 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iSCSI : Concerns regarding DataRN in READ Data PDU. Julian/All, Can anybody comment on the true benefit that the DataRN feature provides that justifies the complexity and performance issues it introduces to the protocol ? I see the following issues with DataRN : 1) It adds a performance penalty in that on every READ Data PDU targets will have to initialize one additional 32-bit word in the header. 2) Extra traffic on the outbound context from initiators sending NOP-OUTs to acknowledge the DataRNs. 3) Performance path checks for the "P" bit on every READ PDU. (Alternatively, checks for the DataNumber field from the Login Text Keys). 4) Implementing DataRNs per task is too short-lived to provide any real benefit. Implementing them per connection or session can result in quick use up of the 32-bit DataRN space (since each frame consumes one DataRN), adding additional complexity to the target code to use the "P" bit appropriately. 5) Further, "DataNumber", which is the size of the un-acknowledged DataRN window is negotiated at login time, whereas this is really desired to be a dynamic variable based on the target's current I/O load and the number of initiators speaking to it. Thus, DataRN can end up being under-utilized resulting in too much NOP-OUT traffic, or it can be over-utilized resulting in too much usage of the "P" bit in the READ PDUs, which, again, can cause too much NOP-OUT traffic. 6) Considering that the DataRN generation and acknowledgement are functions that would typically be in SCSI hardware assist engines and there is a need to keep these as simple as possible, adding such features in the spec means that these SCSI assist engines MUST implement DataRN generation and acknowledgement [since initiators have no way to turn it off if a target decides to use it] What is the true benefit of this feature ? Is it intended to be used as : a) Will allow targets to do partial freeing of their memory buffers as and when DataRNs are received ? b) Will allow targets to send back only the un-acknowledged data (as seen from the ExpDataRN received on the last NOP-OUT from the initiator) when the initiators retry commands with the "Retry" bit set ? If the benefit intended is (a), then, in order to derive this benefit, targets will have to advertise their DataNumber login key to be < the avg. number of frames transmitted per READ I/O. Having a DataNumber > the avg. number of frames per READ I/O implies the life of the task ends [and buffers for the task get released] before the initiator sends NOP-OUT, thereby, defeating benefit (a). If the benefit intended is (b) [and i do not see this explicitly stated in the draft], then, this is a dangerous assumption which can lead to data corruption issues. When commands are retried with the "Retry" bit set, are targets allowed to send back partial data based on the last ExpDataRN they received ? If so, this can have multiple issues like : o complexity in handling I/O underrun cases for the retried command which only saw partial data transfer from the target. o If the target only sent back partial data based on its last ExpDataRN received, then, it can open up possible data corruption windows. All in all, I believe that the DataRN is adding too much complexity and potential performance impact to iSCSI for the benefits it may provide. Moreover and most importantly, the draft does not give initiators any control over the usage of this feature and if a target decides to use this feature, initiators will have to support it. This exposes the initiators to all of the above issues without being able to turn off this feature. I would like to therefore ask that : either, a) DataRN support be removed from the iSCSI draft. or b) The support for DataRN be negotiated at login time giving initators control over this feature and allowing them to disable this feature. This will allow SCSI hardware assist engines to skip implementation of this functionality and their associated software can merely turn off DataNumber in the login negotiation. Thanks, Santosh - santoshr.vcf From owner-ips@ece.cmu.edu Thu Jan 11 16:38:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA15149 for ; Thu, 11 Jan 2001 16:38:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0BJ4C014666 for ips-outgoing; Thu, 11 Jan 2001 14:04:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0BJ3ba14649 for ; Thu, 11 Jan 2001 14:03:37 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0BKEg028166; Thu, 11 Jan 2001 12:14:43 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Venkat Rangan" , "Matt Wakeley" , "IPS Reflector" Subject: RE: iSCSI : On the subject of R2T and Task Tags. Date: Thu, 11 Jan 2001 11:02:23 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Venkat, You misunderstood my comment. Matt had indicated a simple solution of counting bytes. As you indicate, the target tag is uncertain. Assumptions about data not overlapping is also incorrect and data overlays may not be sent in sequence. All of this adds to difficulty in confirming completion before returning to the application. Doug > Doug, > > The draft make should not make any statements about how a target > chooses to > assign Target Task Tags. Some may choose to assign it per R2T, others may > choose to assign it per task, based on how they choose to implement the > target. Since the initiator is required to reflect it, I do not believe > there are any interoperability issues. If an implementation wishes to do a > Task-SubTask breakdown, they can choose to partition the 32 bits into > portions that can be used for identifying resources within the target. > > Venkat Rangan > Rhapsody Networks Inc. > http://www.rhapsodynetworks.com > > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Douglas Otis > Sent: Wednesday, January 10, 2001 5:27 PM > To: Matt Wakeley; IPS Reflector > Subject: RE: iSCSI : On the subject of R2T and Task Tags. > > > Matt, > > With independent error checking requiring an additional level of error > handling requesting repeated reads, there could be old responses combined > with the retry from failed data digests. (If requesting > overlapped reads are > prohibited, retry is an exception.) Without some means of > identifying which > retry phase data is associated, a complete set would be difficult to > confirm. As order of each data portion is not mandated and these > transfers > themselves may overlap, byte counts can not reveal completion. > Should does > not mean 'must.' > > Doug > > > Pierre Labat wrote: > > > > > I agree that the draft must not impose to the target to have a > > different tag > > > for each R2T (it is up to the target implementation to decide). > > I disagree > > > to impose a unique target tag for the whole IO. It imposes the > > target to do > > > score boarding > > > to know when all the data have been received. Letting the > > possiblility to the > > > target > > > to have one tag per R2T (or add a subtag identifying the R2T as > > proposed Santosh) > > > > > > allows the target to dispense with the score boarding. The > > target knows that all > > > the > > > data has been received when a data PDU with the "F" bit set has > > been received > > > for each R2T. > > > > There is no need to do scoreboarding. Just count the number of bytes > > received. If they all add up, then you've received all the data. > > I for one > > would not rely on a PDU with the "F" bit set. I would want to > > cross check it > > with the correct number of bytes. > > > > > > > > > > Regards, > > > > > > Pierre > > > > > > > > > > > > > -Matt Wakeley > > Agilent Technologies > > > > > From owner-ips@ece.cmu.edu Thu Jan 11 18:21:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA16257 for ; Thu, 11 Jan 2001 18:21:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0BL9Hn18814 for ips-outgoing; Thu, 11 Jan 2001 16:09:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0BCR6f01325 for ; Thu, 11 Jan 2001 07:27:06 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA29026; Thu, 11 Jan 2001 07:27:00 -0500 (EST) Message-Id: <200101111227.HAA29026@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; CC: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-monia-ips-ifcp-01.txt Date: Thu, 11 Jan 2001 07:27:00 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. Title : iFCP - A Protocol for Internet Fibre Channel Storage Networking Author(s) : R. Mullendore, C. Monia, J. Tseng et al. Filename : draft-monia-ips-ifcp-01.txt Pages : 43 Date : 10-Jan-01 This document specifies iFCP, a gateway-to-gateway protocol for the implementation of a Fibre Channel fabric in which TCP/IP switching and routing elements replace Fibre Channel components. The protocol enables the attachment of existing Fibre Channel storage products to an IP network by supporting the subset of fabric services required by such devices A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-monia-ips-ifcp-01.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-monia-ips-ifcp-01.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-monia-ips-ifcp-01.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20010110121749.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-monia-ips-ifcp-01.txt --OtherAccess Content-Type: Message/External-body; name="draft-monia-ips-ifcp-01.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20010110121749.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Thu Jan 11 18:23:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA16271 for ; Thu, 11 Jan 2001 18:23:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0BL9F918811 for ips-outgoing; Thu, 11 Jan 2001 16:09:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0BAlif29649 for ; Thu, 11 Jan 2001 05:47:44 -0500 (EST) Received: from divyaroot.India.Sun.COM ([129.158.228.35]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id CAA07645 for ; Thu, 11 Jan 2001 02:47:40 -0800 (PST) Received: from helix (helix [129.158.226.51]) by divyaroot.India.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.0) with SMTP id QAA21484 for ; Thu, 11 Jan 2001 16:16:00 +0530 (IST) Message-Id: <200101111046.QAA21484@divyaroot.India.Sun.COM> Date: Thu, 11 Jan 2001 16:17:50 -0500 (GMT) From: Raghavendra Rao Reply-To: Raghavendra Rao Subject: Re: iSCSI : On the subject of R2T and Task Tags. To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: aeDg7SGfss48GOBAm6ihWw== Sender: owner-ips@ece.cmu.edu Precedence: bulk > > > When multiple R2Ts are outstanding, the targets need a tag per R2T to track > their > SCSI State Table Entries for each R2T. While its usage depends on how each > individual programming model implements this feature, I believe the header should > provide support to track each R2T using an id that the target can lookup for its > R2T data buffer pointers. > If a target is doing the I/O in a lot of bits-n-pieces with several R2Ts, it must already be spending a lot of cycles to complete the I/Os - And I can't believe a lookup (if it needs to) is going to be excessively harmful compared to the magnitude of processing it must already be doing (in getting buffers available in piecemeal, allocating, building, and transmiting R2T for each such buffer and tracking the I/O progress). Let us also not overlook the link bandwidth utilization of such examples of doing I/Os. Please explain what is the gain in improving a bad transfer example ? -JP From owner-ips@ece.cmu.edu Thu Jan 11 18:24:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA16299 for ; Thu, 11 Jan 2001 18:24:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0BL9Ci18806 for ips-outgoing; Thu, 11 Jan 2001 16:09:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0BAWUf29392 for ; Thu, 11 Jan 2001 05:32:30 -0500 (EST) Received: from divyaroot.India.Sun.COM ([129.158.225.35]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id CAA04035 for ; Thu, 11 Jan 2001 02:32:27 -0800 (PST) Received: from helix (helix [129.158.226.51]) by divyaroot.India.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.0) with SMTP id QAA19757 for ; Thu, 11 Jan 2001 16:02:22 +0530 (IST) Message-Id: <200101111032.QAA19757@divyaroot.India.Sun.COM> Date: Thu, 11 Jan 2001 16:04:12 -0500 (GMT) From: Raghavendra Rao Reply-To: Raghavendra Rao Subject: Re: iSCSI: Logout for a Session To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: /yDlPIf6N3gLYYMlkR5ESQ== Sender: owner-ips@ece.cmu.edu Precedence: bulk > > The use of a Logout to close the connection [and the session if this is > the last connection being closed] is preferred to just closing the TCP > connection since the logout command PDU provides the other end a "Reason > Code" that explains the reason for the close. > It may not be possible always to send LOGOUT or get the LOGOUT reach the target under pathological conditions - So a target must be prepared to cleanup and perform an equivalent of LOGOUT when the last TCP connection is closed. -JP From owner-ips@ece.cmu.edu Thu Jan 11 19:55:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA16971 for ; Thu, 11 Jan 2001 19:55:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0BNHHu22725 for ips-outgoing; Thu, 11 Jan 2001 18:17:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([208.50.99.84]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0BNGFa22699 for ; Thu, 11 Jan 2001 18:16:15 -0500 (EST) Received: from pclight211 (dummy211.lightsand.com [192.168.1.211]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id PAA04652; Thu, 11 Jan 2001 15:15:56 -0800 (PST) From: "Murali Rajagopal" To: "Kavi, Prabhu" , Subject: RE: Questions about FCIP Date: Thu, 11 Jan 2001 15:17:12 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 In-Reply-To: <6B190B34070BD411ACA000B0D0214E563D3484@newman.tenornet.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Kavi: We will be discussing your second item in the upcoming interim meeting on -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Kavi, Prabhu Sent: Monday, January 08, 2001 10:08 AM To: 'ips@ece.cmu.edu' Subject: Questions about FCIP I have some questions about the Fibre Channel over TCP/IP I-D. I understand IP networks well, but am a Fibre Channel newbie and am asking these questions to learn more about Fibre Channel's requirements rather than to challenge the document. Here are my questions: 1. Why Fibre Channel over TCP, rather than over UDP? Is it because of an expectation of lossy IP networks? Or is it equally valid to put Fibre Channel over UDP (and therefore can be the subject of another I-D). 2. What are the typical values of the ED_TOV and R_A_TOV timers? Do I understand this right to say that these timers effectively limit the latency across the IP network? 3. What is an acceptable loss ratio of Fibre Channel datagrams that still allows the SAN devices to deliver appropriate end user performance (order of magnitude estimate is sufficient). The reasons I ask are: * I believe it is invalid to assume that IP networks are necessarily more lossy than Fibre Channel networks. A well-engineered IP network, using a combination of both Diff-Serv and MPLS, can achieve (close to) zero loss. * Stacking one reliable delivery protocol on top of another can possibly lead to interesting performance problems after a loss occurs, depending upon the timer values for TCP and Fibre Channel. Thanks in advance for your answers. Prabhu Kavi ---------------------------------------------------------------------- Prabhu Kavi Phone: 1-978-264-4900 x125 Director, Adv. Prod. Planning Fax: 1-978-264-0671 Tenor Networks Email: prabhu_kavi@tenornetworks.com 100 Nagog Park WWW: www.tenornetworks.com Acton, MA 01720 From owner-ips@ece.cmu.edu Thu Jan 11 19:58:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA17011 for ; Thu, 11 Jan 2001 19:58:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0BMrI422034 for ips-outgoing; Thu, 11 Jan 2001 17:53:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0BMqJa22005 for ; Thu, 11 Jan 2001 17:52:19 -0500 (EST) Received: from hpbs5001.boi.hp.com (hpbs5001.boi.hp.com [15.2.209.237]) by palrel1.hp.com (Postfix) with ESMTP id 230EB1829 for ; Thu, 11 Jan 2001 14:52:18 -0800 (PST) Received: from xrosebh3.rsvl.itc.hp.com (xrosebh3.rsvl.itc.hp.com [15.34.240.67]) by hpbs5001.boi.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id PAA05115 for ; Thu, 11 Jan 2001 15:52:15 -0700 (MST) Received: by xrosebh3.rsvl.itc.hp.com with Internet Mail Service (5.5.2653.19) id ; Thu, 11 Jan 2001 14:51:03 -0800 Message-ID: <670D895CA648D41198EA00A0C9F2D428038409A9@xrose01.rose.hp.com> From: "ELLIOTT,STEPHEN (HP-Roseville,ex1)" To: ips@ece.cmu.edu Subject: New iFCP draft Date: Thu, 11 Jan 2001 14:41:13 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk With regards to the new iFCP draft 5.3 Trailing iFCP CRC This original Fibre Channel CRC shall not be encapsulated into the iFCP message. Rather, iFCP MAY use a 32-bit field containing a CRC calculated over the data contained in the frame from the iFCP AUGMENTED DATA to the iFCP Payload (inclusive). This includes the Fibre Channel header and payload. The iFCP CRC follows the iFCP payload, and the CRC algorithm is the same used for the Frame Check Sequence (FCS) of IEEE 802.3 Ethernet frames. A bit in the iFCP FLAGS field in the iFCP header indicates the presence or absence of this 32-bit trailing CRC. I would suggest "Rather, iFCP MUST use a 32-bit field containing a CRC ..." unless ethernet provides an equivalent CRC elsewhere. The reason is that I believe gigabit ethernet uses the same physical layer as fibre channel. This physical layer is specified for fibre channel to have a data bit loss rate of 1 in 10^12 or less, and is not "perfect". I think the CRC is designed to throw out these imperfect frames instead of resulting in data corruption. Without the CRC, physical layer imperfections within fibre-channel specificaiton may lead to data corruption. Also, I am not intending to show preference for iSCSI or iFCP with this email. I just was skimming over the standard and thought this feedback might be helpful. Stephen Elliott From owner-ips@ece.cmu.edu Thu Jan 11 20:55:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA17431 for ; Thu, 11 Jan 2001 20:55:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0BNsHx23651 for ips-outgoing; Thu, 11 Jan 2001 18:54:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([208.50.99.84]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0BNsBa23642 for ; Thu, 11 Jan 2001 18:54:11 -0500 (EST) Received: from pclight211 (dummy211.lightsand.com [192.168.1.211]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id PAA05593; Thu, 11 Jan 2001 15:53:58 -0800 (PST) From: "Murali Rajagopal" To: "Kavi, Prabhu" , Subject: RE: Questions about FCIP Date: Thu, 11 Jan 2001 15:55:13 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 In-Reply-To: <6B190B34070BD411ACA000B0D0214E563D3484@newman.tenornet.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Kavi: We'll be discussing your second item in the upcoming meeting in Florida (Wed 17th Jan.). Regards, Murali -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Kavi, Prabhu Sent: Monday, January 08, 2001 10:08 AM To: 'ips@ece.cmu.edu' Subject: Questions about FCIP I have some questions about the Fibre Channel over TCP/IP I-D. I understand IP networks well, but am a Fibre Channel newbie and am asking these questions to learn more about Fibre Channel's requirements rather than to challenge the document. Here are my questions: 1. Why Fibre Channel over TCP, rather than over UDP? Is it because of an expectation of lossy IP networks? Or is it equally valid to put Fibre Channel over UDP (and therefore can be the subject of another I-D). 2. What are the typical values of the ED_TOV and R_A_TOV timers? Do I understand this right to say that these timers effectively limit the latency across the IP network? 3. What is an acceptable loss ratio of Fibre Channel datagrams that still allows the SAN devices to deliver appropriate end user performance (order of magnitude estimate is sufficient). The reasons I ask are: * I believe it is invalid to assume that IP networks are necessarily more lossy than Fibre Channel networks. A well-engineered IP network, using a combination of both Diff-Serv and MPLS, can achieve (close to) zero loss. * Stacking one reliable delivery protocol on top of another can possibly lead to interesting performance problems after a loss occurs, depending upon the timer values for TCP and Fibre Channel. Thanks in advance for your answers. Prabhu Kavi ---------------------------------------------------------------------- Prabhu Kavi Phone: 1-978-264-4900 x125 Director, Adv. Prod. Planning Fax: 1-978-264-0671 Tenor Networks Email: prabhu_kavi@tenornetworks.com 100 Nagog Park WWW: www.tenornetworks.com Acton, MA 01720 From owner-ips@ece.cmu.edu Thu Jan 11 20:56:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA17443 for ; Thu, 11 Jan 2001 20:56:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0C06Jb23921 for ips-outgoing; Thu, 11 Jan 2001 19:06:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0C05Ua23902 for ; Thu, 11 Jan 2001 19:05:30 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Thu, 11 Jan 2001 16:05:31 -0800 Message-ID: From: Joshua Tseng To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: iSCSI: Security Comments Date: Thu, 11 Jan 2001 16:04:56 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, The following are security comments to your new -03.txt version of iSCSI: 1) There are several security sections (4.2, 4.3, and 7.) and they should be consolidated into one section. 2) Did we come to a consensus that no encryption was to be included in iSCSI? If yes, then section 7.2.2.4 needs to be deleted. If no, then I now do suggest encryption be delegated to IPSec or TLS. Both of these mechanisms are handled at a lower layer and this allows leverage of existing h/w and s/w implementations. 3) I would suggest that iSCSI consider adding an optional authentication block in each iSCSI PDU. If this is of interest I can work with you further on it. Specific comments to security section Appendix A: a) pg 77, the following: "- Public key algorithm (InitPublicKey,TargetPublicKey)" needs to be replaced by: "- Public key algorithm (PublicKey)" If a per-iSCSI PDU authentication block is to be added, perhaps that can be added to this list with something like: "- PDU-Authentication (AuthAlgorithm:)" Where AuthAlgorithm is the signature algorithm used to sign the authentication block. We would also need a description of new text commands such as: InitDHValue: and TargetDHValue:, which would list parameters for the Diffie Hellman exchange to calculate a shared secret key used for AuthAlgorithm. b) pg 83, see the following text and suggested modifications: "The next example is a public-key authentication. The initiator authenticates itself to the target and no keys are exchanged: " - Need to delete the part "...and no keys are exchanged: ". "If the user was not confirmed, the target sends a login response message with "login reject" to the initiator. Else, it can send a login response with "login accept" and MAY attach a secret: " - Need to delete the part "...and MAY attach a secret:". "The next example is another public-key authentication. The initiator authenticates itself to the target. The target authenticates itself to the initiator and key are exchanged: " - Delete the part "...and key are exchanged: ". " T->Text StartSecure:HERE secret: " - Delete the part "...secret: ". No secret keys should be exchanged in this phase since the login is authenticated only, not encrypted. If secret keys are needed for a PDU authentication block, then Diffie-Hellman should be used using the above text command. c) pg 83, I suggest changing the following text: NB - where the blob stands for the digitally signed hash of the packet header, the user (presumably some form of machine+OS+session name or a certificate issued by a certificate authority) the target salt and using the appropriate digital signature algorithm (DSS). to the following: "...where the blob stands for the digitally signed hash of the iSCSI PDU header, the WWUI of the iSCSI node being authenticated, and the salt provided by the authenticating node, using the appropriate digital signature method (DSS or DSA)." d) pg 84, suggest modifying the following text: where the blob stands for the digitally signed hash of the packet header, the user (presumably some form WWUID name or certificate issued by a certificate authority) the initiator salt and using the appropriate digital signature algorithm (DSS). The target also send a suggested key encrypted with the initiator public key. to the following: "...where the blob stands for the digitally signed hash of the iSCSI PDU header, the WWUI of the iSCSI node being authenticated, and the salt privided by the authenticating node, using the appropriate digital signature method (DSS or RSA). - Delete "Secret:key" from the following: "T-> Text Authenticate:user,blob Secret:key" In the following: where the blob stands for the digitally signed hash of the packet header, the user (presumably some form WWUID name or certificate issued by a certificate authority) the initiator salt and using the appropriate digital signature algorithm (DSS). The target also send a suggested key encrypted with the initiator public key. - delete the last sentence "The target also send....". That's all for now. Josh Tseng From owner-ips@ece.cmu.edu Fri Jan 12 00:22:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA22083 for ; Fri, 12 Jan 2001 00:22:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0C2vOQ27455 for ips-outgoing; Thu, 11 Jan 2001 21:57:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0C2uYa27441 for ; Thu, 11 Jan 2001 21:56:34 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Thu, 11 Jan 2001 18:56:42 -0800 Message-ID: From: Charles Monia To: "ELLIOTT,STEPHEN (HP-Roseville,ex1)" Cc: ips@ece.cmu.edu, Charles Monia Subject: RE: New iFCP draft Date: Thu, 11 Jan 2001 18:56:05 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Stephen: Thanks for the input. A little history: We added the CRCs to buttress end-to-end rather than on-the-wire error detection (although that's an important consideration of course). That is, we wanted to give implementations a way to protect the frame while it traversed the sending and receiving gateways and other way points in between. We had decided to make CRCs optional to accomodate those who felt that the TCP/IP checksum was adequate for this purpose and easier to deal with. In rethinking the issue, my preference now would be to make it mandatory as you suggest. If it were to remain optional, I'd propose to change the spec as follows. 1. All frames will contain CRC fields. As described below, the fields may or may not contain valid CRCs. 2. A sending gateway that supports CRCs can always unconditionally generate and include them in outgoing frames. A CRC VALID bit in the encapsulation header defines whether or not the CRC fields in the frame are valid. 3. A gateway that supports CRCs will conditionally check the frame based on the setting of the CRC VALID bit. A gateway that does not support CRCs will, of course, bypass such checks. With regard to the Fibre Channel undetected error rate you mention, it's my impression that the deployed systems are exteremely solid. As I understand it, the 10^^-12 error rate was set on the basis of testability concerns rather than physical limitations in the technology. Charles PS: In my opinion, any feedback of this type should be interpreted in the light you describe. > -----Original Message----- > From: ELLIOTT,STEPHEN (HP-Roseville,ex1) > [mailto:stephen_elliott@hp.com] > Sent: Thursday, January 11, 2001 2:41 PM > To: ips@ece.cmu.edu > Subject: New iFCP draft > > > With regards to the new iFCP draft > > 5.3 Trailing iFCP CRC > > This original Fibre Channel CRC shall not be encapsulated into the > iFCP message. Rather, iFCP MAY use a 32-bit field > containing a CRC > calculated over the data contained in the frame from the iFCP > AUGMENTED DATA to the iFCP Payload (inclusive). This includes the > Fibre Channel header and payload. The iFCP CRC follows the iFCP > payload, and the CRC algorithm is the same used for the > Frame Check > Sequence (FCS) of IEEE 802.3 Ethernet frames. A bit in the iFCP > FLAGS field in the iFCP header indicates the presence or > absence of > this 32-bit trailing CRC. > > I would suggest "Rather, iFCP MUST use a 32-bit field > containing a CRC ..." > unless ethernet provides an equivalent CRC elsewhere. The > reason is that I > believe gigabit ethernet uses the same physical layer as > fibre channel. > This physical layer is specified for fibre channel to have a > data bit loss > rate of 1 in 10^12 or less, and is not "perfect". I think the CRC is > designed to throw out these imperfect frames instead of > resulting in data > corruption. Without the CRC, physical layer imperfections within > fibre-channel specificaiton may lead to data corruption. > > Also, I am not intending to show preference for iSCSI or iFCP > with this > email. I just was skimming over the standard and thought > this feedback > might be helpful. > > Stephen Elliott > > From owner-ips@ece.cmu.edu Fri Jan 12 03:10:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA06014 for ; Fri, 12 Jan 2001 03:10:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0C5jVs00752 for ips-outgoing; Fri, 12 Jan 2001 00:45:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0C5iba00716 for ; Fri, 12 Jan 2001 00:44:38 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id 091CF1629 for ; Thu, 11 Jan 2001 21:44:37 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id VAA03444; Thu, 11 Jan 2001 21:44:35 -0800 (PST) Message-ID: <3A5E99D2.B52E2D8E@hp.com> Date: Thu, 11 Jan 2001 21:44:50 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: Connection replacement References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Venkat Rangan wrote: > Pierre, > > There is another aspect to connection replacement - any in progress > data/status transfers on the original connection may now be sent on the new > connection, thus "breaking" the connection allegiance requirementg. Venkat, Hmmm, it depends on how you look at it. On the new connection before receiving the completion of a command initiated on the failed connection, you need to send the command with the retry bit. Hence on the new connection the initiator sends a command and receive the completion hence there is allegiance there. > If we > allowed retaining the same CID as the failed connection, and sent that along > in the Login of the replacing connection (along with the RecoverId as you > suggest), we could have the target maintain connection allegiance with > respect to the CID (although not with respect to TCP connection). Why not, the RecoverID and CID could be the same. But the rule is that the target can not send anything concerning the commands associated to the failed connection till it receives the logout and the "retries". > Do you > have a suggestion on how the in-progress I/Os can be handled? As described in the draft, to summarize: a) a connection drops b) the initiator sends a Logout on another connection c) when the target receives the Logout it drops the failed connection hence it cannot receives anything on this connection from this point. d) the target send the Logout response to the initiator e) the initiator "retry" the commands on the valid connection f) the target when receiving a "retry" must handle the retry in a way that the outcome of the command on the initiator and target is the same as if the connection didn't fail. For example for a READ the target (it is up to it) could resend everything or just what as not be acknowledged, because the outcome is the same. For a "FORMAT MEDIUM" command, the command will not be delivered a second time to the device server. When the command is over the completion is send back on the valid connection. Regards, Pierre > > > Regards, > > Venkat Rangan > Rhapsody Networks Inc. > http://www.rhapsodynetworks.com > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Pierre Labat > Sent: Wednesday, January 10, 2001 6:23 PM > To: ips@ece.cmu.edu > Subject: iSCSI: Connection replacement > > Julian, > > Where > ===== > 2.10 Login command > > Problem > ====== > > In the case where > - the maximum number of connections/session is reached > - the initiator is faster than the target to detect a failed TCP > connection, > > the establishement of a new connection to replace the bad > one will fail. > > The Login on the new connection will be rejected by the target > because the maximum number of connections is reached and the target > has not yet detected that there were a failed connection. > > In the event of a server adapter failure this problem will almost > always happen. > As soon as the harware fails, the server open a new connection > using a new adapter. The target will not be fast enough to > realize the connection is bad. > > Solution > ====== > > a) in the Login message add a field (RecoverID) to inform the target > that this connection is to replace a failed one. The value 0xffff > means > it is not a connection used to replace an old one. > > b) After the login phase, the first message sent to the target on this > new connection must be a Logout for the failed connection. If not > the target close the connection. > > Regards, > > Pierre From owner-ips@ece.cmu.edu Fri Jan 12 07:42:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA07607 for ; Fri, 12 Jan 2001 07:42:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CAMka06802 for ips-outgoing; Fri, 12 Jan 2001 05:22:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CALsa05503 for ; Fri, 12 Jan 2001 05:21:54 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id LAA114210 for ; Fri, 12 Jan 2001 11:21:48 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA112580 for ; Fri, 12 Jan 2001 11:21:47 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D2.0038D056 ; Fri, 12 Jan 2001 11:20:33 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 12 Jan 2001 09:41:36 +0200 Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, A machine producing a format error is DEFECTIVE and this a reported as hardware error by SCSI (as SCSI does not distinguish between hardware and firmware and rightfully so). And an error in the iSCSI layer gets reported by the next layer - that is the regular layering technique (and BTW I am getting a bit uneasy about all this preaching on layering when it is not obvious that you understand all the implications of the point). Julo Santosh Rao on 10/01/2001 05:05:43 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iSCSI : Initiators expected to fake CHECK CONDITIONS. Julian, My comments embedded below. Thanks, Santosh > Section 4.4. Format Errors > ==================== > Quoting from the draft : > "When a session is active whenever an initiator receives an iSCSI PDU > with a format error, for which it has an outstanding task, it MUST > abort the target task and report the error as a SCSI check condition > status with a sense key of 4h (hardware error)." > What is the reasoning behind this ? It seems to complicate Format Error > Handling with no clear benefit. Is it the expectation that iSCSI > initiators > will fake a format error as a CHECK CONDITION back to the SCSI >Layer ? If so, why ? And why the choice of "Hardware Error" ? [js} the error was generated by a faulty controller and I did not find any other SCSI sense fit for it[/js] A clean layering should be maintained between iSCSI layer errors and SCSI layer errors. A format error on a iSCSI PDU header does not constitute a SCSI error. The specific error returned back to the SCSI upper layer driver in this case is really O.S. specific since each O.S. has its own error codes to deal with interconnect transport errors. Expecting initiators to fabricate sense data on behalf of a target and faking a CHECK CONDITION back to the upper layers is : a) violating layering b/n iSCSI and SCSI layers. b) adds un-necessary complexity to the initiator which now needs to build sense data internally on behalf of a target. c) will cause different error recovery paths to be taken in the upper layer SCSI driver which is expecting to see some std O.S. defined error codes to be returned on interconnect transport errors. > Last, but not the least, the above statement is referring to all iSCSI > PDUs and advocates that iSCSI initiators must fake a CHECK CONDITION > back to the SCSI layer on any format error of any iSCSI PDU. [js] only if a task is running [/js] I'm somewhat confused here. Let us say a format error occurs on a Logout. What is the error recovery to be done in this case if there are other outstanding tasks ? Is this section advocating erroring ALL the outstanding tasks [that did not encounter format error] ? > This is not applicable for non-scsi iSCSI PDUs which have no SCSI > context. I believe the REJECT PDU should only be used for non-scsi > PDUs since Rejects to SCSI PDUs will need a different type of reason > code/explanation information best fitted into the existing SCSI Response - santoshr.vcf From owner-ips@ece.cmu.edu Fri Jan 12 07:42:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA07618 for ; Fri, 12 Jan 2001 07:42:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CALfL05483 for ips-outgoing; Fri, 12 Jan 2001 05:21:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CAKwa05465 for ; Fri, 12 Jan 2001 05:20:58 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id LAA25942 for ; Fri, 12 Jan 2001 11:20:52 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA236858 for ; Fri, 12 Jan 2001 11:20:51 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D2.0038D533 ; Fri, 12 Jan 2001 11:20:45 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 12 Jan 2001 11:52:30 +0200 Subject: Re: iSCSI: Question on MaxCmdRN update rule Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, What version of the document are you referring to? Julo Charles Monia on 11/01/2001 05:15:08 Please respond to Charles Monia To: Julian Satran/Haifa/IBM@IBMIL cc: "Ips (E-mail)" Subject: iSCSI: Question on MaxCmdRN update rule Hi Julo: I don't understand the MaxCmdRN update policy as defined below: "2.3.10 MaxCmdRN - maximum CmdRN acceptable from this initiator MaxCmdRN is a reference number that the target iSCSI returns to the initiator to indicate the maximum CmdRN the initiator can send. The initiator must ignore values not between the current value of the ExpCmdRN and MaxCmdRN; this may be required when updates arrive out of order (they travel on different TCP connections)." As I interpret the above, values not within the upper and lower limits given above must be discarded. I would think this parameter should be updated when a new value greater than the current MaxCmdRN is received. Charles From owner-ips@ece.cmu.edu Fri Jan 12 07:42:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA07630 for ; Fri, 12 Jan 2001 07:42:56 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CAMge06767 for ips-outgoing; Fri, 12 Jan 2001 05:22:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CALua05506 for ; Fri, 12 Jan 2001 05:21:56 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id LAA59536 for ; Fri, 12 Jan 2001 11:21:48 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA183218 for ; Fri, 12 Jan 2001 11:21:48 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D2.0038CEA8 ; Fri, 12 Jan 2001 11:20:29 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 12 Jan 2001 08:06:15 +0200 Subject: Re: iSCSI : On the subject of R2T and Task Tags. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk 64 bit tags - that is a bit bloated. Will have to look into it and consider has strong the mandate is. I assume people where thinking of using addresses as tags. Julo Raghavendra Rao on 09/01/2001 21:18:49 Please respond to Raghavendra Rao To: ips@ece.cmu.edu cc: Subject: Re: iSCSI : On the subject of R2T and Task Tags. > Since a READ Data PDU has no Target Task Tag, is it the expectation that > iSCSI targets should perform a lookup on the Initiator Task Tag to obtain > their per-task data structure ? This is a performance penalty, if so. > For a SCSI READ command, I don't think it'll be really useful to have a target task tag. > Again, with WRITE Data PDUs there is no target assigned > per-task tag. While one could argue about vertically encoding the Target There is - this is done during R2T time and every solicited data PDU from the initiator goes out with this tag. BTW, SAM-2 rev 14 has 64 bit task tags; How are we handling situations like these ? -JP From owner-ips@ece.cmu.edu Fri Jan 12 07:43:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA07642 for ; Fri, 12 Jan 2001 07:43:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CALcq05480 for ips-outgoing; Fri, 12 Jan 2001 05:21:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CAKxa05468 for ; Fri, 12 Jan 2001 05:20:59 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id LAA35698 for ; Fri, 12 Jan 2001 11:20:52 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA236864 for ; Fri, 12 Jan 2001 11:20:52 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D2.0038D46C ; Fri, 12 Jan 2001 11:20:43 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 12 Jan 2001 11:34:32 +0200 Subject: Re: iSCSI: CmdRN mandatory Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I support this. Julo Pierre Labat on 11/01/2001 03:27:47 Please respond to Pierre Labat To: ips@ece.cmu.edu cc: Subject: iSCSI: CmdRN mandatory Julian, Where: ====== 1.2.2.1 Command numbering "Command numbering for sessions that will only be made up of one connection is optional." Problem ====== The targets have to be built to handle the case where a session doesn't use the command numbering and the case where a session use it. It is : - two code paths to test - a code bloat - add complexity hence potential errors Solution ====== Make CmdRN mandatory even with one connection per session. Regards, Pierre From owner-ips@ece.cmu.edu Fri Jan 12 07:44:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA07653 for ; Fri, 12 Jan 2001 07:44:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CALhd05488 for ips-outgoing; Fri, 12 Jan 2001 05:21:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CAKwa05466 for ; Fri, 12 Jan 2001 05:20:58 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id LAA13184 for ; Fri, 12 Jan 2001 11:20:52 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA236860 for ; Fri, 12 Jan 2001 11:20:52 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D2.0038D4EA ; Fri, 12 Jan 2001 11:20:45 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 12 Jan 2001 11:47:06 +0200 Subject: Re: iSCSI: Connection replacement Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk What I have on my todo list is a "restart connection" (use the X bit) to mean do first a logout and then do the login keeping all the way the same CID. Julo Pierre Labat on 11/01/2001 04:23:20 Please respond to Pierre Labat To: ips@ece.cmu.edu cc: Subject: iSCSI: Connection replacement Julian, Where ===== 2.10 Login command Problem ====== In the case where - the maximum number of connections/session is reached - the initiator is faster than the target to detect a failed TCP connection, the establishement of a new connection to replace the bad one will fail. The Login on the new connection will be rejected by the target because the maximum number of connections is reached and the target has not yet detected that there were a failed connection. In the event of a server adapter failure this problem will almost always happen. As soon as the harware fails, the server open a new connection using a new adapter. The target will not be fast enough to realize the connection is bad. Solution ====== a) in the Login message add a field (RecoverID) to inform the target that this connection is to replace a failed one. The value 0xffff means it is not a connection used to replace an old one. b) After the login phase, the first message sent to the target on this new connection must be a Logout for the failed connection. If not the target close the connection. Regards, Pierre From owner-ips@ece.cmu.edu Fri Jan 12 07:44:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA07664 for ; Fri, 12 Jan 2001 07:44:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CALnI05499 for ips-outgoing; Fri, 12 Jan 2001 05:21:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CAKxa05467 for ; Fri, 12 Jan 2001 05:20:59 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id LAA19150 for ; Fri, 12 Jan 2001 11:20:52 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA236862 for ; Fri, 12 Jan 2001 11:20:52 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D2.0038D4AA ; Fri, 12 Jan 2001 11:20:44 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 12 Jan 2001 11:33:00 +0200 Subject: RE: iSCSI : Concerns regarding DataRN in READ Data PDU. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk This feature was meant for long lasting operations (tons of data) that could then recover with ease. As for complexity... it is local counter. If that is not available all long lasting commands will have to provide their own checkpointing mechanisms or there will be no long-lasting commands. Julo "VOIGT,DOUG (HP-Boise,ex1)" on 11/01/2001 03:11:41 Please respond to "VOIGT,DOUG (HP-Boise,ex1)" To: IPS Reflector cc: Subject: RE: iSCSI : Concerns regarding DataRN in READ Data PDU. It seems to me that this feature is redundant with TCP's own delivery guarantees, in an effort to minimize work loss in the event of TCP connection failure. I too believe this level of complexity and potential normal performance impact compared with accelerated recovery from improbable error events is not likely to yield a net gain. Is recoverable TCP connection loss more common than disk array controller failure? Since different controllers in the same array are (quite reasonably) viewed as different targets (pending SAM change), iSCSI level failover won't compensate for controller failure. This example leads me to suggest that the benefit we are shooting for has a high likelihood of being overshadowed by other failure modes whose recovery is less graceful. Doug Voigt > -----Original Message----- > From: Santosh Rao [mailto:santoshr@cup.hp.com] > Sent: Monday, January 08, 2001 9:03 PM > To: IPS Reflector > Subject: iSCSI : Concerns regarding DataRN in READ Data PDU. > > > Julian/All, > > Can anybody comment on the true benefit that the DataRN > feature provides > that justifies the complexity and performance issues it introduces to > the protocol ? > > I see the following issues with DataRN : > > 1) It adds a performance penalty in that on every READ Data PDU > targets will have to initialize one additional 32-bit word in the > header. > > 2) Extra traffic on the outbound context from initiators sending > NOP-OUTs to acknowledge the DataRNs. > > 3) Performance path checks for the "P" bit on every READ PDU. > (Alternatively, checks for the DataNumber field from the Login > Text Keys). > > 4) Implementing DataRNs per task is too short-lived to provide any > real benefit. Implementing them per connection or session can result > in quick use up of the 32-bit DataRN space (since each frame consumes > one DataRN), adding additional complexity to the target code > to use the > "P" > bit appropriately. > > 5) Further, "DataNumber", which is the size of the un-acknowledged > DataRN window is negotiated at login time, whereas this is really > desired to be a dynamic variable based on the target's > current I/O load > and the number of initiators speaking to it. Thus, DataRN can end up > being > under-utilized resulting in too much NOP-OUT traffic, or it can be > over-utilized resulting in too much usage of the "P" bit in the READ > PDUs, which, again, can cause too much NOP-OUT traffic. > > 6) Considering that the DataRN generation and acknowledgement are > functions > that would typically be in SCSI hardware assist engines and there is a > need to > keep these as simple as possible, adding such features in the > spec means > > that these SCSI assist engines MUST implement DataRN generation > and acknowledgement [since initiators have no way to turn it off if a > target decides to use it] > > What is the true benefit of this feature ? Is it intended to > be used as > : > > a) Will allow targets to do partial freeing of their memory buffers as > and when DataRNs are received ? > > b) Will allow targets to send back only the un-acknowledged data (as > seen from the ExpDataRN received on the last NOP-OUT from the > initiator) > when the initiators retry commands with the "Retry" bit set ? > > If the benefit intended is (a), then, in order to derive this benefit, > targets will have to advertise their DataNumber login key to be < the > avg. number of frames transmitted per READ I/O. Having a DataNumber > > the avg. number of frames per READ I/O implies the life of > the task ends > [and buffers for the task get released] before the initiator sends > NOP-OUT, thereby, defeating benefit (a). > > If the benefit intended is (b) [and i do not see this > explicitly stated > in the draft], then, this is a dangerous assumption which can lead to > data corruption issues. > > When commands are retried with the "Retry" bit set, are > targets allowed > to send back partial data based on the last ExpDataRN they > received ? If > so, this can have multiple issues like : > > o complexity in handling I/O underrun cases for the retried command > which only saw partial data transfer from the target. > > o If the target only sent back partial data based > on its last ExpDataRN received, then, it can open up > possible data > > corruption windows. > > All in all, I believe that the DataRN is adding too much > complexity and > potential performance impact to iSCSI for the benefits it may provide. > > Moreover and most importantly, the draft does not give initiators any > control over the usage of this feature and if a target decides to use > this feature, initiators will have to support it. This exposes the > initiators to all of the above issues without being able to turn off > this feature. > > I would like to therefore ask that : > > either, > a) DataRN support be removed from the iSCSI draft. > > or > b) The support for DataRN be negotiated at login time giving initators > control over this feature and allowing them to disable this feature. > This will allow SCSI hardware assist engines to skip implementation of > this functionality and their associated software can merely turn off > DataNumber in the login negotiation. > > Thanks, > Santosh > > > > From owner-ips@ece.cmu.edu Fri Jan 12 07:45:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA07685 for ; Fri, 12 Jan 2001 07:45:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CAKcN05450 for ips-outgoing; Fri, 12 Jan 2001 05:20:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CAKaa05445 for ; Fri, 12 Jan 2001 05:20:36 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id LAA117016 for ; Fri, 12 Jan 2001 11:20:30 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA232792 for ; Fri, 12 Jan 2001 11:20:30 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D2.0038CC11 ; Fri, 12 Jan 2001 11:20:22 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 12 Jan 2001 07:54:35 +0200 Subject: Re: iSCSI: target resource "leak"/logout command Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Pierre, You are correct in bot the issue and the solution. Will fix in 04 (it came too late for 03). Thanks, Julo Pierre Labat on 09/01/2001 12:15:59 Please respond to Pierre Labat To: ips@ece.cmu.edu cc: Subject: iSCSI: target resource "leak"/logout command Julian, There is a corner case where - a [group of] command completion[s] will never be acked (ExpStatRN) - the corresponding task[s] are not "retried" because they are perfectly completed from the initiator point of view and - the session continues to work fine. Thus the resource for the task[s] may never be released on the target (the target waits for ExpStatRN acking the completion before releasing task resources) because on the target side the ack of the completion is not received and on the initiator will not retry the command. A resource (memory) "leak" happens on the target. First i described the scenario, then the solution. Scenario ======= Session with 2 TCP connections 1 and 2 1) A command completion (StatRN=10) is received by the initiator (over cx 1) for the command 5. 2) The initiator sends back ExpStatRN=11 with a command over TCP cx 1 3) The TCP cx 1 drops and the command 5 (so ExpStatRN=11) will never make it to the target. 4) The command 5 is not retried on the connection 2 because for the initiator it is perfectly completed. 5) The target never gets the completion acked and doesn't know when to release resources. Solution ======= - Add the field ExpStatRN in the logout command. - The initiator puts in this field the last value of ExpStatRN of the failed cx. (Not the ExpStatRN value of the connection used to transport the logout) - The target when receiving the logout, reads ExpStatRN and is able to free up the resources. Regards, Pierre From owner-ips@ece.cmu.edu Fri Jan 12 07:45:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA07696 for ; Fri, 12 Jan 2001 07:45:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CAKhR05456 for ips-outgoing; Fri, 12 Jan 2001 05:20:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CAKaa05444 for ; Fri, 12 Jan 2001 05:20:36 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id LAA143394 for ; Fri, 12 Jan 2001 11:20:30 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA232794 for ; Fri, 12 Jan 2001 11:20:30 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D2.0038CBC2 ; Fri, 12 Jan 2001 11:20:21 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 12 Jan 2001 07:23:17 +0200 Subject: RE: iFCP as an IP Storage Work Item Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Josh, iFCP as way to keep your investment in FCP stacks is a very weak argument. FCP stacks are not that stable neither that prevalent (there is none in the most widespread OS family - Windows). A gateway for a single device should be the exception rather than the rule. I can support it as a work item ONLY if it plays a real gateway role and can coexist with FCIP is some synergistic fashion. As a end-to-end proposal is has little value IMHO. Julo Joshua Tseng on 09/01/2001 07:39:18 Please respond to Joshua Tseng To: Venkat Rangan cc: "Ips@Ece. Cmu. Edu" Subject: RE: iFCP as an IP Storage Work Item Venkat, > > Josh, > > Thanks for the clarification that iFCP is only presented as a gateway > protocol. The one comment we would make is that we have FC to > SCSI gateways > already in place, without the need for any standards body > standardizing a > new protocol. The function of the gateway is defined by the > standards for > the two protocols being "connected", and gateway details are left as > implementation details. Actually, what I meant by gateway protocol is that it is a protocol spoken by gateways. It doesn't specify anything about how to implement the protocol, which is internal to the gateway device as you point out. iFCP is the protocol that shows up on the IP network when at least two iFCP gateways transport storage data between them. > > On another note, it should be feasible to build a gateway > that receives FCP > frames from an N_Port or NL_Port of a SCSI Initiator and map > the FCP frames > into iSCSI frames. The frames are sent on an IP interface and > routed by a > normal IP network and another gateway reconverts the iSCSI > PDUs back to FCP > frames and sends them to the target. You will notice that > this does not > require any routing in the FC Plane and accomplishes the same > end goals as > iFCP. Also, this does not require any further standards work, > besides the > usual FCP, iSCSI and related naming protocols. This also > provides the same > scalability of number of nodes on the network, because the > conversion from > locally significant S_ID and D_ID to iSCSI IP addresses can > be done, with > help from a standardized naming effort such as iSNS. Yes, I agree. But this is a different gateway which has nothing to do with an iFCP gateway. Definitely these types of gateways will be needed as we transition to iSCSI. The new iSNS draft has a diagram showing both iSCSI-FC gateways and iFCP gateways. They have different roles. > > Based on these, we believe the need for IP Storage working > group to pick up > iFCP as a work item is reduced. hmmm....you lost me here. An iFCP gateway has nothing to do with iSCSI. They are completely separate. In order to use iSCSI, you need one of the following: 1) Two iSCSI devices 2) One iSCSI device, one FC device, and one iSCSI-FC gateway (which you described above) The situation of two FC devices and two iSCSI-FC gateways is clearly not a design objective of iSCSI. Of course it can be done, but we believe iFCP is clearly the best solution here. Fibre Channel has its issues, but one thing is certain: the FCP driver stacks are very stable and can provide excellent performance for storage applications. But the drawback is FC's networking capabilities leave much to be desired. On the other hand, IP networking capabilities are quite stable and work exceedingly well, but the iSCSI driver stack does not exist yet. So, iFCP's objective is to marry the best of both worlds--to take the existing stable, high-performance driver stacks of Fibre Channel and directly internetwork their individual N_PORTs using TCP/IP. Therefore, iFCP is an opportunity to leverage the existing and proven Fibre Channel driver stacks and combine them with the capabilities that IP networking can provide. Until the day we have stable iSCSI driver stacks everywhere, this may prove to be an attractive alternative for many end users. Another comparison I liken it to is the need for NAT until we have IPv6 everywhere. Josh > > Regards, > > Venkat Rangan > Rhapsody Networks Inc. > http://www.rhapsodynetworks.com > > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Joshua Tseng > Sent: Thursday, January 04, 2001 10:54 AM > To: Ips@Ece. Cmu. Edu > Subject: RE: iFCP as an IP Storage Work Item > > > I don't want to stifle any creative technical discussion here, > but I feel the need to remind everybody that iFCP is positioned > as a gateway technology only. While the thought of "native" > iFCP HBA's might be interesting, this discussion is > completely irrelevant with regard to whether iFCP should > or should not become an IPS work item. iFCP is being proposed > as an IPS work item purely on its merits as a gateway technology. > > Regards, > Josh > > > -----Original Message----- > > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > > Sent: Thursday, January 04, 2001 5:47 AM > > To: 'ips@ece.cmu.edu' > > Subject: FW: iFCP as an IP Storage Work Item > > > > > > > > > > -----Original Message----- > > From: Stephen Byan > > Sent: Thursday, January 04, 2001 8:40 AM > > To: 'Bill Terrell' > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > It's all the FC stuff that lets iFCP work over an unreliable > > data transport > > like UDP. It's redundant when running over TCP/IP. > > > > Regards, > > -Steve > > > > > -----Original Message----- > > > From: Bill Terrell [mailto:terrell@troikanetworks.com] > > > Sent: Wednesday, January 03, 2001 6:10 PM > > > To: 'Stephen Byan' > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > >The downside of this advantage is that native iFCP > devices would be > > > burdened > > > >with greater complexity and cost. I therefor think iFCP > > > should not be an IP > > > >Storage work item. > > > > > > > >Regards, > > > >-Steve > > > > > > How is a native iFCP endpoint (initiator or target) more > > > complex or costly > > > than an iSCSI native endpoint? What are the specific > > > difficulties inherent > > > to native iFCP devices versus native iSCSI devices? > > > > > > Bill > > > > > > From owner-ips@ece.cmu.edu Fri Jan 12 07:45:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA07707 for ; Fri, 12 Jan 2001 07:45:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CALjn05491 for ips-outgoing; Fri, 12 Jan 2001 05:21:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CAKxa05469 for ; Fri, 12 Jan 2001 05:20:59 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id LAA214618 for ; Fri, 12 Jan 2001 11:20:52 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA236866 for ; Fri, 12 Jan 2001 11:20:52 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D2.0038D42C ; Fri, 12 Jan 2001 11:20:43 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 12 Jan 2001 11:14:14 +0200 Subject: RE: iSCSI: Logout for a Session Mime-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=OTUu7wxOIYBLXab0qFXQ2wC6reVtvh7XGJjqsJfNyRmGHNtN74ZtEdlt" Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=OTUu7wxOIYBLXab0qFXQ2wC6reVtvh7XGJjqsJfNyRmGHNtN74ZtEdlt Content-type: text/plain; charset=us-ascii Content-Disposition: inline Not necessarily. After a while yes but for a brief interval you may want to open another connection and recover. Julo "Grun, Paul" on 11/01/2001 00:16:46 Please respond to "Grun, Paul" To: "'Matt Wakeley'" , IPS Reflector cc: Subject: RE: iSCSI: Logout for a Session --0__=OTUu7wxOIYBLXab0qFXQ2wC6reVtvh7XGJjqsJfNyRmGHNtN74ZtEdlt Content-type: text/plain; charset=iso-8859-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Isn't a session terminated when the last TCP connection is closed? -paul Paul Grun Intel Corporation Enterprise=A0Platform Group Fabric Components Division (503) 677-6768 paul.grun@intel.com > -----Original Message----- > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > Sent: Wednesday, January 10, 2001 12:59 PM > To: IPS Reflector > Subject: iSCSI: Logout for a Session > > > Throughout the iSCSI document, it talks about the logout > command closing a TCP > connection. Especially for the purposes of error recover to > terminate a TCP > connection and "clean up" so that another TCP connection can > be established to > continue the session. There seems to be no way to indicate > that a session is > being terminated. > > There should be a flag in the logout command that indicates > that the *session* > is being terminated (as well as the TCP connection). > > -Matt Wakeley > Agilent Technologies > = --0__=OTUu7wxOIYBLXab0qFXQ2wC6reVtvh7XGJjqsJfNyRmGHNtN74ZtEdlt-- From owner-ips@ece.cmu.edu Fri Jan 12 07:46:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA07718 for ; Fri, 12 Jan 2001 07:46:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CAMiS06786 for ips-outgoing; Fri, 12 Jan 2001 05:22:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CALsa05502 for ; Fri, 12 Jan 2001 05:21:54 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id LAA10368 for ; Fri, 12 Jan 2001 11:21:48 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA183214 for ; Fri, 12 Jan 2001 11:21:48 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D2.0038CF80 ; Fri, 12 Jan 2001 11:20:31 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 12 Jan 2001 09:10:39 +0200 Subject: Re: iSCSI : Further review feedback on latest iSCSI-02.txt draft. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Well then Task does not exist will get all enough differentiation. If you could not agree with reject (not enough differentiation) another use can have difficulty in accepting the other extreme - completed when a task is really not found. Julo Pierre Labat on 10/01/2001 01:00:01 Please respond to Pierre Labat To: ips@ece.cmu.edu cc: Subject: iSCSI : Further review feedback on latest iSCSI-02.txt draft. Julian >>>>>>>>> Section 2.6. SCSI Task Management Response ================================== The referenced Task Tag field can be removed since the "No Task Found" response error is not valid. (A Target should respond with an accept for an Abort Task request specifying an invalid Initator Task Tag). [js]Different people have different opinions on this - even with HP - talk to Pierre! [/js] <<<<<<<<< We are in agreement, it is just a point of terminology. Some month ago, we had a discussion on the IPS about what to do in case the target doesn't find the task to abort. I requested that the Abort response returns "OK" (something different than Function rejected). You added the value "Task not found". But "Function complete" can works too. Regards, Pierre From owner-ips@ece.cmu.edu Fri Jan 12 07:46:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA07729 for ; Fri, 12 Jan 2001 07:46:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CAMdL06750 for ips-outgoing; Fri, 12 Jan 2001 05:22:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CALta05505 for ; Fri, 12 Jan 2001 05:21:55 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id LAA05614; Fri, 12 Jan 2001 11:21:48 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA112582; Fri, 12 Jan 2001 11:21:47 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D2.0038CFDE ; Fri, 12 Jan 2001 11:20:32 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: "Victor Firoiu" cc: ips@ece.cmu.edu Message-ID: Date: Fri, 12 Jan 2001 09:30:10 +0200 Subject: Re: Performance of iSCSI, FCIP and iFCP Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Victor, Thanks for your excellent analysis. We became also aware about your paper on TCP performance in face of failures. Using effectively links was very much on our minds when advocating multiple connections but so was also the need to use more than a link. I have to admit that we did not have the model to support numerically our contention that multiple links will behave better in the presence of errors and performance will degrade more gracefully even on congestion. Regards, Julo From owner-ips@ece.cmu.edu Fri Jan 12 07:46:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA07741 for ; Fri, 12 Jan 2001 07:46:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CALlR05494 for ips-outgoing; Fri, 12 Jan 2001 05:21:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CAKwa05464 for ; Fri, 12 Jan 2001 05:20:58 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id LAA30842 for ; Fri, 12 Jan 2001 11:20:51 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA236856 for ; Fri, 12 Jan 2001 11:20:51 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D2.0038D573 ; Fri, 12 Jan 2001 11:20:46 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 12 Jan 2001 11:58:46 +0200 Subject: Re: iSCSI : On the subject of R2T and Task Tags. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk That is taking a narrow view on the capabilities of implementers and protocol analyzer designers. Another viewpoint is have a minimalist design - do no specify what you don't have to! Julo Santosh Rao on 11/01/2001 09:20:20 Please respond to Santosh Rao To: venkat@rhapsodynetworks.com (Venkat Rangan) cc: dotis@sanlight.net (Douglas Otis), matt_wakeley@agilent.com (Matt Wakeley), ips@ece.cmu.edu (IPS Reflector) Subject: Re: iSCSI : On the subject of R2T and Task Tags. The draft must provide for seperate fields to track the task and the R2T along the lines of RX_ID and SEQ_ID in Fibre Channel. Implementations are free to use them as they please and must use 0xFFFFFFFF as their un-used values in these fields. The advantage of defining the fields explicitly in the header as opposed to allowing targets to vertically encode a portion of the tag field for tracking the R2T is that these standard fields are interpret-able by iSCSI Analyzers and will make debugging and tracking the sequence of the I/O easier than implementation specific vertical encoding into a single field. Merging the 2 into a single field is akin to attempting to merge the OX_ID, SEQ_ID and SEQ_CNT fields of FC into a single field and allowing implementation specific vertical encoding. Standard defined fields make for easier introp verification and debugging. - Santosh > > Doug, > > The draft make should not make any statements about how a target chooses to > assign Target Task Tags. Some may choose to assign it per R2T, others may > choose to assign it per task, based on how they choose to implement the > target. Since the initiator is required to reflect it, I do not believe > there are any interoperability issues. If an implementation wishes to do a > Task-SubTask breakdown, they can choose to partition the 32 bits into > portions that can be used for identifying resources within the target. > > Venkat Rangan > Rhapsody Networks Inc. > http://www.rhapsodynetworks.com > > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Douglas Otis > Sent: Wednesday, January 10, 2001 5:27 PM > To: Matt Wakeley; IPS Reflector > Subject: RE: iSCSI : On the subject of R2T and Task Tags. > > > Matt, > > With independent error checking requiring an additional level of error > handling requesting repeated reads, there could be old responses combined > with the retry from failed data digests. (If requesting overlapped reads are > prohibited, retry is an exception.) Without some means of identifying which > retry phase data is associated, a complete set would be difficult to > confirm. As order of each data portion is not mandated and these transfers > themselves may overlap, byte counts can not reveal completion. Should does > not mean 'must.' > > Doug > > > Pierre Labat wrote: > > > > > I agree that the draft must not impose to the target to have a > > different tag > > > for each R2T (it is up to the target implementation to decide). > > I disagree > > > to impose a unique target tag for the whole IO. It imposes the > > target to do > > > score boarding > > > to know when all the data have been received. Letting the > > possiblility to the > > > target > > > to have one tag per R2T (or add a subtag identifying the R2T as > > proposed Santosh) > > > > > > allows the target to dispense with the score boarding. The > > target knows that all > > > the > > > data has been received when a data PDU with the "F" bit set has > > been received > > > for each R2T. > > > > There is no need to do scoreboarding. Just count the number of bytes > > received. If they all add up, then you've received all the data. > > I for one > > would not rely on a PDU with the "F" bit set. I would want to > > cross check it > > with the correct number of bytes. > > > > > > > > > > Regards, > > > > > > Pierre > > > > > > > > > > > > > -Matt Wakeley > > Agilent Technologies > > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Fri Jan 12 07:49:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA07768 for ; Fri, 12 Jan 2001 07:49:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CAKfq05453 for ips-outgoing; Fri, 12 Jan 2001 05:20:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CAKaa05443 for ; Fri, 12 Jan 2001 05:20:36 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id LAA143392 for ; Fri, 12 Jan 2001 11:20:29 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA232790 for ; Fri, 12 Jan 2001 11:20:29 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D2.0038CC57 ; Fri, 12 Jan 2001 11:20:23 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 12 Jan 2001 07:38:09 +0200 Subject: Re: iSCSI: Abort task Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Pierre, I am not sure that I follow completely (and accurately) your line of thought but if you have a multiple connection session the command numbering is meant to offer a total ordering. Any of the scenarios you mention can't happen unless the target decides to deliver commands to the "SCSI layer" out of order. And even if it does deliver command out of order (there might be good reasons for it) it should build a "sync barrier" for those commands that mandate ordering. This type of problem exist also in FCP (on a single connection!) and I assume that the thinking there is to use the per LU command ordering to avoid task management PDUs (like clear a queue) arriving before the commands they are referring to. I hope this clarifies the issue. Julo Pierre Labat on 09/01/2001 11:39:53 Please respond to Pierre Labat To: ips@ece.cmu.edu cc: Subject: iSCSI: Abort task Julian, It seems that we need to specify more the "Abort task" to avoid some problems. Below are 3 points we should add in the spec: 1) An "Abort task" must be sent on the TCP connection that was used to send the command. This does not apply if the connection used for the command has been "logged out". In this case of course, the "Abort task" can be send on another connection and it will not hurt. It is to avoid that a command that was delayed in the network, finally make it after the "Abort task" as been processed. 2) When the initiator receives the "Abort task" response, if the ExpCmdRN returned in the "Abort task" response is <= CmdRN of the aborted task, the initiator must re-use this CmdRN for another task (any). 3) The target when receiving an "Abort task" for a task whose the CmdRN (retreived from the initiator task tag) is >= ExpCmdRN must not bridge this CmdRN. I mean by bridge: allow ExpCmdRN to pass the CmdRN value of the aborted task. It will be up to the initiator to send another command with the same CmdRN in order to allow the target to move ExpCmdRN over the CmdRN value of the aborted task. "3)" Is to prevent to following scenario: Two TCP connections 1 and 2 in the session ExpCmdRN = 2 a) the command with CmdRN = 3 is sent over connection1 b) the command with CmdRN = 4 is sent over connection 2 c) the command CmdRN = 4 reaches the target d) the initiator aborts the command 4 and bridges CmdRN=4 e) the target receives the command 3, and as CmdRN=4 is bridged, advances ExpCmdRN to 5 f) the initiator sends another command (not related with the aborted one) with CmdRN=4 g) the target drops this command because CmdRN is less than ExpCmdRN. "2)" needs to exist because if the initiator doesn't re-use the CmdRN the target will deadlock because ExpCmdRN will be stuck behind the CmdRN of the aborted command. An alternative to the points 2) and 3) would be the initiator not re-using the CmdRN of the aborted commands. But this requires the target to bridge all the CmdRNs of the Aborted tasks (need extra states) and anyway there is a scenario where it doesn't work. Hence this is a bad alternative. Scenario where it doesn't work: Session with 2 connections 1 and 2. ExpCmdRN=1 a) Command with CmdRN=4 is sent over connection1 b) Connection 1 drops and the command 4 will never make it to the target. c) A logout for the connection 1 is sent over connection 2 d) The initiator sends an "Abort task" (for the task CmdRN=4) over the connection 2 e) The target, from the referenced initiator task tag cannot find the CmdRN of the aborted task because the initiator task tag corresponds to nothing because the command has never been received. Hence the target is not able to bridge CmdRN=4 f) As the initiator does not re-use CmdRN=4 the target will deadlock when ExpCmdRN will be 4. Regards, Pierre From owner-ips@ece.cmu.edu Fri Jan 12 10:32:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA10665 for ; Fri, 12 Jan 2001 10:32:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CEBmh18848 for ips-outgoing; Fri, 12 Jan 2001 09:11:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CEBMa18834 for ; Fri, 12 Jan 2001 09:11:23 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id PAA17332 for ; Fri, 12 Jan 2001 15:11:15 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id PAA177180 for ; Fri, 12 Jan 2001 15:09:53 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D2.004DCCE2 ; Fri, 12 Jan 2001 15:09:47 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 12 Jan 2001 16:05:31 +0200 Subject: Re: iSCSI: Security Comments Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua, Thanks for your fast reading. -Consolidation? I am not sure... -Encryption is in (as IPsec - see the appendix) and we are seriously considering reviving TLS - that was previously in and subsuming (relegating) part of the clutter to TLS -Authentication is assumed to be accomplished by the more complex digests by including in the hash a key using the Secret negotiated -page 77 - a vestige from a working draft (will fix in 04) -The exchange examples have some errors that survived my review> I've fixed them already -I've added your comment on WWUI of the note instead of OS+ (I assume that that reflects the whole NDT position) Thanks, Julo Joshua Tseng on 12/01/2001 02:04:56 Please respond to Joshua Tseng To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: iSCSI: Security Comments Julian, The following are security comments to your new -03.txt version of iSCSI: 1) There are several security sections (4.2, 4.3, and 7.) and they should be consolidated into one section. 2) Did we come to a consensus that no encryption was to be included in iSCSI? If yes, then section 7.2.2.4 needs to be deleted. If no, then I now do suggest encryption be delegated to IPSec or TLS. Both of these mechanisms are handled at a lower layer and this allows leverage of existing h/w and s/w implementations. 3) I would suggest that iSCSI consider adding an optional authentication block in each iSCSI PDU. If this is of interest I can work with you further on it. Specific comments to security section Appendix A: a) pg 77, the following: "- Public key algorithm (InitPublicKey,TargetPublicKey)" needs to be replaced by: "- Public key algorithm (PublicKey)" If a per-iSCSI PDU authentication block is to be added, perhaps that can be added to this list with something like: "- PDU-Authentication (AuthAlgorithm:)" Where AuthAlgorithm is the signature algorithm used to sign the authentication block. We would also need a description of new text commands such as: InitDHValue: and TargetDHValue:, which would list parameters for the Diffie Hellman exchange to calculate a shared secret key used for AuthAlgorithm. b) pg 83, see the following text and suggested modifications: "The next example is a public-key authentication. The initiator authenticates itself to the target and no keys are exchanged: " - Need to delete the part "...and no keys are exchanged: ". "If the user was not confirmed, the target sends a login response message with "login reject" to the initiator. Else, it can send a login response with "login accept" and MAY attach a secret: " - Need to delete the part "...and MAY attach a secret:". "The next example is another public-key authentication. The initiator authenticates itself to the target. The target authenticates itself to the initiator and key are exchanged: " - Delete the part "...and key are exchanged: ". " T->Text StartSecure:HERE secret: " - Delete the part "...secret: ". No secret keys should be exchanged in this phase since the login is authenticated only, not encrypted. If secret keys are needed for a PDU authentication block, then Diffie-Hellman should be used using the above text command. c) pg 83, I suggest changing the following text: NB - where the blob stands for the digitally signed hash of the packet header, the user (presumably some form of machine+OS+session name or a certificate issued by a certificate authority) the target salt and using the appropriate digital signature algorithm (DSS). to the following: "...where the blob stands for the digitally signed hash of the iSCSI PDU header, the WWUI of the iSCSI node being authenticated, and the salt provided by the authenticating node, using the appropriate digital signature method (DSS or DSA)." d) pg 84, suggest modifying the following text: where the blob stands for the digitally signed hash of the packet header, the user (presumably some form WWUID name or certificate issued by a certificate authority) the initiator salt and using the appropriate digital signature algorithm (DSS). The target also send a suggested key encrypted with the initiator public key. to the following: "...where the blob stands for the digitally signed hash of the iSCSI PDU header, the WWUI of the iSCSI node being authenticated, and the salt privided by the authenticating node, using the appropriate digital signature method (DSS or RSA). - Delete "Secret:key" from the following: "T-> Text Authenticate:user,blob Secret:key" In the following: where the blob stands for the digitally signed hash of the packet header, the user (presumably some form WWUID name or certificate issued by a certificate authority) the initiator salt and using the appropriate digital signature algorithm (DSS). The target also send a suggested key encrypted with the initiator public key. - delete the last sentence "The target also send....". That's all for now. Josh Tseng From owner-ips@ece.cmu.edu Fri Jan 12 10:38:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA10713 for ; Fri, 12 Jan 2001 10:38:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CDbsp18096 for ips-outgoing; Fri, 12 Jan 2001 08:37:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from exchange-1.xiotech.com (ftp.xiotech.com [209.46.118.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CDbKa18068 for ; Fri, 12 Jan 2001 08:37:20 -0500 (EST) Message-ID: From: "Peglar, Robert" To: "'julian_satran@il.ibm.com'" Cc: "'ips@ece.cmu.edu'" Subject: RE: iSCSI : Concerns regarding DataRN in READ Data PDU. Date: Fri, 12 Jan 2001 07:36:33 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a very important feature indeed. Think of block-sequential operations on tens- (or even hundreds-) of TB of data. Think of sessions that perform in-band continuous management functions; they have to live forever, by definition. Think of sessions between virtual initiators and virtual targets. Forever, potentially. Thank you, Rob Rob Peglar Director, Storage Architecture XIOtech Corporation (314) 308-6983 > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Friday, January 12, 2001 3:33 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI : Concerns regarding DataRN in READ Data PDU. > > > > > This feature was meant for long lasting operations (tons of data) that > could then recover > with ease. As for complexity... it is local counter. If that is not > available all long lasting commands will have to provide their own > checkpointing mechanisms or there will be no long-lasting commands. > > Julo > > "VOIGT,DOUG (HP-Boise,ex1)" on 11/01/2001 03:11:41 > > Please respond to "VOIGT,DOUG (HP-Boise,ex1)" > > To: IPS Reflector > cc: > Subject: RE: iSCSI : Concerns regarding DataRN in READ Data PDU. > > > > > It seems to me that this feature is redundant with TCP's own delivery > guarantees, in an effort to minimize work loss in the event of TCP > connection failure. I too believe this level of complexity > and potential > normal performance impact compared with accelerated recovery from > improbable > error events is not likely to yield a net gain. > > Is recoverable TCP connection loss more common than disk > array controller > failure? Since different controllers in the same array are (quite > reasonably) viewed as different targets (pending SAM change), > iSCSI level > failover won't compensate for controller failure. This > example leads me to > suggest that the benefit we are shooting for has a high > likelihood of being > overshadowed by other failure modes whose recovery is less graceful. > > Doug Voigt > > > -----Original Message----- > > From: Santosh Rao [mailto:santoshr@cup.hp.com] > > Sent: Monday, January 08, 2001 9:03 PM > > To: IPS Reflector > > Subject: iSCSI : Concerns regarding DataRN in READ Data PDU. > > > > > > Julian/All, > > > > Can anybody comment on the true benefit that the DataRN > > feature provides > > that justifies the complexity and performance issues it > introduces to > > the protocol ? > > > > I see the following issues with DataRN : > > > > 1) It adds a performance penalty in that on every READ Data PDU > > targets will have to initialize one additional 32-bit word in the > > header. > > > > 2) Extra traffic on the outbound context from initiators sending > > NOP-OUTs to acknowledge the DataRNs. > > > > 3) Performance path checks for the "P" bit on every READ PDU. > > (Alternatively, checks for the DataNumber field from the Login > > Text Keys). > > > > 4) Implementing DataRNs per task is too short-lived to provide any > > real benefit. Implementing them per connection or session can result > > in quick use up of the 32-bit DataRN space (since each > frame consumes > > one DataRN), adding additional complexity to the target code > > to use the > > "P" > > bit appropriately. > > > > 5) Further, "DataNumber", which is the size of the un-acknowledged > > DataRN window is negotiated at login time, whereas this is really > > desired to be a dynamic variable based on the target's > > current I/O load > > and the number of initiators speaking to it. Thus, DataRN can end up > > being > > under-utilized resulting in too much NOP-OUT traffic, or it can be > > over-utilized resulting in too much usage of the "P" bit in the READ > > PDUs, which, again, can cause too much NOP-OUT traffic. > > > > 6) Considering that the DataRN generation and acknowledgement are > > functions > > that would typically be in SCSI hardware assist engines and > there is a > > need to > > keep these as simple as possible, adding such features in the > > spec means > > > > that these SCSI assist engines MUST implement DataRN generation > > and acknowledgement [since initiators have no way to turn > it off if a > > target decides to use it] > > > > What is the true benefit of this feature ? Is it intended to > > be used as > > : > > > > a) Will allow targets to do partial freeing of their memory > buffers as > > and when DataRNs are received ? > > > > b) Will allow targets to send back only the un-acknowledged data (as > > seen from the ExpDataRN received on the last NOP-OUT from the > > initiator) > > when the initiators retry commands with the "Retry" bit set ? > > > > If the benefit intended is (a), then, in order to derive > this benefit, > > targets will have to advertise their DataNumber login key > to be < the > > avg. number of frames transmitted per READ I/O. Having a > DataNumber > > > the avg. number of frames per READ I/O implies the life of > > the task ends > > [and buffers for the task get released] before the initiator sends > > NOP-OUT, thereby, defeating benefit (a). > > > > If the benefit intended is (b) [and i do not see this > > explicitly stated > > in the draft], then, this is a dangerous assumption which > can lead to > > data corruption issues. > > > > When commands are retried with the "Retry" bit set, are > > targets allowed > > to send back partial data based on the last ExpDataRN they > > received ? If > > so, this can have multiple issues like : > > > > o complexity in handling I/O underrun cases for the > retried command > > which only saw partial data transfer from the target. > > > > o If the target only sent back partial data based > > on its last ExpDataRN received, then, it can open up > > possible data > > > > corruption windows. > > > > All in all, I believe that the DataRN is adding too much > > complexity and > > potential performance impact to iSCSI for the benefits it > may provide. > > > > Moreover and most importantly, the draft does not give > initiators any > > control over the usage of this feature and if a target > decides to use > > this feature, initiators will have to support it. This exposes the > > initiators to all of the above issues without being able to turn off > > this feature. > > > > I would like to therefore ask that : > > > > either, > > a) DataRN support be removed from the iSCSI draft. > > > > or > > b) The support for DataRN be negotiated at login time > giving initators > > control over this feature and allowing them to disable this feature. > > This will allow SCSI hardware assist engines to skip > implementation of > > this functionality and their associated software can merely turn off > > DataNumber in the login negotiation. > > > > Thanks, > > Santosh > > > > > > > > > > > From owner-ips@ece.cmu.edu Fri Jan 12 11:49:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA12175 for ; Fri, 12 Jan 2001 11:49:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CEdoM19885 for ips-outgoing; Fri, 12 Jan 2001 09:39:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from exchange-1.xiotech.com (ftp.xiotech.com [209.46.118.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CEdaa19875 for ; Fri, 12 Jan 2001 09:39:36 -0500 (EST) Message-ID: From: "Peglar, Robert" To: "'julian_satran@il.ibm.com'" , ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 12 Jan 2001 08:38:50 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk As far as FCP stacks go, it is a truism that no OS (host) has a native FCP stack. Today. (Stay tuned, though; Blackcomb AS is going to be rather interesting) That's why the HBA rules. It's the entity performing all the translation between what the OSes see as a pure SCSI universe and the real (or virtual) device FC universe. Had we native motherboard FC physical interfaces and OS FCP stacks, we would not need FC HBAs. But, that's hosts. Consider the world of devices for a moment. No one can argue that there are not a plethora of FC devices today, all with FCP stacks by definition. Plus, many FC devices today are not just one FC port but multiple, with multiple WWPN and one WWNN to achieve higher RAS characteristics. So while your statement > iFCP as way to keep your investment in FCP stacks is a very > weak argument. is certainly true for hosts (initiators), it is certainly not true for devices (targets). We even have several current implementations of FC tape devices (viz. the need for FCP-2, amongst other reasons). Some marketing-buzzheads say that for disk, 80% of the bytes placed into enterprise storage in 2000 were FC. Seagate alone produces one million (!) fully assembled disk units per week, a goodly proportion of which are FC. FWIW. In any event, even though your argument is strong for initiators, it is weak for targets. That alone is enough to keep (at least the) debate going over if iFCP is worthy for discussion. If the WG wants to keep iFCP on the back burner, that is one thing, but I must say to quash the debate now and invoke cloture is surely the wrong thing to do now given the IETF way of doing things (rough consensus, working code). There is little doubt that as a draft that iFCP has a long way to go, but that is to be expected. To your point that 'can iFCP co-exist with FCIP', there is no technical reason (or even non-technical) why it cannot. After all, we have had BGP and OSPF co-existing for approaching a decade now. Also, there is no doubt that iFCP is a gateway-oriented proposal, just as there is no doubt that well-written FCP (or FCP-2) device stacks are very reliable. Having said that, I believe that there probably will be more initial implementations of FCIP than iFCP, but that is surely not - among reasonable IETF people - a reason to quash discussion. Thank you, Rob Rob Peglar Director, Storage Architecture XIOtech Corporation (314) 308-6983 > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Thursday, January 11, 2001 11:23 PM > To: ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > > > Josh, > > iFCP as way to keep your investment in FCP stacks is a very > weak argument. > FCP stacks are not that stable neither that prevalent (there > is none in the > most widespread OS family - Windows). > > A gateway for a single device should be the exception rather > than the rule. > > I can support it as a work item ONLY if it plays a real > gateway role and > can coexist with FCIP is some synergistic fashion. > As a end-to-end proposal is has little value IMHO. > > Julo From owner-ips@ece.cmu.edu Fri Jan 12 13:09:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA14213 for ; Fri, 12 Jan 2001 13:09:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CFR5O22837 for ips-outgoing; Fri, 12 Jan 2001 10:27:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CFQfa22823 for ; Fri, 12 Jan 2001 10:26:41 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Fri, 12 Jan 2001 07:26:54 -0800 Message-ID: From: Joshua Tseng To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI: Security Comments Date: Fri, 12 Jan 2001 07:26:05 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, > > Joshua, > > Thanks for your fast reading. > > -Consolidation? I am not sure... > > -Encryption is in (as IPsec - see the appendix) and we are seriously > considering reviving TLS - that was previously in > and subsuming (relegating) part of the clutter to TLS If you are delegating encryption to IPSec, the shared secret should not be exchanged via iSCSI. Rather, ISAKMP and IKE should be leveraged to set up the IPSec SA's, and the entire login process will already have been encrypted by IPSec. IPSec is independent of iSCSI and anything else above IP. Similarly, TLS will require distribution of PK certificates which is outside the scope of iSCSI. My question was whether iSCSI itself will have an encryption mechanism. If it does, I don't see it anywhere. If it doesn't, then section 7.2.2.4 needs to be removed. > > -Authentication is assumed to be accomplished by the more > complex digests > by including in the hash a key using the Secret negotiated If iSCSI needs to negotiate a shared secret key, I recommend using Diffie Hellman to do it. This removes the requirement for each side to have a public/private key pair, since you are relying on other methods than public key. > > -page 77 - a vestige from a working draft (will fix in 04) > > -The exchange examples have some errors that survived my > review> I've fixed > them already > -I've added your comment on WWUI of the note instead of OS+ > (I assume that > that reflects the whole NDT position) > > Thanks, > Julo > > Joshua Tseng on 12/01/2001 02:04:56 > > Please respond to Joshua Tseng > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: iSCSI: Security Comments > > > > > Julian, > > The following are security comments to your new -03.txt version > of iSCSI: > > 1) There are several security sections (4.2, 4.3, and 7.) and > they should be consolidated into one section. > > 2) Did we come to a consensus that no encryption was to be > included in iSCSI? If yes, then section 7.2.2.4 needs to be > deleted. If no, then I now do suggest encryption be delegated > to IPSec or TLS. Both of these mechanisms are handled at a > lower layer and this allows leverage of existing h/w and s/w > implementations. > > 3) I would suggest that iSCSI consider adding an optional > authentication block in each iSCSI PDU. If this is of interest > I can work with you further on it. > > Specific comments to security section Appendix A: > > a) pg 77, the following: > > "- Public key algorithm (InitPublicKey,TargetPublicKey)" > > needs to be replaced by: > > "- Public key algorithm (PublicKey)" > > If a per-iSCSI PDU authentication block is to be added, perhaps > that can be added to this list with something like: > > "- PDU-Authentication (AuthAlgorithm:)" > > Where AuthAlgorithm is the signature algorithm used to sign the > authentication block. > > We would also need a description of new text commands such as: > > InitDHValue: and TargetDHValue:, which would list parameters for the > Diffie Hellman exchange to calculate a shared secret key used for > AuthAlgorithm. > > b) pg 83, see the following text and suggested modifications: > > "The next example is a public-key authentication. The initiator > authenticates itself to the target and no keys are exchanged: " > > - Need to delete the part "...and no keys are exchanged: ". > > "If the user was not confirmed, the target sends a login > response message with "login reject" to the initiator. Else, > it can send a login response with "login accept" and MAY > attach a secret: " > > - Need to delete the part "...and MAY attach a secret:". > > "The next example is another public-key authentication. The initiator > authenticates itself to the target. The target authenticates itself > to the initiator and key are exchanged: " > > - Delete the part "...and key are exchanged: ". > > " T->Text StartSecure:HERE secret: " > > - Delete the part "...secret: ". > > No secret keys should be exchanged in this phase since the login > is authenticated only, not encrypted. If secret keys are needed for > a PDU authentication block, then Diffie-Hellman should be used using > the above text command. > > c) pg 83, > > I suggest changing the following text: > > NB - where the blob stands for the digitally signed hash of > the packet header, the user (presumably some form of > machine+OS+session name or a certificate issued by a > certificate authority) the target salt and using the > appropriate digital signature algorithm (DSS). > > to the following: > > "...where the blob stands for the digitally signed hash of > the iSCSI PDU header, the WWUI of the iSCSI node being authenticated, > and the salt provided by the authenticating node, using > the appropriate digital signature method (DSS or DSA)." > > d) pg 84, suggest modifying the following text: > > where the blob stands for the digitally signed hash of the > packet header, the user (presumably some form WWUID name or > certificate issued by a certificate authority) the initiator > salt and using the appropriate digital signature algorithm > (DSS). The target also send a suggested key encrypted with the > initiator public key. > > to the following: > > "...where the blob stands for the digitally signed hash of the > iSCSI PDU header, the WWUI of the iSCSI node being authenticated, > and the salt privided by the authenticating node, using the > appropriate digital signature method (DSS or RSA). > > - Delete "Secret:key" from the following: > > "T-> Text Authenticate:user,blob Secret:key" > > In the following: > > where the blob stands for the digitally signed hash of the > packet header, the user (presumably some form WWUID name or > certificate issued by a certificate authority) the initiator > salt and using the appropriate digital signature algorithm > (DSS). The target also send a suggested key encrypted with the > initiator public key. > > - delete the last sentence "The target also send....". > > That's all for now. > > Josh Tseng > > > > From owner-ips@ece.cmu.edu Fri Jan 12 14:10:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA15301 for ; Fri, 12 Jan 2001 14:10:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CHJth29953 for ips-outgoing; Fri, 12 Jan 2001 12:19:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CHJ1a29928 for ; Fri, 12 Jan 2001 12:19:01 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id 14EF5CD8 for ; Fri, 12 Jan 2001 09:18:56 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id JAA06737; Fri, 12 Jan 2001 09:18:54 -0800 (PST) Message-ID: <3A5F3C8D.6A92625D@hp.com> Date: Fri, 12 Jan 2001 09:19:09 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI : Concerns regarding DataRN in READ Data PDU. References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > This feature was meant for long lasting operations (tons of data) that > could then recover > with ease. As for complexity... it is local counter. If that is not > available all long lasting commands will have to provide their own > checkpointing mechanisms or there will be no long-lasting commands. Julian, The target can use the TCP acks to know what data reached the initiator. From the TCP ack the target can find the corresponding data in the READ cmd that have been received by the initiator. And it can free the resources for this data and in case of connection failure it can restart from there. It implies that the initiator sends the TCP ack only after the data has been copied out of the adapter (in sever memory or alike). It's why the DATARN doesn't add value, it is because using TCP you can get the same benefit. Regards, Pierre > > > Julo > > "VOIGT,DOUG (HP-Boise,ex1)" on 11/01/2001 03:11:41 > > Please respond to "VOIGT,DOUG (HP-Boise,ex1)" > > To: IPS Reflector > cc: > Subject: RE: iSCSI : Concerns regarding DataRN in READ Data PDU. > > It seems to me that this feature is redundant with TCP's own delivery > guarantees, in an effort to minimize work loss in the event of TCP > connection failure. I too believe this level of complexity and potential > normal performance impact compared with accelerated recovery from > improbable > error events is not likely to yield a net gain. > > Is recoverable TCP connection loss more common than disk array controller > failure? Since different controllers in the same array are (quite > reasonably) viewed as different targets (pending SAM change), iSCSI level > failover won't compensate for controller failure. This example leads me to > suggest that the benefit we are shooting for has a high likelihood of being > overshadowed by other failure modes whose recovery is less graceful. > > Doug Voigt > > > -----Original Message----- > > From: Santosh Rao [mailto:santoshr@cup.hp.com] > > Sent: Monday, January 08, 2001 9:03 PM > > To: IPS Reflector > > Subject: iSCSI : Concerns regarding DataRN in READ Data PDU. > > > > > > Julian/All, > > > > Can anybody comment on the true benefit that the DataRN > > feature provides > > that justifies the complexity and performance issues it introduces to > > the protocol ? > > > > I see the following issues with DataRN : > > > > 1) It adds a performance penalty in that on every READ Data PDU > > targets will have to initialize one additional 32-bit word in the > > header. > > > > 2) Extra traffic on the outbound context from initiators sending > > NOP-OUTs to acknowledge the DataRNs. > > > > 3) Performance path checks for the "P" bit on every READ PDU. > > (Alternatively, checks for the DataNumber field from the Login > > Text Keys). > > > > 4) Implementing DataRNs per task is too short-lived to provide any > > real benefit. Implementing them per connection or session can result > > in quick use up of the 32-bit DataRN space (since each frame consumes > > one DataRN), adding additional complexity to the target code > > to use the > > "P" > > bit appropriately. > > > > 5) Further, "DataNumber", which is the size of the un-acknowledged > > DataRN window is negotiated at login time, whereas this is really > > desired to be a dynamic variable based on the target's > > current I/O load > > and the number of initiators speaking to it. Thus, DataRN can end up > > being > > under-utilized resulting in too much NOP-OUT traffic, or it can be > > over-utilized resulting in too much usage of the "P" bit in the READ > > PDUs, which, again, can cause too much NOP-OUT traffic. > > > > 6) Considering that the DataRN generation and acknowledgement are > > functions > > that would typically be in SCSI hardware assist engines and there is a > > need to > > keep these as simple as possible, adding such features in the > > spec means > > > > that these SCSI assist engines MUST implement DataRN generation > > and acknowledgement [since initiators have no way to turn it off if a > > target decides to use it] > > > > What is the true benefit of this feature ? Is it intended to > > be used as > > : > > > > a) Will allow targets to do partial freeing of their memory buffers as > > and when DataRNs are received ? > > > > b) Will allow targets to send back only the un-acknowledged data (as > > seen from the ExpDataRN received on the last NOP-OUT from the > > initiator) > > when the initiators retry commands with the "Retry" bit set ? > > > > If the benefit intended is (a), then, in order to derive this benefit, > > targets will have to advertise their DataNumber login key to be < the > > avg. number of frames transmitted per READ I/O. Having a DataNumber > > > the avg. number of frames per READ I/O implies the life of > > the task ends > > [and buffers for the task get released] before the initiator sends > > NOP-OUT, thereby, defeating benefit (a). > > > > If the benefit intended is (b) [and i do not see this > > explicitly stated > > in the draft], then, this is a dangerous assumption which can lead to > > data corruption issues. > > > > When commands are retried with the "Retry" bit set, are > > targets allowed > > to send back partial data based on the last ExpDataRN they > > received ? If > > so, this can have multiple issues like : > > > > o complexity in handling I/O underrun cases for the retried command > > which only saw partial data transfer from the target. > > > > o If the target only sent back partial data based > > on its last ExpDataRN received, then, it can open up > > possible data > > > > corruption windows. > > > > All in all, I believe that the DataRN is adding too much > > complexity and > > potential performance impact to iSCSI for the benefits it may provide. > > > > Moreover and most importantly, the draft does not give initiators any > > control over the usage of this feature and if a target decides to use > > this feature, initiators will have to support it. This exposes the > > initiators to all of the above issues without being able to turn off > > this feature. > > > > I would like to therefore ask that : > > > > either, > > a) DataRN support be removed from the iSCSI draft. > > > > or > > b) The support for DataRN be negotiated at login time giving initators > > control over this feature and allowing them to disable this feature. > > This will allow SCSI hardware assist engines to skip implementation of > > this functionality and their associated software can merely turn off > > DataNumber in the login negotiation. > > > > Thanks, > > Santosh > > > > > > > > From owner-ips@ece.cmu.edu Fri Jan 12 14:11:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA15328 for ; Fri, 12 Jan 2001 14:11:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CGTsQ28567 for ips-outgoing; Fri, 12 Jan 2001 11:29:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d06lmsgate-2.uk.ibm.com (d06lmsgate-2.uk.ibm.com [195.212.29.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CGT7a28548 for ; Fri, 12 Jan 2001 11:29:09 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d06lmsgate-2.uk.ibm.com (1.0.0) with ESMTP id QAA140970 for ; Fri, 12 Jan 2001 16:14:46 GMT From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA38778 for ; Fri, 12 Jan 2001 17:21:53 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D2.0059E22A ; Fri, 12 Jan 2001 17:21:46 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 12 Jan 2001 18:17:30 +0200 Subject: RE: iSCSI: Security Comments Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Josh, The IPsec is mentioned rightfully and then on in the appendix you have an (attempt) at a synchronous start of it. In case of IPsec no negotiation is performed at the iSCSI level - although iSCSI authentication is kept separate. Alternatively IPsec can be established before iSCSI and the parties may want only to confirm to each other that it is in place. The secret can be used for other things as I mentioned before. There are 2 new keys specific for IPsec. Julo Joshua Tseng on 12/01/2001 17:26:05 Please respond to Joshua Tseng To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI: Security Comments Julo, > > Joshua, > > Thanks for your fast reading. > > -Consolidation? I am not sure... > > -Encryption is in (as IPsec - see the appendix) and we are seriously > considering reviving TLS - that was previously in > and subsuming (relegating) part of the clutter to TLS If you are delegating encryption to IPSec, the shared secret should not be exchanged via iSCSI. Rather, ISAKMP and IKE should be leveraged to set up the IPSec SA's, and the entire login process will already have been encrypted by IPSec. IPSec is independent of iSCSI and anything else above IP. Similarly, TLS will require distribution of PK certificates which is outside the scope of iSCSI. My question was whether iSCSI itself will have an encryption mechanism. If it does, I don't see it anywhere. If it doesn't, then section 7.2.2.4 needs to be removed. > > -Authentication is assumed to be accomplished by the more > complex digests > by including in the hash a key using the Secret negotiated If iSCSI needs to negotiate a shared secret key, I recommend using Diffie Hellman to do it. This removes the requirement for each side to have a public/private key pair, since you are relying on other methods than public key. > > -page 77 - a vestige from a working draft (will fix in 04) > > -The exchange examples have some errors that survived my > review> I've fixed > them already > -I've added your comment on WWUI of the note instead of OS+ > (I assume that > that reflects the whole NDT position) > > Thanks, > Julo > > Joshua Tseng on 12/01/2001 02:04:56 > > Please respond to Joshua Tseng > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: iSCSI: Security Comments > > > > > Julian, > > The following are security comments to your new -03.txt version > of iSCSI: > > 1) There are several security sections (4.2, 4.3, and 7.) and > they should be consolidated into one section. > > 2) Did we come to a consensus that no encryption was to be > included in iSCSI? If yes, then section 7.2.2.4 needs to be > deleted. If no, then I now do suggest encryption be delegated > to IPSec or TLS. Both of these mechanisms are handled at a > lower layer and this allows leverage of existing h/w and s/w > implementations. > > 3) I would suggest that iSCSI consider adding an optional > authentication block in each iSCSI PDU. If this is of interest > I can work with you further on it. > > Specific comments to security section Appendix A: > > a) pg 77, the following: > > "- Public key algorithm (InitPublicKey,TargetPublicKey)" > > needs to be replaced by: > > "- Public key algorithm (PublicKey)" > > If a per-iSCSI PDU authentication block is to be added, perhaps > that can be added to this list with something like: > > "- PDU-Authentication (AuthAlgorithm:)" > > Where AuthAlgorithm is the signature algorithm used to sign the > authentication block. > > We would also need a description of new text commands such as: > > InitDHValue: and TargetDHValue:, which would list parameters for the > Diffie Hellman exchange to calculate a shared secret key used for > AuthAlgorithm. > > b) pg 83, see the following text and suggested modifications: > > "The next example is a public-key authentication. The initiator > authenticates itself to the target and no keys are exchanged: " > > - Need to delete the part "...and no keys are exchanged: ". > > "If the user was not confirmed, the target sends a login > response message with "login reject" to the initiator. Else, > it can send a login response with "login accept" and MAY > attach a secret: " > > - Need to delete the part "...and MAY attach a secret:". > > "The next example is another public-key authentication. The initiator > authenticates itself to the target. The target authenticates itself > to the initiator and key are exchanged: " > > - Delete the part "...and key are exchanged: ". > > " T->Text StartSecure:HERE secret: " > > - Delete the part "...secret: ". > > No secret keys should be exchanged in this phase since the login > is authenticated only, not encrypted. If secret keys are needed for > a PDU authentication block, then Diffie-Hellman should be used using > the above text command. > > c) pg 83, > > I suggest changing the following text: > > NB - where the blob stands for the digitally signed hash of > the packet header, the user (presumably some form of > machine+OS+session name or a certificate issued by a > certificate authority) the target salt and using the > appropriate digital signature algorithm (DSS). > > to the following: > > "...where the blob stands for the digitally signed hash of > the iSCSI PDU header, the WWUI of the iSCSI node being authenticated, > and the salt provided by the authenticating node, using > the appropriate digital signature method (DSS or DSA)." > > d) pg 84, suggest modifying the following text: > > where the blob stands for the digitally signed hash of the > packet header, the user (presumably some form WWUID name or > certificate issued by a certificate authority) the initiator > salt and using the appropriate digital signature algorithm > (DSS). The target also send a suggested key encrypted with the > initiator public key. > > to the following: > > "...where the blob stands for the digitally signed hash of the > iSCSI PDU header, the WWUI of the iSCSI node being authenticated, > and the salt privided by the authenticating node, using the > appropriate digital signature method (DSS or RSA). > > - Delete "Secret:key" from the following: > > "T-> Text Authenticate:user,blob Secret:key" > > In the following: > > where the blob stands for the digitally signed hash of the > packet header, the user (presumably some form WWUID name or > certificate issued by a certificate authority) the initiator > salt and using the appropriate digital signature algorithm > (DSS). The target also send a suggested key encrypted with the > initiator public key. > > - delete the last sentence "The target also send....". > > That's all for now. > > Josh Tseng > > > > From owner-ips@ece.cmu.edu Fri Jan 12 14:17:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA15425 for ; Fri, 12 Jan 2001 14:17:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CH7vo29612 for ips-outgoing; Fri, 12 Jan 2001 12:07:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CH7Da29592 for ; Fri, 12 Jan 2001 12:07:13 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Fri, 12 Jan 2001 09:07:27 -0800 Message-ID: From: Joshua Tseng To: "Peglar, Robert" , ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 12 Jan 2001 09:06:43 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Rob, Thanks for your good insights. Regarding the following > To your point that 'can iFCP co-exist with FCIP', there > is no technical reason (or even non-technical) why it > cannot. After all, we have had BGP and OSPF co-existing > for approaching a decade now. Also, there is no doubt > that iFCP is a gateway-oriented proposal, just as there is > no doubt that well-written FCP (or FCP-2) device stacks are > very reliable. Having said that, I believe that there probably > will be more initial implementations of FCIP than iFCP, but > that is surely not - among reasonable IETF people - a reason > to quash discussion. I believe the comparison between FCIP and iFCP is more like that of RIP and OSPF, not BGP and OSPF. BGP and OSPF coexist because they perform completely different roles. BGP is used to route between autonomous systems, and OSPF is used for intra-autonomous system routing. The Internet could not possibly exist today without one or the other. RIP provides a subset OSPF's capabilities, just as Murali pointed out that FCIP provides a subset of iFCP's capabilities. Could you do both in the same network? Of course you can, but the chances are if you're using OSPF, you would not need RIP. Does that mean RIP is not viable? Of course not! RIP exists because it is simple and can be used to achieve limited objectives. Josh > > Thank you, > Rob > > Rob Peglar > Director, Storage Architecture > XIOtech Corporation > (314) 308-6983 > > > > -----Original Message----- > > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > > Sent: Thursday, January 11, 2001 11:23 PM > > To: ips@ece.cmu.edu > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > > Josh, > > > > iFCP as way to keep your investment in FCP stacks is a very > > weak argument. > > FCP stacks are not that stable neither that prevalent (there > > is none in the > > most widespread OS family - Windows). > > > > A gateway for a single device should be the exception rather > > than the rule. > > > > I can support it as a work item ONLY if it plays a real > > gateway role and > > can coexist with FCIP is some synergistic fashion. > > As a end-to-end proposal is has little value IMHO. > > > > Julo > From owner-ips@ece.cmu.edu Fri Jan 12 16:30:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA25035 for ; Fri, 12 Jan 2001 16:30:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CId0h02180 for ips-outgoing; Fri, 12 Jan 2001 13:39:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CIc9a02164 for ; Fri, 12 Jan 2001 13:38:11 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0CJlO029618; Fri, 12 Jan 2001 11:48:01 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "ELLIOTT,STEPHEN \(HP-Roseville,ex1\)" , Subject: RE: New iFCP draft Date: Fri, 12 Jan 2001 10:35:06 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <670D895CA648D41198EA00A0C9F2D428038409A9@xrose01.rose.hp.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Elliott, While correct about the need for CRC on Ethernet, IP is more than just Ethernet. The Ethernet-Fibre-Channel-FDDI CRC added is not to improve error detection for Ethernet. While moving through DSLAMs, Switches, Routers, and the like, errors happen without the aid of CRC to catch this event as the CRC is unfortunately stripped and regenerated as the world is not entirely Ethernet-Fibre-Channel-FDDI. This helps explain the Digest added to iSCSI. iFCP could have added a separate CRC to protect their header. FCIP does not add any additional information yet and so they encapsulate the CRC found on Fibre-Channel. This is one area where we just can't get along. Doug > With regards to the new iFCP draft > > 5.3 Trailing iFCP CRC > > This original Fibre Channel CRC shall not be encapsulated into the > iFCP message. Rather, iFCP MAY use a 32-bit field containing a CRC > calculated over the data contained in the frame from the iFCP > AUGMENTED DATA to the iFCP Payload (inclusive). This includes the > Fibre Channel header and payload. The iFCP CRC follows the iFCP > payload, and the CRC algorithm is the same used for the Frame Check > Sequence (FCS) of IEEE 802.3 Ethernet frames. A bit in the iFCP > FLAGS field in the iFCP header indicates the presence or absence of > this 32-bit trailing CRC. > > I would suggest "Rather, iFCP MUST use a 32-bit field containing > a CRC ..." > unless ethernet provides an equivalent CRC elsewhere. The reason > is that I > believe gigabit ethernet uses the same physical layer as fibre channel. > This physical layer is specified for fibre channel to have a data bit loss > rate of 1 in 10^12 or less, and is not "perfect". I think the CRC is > designed to throw out these imperfect frames instead of resulting in data > corruption. Without the CRC, physical layer imperfections within > fibre-channel specificaiton may lead to data corruption. > > Also, I am not intending to show preference for iSCSI or iFCP with this > email. I just was skimming over the standard and thought this feedback > might be helpful. > > Stephen Elliott > > From owner-ips@ece.cmu.edu Fri Jan 12 16:33:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA25098 for ; Fri, 12 Jan 2001 16:33:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CIr1j02594 for ips-outgoing; Fri, 12 Jan 2001 13:53:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CIqYa02583 for ; Fri, 12 Jan 2001 13:52:34 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id NAA57546 for ; Fri, 12 Jan 2001 13:45:48 -0500 Received: from d03nmx41.almaden.ibm.com (d03nmx41.almaden.ibm.com [9.1.26.87]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.95) with ESMTP id f0CIqTo34936 for ; Fri, 12 Jan 2001 11:52:29 -0700 Importance: Normal Subject: iSCSI: Naming and Discovery Document To: ips@ece.cmu.edu From: "Kaladhar Voruganti" Date: Fri, 12 Jan 2001 10:52:27 -0800 Message-ID: X-MIMETrack: Serialize by Router on D03NMX41/03/M/IBM(Release 5.0.4a |July 24, 2000) at 01/12/2001 10:52:29 AM MIME-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=882569D200677A018f9e8a93df938690918c882569D200677A01" Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=882569D200677A018f9e8a93df938690918c882569D200677A01 Content-type: text/plain; charset=us-ascii The following attached document has been submitted to IETF. This naming and discovery document has been worked on by the iSCSI naming and discovery team. (See attached file: draft-ietf-ips-iscsi-disc-reqts-01.txt) regards, Kaladhar Voruganti --0__=882569D200677A018f9e8a93df938690918c882569D200677A01 Content-type: text/plain; name="draft-ietf-ips-iscsi-disc-reqts-01.txt" Content-Description: Text-ASCII-7bit Content-Transfer-Encoding: base64 DQoNCg0KDQoNCiAgICAgICAgICANCiAgICAgICAgICANCiAgICAgICAgIGlTQ1NJICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBNYXJrIEJha2tl ICANCiAgICAgICAgIEludGVybmV0IERyYWZ0ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIENpc2NvICANCiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBK b2UgQ3phcCAgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIElCTSAgDQogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgSmltIEhhZm5lciAgDQogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIElC TSAgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQogICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBIb3dhcmQgSGFsbCAg DQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBQaXJ1cyAgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIEphY2sgSGFyd29vZCAgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEVNQyAgDQogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEpvaG4gSHVmZmVyZCAgDQogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIElCTSAgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBZYXJvbiBLbGVpbiAgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFNhbnJhZCAgDQogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIExhd3JlbmNlIExhbWVycyAgDQogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFNhbiBWYWxsZXkg U3lzdGVtcyAgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEpvc2h1YSBU c2VuZyAgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIE5pc2hhbiAgDQogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIEthbGFkaGFyIFZvcnVnYW50aSAgDQogICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIElCTSAg DQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgDQogICAgICAgICBkcmFmdC1pZXRmLWlwcy1pc2NzaS1k aXNjLXJlcXRzLTAxLnR4dCAgICAgICAgICAgICAgSmFudWFyeSwgMjAwMSAgDQogICAgICAgICBF eHBpcmVzIEp1bHkgMjAwMSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIA0KICAgICAgICAgICANCiAgICAgICAgICAgICAgICAgICAgICBpU0NT SSBOYW1pbmcgYW5kIERpc2NvdmVyeSBSZXF1aXJlbWVudHMgDQogICAgICAgICAgIA0KICAgICAg ICAgU3RhdHVzIG9mIHRoaXMgTWVtbyAgDQogICAgICAgICAgIA0KICAgICAgICAgICAgICANCiAg ICAgICAgICAgIFRoaXMgZG9jdW1lbnQgaXMgYW4gSW50ZXJuZXQtRHJhZnQgYW5kIGlzIGluIGZ1 bGwgY29uZm9ybWFuY2Ugd2l0aCAgDQogICAgICAgICAgICBhbGwgcHJvdmlzaW9ucyBvZiBTZWN0 aW9uIDEwIG9mIFJGQzIwMjYgZXhjZXB0IHRoYXQgdGhlIHJpZ2h0IHRvICANCiAgICAgICAgICAg IHByb2R1Y2UgZGVyaXZhdGl2ZSB3b3JrcyBpcyBub3QgZ3JhbnRlZC4gICANCiAgICAgICAgICAg ICAgDQogICAgICAgICAgICBJbnRlcm5ldC1EcmFmdHMgYXJlIHdvcmtpbmcgZG9jdW1lbnRzIG9m IHRoZSBJbnRlcm5ldCBFbmdpbmVlcmluZyAgDQogICAgICAgICAgICBUYXNrIEZvcmNlIChJRVRG KSwgaXRzIGFyZWFzLCBhbmQgaXRzIHdvcmtpbmcgZ3JvdXBzLiBOb3RlIHRoYXQgIA0KICAgICAg ICAgICAgb3RoZXIgZ3JvdXBzIG1heSBhbHNvIGRpc3RyaWJ1dGUgd29ya2luZyBkb2N1bWVudHMg YXMgSW50ZXJuZXQtIA0KICAgICAgICAgICAgRHJhZnRzLiBJbnRlcm5ldC1EcmFmdHMgYXJlIGRy YWZ0IGRvY3VtZW50cyB2YWxpZCBmb3IgYSBtYXhpbXVtIG9mICANCiAgICAgICAgICAgIHNpeCBt b250aHMgYW5kIG1heSBiZSB1cGRhdGVkLCByZXBsYWNlZCwgb3Igb2Jzb2xldGVkIGJ5IG90aGVy ICANCiAgICAgICAgICAgIGRvY3VtZW50cyBhdCBhbnkgdGltZS4gSXQgaXMgaW5hcHByb3ByaWF0 ZSB0byB1c2UgSW50ZXJuZXQtIERyYWZ0cyAgDQogICAgICAgICAgICBhcyByZWZlcmVuY2UgbWF0 ZXJpYWwgb3IgdG8gY2l0ZSB0aGVtIG90aGVyIHRoYW4gYXMgIndvcmsgaW4gIA0KICAgICAgICAg ICAgcHJvZ3Jlc3MuIiAgIA0KICAgICAgICAgICAgVGhlIGxpc3Qgb2YgY3VycmVudCBJbnRlcm5l dC1EcmFmdHMgY2FuIGJlIGFjY2Vzc2VkIGF0ICANCiAgICAgICAgICAgIGh0dHA6Ly93d3cuaWV0 Zi5vcmcvaWV0Zi8xaWQtYWJzdHJhY3RzLnR4dCAgIA0KICAgICAgICAgICAgDA0KICAgICAgICAg IA0KICAgICAgICAgVm9ydWdhbnRpICAgICAgICAgIEludGVybmV0IERyYWZ0IEV4cGlyZXMgSnVs eSAyMDAxICAgICAgIDEgIA0KICAgICAgICAgIA0KICAgICAgICAgIA0KICAgICAgICAgICAgICAg ICAgICAgICAgICAgIGlTQ1NJIE5hbWluZyBhbmQgRGlzY292ZXJ5ICAgICAgICBOb3ZlbWJlciAy MDAwIA0KICAgICAgICAgICANCiAgICAgICAgICAgDQogICAgICAgICAgICBUaGUgbGlzdCBvZiBJ bnRlcm5ldC1EcmFmdCBTaGFkb3cgRGlyZWN0b3JpZXMgY2FuIGJlIGFjY2Vzc2VkIGF0ICANCiAg ICAgICAgICAgIGh0dHA6Ly93d3cuaWV0Zi5vcmcvc2hhZG93Lmh0bWwuICANCiAgICAgICAgICAg ICAgDQogICAgICAgICAgICAgIA0KICAgICAgICAgICAgICANCiAgICAgICAgIENvbW1lbnRzICAN CiAgICAgICAgIENvbW1lbnRzIHNob3VsZCBiZSBzZW50IHRvIHRoZSBpcHMgbWFpbGluZyBsaXN0 IChpcHNAZWNlLmNtdS5lZHUpIG9yIHRvICAgDQogICAgICAgICBrYWxhZGhhckB1cy5pYm0uY29t ICANCiAgICAgICAgICANCiAgICAgICAgICANCiAgICAgICAgIDEuIEFic3RyYWN0IA0KICAgICAg ICAgIA0KICAgICAgICAgVGhpcyBkb2N1bWVudCBkZXNjcmliZXMgdGhlICBpU0NTSSBbN10gbmFt aW5nIGFuZCBkaXNjb3ZlcnkgcmVxdWlyZW1lbnRzLiBUaGUgIA0KICAgICAgICAgcmVxdWlyZW1l bnRzIHByZXNlbnRlZCBpbiB0aGlzIGRvY3VtZW50IGhhdmUgYmVlbiBhZ3JlZWQgdG8gYnkgdGhl IG1lbWJlcnMgb2YgIA0KICAgICAgICAgdGhlIGlTQ1NJIG5hbWluZyBhbmQgZGlzY292ZXJ5IHRl YW0uIFRoaXMgZG9jdW1lbnQgY29tcGxlbWVudHMgdGhlIGlTQ1NJIElFVEYgIA0KICAgICAgICAg ZHJhZnQuIEZsZXhpYmlsaXR5IGlzIHRoZSBrZXkgZ3VpZGluZyBwcmluY2lwbGUgYmVoaW5kIHRo aXMgcmVxdWlyZW1lbnRzICANCiAgICAgICAgIGRvY3VtZW50LiBUaGF0IGlzLCBhbiBlZmZvcnQg aGFzIGJlZW4gbWFkZSB0byBzYXRpc2Z5IHRoZSBuZWVkcyBvZiBib3RoIHNtYWxsICANCiAgICAg ICAgIGlzb2xhdGVkIGVudmlyb25tZW50cywgYXMgd2VsbCBhcyBsYXJnZSBlbnZpcm9ubWVudHMg cmVxdWlyaW5nIHNlY3VyZS9zY2FsYWJsZSAgDQogICAgICAgICBzb2x1dGlvbnMuIA0KICAgICAg ICAgIA0KICAgICAgICAgVGhpcyBkb2N1bWVudCBoYXMgYmVlbiBvcmdhbml6ZWQgaW50byB0aGUg Zm9sbG93aW5nIHNlY3Rpb25zOiANCiAgICAgICAgIGEpIFNlY3Rpb24gMyBwcmVzZW50cyB0aGUg bmFtaW5nIHJlcXVpcmVtZW50cy4gDQogICAgICAgICBiKSBTZWN0aW9uIDQgZGlzY3Vzc2VzIHRo ZSBkaXNjb3ZlcnkgcmVxdWlyZW1lbnRzLiANCiAgICAgICAgIGMpIFNlY3Rpb24gNSBwcmVzZW50 cyBTdG9yYWdlIE5hbWUgU2VydmVyIChTTlMpIHJlcXVpcmVtZW50cy4gDQogICAgICAgICBkKSBT ZWN0aW9uIDYgYnJpZWZseSBkaXNjdXNzZXMgb3RoZXIgZXhpc3RpbmcgZGlzY292ZXJ5IHByb3Rv Y29scy4gDQogICAgICAgICAgDQogICAgICAgICAgDQogICAgICAgICAyLiBDb252ZW50aW9ucyB1 c2VkIGluIHRoaXMgZG9jdW1lbnQgDQogICAgICAgICAgDQogICAgICAgICAgICBUaGUga2V5IHdv cmRzICJNVVNUIiwgIk1VU1QgTk9UIiwgIlJFUVVJUkVEIiwgIlNIQUxMIiwgIlNIQUxMIE5PVCIs IA0KICAgICAgICAgICAgIlNIT1VMRCIsICJTSE9VTEQgTk9UIiwgIlJFQ09NTUVOREVEIiwgICJN QVkiLCBhbmQgIk9QVElPTkFMIiBpbiANCiAgICAgICAgICAgIHRoaXMgZG9jdW1lbnQgYXJlIHRv IGJlIGludGVycHJldGVkIGFzIGRlc2NyaWJlZCBpbiBSRkMtMjExOS4gDQogICAgICAgICAgDQog ICAgICAgICAgDQogICAgICAgICAzLiBOYW1pbmcgUmVxdWlyZW1lbnRzICANCiAgICAgICAgIElu IG9yZGVyIGZvciBhbiBpU0NTSSBpbml0aWF0b3IgdG8gY29ubmVjdCB0byBhbiBpU0NTSSB0YXJn ZXQsIHRoZSBpbml0aWF0b3IgIA0KICAgICAgICAgbmVlZHMgdG8gcHJvdmlkZSBpbmZvcm1hdGlv biBhYm91dCB0aGUgTmV0d29yayBFbnRpdHkgb2JqZWN0LCBQb3J0YWwgT2JqZWN0IGFuZCAgDQog ICAgICAgICB0aGUgdGFyZ2V0IFN0b3JhZ2UgTm9kZSBvYmplY3QuIFRoZSBkZXRhaWxzIG9mIHRo ZXNlIHRocmVlIGlTQ1NJIG9iamVjdHMgYXJlIGFzICANCiAgICAgICAgIGZvbGxvd3M6IA0KICAg ICAgICAgIA0KICAgICAgICAgYSkgTmV0d29yayBFbnRpdHkgT2JqZWN0IA0KICAgICAgICAgVGhl IE5ldHdvcmsgRW50aXR5IG9iamVjdCByZXByZXNlbnRzIGEgZGV2aWNlIG9yIGdhdGV3YXkgdGhh dCBpcyBhY2Nlc3NpYmxlIGZyb20gIA0KICAgICAgICAgdGhlIElQIG5ldHdvcmsuIFRoaXMgZGV2 aWNlIG9yIGdhdGV3YXkgbWF5IHN1cHBvcnQgb25lIG9yIG1vcmUgaW5pdGlhdG9ycyBvciAgDQog ICAgICAgICB0YXJnZXRzIHRoYXQgYXJlIGVpdGhlciBpbnRlcm5hbCB0byB0aGUgc3RvcmFnZSBk ZXZpY2Ugb3IgYWNjZXNzaWJsZSB0aHJvdWdoIGEgIA0KICAgICAgICAgbmV0d29yayBiZWhpbmQg dGhlIGdhdGV3YXkuIEVhY2ggaW5pdGlhdG9yIG9yIHRhcmdldCBpcyByZXByZXNlbnRlZCBieSAg DQogICAgICAgICBzdWJvcmRpbmF0ZSBTdG9yYWdlIE5vZGUgb2JqZWN0cy4gVGhlIE5ldHdvcmsg RW50aXR5IG9iamVjdCBpcyBpZGVudGlmaWVkIGJ5IGl0cyAgDQogICAgICAgICBJUCBhZGRyZXNz LiANCiAgICAgICAgICANCiAgICAgICAgIGIpIFBvcnRhbCBPYmplY3QgDQogICAgICAgICBUaGUg UG9ydGFsIG9iamVjdCBpcyBhIHBvcnQgdGhyb3VnaCB3aGljaCBhY2Nlc3MgdG8gYW55IFN0b3Jh Z2UgTm9kZSAMDQogICAgICAgICBvYmplY3Qgd2l0aGluIHRoZSBOZXR3b3JrIEVudGl0eSBvYmpl Y3QgY2FuIGJlIG9idGFpbmVkLiBBIE5ldHdvcmsgRW50aXR5IG9iamVjdCAgDQogICAgICAgICBt dXN0IGhhdmUgb25lIG9yIG1vcmUgUG9ydGFsIG9iamVjdHMsIGVhY2ggb2Ygd2hpY2ggaXMgdXNh YmxlIGJ5IFN0b3JhZ2UgTm9kZSAgDQogICAgICAgICBvYmplY3RzIGNvbnRhaW5lZCBpbiB0aGF0 IE5ldHdvcmsgRW50aXR5IG9iamVjdCB0byBnYWluIGFjY2VzcyB0byB0aGUgSVAgIA0KICAgICAg ICAgbmV0d29yay4gVGhlIFBvcnRhbCBvYmplY3QgaXMgaWRlbnRpZmllZCBieSBpdHMgSVAgYWRk cmVzcyBhbmQgUG9ydCBudW1iZXIuIFRoZSAgDQogICAgICAgICBQb3J0YWwgb2JqZWN0J3MgSVAg YWRkcmVzcyBjYW4gYmUgZGlmZmVyZW50IHRoYW4gdGhlIE5ldHdvcmsgRW50aXR5IElQIGFkZHJl c3MuIA0KICAgICAgICAgVGhlcmUgaXMgYSBjYW5vbmljYWwgaVNDU0kgVENQIHBvcnQgcHJlc2Vu dCBhdCBlYWNoIE5ldHdvcmsgRW50aXR5IG9iamVjdC4gDQogICAgICAgICBIb3dldmVyLCBTdG9y YWdlIE5vZGUgb2JqZWN0cyBjYW4gYWxzbyBiZSBhY2Nlc3NlZCB2aWEgbm9uLWNhbm9uaWNhbCAN CiAgICAgICAgIGlTQ1NJIFRDUCBwb3J0cy4gDQogICAgICAgICAgDQogICAgICAgICBjKSBTdG9y YWdlIE5vZGUgT2JqZWN0IA0KICAgICAgICAgVGhlIFN0b3JhZ2UgTm9kZSBvYmplY3QgZGVmaW5l cyBhbiBpbmRpdmlkdWFsIGlTQ1NJIGluaXRpYXRvciBvciB0YXJnZXQuIA0KICAgICAgICAgVGhl cmUgbWF5IGJlIG9uZSBvciBtb3JlIFN0b3JhZ2UgTm9kZSBvYmplY3RzIHdpdGhpbiB0aGUgTmV0 d29yayBFbnRpdHkgDQogICAgICAgICBvYmplY3QuIEEgU3RvcmFnZSBOb2RlIG9iamVjdCBpcyBp ZGVudGlmaWVkIGJ5IGl0cyB3b3JsZCB3aWRlIHVuaXF1ZSANCiAgICAgICAgIGlkZW50aWZpZXIg KFdXVUkpLiBUaGVyZSBpcyBhIHJlcXVpcmVtZW50IHRvIGhhdmUgdGhlIGFiaWxpdHkgdG8gZ2Vu ZXJhdGUgIA0KICAgICAgICAgd29ybGQgd2lkZSB1bmlxdWUgaWRlbnRpZmllcnMgKFdXVUlzKSBm b3IgYm90aCBpU0NTSSBpbml0aWF0b3JzIGFuZCB0YXJnZXRzLiANCiAgICAgICAgIEhvd2V2ZXIs IGl0IGlzIG5vdCBtYW5kYXRvcnkgZm9yIHRoZSBpbml0aWF0b3JzIGFuZCB0YXJnZXRzIHRvIHVz ZSBXV1VJcyANCiAgICAgICAgIGJlY2F1c2UgYSBnbG9iYWxseSB1bmlxdWUgaWRlbnRpZmllciBt aWdodCBub3QgYmUgcmVxdWlyZWQgaW4gc29tZSBzaW1wbGUsICANCiAgICAgICAgIGlzb2xhdGVk IGlTQ1NJIGNvbmZpZ3VyYXRpb25zLiBXV1VJcyBhcmUgdXNlZnVsIGJlY2F1c2UgaW4gc29tZSBj YXNlcyAoZS5nLiB3aGVuICANCiAgICAgICAgIERIQ1Agc2VydmljZXMgWzZdIGFyZSB1c2VkIGV0 YyksIHRoZSBjb21iaW5hdGlvbiBvZiBJUCBhZGRyZXNzIGFuZCBwb3J0IG51bWJlciAgDQogICAg ICAgICBbNl0gY2Fubm90IHVuaXF1ZWx5IGlkZW50aWZ5IGFuIGluaXRpYXRvciBvciBhIHRhcmdl dC4gDQogICAgICAgICAgDQogICAgICAgICAgDQogICAgICAgICAgDQogICAgICAgICBUaGVyZSBp cyBhIGRlZmF1bHQgU3RvcmFnZSBOb2RlIG9iamVjdCBwcmVzZW50IGF0IGV2ZXJ5IHRhcmdldCBu ZXR3b3JrIGVudGl0eSAgDQogICAgICAgICB0aGF0IGNhbiBiZSBhY2Nlc3NlZCB3aXRob3V0IHNw ZWNpZnlpbmcgdGhlIFdXVUkuIEhvd2V2ZXIsIGlmIHRoZXJlIGFyZSBtdWx0aXBsZSANCiAgICAg ICAgIGlTQ1NJIHRhcmdldCBTdG9yYWdlIE5vZGVzIHRoYXQgYXJlIHNlcnZpY2VkIGJ5IGEgc2lu Z2xlIE5ldHdvcmsgRW50aXR5IGFuZCAgDQogICAgICAgICBQb3J0YWwgb2JqZWN0cywgdGhlbiBp dCBpcyBuZWNlc3NhcnkgZm9yIHRoZSBpbml0aWF0b3IgdG8gc3BlY2lmeSB0aGUgdGFyZ2V0ICAN CiAgICAgICAgIFN0b3JhZ2UgTm9kZSBXV1VJIHRvIHVuaXF1ZWx5IGlkZW50aWZ5IHRoZSB0YXJn ZXQgc3RvcmFnZSBub2RlLiBBbiBhbGlhcyBzdHJpbmcgIA0KICAgICAgICAgY291bGQgYWxzbyBi ZSBhc3NvY2lhdGVkIHdpdGggYSB0YXJnZXQgc3RvcmFnZSBub2RlLiBUaGUgdGFyZ2V0IGFsaWFz IGhlbHBzIGFuICANCiAgICAgICAgIG9yZ2FuaXphdGlvbiB0byBhc3NvY2lhdGUgdGhlaXIgb3du IHNlbWFudGljIG1lYW5pbmcgd2l0aCB0aGUgdGFyZ2V0IGFsaWFzICANCiAgICAgICAgIHN0cmlu Zy4gRm9yIGV4YW1wbGUsIHRoZSBhbGlhcyBzdHJpbmcgY291bGQgcmVwcmVzZW50IHRoZSBvcmdh bml6YXRpb25hbCAgDQogICAgICAgICBoaWVyYXJjaHkgaW4gd2hpY2ggdGhlIHN0b3JhZ2UgZGV2 aWNlIHJlc2lkZXMgc3VjaCBhczogDQogICAgICAgICBDb21wYW55WFhYLmNvbS9yZXNlYXJjaC9k ZXB0MS9pbmRpdmlkdWFsL3N0b3JhZ2VfZGV2aWNlMSANCiAgICAgICAgIEhvd2V2ZXIsIHRoZSB0 YXJnZXQgYWxpYXMgc3RyaW5nIGlzIG5vdCBhIHN1YnN0aXR1dGUgZm9yIHRoZSB0YXJnZXQgV1dV SS4gDQogICAgICAgICAgDQogICAgICAgICAgDQogICAgICAgICAgDQogICAgICAgICAgDQogICAg ICAgICAzLjEgV29ybGQgV2lkZSBVbmlxdWUgSWRlbnRpZmllciANCiAgICAgICAgICANCiAgICAg ICAgIFRoZSBXV1VJIHVuaXF1ZWx5IGlkZW50aWZpZXMgaVNDU0kgaW5pdGlhdG9ycyBhbmQgdGFy Z2V0cy4gVGhlIGluaXRpYXRvciBXV1VJICANCiAgICAgICAgIGNvcnJlc3BvbmRzIHRvIHRoZSBs b2dpY2FsIG9wZXJhdGluZyBzeXN0ZW0gb24gd2hpY2ggdGhlIGluaXRpYXRvciBpcyBydW5uaW5n LCAgDQogICAgICAgICBhbmQgdGhlIHRhcmdldCBXV1VJIGNvcnJlc3BvbmRzIHRvIHRoZSB0YXJn ZXQgU3RvcmFnZSBOb2RlIGVudGl0eS4gIFRoZSBXV1VJIG1heSAgDQogICAgICAgICBiZSBkaXNw bGF5ZWQgYnkgdXNlciBpbnRlcmZhY2VzLCBidXQgaXMgZ2VuZXJhbGx5IHVuaW50ZXJwcmV0ZWQg YW5kIHVzZWQgYXMgYW4gIA0KICAgICAgICAgb3BhcXVlIGJpbmFyeSBzdHJpbmcgZm9yIGNvbXBh cmlzb24gd2l0aCBvdGhlciBXV1VJIHZhbHVlcy4gDQogICAgICAgICAgDQogICAgICAgICBUaGUg dXNlIG9mIHRoZSBuYW1pbmcgYXV0aG9yaXR5IG1lYW5zIHRoYXQgV1dVSXMgY2FuIGJlIGFzc2ln bmVkIGJ5IHZpcnR1YWxseSAgDQogICAgICAgICBhbnkgdW5pcXVlbmVzcyBzY2hlbWUgdGhhdCBj YW4gYmUgZGV2aXNlZCBieSBPUyB2ZW5kb3JzLCBkcml2ZXIgb3IgaVNDU0kgTklDICANCiAgICAg ICAgIHZlbmRvcnMsIGRldmljZSB2ZW5kb3JzLCBnYXRld2F5IHZlbmRvcnMsIGFuZCBldmVuIHRo ZSBjdXN0b21lci4gDQogICAgICAgICAgDQogICAgICAgICAgDQogICAgICAgICBUaGUgZm9ybWF0 IG9mIHRoZSBpU0NTSSBXV1VJIGlzIGFzIGZvbGxvd3M6IA0KICAgICAgICAgIA0KICAgICAgICAg V1dVSSA9IExlbmd0aCArIFR5cGUgKyBUeXBlLWRlcGVuZGVudCBmb3JtYXQgDQogICAgICAgICAg DQogICAgICAgICBMZW5ndGggaXMgMSBieXRlIGFuZCBpbmNsdWRlcyBUeXBlIGFuZCB0aGUgcmVz dCBvZiB0aGUgV1dVSSwgYnV0IG5vdCBpdHNlbGYuICAgDQogICAgICAgICBUaGUgbWF4aW11bSBs ZW5ndGggZmllbGQgdmFsdWUgaXMgMjU1LCBtYWtpbmcgYSBtYXhpbXVtIHRvdGFsIFdXVUkgb2Yg MjU2IGJ5dGVzICAMDQogICAgICAgICAoaW5jbHVkaW5nIExlbmd0aCksIGFuZCBhIG1heGltdW0g dHlwZS1kZXBlbmRlbnQgZm9ybWF0IG9mIDI1NCBieXRlcy4gDQogICAgICAgICAgDQogICAgICAg ICBUaGUgbWluaW11bSBsZW5ndGggb2YgYSBXV1VJIGlzIDI7IHRoZSBXV1VJIHdvdWxkIGNvbnNp c3Qgb2YganVzdCANCiAgICAgICAgIHRoZSBMZW5ndGggZmllbGQgKD09IDEpLCBhbmQgYSBUeXBl IGZpZWxkLiANCiAgICAgICAgICANCiAgICAgICAgIFR5cGUgaXMgMSBieXRlIGFuZCBpcyBhcyBm b2xsb3dzIChzaW1pbGFyLCBidXQgbm90IGlkZW50aWNhbCB0byBTUEMtMiBWUEQpIA0KICAgICAg ICAgIA0KICAgICAgICAgICAgICAgMDAgLSBOb19BdXRob3JpdHkgKG5vdCBndWFyYW50ZWVkIHRv IGJlIHVuaXF1ZSkgDQogICAgICAgICAgICAgICAwMSAtIEFTQ0lJICh1c2luZyByZXZlcnNlZCBE TlMgbmFtZSBhcyBOYW1pbmcgQXV0aG9yaXR5KSANCiAgICAgICAgICAgICAgIDAyIC0gSUVFRSBF VUktNjQgDQogICAgICAgICAgICAgICAwMyAtIFVuaWNvZGUgKEROUyBuYW1pbmcgYXV0aG9yaXR5 KSANCiAgICAgICAgICAgICAgIDA0IC0gR2VuZXJpYyBCaW5hcnkgV1dVSSAodG8gYmUgY29uc2lk ZXJlZCkgDQogICAgICAgICAgDQogICAgICAgICAgIEFkZGl0aW9uIG9mIG5ldyB0eXBlcyByZXF1 aXJlcyBhcHByb3ZhbCB0byBiZWNvbWUgYW4gaVNDU0kgc3RhbmRhcmQuIA0KICAgICAgICAgIA0K ICAgICAgICAgIA0KICAgICAgICAgT3BlbiBRdWVzdGlvbjogIFNob3VsZCBhbGwgb2NjdXJlbmNl cyBvZiAiQVNDSUkiIGluIHRoaXMgDQogICAgICAgICAgICAgICAgICAgICAgICAgZG9jdW1lbnQg YmUgcmVwbGFjZWQgd2l0aCAiVVRGLTgiPyAgU28gZmFyLCB3ZSANCiAgICAgICAgICAgICAgICAg ICAgICAgICBoYXZlIGhhZCBubyB2b3RlcyBmb3IgVVRGLTguIA0KICAgICAgICAgIA0KICAgICAg ICAgT3BlbiBRdWVzdGlvbjogIFNob3VsZCB0aGUgV1dVSSBiZSBwYWRkZWQgdG8gYSA0LWJ5dGUg Ym91bmRhcnk/IA0KICAgICAgICAgICAgICAgICAgICAgICAgIFBsZWFzZSBzZWUgZGlzY3Vzc2lv biBvbiB0cmFuc3BvcnRpbmcgYSBXV1VJLiANCiAgICAgICAgICANCiAgICAgICAgICANCiAgICAg ICAgIFVzZSBvZiB0aGUgQVNDSUkgZm9ybWF0IGlzIHJlY29tbWVuZGVkIHdoZW4gcG9zc2libGUg Zm9yIHRoZSBmb2xsb3dpbmcgDQogICAgICAgICByZWFzb25zOiANCiAgICAgICAgICANCiAgICAg ICAgIC0gYW4gQVNDSUkgV1dVSSBpcyBlYXNpZXIgdG8gdHlwZSBhbmQgZGlmZmVyZW50aWF0ZSBp biBhIHVzZXIgDQogICAgICAgICAgIGludGVyZmFjZS4gDQogICAgICAgICAgDQogICAgICAgICAt IEFuIEFTQ0lJIFdXVUkgY2FuIHVzZSBhIEROUyBuYW1lIGFzIGEgbmFtaW5nIGF1dGhvcml0eS4g IEl0IGNhbiANCiAgICAgICAgICAgYmUgYXNzdW1lZCB0aGF0IGFueW9uZSB3aG8gd2FudHMgdG8g bmFtZSB0YXJnZXRzIG9yIGluaXRpYXRvcnMgDQogICAgICAgICAgIG93bnMgYSBETlMgbmFtZS4g IFRoZSBzYW1lIGlzIG5vdCB0cnVlIGZvciBlaXRoZXIgT1VJIG9yIFNDU0kgDQogICAgICAgICAg IFZlbmRvciBJRC4gIFRoaXMgYWxzbyBtZWFucyB0aGF0IGVuZCB1c2VycyBjYW4gbmFtZSB0aGVp ciBvd24gDQogICAgICAgICAgIHRhcmdldHMgYW5kIGluaXRpYXRvcnMsIGZvciB3aGF0ZXZlciB0 aGVpciBwdXJwb3NlcyBtYXkgYmUuIA0KICAgICAgICAgIA0KICAgICAgICAgLSBXV1VJcyBhcmUg b25seSB1c2VkIGR1cmluZyBsb2dpbiBhbmQgZGlzY292ZXJ5IHBoYXNlcywgc28gdGhlIA0KICAg ICAgICAgICBvdmVyaGVhZCBkb2VzIG5vdCBnZXQgaW4gdGhlIHdheSBvZiB0aGUgZGF0YSBwYXRo LiANCiAgICAgICAgICANCiAgICAgICAgICANCiAgICAgICAgIFRoZSBJRUVFIGZvcm1hdCBpcyBy ZWNvbW1lbmRlZCB3aGVuOiANCiAgICAgICAgICANCiAgICAgICAgIC0gVGhlcmUgaXMgYW4gZXhp c3RpbmcgSUVFRSB1bmlxdWUgbmFtZSB0aGF0IG11c3QgYmUgY29tbXVuaWNhdGVkIA0KICAgICAg ICAgICB0byBpU0NTSS4gDQogICAgICAgICAgDQogICAgICAgICAgDQogICAgICAgICBUaGUgVW5p Y29kZSBmb3JtYXQgaXMgcmVjb21tZW5kZWQgaW4gcGxhY2Ugb2YgQVNDSUkgd2hlbjogDQogICAg ICAgICAgDQogICAgICAgICAtIEh1bWFuLXJlYWRhYmxlIGZvcm1hdCBpcyBkZXNpcmVkLCBhbmQg YSBjaGFyYWN0ZXIgc2V0IG90aGVyIA0KICAgICAgICAgICB0aGFuIEFTQ0lJIGlzIG5lZWRlZC4g DQogICAgICAgICAgDQogICAgICAgICAgDQogICAgICAgICBXZSBtYXkgYWxzbyBjb25zaWRlciBh ZGRpbmcgYSBnZW5lcmljIGJpbmFyeSBzdHJpbmcgZm9ybWF0IHVzaW5nIGEgDQogICAgICAgICBt YW51ZmFjdHVyZXIncyBPVUkgYXMgYSBuYW1pbmcgYXV0aG9yaXR5LiANCiAgICAgICAgICANCiAg ICAgICAgICANCiAgICAgICAgIFR5cGUgZGV0ZXJtaW5lcyB0aGUgcmVtYWluZGVyIG9mIHRoZSBX V1VJIGZvcm1hdCBhbmQgaXQgY2FuIGJlIGluIHRoZSAMDQogICAgICAgICBmb2xsb3dpbmcgZm9y bWF0czogDQogICAgICAgICAgDQogICAgICAgICAgDQogICAgICAgICBOb19XV1VJIEZvcm1hdCAN CiAgICAgICAgICANCiAgICAgICAgICAgICstLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0rIA0KICAg ICAgICAgICAgfCBMZW5ndGggPSAxIHwgVHlwZSA9IDAwIHwgDQogICAgICAgICAgICArLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tKyANCiAgICAgICAgICANCiAgICAgICAgICAgIFRoaXMgZm9ybWF0 IGlzIHVzZWQgdG8gaW5kaWNhdGUgYSBOVUxMIFdXVUkuIA0KICAgICAgICAgIA0KICAgICAgICAg IA0KICAgICAgICAgQVNDSUlfV1dVSSBGb3JtYXQgDQogICAgICAgICAgDQogICAgICAgICAgICAr LS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLSANCiAgICAg ICAgICAgIHwgTGVuZ3RoID0gICAgICAgICB8IFR5cGUgPSAwMSB8IHN0cmluZyANCiAgICAgICAg ICAgIHwgMStzdHJsZW4oc3RyaW5nKSB8ICAgICAgICAgICB8IA0KICAgICAgICAgICAgKy0tLS0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0gDQogICAgICAgICAg DQogICAgICAgICAgICBUaGUgQVNDSUkgV1dVSSBzdHJpbmcgaXMgZGVmaW5lZCBhcyBmb2xsb3dz OiANCiAgICAgICAgICANCiAgICAgICAgICAgIFN0cmluZyBzdGFydHMgd2l0aCBhIGJhY2t3YXJk cyBkb21haW4gbmFtZSBzcGVjaWZ5aW5nIHRoZSBOYW1pbmcgDQogICAgICAgICAgICBBdXRob3Jp dHksIHVzaW5nIGRvdHMgYXMgc2VwYXJhdG9ycywganVzdCBhcyBpbiBhIHJlZ3VsYXIgZG9tYWlu IA0KICAgICAgICAgICAgbmFtZS4gIEl0J3MgYmFja3dhcmRzLCBzaW5jZSBpdCBpcyBub3QgcmVh bGx5IHVzZWQgYXMgYSBmdWxseSANCiAgICAgICAgICAgIHF1YWxpZmllZCBob3N0IG5hbWU7IG9u bHkgdGhlIG5lY2Vzc2FyeSB0b3AgbGV2ZWxzIG5lZWQgYnkgdXNlZC4gDQogICAgICAgICAgDQog ICAgICAgICAgICBCYXNpY2FsbHksIGV2ZXJ5dGhpbmcgYWZ0ZXIgdGhlIGJhY2t3YXJkcyBkb21h aW4gbmFtZSwgZm9sbG93ZWQgDQogICAgICAgICAgICBieSBhbm90aGVyIGRvdCAiLiIsIGNhbiBi ZSBhc3NpZ25lZCBhcyBuZWVkZWQgYnkgdGhlIG93bmVyIG9mIA0KICAgICAgICAgICAgdGhlIGRv bWFpbiBuYW1lLiANCiAgICAgICAgICANCiAgICAgICAgICAgIEhlcmUgaXMgYW4gZXhhbXBsZSBB U0NJSSBXV1VJIHN0cmluZzogDQogICAgICAgICAgDQogICAgICAgICAgICAgIDMyMDFjb20uYWNt ZS5kaXNrYXJyYXlzLnNuLmE4Njc1MzA5IA0KICAgICAgICAgIA0KICAgICAgICAgICAgV2hlcmU6 IA0KICAgICAgICAgICAgICAzMiAgaXMgdGhlIGxlbmd0aCBvZiB0aGUgc3RyaW5nICsgbGVuZ3Ro IG9mIFR5cGUgDQogICAgICAgICAgICAgIDAxICByZWZlcnMgdG8gQVNDSUkgV1dVSSB0eXBlIHN0 cmluZyANCiAgICAgICAgICANCiAgICAgICAgICAgICAgSW4gdGhlIHJlc3Qgb2YgdGhpcyBkb2N1 bWVudCBldmVuIHRob3VnaCB0aGUgbGVuZ3RoIGZpZWxkIGFuZCB0aGUgdHlwZSANCiAgICAgICAg ICAgICAgZmllbGQgdmFsdWVzIGFyZSBpbiBmcm9udCBvZiB0aGUgV1dVSSBzdHJpbmcsIHRoZXkg YXJlIG5vdCBiZWluZyAgDQogICAgICAgICBzaG93biBmb3IgDQogICAgICAgICAgICAgIHJlYWRh YmlsaXR5IHNha2UuIA0KICAgICAgICAgIA0KICAgICAgICAgICAgICAiY29tLmFjbWUiIGRlZmlu ZXMgdGhlIE5hbWluZyBBdXRob3JpdHkuICBUaGUgb3duZXIgb2YgdGhlIEROUyANCiAgICAgICAg ICAgICAgbmFtZSAiYWNtZS5jb20iIGhhcyB0aGUgc29sZSByaWdodCBvZiB1c2Ugb2YgdGhpcyBu YW1lIHdpdGhpbiANCiAgICAgICAgICAgICAgYSBXV1VJLiAgSW4gdGhpcyBjYXNlLCBhY21lLmNv bSBoYXBwZW5zIHRvIG1hbnVmYWN0dXJlIGRpc2sgDQogICAgICAgICAgICAgIGFycmF5cy4gDQog ICAgICAgICAgDQogICAgICAgICAgICAgICJkaXNrYXJyYXlzIiB3YXMgcGlja2VkIGFyYml0cmFy aWx5IGJ5IGFjbWUuY29tIHRvIHVzZSB0byANCiAgICAgICAgICAgICAgaWRlbnRpZnkgdGhlIGRp c2sgYXJyYXlzIHRoZXkgbWFudWZhY3R1cmUuICBBbm90aGVyIHByb2R1Y3QgDQogICAgICAgICAg ICAgIHRoYXQgQUNNRSBtYWtlcyB3b3VsZCB1c2UgYSBkaWZmZXJlbnQgbmFtZSwgYW5kIGhhdmUg dGhlaXIgDQogICAgICAgICAgICAgIG93biBuYW1lc3BhY2UgaW5kZXBlbmRlbnQgb2YgdGhlIGRp c2sgYXJyYXkgZ3JvdXAuIA0KICAgICAgICAgIA0KICAgICAgICAgICAgICAic24iIHdhcyBwaWNr ZWQgYnkgdGhlIGRpc2sgYXJyYXkgZ3JvdXAgb2YgQWNtZSB0byBzaG93IHRoYXQgDQogICAgICAg ICAgICAgIHdoYXQgZm9sbG93cyBpcyBhIHNlcmlhbCBudW1iZXIuICBUaGV5IGNvdWxkIGhhdmUg anVzdCBhc3N1bWVkIA0KICAgICAgICAgICAgICB0aGF0IGFsbCBXV1VJcyBhcmUgYmFzZWQgb24g c2VyaWFsIG51bWJlcnMsIGJ1dCB0aGV5IHRob3VnaHQgDQogICAgICAgICAgICAgIHRoYXQgcGVy aGFwcyBsYXRlciBwcm9kdWN0cyBtaWdodCBiZSBiZXR0ZXIgaWRlbnRpZmllZCBieSAMDQogICAg ICAgICAgICAgIHNvbWV0aGluZyBlbHNlLiAgQWRkaW5nICJzbiIgd2FzIGEgZnV0dXJlLXByb29m IG1lYXN1cmUuIA0KICAgICAgICAgIA0KICAgICAgICAgICAgICAiYTg2NzUzMDkiIGlzIHRoZSBz ZXJpYWwgbnVtYmVyIG9mIHRoZSBkaXNrIGFycmF5LCB1bmlxdWVseSANCiAgICAgICAgICAgICAg aWRlbnRpZnlpbmcgaXQgZnJvbSBhbGwgb3RoZXIgYXJyYXlzLiANCiAgICAgICAgICANCiAgICAg ICAgICAgIFBsZWFzZSBub3RlIHRoYXQgV1dVSSBpcyBOT1QgYW4gYWRkcmVzcyAtIGV2ZW4gdGhv dWdoIGl0IHVzZXMgYSBETlMgDQogICAgICAgICAgICBuYW1lLCB0aGlzIGlzIGZvciB0aGUgbmFt aW5nIGF1dGhvcml0eSBvbmx5OyBpdCBpcyBub3QgYW4gYWRkcmVzcyANCiAgICAgICAgICAgIHVz ZWQgdG8gZGlzY292ZXIgYW55dGhpbmcuIA0KICAgICAgICAgIA0KICAgICAgICAgICAgTm90ZSB0 aGF0IHdlIGNvdWxkIGhhdmUgdXNlZCB0aGUgQVNDSUkgVmVuZG9yIElEIGFzIGEgbmFtaW5nIA0K ICAgICAgICAgICAgYXV0aG9yaXR5LiAgSG93ZXZlciwgc29tZSBsYXJnZSBjdXN0b21lcnMgYW5k IHNlcnZpY2UgcHJvdmlkZXJzIA0KICAgICAgICAgICAgbWF5IHdpc2ggdG8gdXNlIHRoZWlyIG93 biBpZGVudGlmaWNhdGlvbiBzY2hlbWUsIHJhdGhlciB0aGFuIA0KICAgICAgICAgICAgdGhhdCBw cm92aWRlZCBieSB0aGUgbWFudWZhY3R1cmVyLiAgVGhlc2UgY3VzdG9tZXJzIHdvdWxkIG5vdCAN CiAgICAgICAgICAgIGxpa2VseSBoYXZlIGEgcmVnaXN0ZXJlZCBWZW5kb3IgSUQsIGJ1dCB0aGUg ZG9tYWluIG5hbWUgd2UgDQogICAgICAgICAgICB1c2VkIGlzIHViaXF1aXRvdXMsIGFuZCBzZWVt ZWQgbW9yZSBhcHByb3ByaWF0ZS4gDQogICAgICAgICAgDQogICAgICAgICAgICBGdXJ0aGVyIGV4 YW1wbGVzIG9mIEFTQ0lJIFdXVUlzIGFyZSBnaXZlbiBhdCB0aGUgZW5kIG9mIHRoaXMgDQogICAg ICAgICAgICBkb2N1bWVudC4gDQogICAgICAgICAgDQogICAgICAgICAgDQogICAgICAgICBJRUVF X1dXVUkgDQogICAgICAgICAgDQogICAgICAgICAgICArLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLS0tLS0tLSsgDQogICAgICAgICAgICB8IExlbmd0aCA9IDkgfCBUeXBl ID0gMDIgfCBJRUVFIEVVSS02NCBBZGRyZXNzIHwgDQogICAgICAgICAgICArLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLSsgDQogICAgICAgICAgDQogICAgICAg ICAgICBUaGUgSUVFRSBXV1VJIG1pZ2h0IGJlIHVzZWQgd2hlbiBhIG1hbnVmYWN0dXJlciBpcyBh bHJlYWR5IA0KICAgICAgICAgICAgYmFzaW5nIHVuaXF1ZSBpZGVudGlmaWVycyBvbiBXb3JsZC1X aWRlIE5hbWVzIGFzIGRlZmluZWQgaW4gDQogICAgICAgICAgICB0aGUgU0NTSSBTUEMtMiBzcGVj aWZpY2F0aW9uLiANCiAgICAgICAgICANCiAgICAgICAgICAgIEl0IG1heSBhbHNvIGJlIHVzZWQg YnkgYSBnYXRld2F5IHJlcHJlc2VudGluZyBhIEZpYnJlIENoYW5uZWwgDQogICAgICAgICAgICBv ciBTQ1NJIGRldmljZSB0aGF0IGlzIGFscmVhZHkgYWRlcXVhdGVseSBpZGVudGlmaWVkIHVzaW5n IGEgDQogICAgICAgICAgICB3b3JsZC13aWRlIG5hbWUuIA0KICAgICAgICAgIA0KICAgICAgICAg VW5pY29kZV9XV1VJIA0KICAgICAgICAgIA0KICAgICAgICAgICAgKy0tLS0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0gDQogICAgICAgICAgICB8IExlbmd0aCA9 ICAgICAgICAgfCBUeXBlID0gMDMgfCBVbmljb2RlIHN0cmluZyANCiAgICAgICAgICAgIHwgMStz dHJsZW4oc3RyaW5nKSB8ICAgICAgICAgICB8IA0KICAgICAgICAgICAgKy0tLS0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0gDQogICAgICAgICAgDQogICAgICAg ICAgIFRoaXMgZm9ybWF0IGlzIGlkZW50aWNhbCB0byB0aGUgQVNDSUkgZm9ybWF0LCBpbmNsdWRp bmcgdGhlIA0KICAgICAgICAgICB1c2Ugb2YgdGhlIHJldmVyc2VkIGRvbWFpbiBuYW1lIGFzIHRo ZSBuYW1pbmcgYXV0aG9yaXR5LCBleGNlcHQgDQogICAgICAgICAgIHRoYXQgVW5pY29kZSBpcyB1 c2VkIGluc3RlYWQgb2YgQVNDSUkuIA0KICAgICAgICAgIA0KICAgICAgICAgIA0KICAgICAgICAg QmluYXJ5X1dXVUkgRm9ybWF0ICh0byBiZSBjb25zaWRlcmVkKSANCiAgICAgICAgICANCiAgICAg ICAgICAgICstLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0t IA0KICAgICAgICAgICAgfCBMZW5ndGggPSAgICAgICAgIHwgVHlwZSA9IDA0IHwgT1VJICAgICB8 IGJpbmFyeSBVSSANCiAgICAgICAgICAgIHwgMStsZW4oYmluYXJ5IFVJKSB8ICAgICAgICAgICB8 IDMgYnl0ZXMgfCANCiAgICAgICAgICAgICstLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tLS0tIA0KICAgICAgICAgIA0KICAgICAgICAgIA0KICAgICAgICAgIA0K ICAgICAgICAgSW5pdGlhdG9yIGFuZCBUYXJnZXQgUmVxdWlyZW1lbnRzIGZvciBXV1VJIHN1cHBv cnQ6IA0KICAgICAgICAgIAwNCiAgICAgICAgICAgQm90aCBzaGFsbCBzdXBwb3J0IFdXVUlzIG9m IHVwIHRvIHRoZSBtYXhpbXVtIGxlbmd0aC4gDQogICAgICAgICAgDQogICAgICAgICAgIEluaXRp YXRvcnMgYW5kIHRhcmdldHMgc2hhbGwgcHJlc2VudCB0aGVpciBvd24gV1dVSSBhcyBwYXJ0IG9m IA0KICAgICAgICAgICB0aGUgcHJvdG9jb2xzIGRlZmluZWQgZWxzZXdoZXJlLiANCiAgICAgICAg ICANCiAgICAgICAgICAgVXNlciBpbnRlcmZhY2VzIHNob3VsZCBkaXNwbGF5IGFueSBBU0NJSSB0 eXBlIFdXVUkgYXMgYW4gDQogICAgICAgICAgIEFTQ0lJIHN0cmluZywgYW55IGJpbmFyeSBmb3Jt YXQgV1dVSSBhcyBhIHN0cmluZyBvZiBoZXggZGlnaXRzLCANCiAgICAgICAgICAgYW5kIGFsbCB0 eXBlcyB1bmtub3duIHRvIHRoZSBpbXBsZW1lbnRhdGlvbiBhcyBpZiB0aGUgZm9ybWF0IA0KICAg ICAgICAgICB3ZXJlIGJpbmFyeS4gDQogICAgICAgICAgDQogICAgICAgICAgDQogICAgICAgICBT b21lIFdXVUkgRXhhbXBsZXMgZm9yIFRhcmdldHMgDQogICAgICAgICAgDQogICAgICAgICAtIEFz c2lnbiB0byBhIHRhcmdldCBiYXNlZCBvbiBjb250cm9sbGVyIHNlcmlhbCBudW1iZXIgDQogICAg ICAgICAgDQogICAgICAgICAgICAgIGNvbS5hY21lLmRpc2thcnJheS5zbi44Njc1MzA5IA0KICAg ICAgICAgIA0KICAgICAgICAgICAgICAgICAgU2VlIHRoZSBBU0NJSSBXV1VJIGV4YW1wbGUgYWJv dmUgZm9yIGRpc2N1c3Npb24uIA0KICAgICAgICAgIA0KICAgICAgICAgLSBBc3NpZ24gdG8gYSB0 YXJnZXQgYmFzZWQgb24gc2VyaWFsIG51bWJlciBhbmQgbG9naWNhbCB0YXJnZXQgYWxpYXMgDQog ICAgICAgICAgDQogICAgICAgICAgICAgIGNvbS5hY21lLmRpc2thcnJheS5zbi44Njc1MzA5Lm9y YWNsZV9kYXRhYmFzZV8xIA0KICAgICAgICAgIA0KICAgICAgICAgICAgICAgICAgV2hlcmUgb3Jh Y2xlX2RhdGFiYXNlXzEgbWlnaHQgYmUgYSB0YXJnZXQgYWxpYXMgYXNzaWduZWQgYnkgDQogICAg ICAgICAgICAgICAgICBhIHVzZXIuIA0KICAgICAgICAgIA0KICAgICAgICAgICAgICBUaGlzIHdv dWxkIGJlIHVzZWZ1bCBmb3IgYSBjb250cm9sbGVyIHRoYXQgY2FuIHByZXNlbnQgDQogICAgICAg ICAgICAgICAgICBkaWZmZXJlbnQgbG9naWNhbCB0YXJnZXRzIHRvIGRpZmZlcmVudCBob3N0cy4g DQogICAgICAgICAgDQogICAgICAgICAgIE9idmlvdXNseSwgYW55IG5hbWluZyBhdXRob3JpdHkg bWF5IGNvbWUgdXAgd2l0aCBpdHMgb3duIHNjaGVtZSANCiAgICAgICAgICAgYW5kIGhpZXJhcmNo eSBmb3IgdGhlc2UgbmFtZXMsIGFuZCBiZSBqdXN0IGFzIHZhbGlkLiANCiAgICAgICAgICANCiAg ICAgICAgICAgQSB0YXJnZXQgV1dVSSBzaG91bGQgTkVWRVIgYmUgYXNzaWduZWQgYmFzZWQgb24g aW50ZXJmYWNlIGhhcmR3YXJlLCANCiAgICAgICAgICAgb3Igb3RoZXIgaGFyZHdhcmUgdGhhdCBj YW4gYmUgc3dhcHBlZCBhbmQgbW92ZWQgdG8gb3RoZXIgZGV2aWNlcy4gDQogICAgICAgICAgDQog ICAgICAgICAgDQogICAgICAgICBTb21lIFdXVUkgRXhhbXBsZXMgZm9yIEluaXRpYXRvcnMgDQog ICAgICAgICAgDQogICAgICAgICAtIEFzc2lnbiB0byB0aGUgT1MgaW1hZ2UgYnkgZnVsbHkgcXVh bGlmaWVkIGhvc3QgbmFtZSANCiAgICAgICAgICANCiAgICAgICAgICAgICBjb20ub3N2ZW5kb3Iu ZG5zLmNvbS5jdXN0b21lcjEuaG9zdF9mb3VyIA0KICAgICAgICAgIA0KICAgICAgICAgICAgIE5v dGUgdGhlIHVzZSBvZiB0d28gRlFETnMgLSB0aGF0IG9mIHRoZSBuYW1pbmcgDQogICAgICAgICAg ICAgYXV0aG9yaXR5IGFuZCBhbHNvIHRoYXQgb2YgdGhlIGhvc3QgdGhhdCBpcyBiZWluZyANCiAg ICAgICAgICAgICBuYW1lZC4gIFRoaXMgY2FuIGNhdXNlIHByb2JsZW1zLCBkdWUgdG8gbGltaXRh dGlvbnMgDQogICAgICAgICAgICAgaW1wb3NlZCBvbiB0aGUgc2l6ZSBvZiB0aGUgV1dVSS4gDQog ICAgICAgICAgDQogICAgICAgICAgICAgKCB3cml0ZSBpbiB3aGF0IHRvIGRvIGFib3V0IHRoaXMg KSANCiAgICAgICAgICANCiAgICAgICAgIC0gQXNzaWduIHRvIHRoZSBPUyBpbWFnZSBieSBPUyBp bnN0YWxsIHNlcmlhbCBudW1iZXIgDQogICAgICAgICAgDQogICAgICAgICAgICAgY29tLm9zdmVu ZG9yLm5ld29zNS4xMjM0NS1PRU0tMDA2Nzg5MC0yMzQ1NiANCiAgICAgICAgICANCiAgICAgICAg ICAgICBOb3RlIHRoYXQgdGhpcyBicmVha3MgaWYgYW4gaW5zdGFsbCBDRCBpcyB1c2VkIG1vcmUg DQogICAgICAgICAgICAgdGhhbiBvbmNlLiANCiAgICAgICAgICANCiAgICAgICAgIC0gQXNzaWdu IHRvIHRoZSBPUyBpbWFnZSBieSBhIHNlcnZpY2UgcHJvdmlkZXIgDA0KICAgICAgICAgIA0KICAg ICAgICAgICAgIGNvbS5teWRpc2sudXNlcnMubWJha2tlMDU2NTcgDQogICAgICAgICAgDQogICAg ICAgICAgICAgTm90ZSB0aGF0IHRoaXMgY291bGQgYWxzbyBiZSBhc3NpZ25lZCB0byBhIHBhcnRp Y3VsYXIgDQogICAgICAgICAgICAgaVNDU0kgYWRkcmVzcyBpZiBtb3JlIHRoYW4gb25lIFNQIGlz IHVzZWQuIA0KICAgICAgICAgIA0KICAgICAgICAgU29tZSBXV1VJIEV4YW1wbGVzIGZvciBHYXRl d2F5cyANCiAgICAgICAgICANCiAgICAgICAgICAgICggbmVlZHMgd29yaywgYnV0IGdhdGV3YXkg dmVuZG9ycyBhcmUgYSBjcmVhdGl2ZSBsb3QgKSANCiAgICAgICAgICANCiAgICAgICAgICANCiAg ICAgICAgIEFkZGluZyB0aGUgV1dVSSB0byBTQ1NJIFRoaXJkIFBhcnR5IENvbW1hbmRzIA0KICAg ICAgICAgIA0KICAgICAgICAgICBXb3JrIGRvbmUgb24gYWRkaW5nIHRoZSBXV1VJIGFkZHJlc3Mg dHlwZSB0byBTQ1NJIHRoaXJkIA0KICAgICAgICAgICBwYXJ0eSBjb21tYW5kcywgc3VjaCBhcyBl eHRlbmRlZCBjb3B5LCBpcyBiZWluZyBkb25lIGluIA0KICAgICAgICAgICBUMTAuIA0KICAgICAg ICAgIA0KICAgICAgICAgIA0KICAgICAgICAgVXNpbmcgSW5pdGlhdG9yIGFuZCBUYXJnZXQgV1dV SSBEdXJpbmcgTG9naW4gDQogICAgICAgICAgDQogICAgICAgICAgIFRoZSBJbml0aWF0b3IgV1dV SSBzaG91bGQgYWx3YXlzIGJlIHNlbnQgZHVyaW5nIGxvZ2luLiAgQXMgYSB0YXJnZXQgDQogICAg ICAgICAgIG1heSB1c2UgdGhlIEluaXRpYXRvciBXV1VJIGFzIHBhcnQgb2YgaXRzIGFjY2VzcyBj b250cm9sIG1lY2hhbmlzbSwgYW4gDQogICAgICAgICAgIGluaXRpYXRvciB0aGF0IGRvZXMgbm90 IHNlbmQgaXRzIFdXVUkgc3RhbmRzIHRoZSByaXNrIHRoYXQgaXQgd2lsbCBiZSANCiAgICAgICAg ICAgZXhjbHVkZWQgZnJvbSBhY2Nlc3Npbmcgc29tZSBvciBhbGwgb2YgaXRzIHRhcmdldHMuIA0K ICAgICAgICAgIA0KICAgICAgICAgMS4gQm90aCB0YXJnZXQgV1dVSSBhbmQgdGhlIHRhcmdldCBh bGlhcyBhcmUgc3BlY2lmaWVkICBJLT5Mb2dpbiBSZXF1ZXN0IA0KICAgICAgICAgICAgICBJbml0 aWF0b3JXV1VJOiBjb20ub3MuaG9zdGlkLjM0NTY3ODkwIA0KICAgICAgICAgICAgICBUYXJnZXRX V1VJOiBjb20uYWNtZS5kaXNrYXJyYXkuc24uODY3NTMwOSANCiAgICAgICAgICAgICAgVGFyZ2V0 QWxpYXM6IGZvbyANCiAgICAgICAgICAgICAgLiANCiAgICAgICAgICAgICAgLiAgdGV4dCBjb21t YW5kcyBmbG93IGhlcmUgZHVyaW5nIGF1dGhlbnRpY2F0aW9uIHBoYXNlIA0KICAgICAgICAgICAg ICAuIA0KICAgICAgICAgICBULT5Mb2dpbiBSZXNwb25zZSANCiAgICAgICAgICAgICAgVGFyZ2V0 V1dVSTogY29tLmFjbWUuZGlza2FycmF5LnNuLjg2NzUzMDkgDQogICAgICAgICAgICAgIFRhcmdl dEFsaWFzOiBmb28gDQogICAgICAgICAgDQogICAgICAgICAyLiBPbmx5IFRhcmdldCBXV1VJIGlz IHNwZWNpZmllZCBhbmQgbm8gYWxpYXMgaXMgc3BlY2lmaWVkLiANCiAgICAgICAgICANCiAgICAg ICAgICAgSS0+TG9naW4gUmVxdWVzdCANCiAgICAgICAgICAgICAgSW5pdGlhdG9yV1dVSTogY29t Lm9zLmhvc3RpZC4zNDU2Nzg5MCANCiAgICAgICAgICAgICAgVGFyZ2V0V1dVSTogY29tLmFjbWUu ZGlza2FycmF5LnNuLjg2NzUzMDkgDQogICAgICAgICAgICAgIC4gDQogICAgICAgICAgICAgIC4g IHRleHQgY29tbWFuZHMgZmxvdyBoZXJlIGR1cmluZyBhdXRoZW50aWNhdGlvbiBwaGFzZSANCiAg ICAgICAgICAgICAgLiANCiAgICAgICAgICAgVC0+TG9naW4gUmVzcG9uc2UgDQogICAgICAgICAg ICAgIFRhcmdldFdXVUk6IGNvbS5hY21lLmRpc2thcnJheS5zbi44Njc1MzA5IA0KICAgICAgICAg ICAgICBUYXJnZXRBbGlhczogZm9vIA0KICAgICAgICAgIA0KICAgICAgICAgMy4gTmVpdGhlciB0 YXJnZXQgYWxpYXMgbm9yIFdXVUkgaXMgc3BlY2lmaWVkLiAgSWYgdGhlcmUgaXMganVzdCANCiAg ICAgICAgICAgIG9uZSB0YXJnZXQsIG9yIGEgZGVmYXVsdCB0YXJnZXQsIGF0IHRoZSBJUCBBZGRy ZXNzIGFuZCBwb3J0LCANCiAgICAgICAgICAgIHRoaXMgd2lsbCB3b3JrLiAgVGhlIHRhcmdldCBy ZXR1cm5zIGl0cyBXV1VJIHNvIHRoZSBpbml0aWF0b3IgDQogICAgICAgICAgICBjYW4ga2VlcCBp dCBmb3IgZnV0dXJlIHVzZS4gDQogICAgICAgICAgDQogICAgICAgICAgIEktPkxvZ2luIFJlcXVl c3QgDQogICAgICAgICAgICAgIEluaXRpYXRvcldXVUk6IGNvbS5vcy5ob3N0aWQuMzQ1Njc4OTAg DQogICAgICAgICAgICAgIC4gDQogICAgICAgICAgICAgIC4gIHRleHQgY29tbWFuZHMgZmxvdyBo ZXJlIGR1cmluZyBhdXRoZW50aWNhdGlvbiBwaGFzZSAMDQogICAgICAgICAgICAgIC4gDQogICAg ICAgICAgIFQtPkxvZ2luIFJlc3BvbnNlIA0KICAgICAgICAgICAgICBUYXJnZXRXV1VJOiBjb20u YWNtZS5kaXNrYXJyYXkuc24uODY3NTMwOSANCiAgICAgICAgICAgICAgVGFyZ2V0QWxpYXM6IGZv byANCiAgICAgICAgICANCiAgICAgICAgICANCiAgICAgICAgICANCiAgICAgICAgIEFuc3dlcnMg dG8gUG90ZW50aWFsbHkgRnJlcXVlbnRseSBBc2tlZCBRdWVzdGlvbnMgDQogICAgICAgICAgDQog ICAgICAgICAgV2hhdCBoYXBwZW5zIGlmIGFuIEluaXRpYXRvciBXV1VJIGlzIG5vdCB1bmlxdWU/ IA0KICAgICAgICAgIA0KICAgICAgICAgICAtIFRhcmdldHMgd2lsbCBhdXRoZW50aWNhdGUgYm90 aCBhcyBzYW1lIGVudGl0eSANCiAgICAgICAgICAgLSBUYXJnZXRzIHdpbGwgYmVsaWV2ZSB0aGF0 IG9uZSBpbml0aWF0b3IgaXMgdXNpbmcgDQogICAgICAgICAgICAgdGhlbSB2aWEgZGlmZmVyZW50 IG5ldHdvcmsgaW50ZXJmYWNlcy4gDQogICAgICAgICAgIC0gSW5pdGlhdG9ycyBtYXkgZW5kIHVw IHNoYXJpbmcgYSBkZXZpY2UgYnkgDQogICAgICAgICAgICAgYWNjaWRlbnQuIA0KICAgICAgICAg IA0KICAgICAgICAgIA0KICAgICAgICAgMy4yIEFsaWFzIFN0cmluZyANCiAgICAgICAgIFRoZSBh bGlhcyBzdHJpbmcgaXMgYW4gQVNDSUkgc3RyaW5nIHRoYXQgaXMgdXNlZCB0byBpZGVudGlmeSBh IFN0b3JhZ2UgTm9kZSAgDQogICAgICAgICBvYmplY3QgdGhhdCBjYW4gYmUgYWNjZXNzZWQgdmlh IGEgcGFydGljdWxhciBOZXR3b3JrIEVudGl0eSBvYmplY3QgYW5kIGEgUG9ydGFsICANCiAgICAg ICAgIG9iamVjdC4gVGhlIGFsaWFzIHN0cmluZyBpcyBhIHZhcmlhYmxlIGxlbmd0aCwgYmV0d2Vl biAwIHRvIDI1NSBieXRlcywgDQogICAgICAgICB1c2VyLXJlYWRhYmxlIEFTQ0lJIHRleHQgc3Ry aW5nLiBUaGUgYWxpYXMgc3RyaW5nIGlzIHRlcm1pbmF0ZWQgd2l0aCBhdCBsZWFzdCAgDQogICAg ICAgICBvbmUgTlVMTCBjaGFyYWN0ZXIuIFRoZSBhbGlhcyBzdHJpbmcgZm9ybWF0IGlzIHNpbWls YXIgdG8gdGhhdCBvZiB0aGUgVU5JWCBmaWxlICANCiAgICAgICAgIGFkZHJlc3MgZm9ybWF0LiAN CiAgICAgICAgICANCiAgICAgICAgICANCiAgICAgICAgIDQuIGlTQ1NJIERpc2NvdmVyeSANCiAg ICAgICAgIEFuIGlTQ1NJIGluaXRpYXRvciBTdG9yYWdlIE5vZGUgY2FuIGRpc2NvdmVyIGFuIGlT Q1NJIHRhcmdldCBTdG9yYWdlIE5vZGUgDQogICAgICAgICBpbiB0aGUgZm9sbG93aW5nIGRpZmZl cmVudCB3YXlzOiANCiAgICAgICAgIGEpIFRhcmdldCBpbmZvcm1hdGlvbiBpcyBoYXJkLWNvZGVk IGF0IHRoZSBpbml0aWF0b3IuIA0KICAgICAgICAgYikgSW5pdGlhdG9yIHF1ZXJpZXMgc3RvcmFn ZSBuYW1lIHNlcnZlcnMuIA0KICAgICAgICAgYykgSW5pdGlhdG9yIGlzc3VlcyBhIG11bHRpY2Fz dCBkaXNjb3ZlcnkgbWVzc2FnZSB0byB0aGUgdGFyZ2V0cyBhbmQgdGhlIA0KICAgICAgICAgU05T LiANCiAgICAgICAgIGQpIEluaXRpYXRvciBxdWVyaWVzIGEgY2Fub25pY2FsIGlTQ1NJIHRhcmdl dCBTdG9yYWdlIE5vZGUgb2JqZWN0IGF0IGEgTmV0d29yayANCiAgICAgICAgIEVudGl0eSBvYmpl Y3QgZm9yIGEgbGlzdCBvZiB0YXJnZXRzLiANCiAgICAgICAgICANCiAgICAgICAgIDQuMSBUYXJn ZXQgSW5mb3JtYXRpb24gaXMgaGFyZC1jb2RlZCANCiAgICAgICAgIFRoZSBleGFjdCBtYW5uZXIg aW4gd2hpY2ggdGhlIHRhcmdldCBpbmZvcm1hdGlvbiBpcyBoYXJkLWNvZGVkIGF0IHRoZSBpbml0 aWF0b3IgIA0KICAgICAgICAgaXMgYW4gaW1wbGVtZW50YXRpb24gZGV0YWlsLiBUaGUgaW5mb3Jt YXRpb24gY291bGQgYmUgcHJlc2VudCBpbiBzb21lIHBlcnNpc3RlbnQgIA0KICAgICAgICAgbG9j YXRpb24gKHN1Y2ggYXMgYSBmaWxlKSB0aGF0IGNhbiBiZSBhY2Nlc3NlZCBieSB0aGUgaW5pdGlh dG9yLiANCiAgICAgICAgICANCiAgICAgICAgIDQuMiBJbml0aWF0b3IgcXVlcmllcyBhIFN0b3Jh Z2UgTmFtZSBTZXJ2ZXIgKFNOUykgDQogICAgICAgICBUaGUgaW5pdGlhdG9yIGNhbiBxdWVyeSBh IFNOUyBmb3IgYSBsaXN0IG9mIHRoZSB0YXJnZXRzIHRoYXQgaXQgY2FuIGFjY2Vzcy4gDQogICAg ICAgICBUaGUgdHlwZSBvZiBpbmZvcm1hdGlvbiB0aGF0IGlzIHN0b3JlZCBhdCB0aGUgU05TLCBh bmQgdGhlIGxpc3Qgb2YgcXVlcnkgYW5kIA0KICAgICAgICAgcmVnaXN0cmF0aW9uIEFQSXMgdGhh dCBzaG91bGQgYmUgc3VwcG9ydGVkIGJ5IHRoZSBTTlMgc2VydmVyIGFyZSBkZXNjcmliZWQgaW4g DQogICAgICAgICBTZWN0aW9uIDUgYmVsb3cuIFRoZSBpbXBsZW1lbnRhdGlvbiBkZXRhaWxzIG9m IHRoZSBTTlMgYXJlIGJleW9uZCB0aGUgc2NvcGUgb2YgDQogICAgICAgICB0aGlzIGRvY3VtZW50 LiANCiAgICAgICAgICANCiAgICAgICAgIDQuMyBJbml0aWF0b3IgSXNzdWVzIGEgTXVsdGljYXN0 IE1lc3NhZ2UgDQogICAgICAgICBBbiBpbml0aWF0b3IgY2FuIHNlbmQgYSBtdWx0aWNhc3QgbWVz c2FnZSB0byBib3RoIHN0b3JhZ2UgbmFtZSBzZXJ2ZXJzIGFuZCBpU0NTSSANCiAgICAgICAgIHRh cmdldHMuIEFuIGluaXRpYXRvciBNQVkgc2VuZCBhIG11bHRpY2FzdCAiU05TIGRpc2NvdmVyeSIg bWVzc2FnZSB0byB0aGUgKFRCRCkgDQogICAgICAgICBpU0NTSSBkaXNjb3ZlcnkgbXVsdGljYXN0 IGFkZHJlc3Mgb24gYSAoVEJEKSB3ZWxsLWtub3duIGlTQ1NJIFVEUCBwb3J0LiBBbiBpU0NTSSAN CiAgICAgICAgIFNOUyBNVVNUIHJlZ2lzdGVyIGFzIHBhcnQgb2YgdGhlIGlTQ1NJIGRpc2NvdmVy eSBtdWx0aWNhc3QgZ3JvdXAgYW5kIFNIQUxMIA0KICAgICAgICAgcmVzcG9uZCB0byB0aGlzIG1l c3NhZ2UgaW5kaWNhdGluZyB0aGF0IGl0IGZ1bmN0aW9ucyBhcyBhbiBTTlMuICBUYXJnZXRzIE1B WSANCiAgICAgICAgIHJlZ2lzdGVyIGFzIHBhcnQgb2YgdGhpcyBtdWx0aWNhc3QgZ3JvdXAgYnV0 IFNIQUxMIE5PVCByZXNwb25kIHRvIHRoaXMgbWVzc2FnZS4gDA0KICAgICAgICAgQWx0ZXJuYXRp dmVseSwgYW4gaW5pdGlhdG9yIE1BWSBzZW5kIGEgbXVsdGljYXN0ICJhbGwgc3RvcmFnZSBkaXNj b3ZlcnkiIG1lc3NhZ2UgDQogICAgICAgICB0byB0aGUgc2FtZSBtdWx0aWNhc3QgYWRkcmVzcy4g IEEgc3RvcmFnZSBuYW1lIHNlcnZlciBNVVNUIHJlc3BvbmQgdG8gdGhpcyANCiAgICAgICAgIG1l c3NhZ2UgYXMgaWYgdGhlIG1lc3NhZ2Ugd2VyZSB0aGUgIlNOUyBkaXNjb3ZlcnkgbWVzc2FnZSIu ICAgQSByZWdpc3RlcmVkIA0KICAgICAgICAgdGFyZ2V0IE1BWSByZXNwb25kIHRvIHRoaXMgbWVz c2FnZSBpbmRpY2F0aW5nIHRoYXQgaXQgaXMgYW4gaVNDU0kgdGFyZ2V0LiANCiAgICAgICAgIEEg ZGV2aWNlIHRoYXQgcHJvdmlkZXMgYm90aCBpU0NTSSB0YXJnZXQgYW5kIHN0b3JhZ2UgbmFtZSBz ZXJ2ZXIgZnVuY3Rpb25zIFNIQUxMIA0KICAgICAgICAgcmVzcG9uZCB3aXRoIGEgbWVzc2FnZSBp bmRpY2F0aW5nIHRoYXQgaXQgcHJvdmlkZXMgYm90aCBzZXJ2aWNlcy4gRmluYWxseSwgDQogICAg ICAgICB0aGUgaW5pdGlhdG9yIE1BWSBzZW5kIGEgbXVsdGljYXN0ICJpU0NTSSB0YXJnZXRzIG9u bHkiIG1lc3NhZ2UgdG8gdGhlIHNhbWUgDQogICAgICAgICBtdWx0aWNhc3QgYWRkcmVzcywgYW5k IG9ubHkgdGhlIGlTQ1NJIHRhcmdldHMgYW5kIHRoZSBpU0NTSSBkZXZpY2VzIHRoYXQgcHJvdmlk ZSANCiAgICAgICAgIGJvdGggaVNDU0kgdGFyZ2V0IGFuZCBzdG9yYWdlIG5hbWUgc2VydmVyIGZ1 bmN0aW9ucyBNQVkgcmVzcG9uZCB0byB0aGlzIG1lc3NhZ2UuIA0KICAgICAgICAgVGhlIGNob2lj ZSBvZiBzdGF0aWMgY29uZmlndXJhdGlvbiwgU05TIGRpc2NvdmVyeSBvciBhbGwgc3RvcmFnZSBk aXNjb3ZlcnkgDQogICAgICAgICBwcm90b2NvbHMgaXMgYSBjb25maWd1cmF0aW9uIGNob2ljZSBv ZiB0aGUgaW5pdGlhdG9yLiAgVGhlcmUgaXMgbm8gDQogICAgICAgICBhdXRoZW50aWNhdGlvbiBw cm9jZXNzIGFzc29jaWF0ZWQgd2l0aCB0aGUgaVNDU0kgZGlzY292ZXJ5IG11bHRpY2FzdCANCiAg ICAgICAgIG1lc3NhZ2VzLiANCiAgICAgICAgICANCiAgICAgICAgIElmIHRoZSBpbml0aWF0b3Ig cmVjZWl2ZXMgb25lIG9yIG1vcmUgcmVzcG9uc2VzIHRvIHRoZSAiU05TIGRpc2NvdmVyeSIgbWVz c2FnZSwgDQogICAgICAgICBpdCBtYXkgaW50ZXJhY3Qgd2l0aCB0aG9zZSBkZXZpY2UgZm9yIGl0 cyB0YXJnZXQgZGlzY292ZXJ5IHNlcnZpY2VzLiAgSWYgYW4gDQogICAgICAgICBpbml0aWF0b3Ig cmVjZWl2ZXMgcmVzcG9uc2VzIHRvIHRoZSAiYWxsIHN0b3JhZ2UgZGlzY292ZXJ5IiBtZXNzYWdl IGZyb20gb25seSANCiAgICAgICAgIHRhcmdldHMsIGl0IG1heSBhdHRlbXB0IExvZ2luIHdpdGgg ZWFjaCBvZiB0aG9zZSBkZXZpY2VzLiBJZiBhbiBpbml0aWF0b3IgDQogICAgICAgICByZWNlaXZl cyByZXNwb25zZXMgdG8gYW4gImFsbCBzdG9yYWdlIGRpc2NvdmVyeSIgbWVzc2FnZSBmcm9tIGJv dGggdGFyZ2V0cyBhbmQgDQogICAgICAgICBzdG9yYWdlIG5hbWUgc2VydmVycywgaXQgbWF5IGNo b29zZSB0byBpbnRlcmFjdCB3aXRoIHRoZSBzdG9yYWdlIG5hbWUgc2VydmVycyANCiAgICAgICAg IGZvciB0YXJnZXQgZGlzY292ZXJ5IHNlcnZpY2VzIGFuZC9vciBhdHRlbXB0IExvZ2luIGRpcmVj dGx5IHdpdGggcmVzcG9uZGluZyANCiAgICAgICAgIHJlZ2lzdGVyZWQgdGFyZ2V0cy4gDQogICAg ICAgICAgDQogICAgICAgICBJbiBzdW1tYXJ5LCB0aGlzIGRpc2NvdmVyeSBhcHByb2FjaCBpcyBm bGV4aWJsZSBpbiB0aGF0IHRoZSBpbml0aWF0b3JzIGhhdmUgdGhlIA0KICAgICAgICAgZnJlZWRv bSB0byBzZWxlY3Qgc3RhdGljIGNvbmZpZ3VyYXRpb24sIGEgbXVsdGljYXN0IGJhc2VkIGRpc2Nv dmVyeSBtZWNoYW5pc20gDQogICAgICAgICBmb3Igc21hbGwsIGlzb2xhdGVkIGlTQ1NJIGVudmly b25tZW50cywgb3IgdGhleSBjYW4gY2hvb3NlIGEgc2NhbGFibGUgc3RvcmFnZSANCiAgICAgICAg IG5hbWUgc2VydmVyIGJhc2VkIGRpc2NvdmVyeSBtZWNoYW5pc20gZm9yIGxhcmdlIGlTQ1NJIGVu dmlyb25tZW50cy4gIA0KICAgICAgICAgQWRkaXRpb25hbGx5LCB0YXJnZXRzIG1heSBiZSBjb25m aWd1cmVkIHRvIHBhcnRpY2lwYXRlIG9yIG5vdCANCiAgICAgICAgIHBhcnRpY2lwYXRlIGluIHRo ZSBtdWx0aWNhc3QgZ3JvdXAgKGUuZy4sIGlmIHRoZXJlIGlzIGFuIFNOUyBhdmFpbGFibGUsIHRo ZW4gDQogICAgICAgICB0aGV5IG1heSBjaG9zZSBlaXRoZXIgZHluYW1pY2FsbHkgb3IgYnkgY29u ZmlndXJhdGlvbiBub3QgdG8gcmVnaXN0ZXIgaW4gdGhlIA0KICAgICAgICAgZ3JvdXApLiANCiAg ICAgICAgICANCiAgICAgICAgICANCiAgICAgICAgICANCiAgICAgICAgIDQuNCBTZW5kVGFyZ2V0 cyBDb21tYW5kIA0KICAgICAgICAgIA0KICAgICAgICAgQW4gaW5pdGlhdG9yIG1heSwgYWZ0ZXIg dGhlIExvZ2luIHByb2Nlc3MsIGNvbm5lY3QgdG8gYW4gaVNDU0kgDQogICAgICAgICBjYW5vbmlj YWwgdGFyZ2V0IGFuZCByZXF1ZXN0IGZvciBhIGxpc3Qgb2YgdGFyZ2V0IFdXVUlzLCB2aWEgYSBz ZXBhcmF0ZSANCiAgICAgICAgIFNlbmRUYXJnZXRzIGNvbW1hbmQsIGF0IHRoZSBwYXJ0aWN1bGFy IE5ldHdvcmsgRW50aXR5IG9iamVjdCBhbmQgdGhlIFBvcnRhbCANCiAgICAgICAgIG9iamVjdC4g VGhlIHJldHVybmVkIGRhdGEgZm9yIHRoaXMgcmVxdWVzdCBzaGFsbCBjb250YWluIGEgbGlzdCAN CiAgICAgICAgIG9mIHR1cGxlcywgd2hlcmUgZWFjaCB0dXBsZSBjb25zaXN0cyBvZiBhIHRhcmdl dCBXV1VJIGFuZCBhbiBJUCANCiAgICAgICAgIGFkZHJlc3M6UG9ydCBhbmQgYW4gb3B0aW9uYWwg YWxpYXMgc3RyaW5nLiAgVGhlIGNhbm9uaWNhbCB0YXJnZXQgTVVTVCBzdXBwb3J0IA0KICAgICAg ICAgdGhpcyByZXF1ZXN0IGFuZCB0aGUgcmV0dXJuZWQgbGlzdCBNVVNUIGNvbnRhaW4gYXQgbGVh c3Qgb25lIGVudHJ5IGZvciB0aGUgDQogICAgICAgICBjYW5vbmljYWwgdGFyZ2V0IGl0c2VsZi4g IFRoZSBpbml0aWF0b3IgY2FuIHRoZW4gYXR0ZW1wdCBpU0NTSSBMb2dpbiB0byBlYWNoIG9mIA0K ICAgICAgICAgdGhlIHRhcmdldHMgc3BlY2lmaWVkIGluIHRoZSByZXR1cm5lZCBsaXN0LiANCiAg ICAgICAgICANCiAgICAgICAgIER1cmluZyB0aGUgbG9naW4gY29tbWFuZCwgdGhlIGluaXRpYXRv ciBzZXRzIHRoZSB0YXJnZXQgYWxpYXMgdG8gImlTQ1NJIiANCiAgICAgICAgIHdpdGggYSBXV1VJ IG9mICIqIi4gIElmIHRoZSBsb2dpbiBzdWNjZWVkcywgdGhlIGluaXRpYXRvciBtYXkgc2VuZCBh IA0KICAgICAgICAgc2VuZFRhcmdldHMgdGV4dCBjb21tYW5kLiANCiAgICAgICAgICANCiAgICAg ICAgIFRoZSByZXNwb25zZSB0byB0aGlzIGNvbW1hbmQgaXMgYSB0ZXh0IHJlc3BvbnNlIGNvbnRh aW5pbmcgYSBsaXN0IG9mIA0KICAgICAgICAgdHVwbGVzLiANCiAgICAgICAgICANCiAgICAgICAg IFRoZSBmb3JtYXQgb2YgdGhpcyB0ZXh0IHN0cmluZyBpcyBhcyBmb2xsb3dzOiANCiAgICAgICAg ICANCiAgICAgICAgIDxUYXJnZXRXV1VJLElQIEFkZHJlc3M6UG9ydCBOdW1iZXIsIEFsaWFzIFN0 cmluZz4gDQogICAgICAgICBUaGUgZXhhY3QgZm9ybWF0IG9mIHRoZSB0ZXh0IHN0cmluZyBpcyBh cyBmb2xsb3dzOiAMDQogICAgICAgICBUYXJnZXRXV1VJOmNvbS5hY21lLmRpc2thcnJheS5zbi44 Njc1MzA5IA0KICAgICAgICAgVGFyZ2V0QWRkcmVzczoxMC4xLjAuNDU6MzAwMCANCiAgICAgICAg IFRhcmdldEFsaWFzOmZvby9kaXNrQ29udHJvbGxlcjEgDQogICAgICAgICBUYXJnZXRXV1VJOmNv bS5hY21lLmRpc2thcnJheS5zbi44ODg4ODg4IA0KICAgICAgICAgVGFyZ2V0QWRkcmVzczoxMC4x LjAuNDY6MzAwMCANCiAgICAgICAgIFRhcmdldEFsaWFzOmZvby9kaXNrQ29udHJvbGxlcjIgDQog ICAgICAgICAgDQogICAgICAgICBBIGxpbmUgY29udGFpbmluZyB0aGUgdGVybSBUYXJnZXRXV1VJ OiBpcyB0aGUgc3RhcnQgb2YgYSB0YXJnZXQ7IGZvbGxvd2VkIGJ5IGl0cyANCiAgICAgICAgIGFk ZHJlc3MgYW5kIGFsaWFzLCB1bnRpbCB0aGUgbmV4dCB0YXJnZXRXV1VJOiBsaW5lLiBJZiBubyB0 YXJnZXQgYWRkcmVzc2VzIGFyZSANCiAgICAgICAgIGdpdmVuLCB0aGUgaW5pdGlhdG9yIGNhbiBs b2cgaW4gdG8gdGhlIHNhbWUgYWRkcmVzcyBhcyB0aGF0IHVzZWQgZm9yIGluIHRoZSANCiAgICAg ICAgIFNlbmRUYXJnZXRzIGNvbW1hbmQsIGFuZCBsb2dpbiB0byB0aGUgZGVmYXVsdCB0YXJnZXQu ICBJZiBtdWx0aXBsZSBwYXRocyB0byB0aGUgDQogICAgICAgICBXV1VJIGFyZSBrbm93biwgbXVs dGlwbGUgYWRkcmVzcyBsaW5lcyBtYXkgYmUgZ2l2ZW4uIA0KICAgICAgICAgIA0KICAgICAgICAg IA0KICAgICAgICAgIA0KICAgICAgICAgIA0KICAgICAgICAgNC40LjEgUG9ydCBSZWRpcmVjdCBD b21tYW5kIA0KICAgICAgICAgRHVyaW5nIHRoZSBMb2dpbiBwcm9jZXNzLCBhIHRhcmdldCBtYXkg cmVkaXJlY3QgdGhlIGluaXRpYXRvciB0byBjb25uZWN0IHRvIA0KICAgICAgICAgYW5vdGhlciBJ UCBhZGRyZXNzOlBvcnQgYW5kIHRoZW4gdGVybWluYXRlIHRoZSBMb2dpbiBjb21tYW5kIChhbmQg aXRzIA0KICAgICAgICAgY29ubmVjdGlvbikuICBBIHRhcmdldCBtaWdodCBkbyB0aGlzIGZvciBs b2FkIGJhbGFuY2luZyBvciBpdCBtaWdodCBkbyB0aGlzIHRvIA0KICAgICAgICAgcHJvdmlkZSBt dWx0aXBsZSB2aXJ0dWFsIHRhcmdldHMgdGhyb3VnaCBhIHNpbXBsZSBpbml0aWF0b3IgZGlzY292 ZXJ5IHByb3RvY29sLiANCiAgICAgICAgIFRoZSB0YXJnZXQncyByZXNwb25zZSBpcyBhIHRleHQg c3RyaW5nIHRoYXQgaXMgaW4gdGhlIGZvbGxvd2luZyBmb3JtYXQ6IA0KICAgICAgICAgIlJFRElS RUNUOiBUYXJnZXRXV1VJOmNvbS5hY21lLmRpc2thcnJheS5zbi45OTk5OTkgDQogICAgICAgICBU YXJnZXRBZGRyZXNzOjEwLjEuMC40OTozMDAwIA0KICAgICAgICAgVGFyZ2V0QWxpYXM6Zm9vL2Rp c2tDb250cm9sbGVyMyIgDQogICAgICAgICAgDQogICAgICAgICAgDQogICAgICAgICA1LiAgU3Rv cmFnZSBOYW1lIFNlcnZlciAoU05TKSANCiAgICAgICAgICANCiAgICAgICAgIFRoZSBmb2xsb3dp bmcgc2VjdGlvbiBkZXNjcmliZXMgcmVxdWlyZW1lbnRzIGZvciBhbnkgU3RvcmFnZSBOYW1lIFNl cnZlciANCiAgICAgICAgIHVzZWQgdG8gc3VwcG9ydCBpU0NTSS4gIEFuIGV4YW1wbGUgb2YgYSBT dG9yYWdlIE5hbWUgU2VydmVyIGlzIHRoZSBpU05TIA0KICAgICAgICAgZGVzY3JpYmVkIGluIHRo ZSBkcmFmdCBkb2N1bWVudCBkcmFmdC1pZXRmLWlwcy1pU05TLTAwLnR4dCBbOF0uIA0KICAgICAg ICAgIA0KICAgICAgICAgNS4xICBPdmVydmlldyANCiAgICAgICAgICANCiAgICAgICAgIFRoZSBT TlMgc2hhbGwgYmUgYXJjaGl0ZWN0ZWQgdXNpbmcgYSBjbGllbnQtc2VydmVyIHBhcmFkaWdtLCB3 aXRoIHRoZSBTTlMgc2VydmVyIA0KICAgICAgICAgcHJlZG9taW5hbnRseSBzZXJ2aW5nIGEgcGFz c2l2ZSByb2xlLiBTTlMgY2xpZW50cyBhY3RpdmVseSByZWdpc3RlciBhbmQgDQogICAgICAgICBt YW5pcHVsYXRlIGVudGl0eSBvYmplY3RzIGFuZCB0aGVpciBhdHRyaWJ1dGVzIGluIHRoZSBTTlMg c2VydmVyLiAgVGhlIFNOUyANCiAgICAgICAgIHNlcnZlciBNQVkgc2VuZCBhc3luY2hyb25vdXMg c3RhdGUgY2hhbmdlIG5vdGlmaWNhdGlvbnMgdG8gcmVnaXN0ZXJlZCBTTlMgDQogICAgICAgICBj bGllbnRzIGluIHJlc3BvbnNlIHRvIGFuIGFjdGlvbiBieSBhIFNOUyBjbGllbnQuICBFeGFtcGxl cyBvZiBTTlMgY2xpZW50cyANCiAgICAgICAgIGluY2x1ZGUgaW5pdGlhdG9ycywgdGFyZ2V0cywg bWFuYWdlbWVudCBzdGF0aW9ucywgYW5kIHN3aXRjaGVzLiAgVGhlIFNOUyBzZXJ2ZXIgDQogICAg ICAgICBjYW4gYmUgaG9zdGVkIG9uIGEgdGFyZ2V0LCBzd2l0Y2gsIG9yIHN0YW5kLWFsb25lIHNl cnZlci4gDQogICAgICAgICAgDQogICAgICAgICA1LjIgIExvZ2luIENvbnRyb2wgYW5kIFpvbmlu ZyANCiAgICAgICAgICANCiAgICAgICAgIFRoZSBTTlMgTVVTVCBzdXBwb3J0IFpvbmluZyBhbmQg TG9naW4gY29udHJvbC4gIFRoZSBTTlMgbXVzdCBwcm92aWRlIFNOUyBjbGllbnRzIA0KICAgICAg ICAgd2l0aCB0aGUgYWJpbGl0eSB0byBlbmZvcmNlIHpvbmluZyBjb25maWd1cmF0aW9ucyB3aGlj aCBtYXkgZXhpc3Qgb24gdGhlIFNOUyANCiAgICAgICAgIHNlcnZlci4gIFRhcmdldHMgYW5kIG1h bmFnZW1lbnQgc3RhdGlvbnMgc2hhbGwgYmUgYWJsZSB0byByZWdpc3RlciAoaS5lLiwgDQogICAg ICAgICB1cGxvYWQpIExvZ2luIENvbnRyb2wgYW5kIFpvbmluZyBjb25maWd1cmF0aW9ucyB0byB0 aGUgaVNOUyBpZiBhdXRob3JpemVkIGJ5IHRoZSANCiAgICAgICAgIGVuZCB1c2VyLiANCiAgICAg ICAgICANCiAgICAgICAgIFpvbmluZyBhbmQgTG9naW4gY29udHJvbCBzdXBwb3J0cyB0d28gc2Vw YXJhdGUgcHVycG9zZXM6IA0KICAgICAgICAgIA0KICAgICAgICAgNS4yLjEgIERpc2NvdmVyeSBE b21haW4gUGFydGl0aW9ucyANCiAgICAgICAgICANCiAgICAgICAgIFRoZSBTTlMgU0hBTEwgc3Vw cG9ydCB0aGUgYWJpbGl0eSB0byBwYXJ0aXRpb24gdGhlIHN0b3JhZ2UgbmV0d29yayBpbnRvIHNl cGFyYXRlIA0KICAgICAgICAgIkRpc2NvdmVyeSBEb21haW5zIi4gIFRoZSBTTlMgc2hhbGwgbm90 IHByb3ZpZGUgaW5mb3JtYXRpb24gaWYgdGhlIFNOUyBjbGllbnQgDA0KICAgICAgICAgcGVyZm9y bWluZyB0aGUgcXVlcnkgaXMgbm90IGluIGEgY29tbW9uIHpvbmUgKGkuZS4sICJEaXNjb3Zlcnkg RG9tYWluIikgYXMgdGhlIA0KICAgICAgICAgU05TIGNsaWVudCB0aGF0IGlzIHRoZSBzdWJqZWN0 IG9mIHRoZSByZXF1ZXN0LiAgVGhpcyBjYXBhYmlsaXR5IHByZXZlbnRzIGFuIA0KICAgICAgICAg aW5pdGlhdG9yIGZyb20gYXR0ZW1wdGluZyBhbiBpU0NTSSBsb2dpbiB0byBldmVyeSBzaW5nbGUg dGFyZ2V0IGluIGEgbGFyZ2UgDQogICAgICAgICBlbnRlcnByaXNlIG5ldHdvcmssIGFuZCBpcyB0 aGUgaVNDU0kgZXF1aXZhbGVudCBvZiAiU29mdCIgem9uaW5nLiANCiAgICAgICAgICANCiAgICAg ICAgIDUuMi4yICBMb2dpbiBDb250cm9sIA0KICAgICAgICAgIA0KICAgICAgICAgVG8gc3VwcG9y dCBsb2dpbiBhY2Nlc3Mgc2VjdXJpdHkgd2hpY2ggaXMgc3BlY2lmaWVkIGluIHRoZSBjdXJyZW50 IGlTQ1NJIGRyYWZ0IA0KICAgICAgICAgKEFwcGVuZGl4IEEpIFs3XSBhbmQgTUFZIGJlIGltcGxl bWVudGVkIGJ5IHRoZSBpU0NTSSB0YXJnZXQuICBUaGUgU05TIHNoYWxsIA0KICAgICAgICAgc3Vw cG9ydCBsb2dpbiBjb250cm9sIGJ5IHN0b3JpbmcgYSBtYXBwaW5nIG9mIGluaXRpYXRvcnMgdGhh dCBhcmUgcGVybWl0dGVkIHRvIA0KICAgICAgICAgYWNjZXNzIGVhY2ggdGFyZ2V0LiAgVGFyZ2V0 cyBzaGFsbCBiZSBhYmxlIHRvIHF1ZXJ5IHRoZSBTTlMgZm9yIA0KICAgICAgICAgYSBsaXN0IG9m IGluaXRpYXRvcnMgdGhhdCBhcmUgYWxsb3dlZCBsb2dpbiBhY2Nlc3MuICBUaGlzIGxpc3Qgc2hh bGwgaW5jbHVkZSANCiAgICAgICAgIHRoZSBrZXkgYXR0cmlidXRlIChlLmcuLCBXV1VJKSB1c2Vk IHRvIGlkZW50aWZ5IHRoZSBpbml0aWF0b3IuICBUaGlzIGNhcGFiaWxpdHkgDQogICAgICAgICBp cyB0aGUgaVNDU0kgZXF1aXZhbGVudCBvZiAiSGFyZCIgem9uaW5nLiANCiAgICAgICAgICANCiAg ICAgICAgIDUuMyAgICBPYmplY3QgTW9kZWwgDQogICAgICAgICAgDQogICAgICAgICBUaGUgU05T IE1VU1Qgc3RvcmUgdGhlIGZvbGxvd2luZyBvYmplY3RzIGFuZCBhdHRyaWJ1dGVzOiANCiAgICAg ICAgICANCiAgICAgICAgICAgICBOZXR3b3JrIEVudGl0eTogDQogICAgICAgICAgICAgICAtICBF bnRpdHkgSWRlbnRpZmllciANCiAgICAgICAgICAgICAgIC0gIE1hbmFnZW1lbnQgSVAgQWRkcmVz cyANCiAgICAgICAgICAgICAgIC0gIEVudGl0eSBUeXBlIChpU0NTSSkgDQogICAgICAgICAgDQog ICAgICAgICAgICAgUG9ydGFsOiANCiAgICAgICAgICAgICAgIC0gIFBvcnRhbCBJbmRleCANCiAg ICAgICAgICAgICAgIC0gIElQIEFkZHJlc3MgDQogICAgICAgICAgICAgICAtICBUQ1AgUG9ydCBO dW1iZXIgDQogICAgICAgICAgDQogICAgICAgICAgICAgU3RvcmFnZSBOb2RlOiANCiAgICAgICAg ICAgICAgIC0gIFdXVUkgDQogICAgICAgICAgICAgICAtICBBbGlhcyANCiAgICAgICAgICAgICAg IC0gIE5vZGUgVHlwZSAodGFyZ2V0IG9yIGluaXRpYXRvciBvciBib3RoKSANCiAgICAgICAgICAN CiAgICAgICAgICAgICBab25lOiANCiAgICAgICAgICAgICAgIC0gIFpvbmUgc3ltYm9saWMgbmFt ZSANCiAgICAgICAgICAgICAgIC0gIFpvbmUgSUQgDQogICAgICAgICAgICAgICAtICBab25lIE1l bWJlcjogIFdXVUkgDQogICAgICAgICAgICAgICAtICBab25lIE1lbWJlcjogIElQIEFkZHJlc3Mg DQogICAgICAgICAgDQogICAgICAgICBBIGRpYWdyYW0gb2YgaG93IHRoZSBhYm92ZSBvYmplY3Rz IGFyZSByZWxhdGVkIGlzIHNob3duIGJlbG93LiANCiAgICAgICAgICANCiAgICAgICAgICAgICAr LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLSsgDQogICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICBJUCBOZXR3 b3JrICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8IA0KICAgICAgICAgICAgICstLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t KyANCiAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgfCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgfCANCiAgICAgICAgICAgICArLS0tLS0rLS0tLS0tKy0t LS0tLSstLS0tLSsgICAgICAgICAgICArLS0tLS0rLS0tLS0tKy0tLS0tLSstLS0tLSsgDQogICAg ICAgICAgICAgfCAgICAgfCBQT1JUQUwgICAgICB8ICAgICB8ICAgICAgICAgICAgfCAgICAgfCBQ T1JUQUwgICAgICB8ICAgICB8IA0KICAgICAgICAgICAgIHwgICAgIHwgLUlQIEFkZHIgMSAgfCAg ICAgfCAgICAgICAgICAgIHwgICAgIHwgLUlQIEFkZHIgMiAgfCAgICAgfCANCiAgICAgICAgICAg ICB8ICAgICB8IC1UQ1AgUG9ydCAxIHwgICAgIHwgICAgICAgICAgICB8ICAgICB8IC1UQ1AgUG9y dCAyIHwgICAgIHwgDQogICAgICAgICAgICAgfCAgICAgKy0tLS0tKyArLS0tLS0rICAgICB8ICAg ICAgICAgICAgfCAgICAgKy0tLS0tKyArLS0tLS0rICAgICB8IA0KICAgICAgICAgICAgIHwgICAg ICAgICAgIHwgfCAgICAgICAgICAgfCAgICAgICAgICAgIHwgICAgICAgICAgIHwgfCAgICAgICAg ICAgfCANCiAgICAgICAgICAgICB8ICAgICAgICAgICB8IHwgICAgICAgICAgIHwgICAgICAgICAg ICB8ICAgICAgICAgICB8IHwgICAgICAgICAgIHwgDQogICAgICAgICAgICAgfCAgKy0tLS0tLS0t KyArLS0tLS0tLS0rICB8ICAgICAgICAgICAgfCAgICstLS0tLS0tKyArLS0tLS0tLS0rICB8IA0K ICAgICAgICAgICAgIHwgIHwgICAgICAgICAgICAgICAgICAgfCAgfCAgICAgICAgICAgIHwgICB8 ICAgICAgICAgICAgICAgICAgfCAgfCANCiAgICAgICAgICAgICB8ICB8ICBTVE9SQUdFIE5PREUg ICAgIHwgIHwgICAgICAgICAgICB8ICAgfCAgU1RPUkFHRSBOT0RFICAgIHwgIHwgDA0KICAgICAg ICAgICAgIHwgIHwgIC1XV1VJICAgICAgICAgICAgfCAgfCAgICAgICAgICAgIHwgICB8ICAgLVdX VUkgICAgICAgICAgfCAgfCANCiAgICAgICAgICAgICB8ICB8ICAtQWxpYXM6ICJzZXJ2ZXIxInwg IHwgICAgICAgICAgICB8ICAgfCAgQWxpYXM6ICJkaXNrMSIgIHwgIHwgDQogICAgICAgICAgICAg fCAgfCAgLVR5cGU6IGluaXRpYXRvciB8ICB8ICAgICAgICAgICAgfCAgIHwgICAtVHlwZTogdGFy Z2V0ICB8ICB8IA0KICAgICAgICAgICAgIHwgIHwgICAgICAgICAgICAgICAgICAgfCAgfCAgICAg ICAgICAgIHwgICB8ICAgICAgICAgICAgICAgICAgfCAgfCANCiAgICAgICAgICAgICB8ICArLS0t LS0tLS0tLS0tLS0tLS0tLSsgIHwgICAgICAgICAgICB8ICAgKy0tLS0tLS0tLS0tLS0tLS0tLSsg IHwgDQogICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAg fCAgICAgICAgICAgICAgICAgICAgICAgICB8IA0KICAgICAgICAgICAgIHwgICAgTkVUV09SSyBF TlRJVFkgICAgICAgfCAgICAgICAgICAgIHwgICAgTkVUV09SSyBFTlRJVFkgICAgICAgfCANCiAg ICAgICAgICAgICB8ICAgLUVudGl0eSBJRCAoRE5TKTogICAgIHwgICAgICAgICAgICB8ICAgLUVu dGl0eSBJRCAoRE5TKTogICAgIHwgDQogICAgICAgICAgICAgfCAgICAic3RyZzEuZm9vLmNvbSIg ICAgICB8ICAgICAgICAgICAgfCAgICAic3RyZzIuYmFyLmNvbSIgICAgICB8IA0KICAgICAgICAg ICAgIHwgICAtVHlwZTogaVNDU0kgICAgICAgICAgfCAgICAgICAgICAgIHwgICAtVHlwZTogaVND U0kgICAgICAgICAgfCANCiAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICAgIHwg ICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICAgIHwgDQogICAgICAgICAgICAgKy0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rICAgICAgICAgICAgKy0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0rIA0KICAgICAgICAgIA0KICAgICAgICAgQSBaT05FIGNvbnRhaW5zIG9uZSBvciBtb3Jl IE5FVFdPUksgRU5USVRZIG9iamVjdHMuICBFYWNoIE5FVFdPUksgRU5USVRZIA0KICAgICAgICAg b2JqZWN0IGNvbnRhaW5zIG9uZSBvciBtb3JlIFBPUlRBTCBvYmplY3RzLCBhbmQgb25lIG9yIG1v cmUgU1RPUkFHRSBOT0RFIA0KICAgICAgICAgb2JqZWN0cy4gDQogICAgICAgICAgDQogICAgICAg ICA1LjQgIFNOUyBNZXNzYWdlIEZvcm1hdCBSZXF1aXJlbWVudHMgDQogICAgICAgICAgDQogICAg ICAgICBUaGUgU05TIHByb3RvY29sIFNIQUxMIHVzZSBhIGZsZXhpYmxlIGFuZCBleHRlbnNpYmxl IG1lc3NhZ2UgZm9ybWF0IHN1Y2ggYXMgDQogICAgICAgICBUTFYgKFRMViBpcyBhbHJlYWR5IHVz ZWQgaW4gbWFueSBuZXR3b3JraW5nIHByb3RvY29scyBzdWNoIGFzIERIQ1ApLiAgVGhlIFNOUyAN CiAgICAgICAgIHByb3RvY29sIHNoYWxsIGFsbG93IG1hbmlwdWxhdGlvbiBvZiBtdWx0aXBsZSBv YmplY3RzIGFuZCBhdHRyaWJ1dGVzIGluIHRoZSBTTlMgDQogICAgICAgICBzZXJ2ZXIgdGhyb3Vn aCBhIHNpbmdsZSBtZXNzYWdlIGFuZCByZXNwb25zZS4gDQogICAgICAgICAgDQogICAgICAgICA1 LjUgIFNOUyBBdXRoZW50aWNhdGlvbiBSZXF1aXJlbWVudHMgDQogICAgICAgICAgDQogICAgICAg ICBUaGUgU05TIHByb3RvY29sIFNIQUxMIGluY2x1ZGUgb3B0aW9uYWwgYXV0aGVudGljYXRpb24g b2YgU05TIHByb3RvY29sIA0KICAgICAgICAgbWVzc2FnZXMgZnJvbSBTTlMgY2xpZW50cy4gVGhl IGF1dGhlbnRpY2F0aW9uIG1lY2hhbmlzbSB3aWxsIGFsbG93IGZvciANCiAgICAgICAgIGF1dGhl bnRpY2F0aW9uIG9mIGJvdGggY2xpZW50IGFuZCBzZXJ2ZXIuIA0KICAgICAgICAgIA0KICAgICAg ICAgNS42IFNOUyBRdWVyeSBhbmQgUmVnaXN0cmF0aW9uIFNlcnZpY2VzIFJlcXVpcmVtZW50cyAN CiAgICAgICAgIFRoZSBTTlMgcHJvdG9jb2wgYWxsb3dzIGluaXRpYXRvcnMgYW5kIHRhcmdldHMg dG8gcmVnaXN0ZXIgdGhlbXNlbHZlcyBhdCAgDQogICAgICAgICB0aGUgU05TIHNlcnZlci4gSW5p dGlhdG9ycyBhbmQgdGFyZ2V0cyBjYW4gYWxzbyBxdWVyeSB0aGUgU05TIHNlcnZlciBmb3IgDQog ICAgICAgICBpbmZvcm1hdGlvbi4gRm9yIGV4YW1wbGUsIHRhcmdldHMgY2FuIHJlZ2lzdGVyIHRo ZW1zZWx2ZXMgYXQgdGhlIFNOUyBzZXJ2ZXIsIGFuZCANCiAgICAgICAgIHRoZSBpbml0aWF0b3Jz IGNhbiBxdWVyeSB0aGUgU05TIHNlcnZlciBhYm91dCB3aGljaCB0YXJnZXRzIHRoZXkgY2FuIGFj Y2Vzcy4gDQogICAgICAgICAgDQogICAgICAgICBEdXJpbmcgcmVnaXN0cmF0aW9uLCB0aGUgaW5p dGlhdG9ycyBhbmQgdGhlIHRhcmdldHMgbXVzdCBwcm92aWRlIHRoZSAgDQogICAgICAgICBmb2xs b3dpbmcgaW5mb3JtYXRpb246IA0KICAgICAgICAgYSkgU3RvcmFnZSBFbnRpdHkgSUQgDQogICAg ICAgICBiKSBQb3J0YWwgb2JqZWN0IGFkZHJlc3MgKElQIGFkZHJlc3MgYW5kIFBvcnQgTnVtYmVy KSANCiAgICAgICAgIGMpIFdXVUkgaW5mb3JtYXRpb24gDQogICAgICAgICBkKSBTdG9yYWdlIG5v ZGUgdHlwZSANCiAgICAgICAgICANCiAgICAgICAgIFRoZXkgY291bGQgb3B0aW9uYWxseSBhbHNv IHByb3ZpZGUgb3RoZXIgaW5mb3JtYXRpb24gc3VjaCBhczogDQogICAgICAgICBhKSBab25lIHJl bGF0ZWQgaW5mb3JtYXRpb24gDQogICAgICAgICBiKSBBbGlhcyBzdHJpbmcgaW5mb3JtYXRpb24g DQogICAgICAgICAgDQogICAgICAgICBXaGVuIHF1ZXJ5aW5nIGFkZHJlc3MgaW5mb3JtYXRpb24g aW4gb3JkZXIgdG8gZXN0YWJsaXNoIGFuIGlTQ1NJICANCiAgICAgICAgIGNvbm5lY3Rpb24sIHRo ZSBxdWVyeSwgDQogICAgICAgICBhcyBhIG1pbmltdW0sIHNob3VsZCByZXR1cm4gdGhlIGZvbGxv d2luZyBpbmZvcm1hdGlvbjogDQogICAgICAgICBhKSBTdG9yYWdlIEVudGl0eSBJUCBhZGRyZXNz IA0KICAgICAgICAgIA0KICAgICAgICAgVGhlIFBvcnRhbCBPYmplY3QgSVAgYWRkcmVzcyBjYW4g YmUgdGhlIHNhbWUgYXMgdGhlIFN0b3JhZ2UgRW50aXR5IElQICANCiAgICAgICAgIGFkZHJlc3Ms IGFuZCB0aGUgUG9ydGFsIE9iamVjdCBwb3J0IG51bWJlciBjYW4gYmUgdGhlIChUQkQpIGRlZmF1 bHQgaVNDU0kgcG9ydCAgDQogICAgICAgICBudW1iZXIuIEZ1cnRoZXJtb3JlLCB0aGUgV1dVSSBv ZiB0aGUgdGFyZ2V0IGRldmljZSBjYW4gYmUgcXVlcmllZCBieSBpc3N1aW5nIHRoZSAgDQogICAg ICAgICBTZW5kVGFyZ2V0IGNvbW1hbmQgdG8gdGhlIGRlZmF1bHQgY2Fub25pY2FsIGlTQ1NJIHRh cmdldCBwcmVzZW50IGF0IHRoZSBJUCANCiAgICAgICAgIGFkZHJlc3MgYW5kIHBvcnQgbnVtYmVy LiAMDQogICAgICAgICAgDQogICAgICAgICAgDQogICAgICAgICA1LjcgIFN0YXRlIENoYW5nZSBO b3RpZmljYXRpb24gUmVxdWlyZW1lbnRzIA0KICAgICAgICAgIA0KICAgICAgICAgQXN5bmNocm9u b3VzIG5vdGlmaWNhdGlvbiAoU3RhdGUgQ2hhbmdlIE5vdGlmaWNhdGlvbnMpOiAgVGhlIFNOUyBt dXN0IGJlIA0KICAgICAgICAgYWJsZSB0byBpbmZvcm0gU05TIGNsaWVudHMgb2YgY2hhbmdlcyB0 byBpdHMgZGF0YWJhc2UsIGluY2x1ZGluZyBjaGFuZ2VzIG9yIA0KICAgICAgICAgbW9kaWZpY2F0 aW9ucyB0byB6b25pbmcgb3IgbG9naW4gY29udHJvbCBwb2xpY2llcyBhbmQgdGhlIA0KICAgICAg ICAgcHJlc2VuY2Ugb3IgYWJzZW5jZSBvZiBpbml0aWF0b3JzIGFuZCB0YXJnZXRzLiAgVGhlc2Ug Y2hhbmdlcyBtYXkgb2NjdXIgYXMgYSANCiAgICAgICAgIHJlc3VsdCBvZiB2YXJpb3VzIGV2ZW50 cywgaW5jbHVkaW5nIGFuIFNOUyBjbGllbnQgYWN0aXZlbHkgbWFuaXB1bGF0aW5nIGNoYW5naW5n IA0KICAgICAgICAgdGhlIFNOUyBkYXRhYmFzZSwgcmVzcG9uc2Ugb3Igbm9uLXJlc3BvbnNlIHRv IGFuIA0KICAgICAgICAgU05TIGhlYXJ0YmVhdCBtZXNzYWdlLCBvciBhIGhhcmR3YXJlIGludGVy cnVwdCBkZWxpdmVyZWQgYnkgdGhlIFNOUyBob3N0IA0KICAgICAgICAgcGxhdGZvcm0gKHN1Y2gg YXMgYSBzd2l0Y2gpLiBBc3luY2hyb25vdXMgbm90aWZpY2F0aW9uIHNoYWxsIGJlIGRlbGl2ZXJl ZCBvbmx5IA0KICAgICAgICAgdG8gU05TIGNsaWVudHMgdGhhdCByZWdpc3RlciBmb3IgdGhlIG5v dGlmaWNhdGlvbiwgYW5kIG9ubHkgZm9yIFNOUyBjbGllbnRzIHRoYXQgDQogICAgICAgICBhcmUg aW4gdGhlIHNhbWUgWm9uZSBhcyB0aGUgZXZlbnQuIA0KICAgICAgICAgIA0KICAgICAgICAgNS44 ICBUaGUgU05TIHByb3RvY29sIFNIQUxMIGJlIGEgbGlnaHR3ZWlnaHQgcHJvdG9jb2wgdGhhdCBj YW4gYmUgc2NhbGVkIGRvd24gDQogICAgICAgICBmb3IgaW1wbGVtZW50YXRpb24gb24gc3dpdGNo ZXMgYW5kIHRhcmdldHMsIG9yIHNjYWxlZCB1cCBmb3IgaW1wbGVtZW50YXRpb24gb24gDQogICAg ICAgICBzZXJ2ZXJzLiANCiAgICAgICAgICANCiAgICAgICAgIDUuOSAgVGhlIFNOUyBTSEFMTCBt ZWV0IHRoZSBpU0NTSSBib290IHJlcXVpcmVtZW50cyAoc2VlIA0KICAgICAgICAgZHJhZnQtaWV0 Zi1pcHMtaXNjc2ktYm9vdC0wMC50eHQpLiANCiAgICAgICAgICANCiAgICAgICAgIDYpIFJlbGF0 ZWQgV29yayANCiAgICAgICAgIEppbmkgWzFdLCBQblAgWzJdIGFuZCBJbnRlcm5ldCBTZXJ2ZXIg TG9jYXRpb24gUHJvdG9jb2wgKFNMUClbM10gYXJlIHNvbWUgb2YgdGhlIA0KICAgICAgICAgb3Ro ZXIgZGlzY292ZXJ5IHByb3RvY29scyB0aGF0IGFyZSBwcmVzZW50IGluIHRoZSBpbmR1c3RyeS4g SXQgaXMgaW1wb3J0YW50IHRvIA0KICAgICAgICAgbm90ZSB0aGF0IHRoZXJlIGlzIG5vIGNvbnNl bnN1cyBpbiB0aGUgaW5kdXN0cnkgYXMgdG8gd2hpY2ggZGlzY292ZXJ5IHByb3RvY29sIA0KICAg ICAgICAgc2hvdWxkIGJlIHVzZWQuIFRoZXJlZm9yZSwgaW5zdGVhZCBvZiBhZG9wdGluZyBhIHNw ZWNpZmljIGV4aXN0aW5nIHByb3RvY29sLCANCiAgICAgICAgIHRoZSBORFQgdGVhbSBoYXMgZW5z dXJlZCB0aGF0IHRoZSBpU0NTSSBkaXNjb3ZlcnkgbWVjaGFuaXNtIGNvbnRhaW5zIHRoZSBrZXkg DQogICAgICAgICBlc3NlbnRpYWwgZmVhdHVyZXMgb2YgdGhlIGFib3ZlIG1lbnRpb25lZCBkaXNj b3ZlcnkgcHJvdG9jb2xzLiBUaGUgbXVsdGljYXN0IA0KICAgICAgICAgZGlzY292ZXJ5IG1lY2hh bmlzbSwgZGVzY3JpYmVkIGFib3ZlLCBwcm92aWRlcyBpU0NTSSB3aXRoIHRoZSBzYW1lIGRpc2Nv dmVyeSANCiAgICAgICAgIGNhcGFiaWxpdGllcyBhcyB0aGVzZSBvdGhlciBkaXNjb3ZlcnkgcHJv dG9jb2xzLiANCiAgICAgICAgICANCiAgICAgICAgICANCiAgICAgICAgIDcuIE91dHN0YW5kaW5n IFdvcmsgSXRlbXMgDQogICAgICAgICBUaGUgZm9sbG93aW5nIHdvcmsgaXRlbXMgYXJlIHN0aWxs IG91dHN0YW5kaW5nOiANCiAgICAgICAgIGEpIEltcGFjdCBvZiBuYW1pbmcgYW5kIGRpc2NvdmVy eSBvbiBpU0NTSSBMb2dpbiBjb21tYW5kLiANCiAgICAgICAgIGIpIFNlY3VyZSBpbnRlcmFjdGlv biBiZXR3ZWVuIHRoZSBzdG9yYWdlIGRpcmVjdG9yIGFuZCB0aGUgaW5pdGlhdG9ycyANCiAgICAg ICAgIGFuZCB0aGUgdGFyZ2V0cy4gDQogICAgICAgICAgDQogICAgICAgICAgDQogICAgICAgICAg DQogICAgICAgICA4LiBSZWZlcmVuY2VzIA0KICAgICAgICAgWzFdIEVkd2FyZHMsIEsuLCAiQ29y ZSBKaW5pOiBJbiBEZXB0aDogRGlzY292ZXJ5IiwgUHJlbnRpY2UgSGFsbCwgMTk5OS4gDQogICAg ICAgICAgDQogICAgICAgICBbMl0gSm9obiwgUi4sICJVUG5QLCBKaW5pIGFuZCBTYWx1dGF0aW9u LSBBIGxvb2sgYXQgc29tZSBwb3B1bGFyIGNvb3JkaW5hdGlvbiANCiAgICAgICAgIGZyYW1ld29y a3MgZm9yIGZ1dHVyZSBuZXR3b3JrZWQgZGV2aWNlcyIsIA0KICAgICAgICAgaHR0cDovL3d3dy5j c3dsLmNvbS93aGl0ZXBwci90ZWNoL3VwbnAuaHRtbCIsIEp1bmUgMTcsIDE5OTkuIA0KICAgICAg ICAgIA0KICAgICAgICAgWzNdIGh0dHA6Ly93d3cuc3J2bG9jLm9yZyANCiAgICAgICAgICANCiAg ICAgICAgIFs0XSBGcmVlZCwgTi4sICJCZWhhdmlvciBvZiBhbmQgUmVxdWlyZW1lbnRzIGZvciBJ bnRlcm5ldCBGaXJld2FsbHMiLCANCiAgICAgICAgIFJGQyAyOTc5LCBPY3RvYmVyIDIwMDAuIA0K ICAgICAgICAgIA0KICAgICAgICAgWzVdIEFOU0kvSUVFRSBTdGQgODAyLTE5OTAsIE5hbWU6IElF RUUgU3RhbmRhcmRzIGZvciBMb2NhbCBhbmQgDQogICAgICAgICBNZXRyb3BvbGl0YW4gQXJlYSBO ZXR3b3JrczogT3ZlcnZpZXcgYW5kIEFyY2hpdGVjdHVyZSANCiAgICAgICAgICANCiAgICAgICAg IFs2XSBLZXNzbGVyLCBHLiBhbmQgU2hlcGFyZCwgUy4sICJBIFByaW1lciBPbiBJbnRlcm5ldCBh bmQgVENQL0lQIFRvb2xzIAwNCiAgICAgICAgIGFuZCBVdGlsaXRpZXMiLCBSRkMgMjE1MSwgSnVu ZSAxOTk3LiANCiAgICAgICAgICANCiAgICAgICAgIFs3XSBTYXRyYW4sIEouLCBTYXB1bnR6YWtp cywgQy4sIFdha2VsZXksIE0uLCBWb24gU3RhbXdpdHosIFAuLCBIYWFnZW5zLCANCiAgICAgICAg IFIuLCBaZWlkbmVyLCBFLiwgRGFsbGUgT3JlLCBMLiwgS2xlaW4sIFkuLCAiaVNDU0kiLCANCiAg ICAgICAgIGRyYWZ0LWlldGYtaXBzLWlzY3NpLTAwLnR4dCwgTm92ZW1iZXIsIDIwMDAuIA0KICAg ICAgICAgIA0KICAgICAgICAgWzhdIEdpYmJvbnMsIEsuLCBUc2VuZywgSi4gYW5kIE1vbmlhLCBD LiwgImlTTlMgSW50ZXJuZXQgU3RvcmFnZSBOYW1lIA0KICAgICAgICAgU2VydmljZSIsIGRyYWZ0 LXRzZW5nLWlwcy1pc25zLTAwLnR4dCwgT2N0b2JlciAyMDAwLiANCiAgICAgICAgICANCiAgICAg ICAgICANCiAgICAgICAgICANCiAgICAgICAgICANCiAgICAgICAgICAgICAgDQogICAgICAgICAg ICAgIA0KICAgICAgICAgIA0KICAgICAgICAgNi4gQ29udGFjdCBBdXRob3IgIA0KICAgICAgICAg S2FsYWRoYXIgVm9ydWdhbnRpICANCiAgICAgICAgIDY1MCBIYXJyeSBSb2FkICANCiAgICAgICAg IElCTSBBbG1hZGVuIFJlc2VhcmNoICANCiAgICAgICAgIFNhbiBKb3NlLCBDQSAgDQogICAgICAg ICBVU0EgIA0KICAgICAgICAgRW1haWw6IGthbGFkaGFyQHVzLmlibS5jb20gIA0KICAgICAgICAg ICAgDQogICAgICAgICBWb3J1Z2FudGkgICAgICAgICAgICBJbnRlcm5ldCBEcmFmdCBFeHBpcmVz IEp1bHkgMjAwMSAgICAgICANCiAgICAgICAgICANCiAgICAgICAgICANCiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBpU0NTSSBOYW1pbmcgYW5kIERpc2NvdmVyeSAgICAgICAgSmFudWFy eSAyMDAxIA0KICAgICAgICAgICANCiAgICAgICAgICAgDQogICAgICAgICAgIA0KICAgICAgICAg ICANCiAgICAgICAgICAgDQogICAgICAgICAgIA0KICAgICAgICAgIkNvcHlyaWdodCAoQykgVGhl IEludGVybmV0IFNvY2lldHkgKGRhdGUpLiBBbGwgUmlnaHRzIFJlc2VydmVkLiBUaGlzICAgDQog ICAgICAgICBkb2N1bWVudCBhbmQgdHJhbnNsYXRpb25zIG9mIGl0IG1heSBiZSBjb3BpZWQgYW5k IGZ1cm5pc2hlZCB0byBvdGhlcnMsICANCiAgICAgICAgIGFuZCBkZXJpdmF0aXZlIHdvcmtzIHRo YXQgY29tbWVudCBvbiBvciBvdGhlcndpc2UgZXhwbGFpbiBpdCBvciBhc3Npc3QgIA0KICAgICAg ICAgaW4gaXRzIGltcGxlbWVudGF0aW9uIG1heSBiZSBwcmVwYXJlZCwgY29waWVkLCBwdWJsaXNo ZWQgYW5kIGRpc3RyaWJ1dGVkLCAgDQogICAgICAgICBpbiB3aG9sZSBvciBpbiBwYXJ0LCB3aXRo b3V0IHJlc3RyaWN0aW9uIG9mIGFueSBraW5kLCBwcm92aWRlZCB0aGF0IHRoZSAgDQogICAgICAg ICBhYm92ZSBjb3B5cmlnaHQgbm90aWNlIGFuZCB0aGlzIHBhcmFncmFwaCBhcmUgaW5jbHVkZWQg b24gYWxsIHN1Y2ggIA0KICAgICAgICAgY29waWVzIGFuZCBkZXJpdmF0aXZlIHdvcmtzLiBIb3dl dmVyLCB0aGlzIGRvY3VtZW50IGl0c2VsZiBtYXkgbm90IGJlICANCiAgICAgICAgIG1vZGlmaWVk IGluIGFueSB3YXksIEZ1bGwgQ29weXJpZ2h0IFN0YXRlbWVudCAgDQogICAgICAgICBzdWNoIGFz IGJ5IHJlbW92aW5nIHRoZSBjb3B5cmlnaHQgbm90aWNlIG9yIHJlZmVyZW5jZXMgdG8gdGhlICAg DQogICAgICAgICBJbnRlcm5ldCBTb2NpZXR5IG9yIG90aGVyIEludGVybmV0IG9yZ2FuaXphdGlv bnMsIGV4Y2VwdCBhcyBuZWVkZWQgZm9yICANCiAgICAgICAgIHRoZSBwdXJwb3NlIG9mIGRldmVs b3BpbmcgSW50ZXJuZXQgc3RhbmRhcmRzIGluIHdoaWNoIGNhc2UgdGhlICANCiAgICAgICAgIHBy b2NlZHVyZXMgZm9yIGNvcHlyaWdodHMgZGVmaW5lZCBpbiB0aGUgSW50ZXJuZXQgU3RhbmRhcmRz IHByb2Nlc3MgbXVzdCAgDQogICAgICAgICBiZSBmb2xsb3dlZCwgb3IgYXMgcmVxdWlyZWQgdG8g dHJhbnNsYXRlIGl0IGludG8gbGFuZ3VhZ2VzIG90aGVyIHRoYW4gIA0KICAgICAgICAgRW5nbGlz aC4gIA0KICAgICAgICAgICANCiAgICAgICAgIFRoZSBsaW1pdGVkIHBlcm1pc3Npb25zIGdyYW50 ZWQgYWJvdmUgYXJlIHBlcnBldHVhbCBhbmQgd2lsbCBub3QgYmUgICANCiAgICAgICAgIHJldm9r ZWQgYnkgdGhlIEludGVybmV0IFNvY2lldHkgb3IgaXRzIHN1Y2Nlc3NvcnMgb3IgYXNzaWducy4g IA0KICAgICAgICAgICANCiAgICAgICAgIFRoaXMgZG9jdW1lbnQgYW5kIHRoZSBpbmZvcm1hdGlv biBjb250YWluZWQgaGVyZWluIGlzIHByb3ZpZGVkIG9uIGFuICANCiAgICAgICAgICJBcyBJUyIg YmFzaXMgYW5kIFRIRSBJTlRFUk5FVCBTT0NJRVRZIEFORCBUSEUgSU5URVJORVQgRU5HSU5FRVJJ TkcgIA0KICAgICAgICAgVEFTSyBGT1JDRSBESVNDTEFJTVMgQUxMIFdBUlJBTlRJRVMsIEVYUFJF U1MgT1IgSU1QTElFRCAsIElOQ0xVRElORyAgDQogICAgICAgICBCVVQgTk9UIExJTUlURUQgVE8g QU5ZIFdBUlJBTlRZIFRIQVQgVEhFIFVTRSBPRiBUSEUgSU5GT1JNQVRJT04gSEVSRUlOICANCiAg ICAgICAgIFdJTEwgTk9UIElORlJJTkdFIEFOWSBSSUdIVFMgT1IgQU5ZIElNUExJRUQgV0FSUkFO VElFUyBPRiANCiAgICAgICAgIE1FUkNIQU5UQUJJTElUWSBPUiBGSVRORVNTIEZPUiBBIFBBUlRJ Q1VMQVIgUFVSUE9TRSIgIAwNCiAgICAgICAgICAgDQogICAgICAgICBFeHBpcmVzIEp1bHkgMjAw MSAgDQogICAgICAgICAgIA0KICAgICAgICAgVm9ydWdhbnRpICBpU0NTSSBOYW1pbmcgYW5kIERp c2NvdmVyeSBEcmFmdCBFeHBpcmVzIEp1bHkgMjAwMSAgDA== --0__=882569D200677A018f9e8a93df938690918c882569D200677A01-- From owner-ips@ece.cmu.edu Fri Jan 12 17:22:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA26060 for ; Fri, 12 Jan 2001 17:22:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CJA7403161 for ips-outgoing; Fri, 12 Jan 2001 14:10:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CJ9Ta03145 for ; Fri, 12 Jan 2001 14:09:31 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id ZJ5LTYG5; Fri, 12 Jan 2001 11:09:24 -0800 From: "Y P Cheng" To: Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 12 Jan 2001 11:08:34 -0800 Message-ID: <002c01c07ccb$0fa12c00$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > As far as FCP stacks go, it is a truism that no OS > (host) has a native FCP stack. Today. (Stay tuned, though; > Blackcomb AS is going to be rather interesting) > > That's why the HBA rules. It's the entity performing > all the translation between what the OSes see as a pure > SCSI universe and the real (or virtual) device FC universe. > Had we native motherboard FC physical interfaces and OS > FCP stacks, we would not need FC HBAs. With native motherboard FC chips, the BIOS has the necessary code to access the chip. After booting, the OS transition from BIOS to the chip device driver. There is no FCP stack. There is only the device driver code supplied by the chip manufacturer. (In other words, different FC chips have different drivers.) > But, that's hosts. Consider the world of devices for > a moment. No one can argue that there are not a > plethora of FC devices today, all with FCP stacks by > definition. Plus, many FC devices today are not just > one FC port but multiple, with multiple WWPN and one WWNN > to achieve higher RAS characteristics. You are confused between the device driver and TCP stack. Most RAID manufacturers use SDK, Software Development Kit, from HBA manufacturers. The SDK allows RTOS to build device drivers for the ICs. The ICs generate FCP packets. Unlike TCP/IP, the Fibre Channel IC device drivers are not aware of the format of the FCP packets. Therefore, they can not be called the FCP stack. > So while your statement > > iFCP as way to keep your investment in FCP stacks is a very > > weak argument. > is certainly true for hosts (initiators), it is certainly > not true for devices (targets). No, Julo is correct. The argument is very weak. The statement is especially weak if an iSCSI adapter uses the same implementation as an FCP adapter. In the adapter implementation, the iSCSI header will be added by the adapter. There won't be an iSCSI stack in the OS for adding the iSCSI headers. Of course, a stack implementation of iSCSI in an OS is likely although not efficiently. Most manufacturers with high performance in mind will choose to put the iSCSI stack inside the adapter, not in the OS. Y.P. Cheng, Connectcom Solutions. From owner-ips@ece.cmu.edu Fri Jan 12 18:23:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA26836 for ; Fri, 12 Jan 2001 18:23:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CKtXw06295 for ips-outgoing; Fri, 12 Jan 2001 15:55:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcars04e.ca.nortel.com (h56s242a129n47.user.nortelnetworks.com [47.129.242.56]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CKsql06252 for ; Fri, 12 Jan 2001 15:54:52 -0500 (EST) Received: from zbl6c012.corpeast.baynetworks.com by zcars04e.ca.nortel.com; Fri, 12 Jan 2001 15:52:50 -0500 Received: from zbl6c002.corpeast.baynetworks.com ([132.245.205.52]) by zbl6c012.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id ZLK0B2CT; Fri, 12 Jan 2001 15:52:49 -0500 Received: from ftravost.nortelnetworks.com (dhcp223-248.engeast.baynetworks.com [192.32.223.248]) by zbl6c002.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id CN99DFK8; Fri, 12 Jan 2001 15:52:48 -0500 Message-Id: <5.0.0.25.2.20010112153316.026ba110@zbl6c002.corpeast.baynetworks.com> X-Sender: travos@zbl6c002.corpeast.baynetworks.com X-Mailer: QUALCOMM Windows Eudora Version 5.0 Date: Fri, 12 Jan 2001 15:37:47 -0500 To: "Victor Firoiu" , Vi Chau From: "Franco Travostino" Subject: Re: Performance of iSCSI, FCIP and iFCP Cc: IPS Reflector In-Reply-To: <3A5CBC58.756953D2@nortelnetworks.com> References: <312419998E3CD211A52900A0C991A47A263157@gordan.pl.gadzoox.com> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="=====================_148330272==_.ALT" X-Orig: Sender: owner-ips@ece.cmu.edu Precedence: bulk --=====================_148330272==_.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed Could an FCIP expert reply to the technical question that Victor raised in this message? thanks -franco At 02:47 PM 1/10/01, Firoiu, Victor [BL60:470-M:EXCH] wrote: >Vi, > >thanks for your comments. Indeed, there is no such limitation in the >FCIP ID. But there is no consideration for multiple TCP connections >either. What specifically governs the use of TCP flow A as opposed to >TCP flow B when two FCIP gateways are communicating? > >As we've seen in our examples, the performance is significantly >(quadratically) impacted by the choice of number of TCP connections. >Fine-grain control over multiple TCP connections thus becomes >mandatory between any two FC islands of non-trivial size. The most >natural way to multiplex storage connections (FC or SCSI) on multiple >TCP connections is to have one or a few TCP connections per storage >connection, that is, to use storage connection awareness for assigning >TCP connections to storage connections (iSCSI, iFCP). > >Surely, it is possible to multiplex storage connections into TCP >connections without any knowledge of the identity of storage >connections. But this brings up issues including load balancing, >interdependency between storage connections sharing TCP connections, >managing storage connections that are striped across TCP connections >(as thoroughly discussed within the iSCSI community). It seems that >the iSCSI and iFCP solution of assigning one (or a few) TCP >connection(s) for each storage connections has many practical and >performance merits that needs a special attention in FCIP also. > >Again, any comments are very welcome. > >Victor Firoiu >----- >Content Internetworking Lab, Technology Center >Nortel Networks, Inc. >600 Technology ParkBillerica, MA 01821 USA > >Vi Chau wrote: > > > > Hi, > > > > There is nothing in the FCIP I-D that limits the > > tunnel between the SAN islands to one TCP connection. The > > FCIP gateways can open as many TCP connections as it can > > support to reach any number of destination gateways. > > > > Vi Chau > > Gadzoox Networks, Inc. > > 16241 Laguna Canyon Road, Suite 100 > > Irvine, CA 92618-3611 > > 949-789-4639 > > --=====================_148330272==_.ALT Content-Type: text/html; charset="us-ascii"
Could an FCIP expert reply to the technical question that Victor raised in this message?

thanks
-franco

At 02:47 PM 1/10/01, Firoiu, Victor [BL60:470-M:EXCH] wrote:

Vi,

thanks for your comments.  Indeed, there is no such limitation in the
FCIP ID.  But there is no consideration for multiple TCP connections
either.  What specifically governs the use of TCP flow A as opposed to
TCP flow B when two FCIP gateways are communicating?

As we've seen in our examples, the performance is significantly
(quadratically) impacted by the choice of number of TCP connections.
Fine-grain control over multiple TCP connections thus becomes
mandatory between any two FC islands of non-trivial size.  The most
natural way to multiplex storage connections (FC or SCSI) on multiple
TCP connections is to have one or a few TCP connections per storage
connection, that is, to use storage connection awareness for assigning
TCP connections to storage connections (iSCSI, iFCP).

Surely, it is possible to multiplex storage connections into TCP
connections without any knowledge of the identity of storage
connections.  But this brings up issues including load balancing,
interdependency between storage connections sharing TCP connections,
managing storage connections that are striped across TCP connections
(as thoroughly discussed within the iSCSI community).  It seems that
the iSCSI and iFCP solution of assigning one (or a few) TCP
connection(s) for each storage connections has many practical and
performance merits that needs a special attention in FCIP also.

Again, any comments are very welcome.

Victor Firoiu
-----
Content Internetworking Lab, Technology Center
Nortel Networks, Inc.
600 Technology ParkBillerica, MA 01821 USA

Vi Chau wrote:
>
> Hi,
>
>    There is nothing in the FCIP I-D that limits the
> tunnel between the SAN islands to one TCP connection. The
> FCIP gateways can open as many TCP connections as it can
> support to reach any number of destination gateways.
>
> Vi Chau
> Gadzoox Networks, Inc.
> 16241 Laguna Canyon Road, Suite 100
> Irvine, CA 92618-3611
> 949-789-4639
>
--=====================_148330272==_.ALT-- From owner-ips@ece.cmu.edu Fri Jan 12 19:47:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA27336 for ; Fri, 12 Jan 2001 19:47:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CMaat12068 for ips-outgoing; Fri, 12 Jan 2001 17:36:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CMaQl12062 for ; Fri, 12 Jan 2001 17:36:26 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Fri, 12 Jan 2001 14:36:42 -0800 Message-ID: From: Charles Monia To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 12 Jan 2001 14:35:52 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Julo: > iFCP as way to keep your investment in FCP stacks is a very > weak argument. > FCP stacks are not that stable neither that prevalent (there > is none in the > most widespread OS family - Windows). > I don't understand where you're coming from. Granted, you won't find many FC adapters on notebooks and desktops. However every enterprise-class operating system, including Windows NT/2000, provides robust Fibre Channel support. What's more HBAs, host driver stacks and middleware for management and failover have been deployed and in-service in mission-critical applications for some time. End users have an enormous investment in qualifying, deploying and supporting these systems. Anyone who's ever had to deal with such users in major accounts will appreciate what's at stake. From that perspective, I believe there's value in an approach to IP migration that protects that investment. Charles > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Thursday, January 11, 2001 9:23 PM > To: ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > > > Josh, > > iFCP as way to keep your investment in FCP stacks is a very > weak argument. > FCP stacks are not that stable neither that prevalent (there > is none in the > most widespread OS family - Windows). > > A gateway for a single device should be the exception rather > than the rule. > > I can support it as a work item ONLY if it plays a real > gateway role and > can coexist with FCIP is some synergistic fashion. > As a end-to-end proposal is has little value IMHO. > > Julo > > Joshua Tseng on 09/01/2001 07:39:18 > > Please respond to Joshua Tseng > > To: Venkat Rangan > cc: "Ips@Ece. Cmu. Edu" > Subject: RE: iFCP as an IP Storage Work Item > > > > > Venkat, > > > > Josh, > > > > Thanks for the clarification that iFCP is only presented as > a gateway > > protocol. The one comment we would make is that we have FC to > > SCSI gateways > > already in place, without the need for any standards body > > standardizing a > > new protocol. The function of the gateway is defined by the > > standards for > > the two protocols being "connected", and gateway details are left as > > implementation details. > > Actually, what I meant by gateway protocol is that it is a protocol > spoken by gateways. It doesn't specify anything about how to > implement the protocol, which is internal to the gateway device as you > point out. iFCP is the protocol that shows up on the IP network when > at least two iFCP gateways transport storage data between them. > > > > > On another note, it should be feasible to build a gateway > > that receives FCP > > frames from an N_Port or NL_Port of a SCSI Initiator and map > > the FCP frames > > into iSCSI frames. The frames are sent on an IP interface and > > routed by a > > normal IP network and another gateway reconverts the iSCSI > > PDUs back to FCP > > frames and sends them to the target. You will notice that > > this does not > > require any routing in the FC Plane and accomplishes the same > > end goals as > > iFCP. Also, this does not require any further standards work, > > besides the > > usual FCP, iSCSI and related naming protocols. This also > > provides the same > > scalability of number of nodes on the network, because the > > conversion from > > locally significant S_ID and D_ID to iSCSI IP addresses can > > be done, with > > help from a standardized naming effort such as iSNS. > > Yes, I agree. But this is a different gateway which has nothing > to do with an iFCP gateway. Definitely these types of gateways > will be needed as we transition to iSCSI. The new iSNS draft has > a diagram showing both iSCSI-FC gateways and iFCP gateways. They > have different roles. > > > > > Based on these, we believe the need for IP Storage working > > group to pick up > > iFCP as a work item is reduced. > > hmmm....you lost me here. An iFCP gateway has nothing to do with > iSCSI. They are completely separate. > > In order to use iSCSI, you need one of the following: > > 1) Two iSCSI devices > 2) One iSCSI device, one FC device, and one iSCSI-FC gateway > (which you described above) > > The situation of two FC devices and two iSCSI-FC gateways is clearly > not a design objective of iSCSI. Of course it can be done, but we > believe iFCP is clearly the best solution here. > > Fibre Channel has its issues, but one thing is certain: the > FCP driver > stacks are very stable and can provide excellent performance > for storage > applications. But the drawback is FC's networking capabilities leave > much to be desired. On the other hand, IP networking capabilities are > quite stable and work exceedingly well, but the iSCSI driver > stack does > not exist yet. > > So, iFCP's objective is to marry the best of both worlds--to take the > existing stable, high-performance driver stacks of Fibre Channel and > directly internetwork their individual N_PORTs using TCP/IP. > > Therefore, iFCP is an opportunity to leverage the existing and proven > Fibre Channel driver stacks and combine them with the > capabilities that > IP networking can provide. Until the day we have stable iSCSI driver > stacks everywhere, this may prove to be an attractive alternative for > many end users. Another comparison I liken it to is the need for > NAT until we have IPv6 everywhere. > > Josh > > > > > Regards, > > > > Venkat Rangan > > Rhapsody Networks Inc. > > http://www.rhapsodynetworks.com > > > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Joshua Tseng > > Sent: Thursday, January 04, 2001 10:54 AM > > To: Ips@Ece. Cmu. Edu > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > I don't want to stifle any creative technical discussion here, > > but I feel the need to remind everybody that iFCP is positioned > > as a gateway technology only. While the thought of "native" > > iFCP HBA's might be interesting, this discussion is > > completely irrelevant with regard to whether iFCP should > > or should not become an IPS work item. iFCP is being proposed > > as an IPS work item purely on its merits as a gateway technology. > > > > Regards, > > Josh > > > > > -----Original Message----- > > > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > > > Sent: Thursday, January 04, 2001 5:47 AM > > > To: 'ips@ece.cmu.edu' > > > Subject: FW: iFCP as an IP Storage Work Item > > > > > > > > > > > > > > > -----Original Message----- > > > From: Stephen Byan > > > Sent: Thursday, January 04, 2001 8:40 AM > > > To: 'Bill Terrell' > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > It's all the FC stuff that lets iFCP work over an unreliable > > > data transport > > > like UDP. It's redundant when running over TCP/IP. > > > > > > Regards, > > > -Steve > > > > > > > -----Original Message----- > > > > From: Bill Terrell [mailto:terrell@troikanetworks.com] > > > > Sent: Wednesday, January 03, 2001 6:10 PM > > > > To: 'Stephen Byan' > > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > > > > >The downside of this advantage is that native iFCP > > devices would be > > > > burdened > > > > >with greater complexity and cost. I therefor think iFCP > > > > should not be an IP > > > > >Storage work item. > > > > > > > > > >Regards, > > > > >-Steve > > > > > > > > How is a native iFCP endpoint (initiator or target) more > > > > complex or costly > > > > than an iSCSI native endpoint? What are the specific > > > > difficulties inherent > > > > to native iFCP devices versus native iSCSI devices? > > > > > > > > Bill > > > > > > > > > > > > From owner-ips@ece.cmu.edu Fri Jan 12 19:54:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA27369 for ; Fri, 12 Jan 2001 19:54:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CMmZN12343 for ips-outgoing; Fri, 12 Jan 2001 17:48:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (man-97-106.Reshall.Berkeley.EDU [169.229.97.106]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CMlsl12328 for ; Fri, 12 Jan 2001 17:47:54 -0500 (EST) Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id E7DF18D59; Fri, 12 Jan 2001 14:47:32 -0800 (PST) To: ips@ece.cmu.edu, rdma@cisco.com Subject: WARP RDMA Architectural Requirements Summary posted Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII From: csapuntz@cisco.com Date: 12 Jan 2001 14:47:32 -0800 Message-ID: Lines: 23 Sender: owner-ips@ece.cmu.edu Precedence: bulk A group of us have been working on an RDMA system called WARP. The RDMA layer aims to provide exact data placement for iSCSI, NFS, CIFS, as well as the VI interface. The RDMA layer is implemented on a generic network adapter which does not have to have specific upper-layer protocols implemented on it (like iSCSI or NFS). The requirements and a sketch of the design is now available at http://www.stanford.edu/~csapuntz/draft-jpink-warp-summary-00.txt. The document has been sent in as an internet draft and should be available as draft-jpink-warp-summary-00.txt by the end of next week. The mailing list rdma@cisco.com has been set up for discussion on the topic of RDMA and WARP RDMA. Please use that list and NOT the ips mailing list for discussion and questions. You can subscribe to the rdma list by sending mail to "mailer@cisco.com" with a blank subject line and the body of the message "subscribe rdma". Thanks, Costa From owner-ips@ece.cmu.edu Fri Jan 12 20:39:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA27722 for ; Fri, 12 Jan 2001 20:39:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0CNadx13612 for ips-outgoing; Fri, 12 Jan 2001 18:36:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CNaIl13603 for ; Fri, 12 Jan 2001 18:36:18 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0D0lQ029850 for ; Fri, 12 Jan 2001 16:47:26 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Ips" Subject: RE: Performance of iSCSI, FCIP and iFCP Date: Fri, 12 Jan 2001 15:35:08 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <5.0.0.25.2.20010112153316.026ba110@zbl6c002.corpeast.baynetworks.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Franco, As a layman, FCP is responsible for ordered delivery but a timestamp would help both merging multiple streams as well as filtering out stale frames stalled by intermittent network disruptions. The layer above IP transport should allow needed freedom for adding additional paths. Doug Could an FCIP expert reply to the technical question that Victor raised in this message? thanks -franco At 02:47 PM 1/10/01, Firoiu, Victor [BL60:470-M:EXCH] wrote: Vi, thanks for your comments. Indeed, there is no such limitation in the FCIP ID. But there is no consideration for multiple TCP connections either. What specifically governs the use of TCP flow A as opposed to TCP flow B when two FCIP gateways are communicating? As we've seen in our examples, the performance is significantly (quadratically) impacted by the choice of number of TCP connections. Fine-grain control over multiple TCP connections thus becomes mandatory between any two FC islands of non-trivial size. The most natural way to multiplex storage connections (FC or SCSI) on multiple TCP connections is to have one or a few TCP connections per storage connection, that is, to use storage connection awareness for assigning TCP connections to storage connections (iSCSI, iFCP). Surely, it is possible to multiplex storage connections into TCP connections without any knowledge of the identity of storage connections. But this brings up issues including load balancing, interdependency between storage connections sharing TCP connections, managing storage connections that are striped across TCP connections (as thoroughly discussed within the iSCSI community). It seems that the iSCSI and iFCP solution of assigning one (or a few) TCP connection(s) for each storage connections has many practical and performance merits that needs a special attention in FCIP also. Again, any comments are very welcome. Victor Firoiu ----- Content Internetworking Lab, Technology Center Nortel Networks, Inc. 600 Technology ParkBillerica, MA 01821 USA Vi Chau wrote: > > Hi, > > There is nothing in the FCIP I-D that limits the > tunnel between the SAN islands to one TCP connection. The > FCIP gateways can open as many TCP connections as it can > support to reach any number of destination gateways. > > Vi Chau > Gadzoox Networks, Inc. > 16241 Laguna Canyon Road, Suite 100 > Irvine, CA 92618-3611 > 949-789-4639 > From owner-ips@ece.cmu.edu Fri Jan 12 20:43:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA27778 for ; Fri, 12 Jan 2001 20:43:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0D04cP14216 for ips-outgoing; Fri, 12 Jan 2001 19:04:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0D041l14197 for ; Fri, 12 Jan 2001 19:04:01 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Fri, 12 Jan 2001 16:04:18 -0800 Message-ID: From: Charles Monia To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI: Question on MaxCmdRN update rule Date: Fri, 12 Jan 2001 16:03:28 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Julo: I was referring to rev 02, which was current as of the posting date. The problem is fixed in rev 03. Charles > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Friday, January 12, 2001 1:53 AM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Question on MaxCmdRN update rule > > > > > Charles, > > What version of the document are you referring to? > > Julo > > Charles Monia on 11/01/2001 05:15:08 > > Please respond to Charles Monia > > To: Julian Satran/Haifa/IBM@IBMIL > cc: "Ips (E-mail)" > Subject: iSCSI: Question on MaxCmdRN update rule > > > > > Hi Julo: > > I don't understand the MaxCmdRN update policy as defined below: > > "2.3.10 MaxCmdRN - maximum CmdRN acceptable from this initiator > > MaxCmdRN is a reference number that the target iSCSI returns to the > initiator to indicate the maximum CmdRN the initiator can send. The > initiator must ignore values not between the current value of the > ExpCmdRN and MaxCmdRN; this may be required when updates arrive out > of order (they travel on different TCP connections)." > > As I interpret the above, values not within the upper and lower limits > given > above must be discarded. > > I would think this parameter should be updated when a new > value greater > than > the current MaxCmdRN is received. > > Charles > > > From owner-ips@ece.cmu.edu Fri Jan 12 21:58:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA29495 for ; Fri, 12 Jan 2001 21:58:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0D1Afl15667 for ips-outgoing; Fri, 12 Jan 2001 20:10:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0D1AXl15662 for ; Fri, 12 Jan 2001 20:10:33 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0D2LN029924; Fri, 12 Jan 2001 18:21:28 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: , , Subject: RE: WARP RDMA Architectural Requirements Summary posted Date: Fri, 12 Jan 2001 17:09:05 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Costa, Would a third-party proxy completion of a command represent the same functionality as that found using VI for SCSI? In other words, allow a separate device to act on behalf of another with respect to exchanging data. An allowable type of spoofing to assist collection of data. SCSI already provides for content directed placement, just not by a third-party as does VI. It would seem a field added to the present structures to identify the child device together with the parent solves this issue. I have already subscribed to this reflector months ago with no activity. Do I need to re-subscribe? Doug > A group of us have been working on an RDMA system called WARP. The > RDMA layer aims to provide exact data placement for iSCSI, NFS, CIFS, > as well as the VI interface. The RDMA layer is implemented on a generic > network adapter which does not have to have specific upper-layer > protocols > implemented on it (like iSCSI or NFS). > > The requirements and a sketch of the design is now available at > http://www.stanford.edu/~csapuntz/draft-jpink-warp-summary-00.txt. The > document has been sent in as an internet draft and should be available > as draft-jpink-warp-summary-00.txt by the end of next week. > > The mailing list rdma@cisco.com has been set up for discussion on the > topic of RDMA and WARP RDMA. Please use that list and NOT the ips > mailing list for discussion and questions. > > You can subscribe to the rdma list by sending mail to > "mailer@cisco.com" with a blank subject line and the body of the > message "subscribe rdma". > > Thanks, > Costa > > From owner-ips@ece.cmu.edu Fri Jan 12 21:59:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA29526 for ; Fri, 12 Jan 2001 21:59:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0D0RdN14728 for ips-outgoing; Fri, 12 Jan 2001 19:27:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0D0RKl14719 for ; Fri, 12 Jan 2001 19:27:20 -0500 (EST) Message-ID: <618471D08ABDD31188B6009027E50093B02285@apollo.pirus.com> From: "Ferrari, Stephen" To: ips@ece.cmu.edu Subject: iSCSI: F bits in Command and DataIn PDUs, too? Date: Fri, 12 Jan 2001 19:27:30 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk According to the spec, the F bit in DataOut PDUs indicates "the last PDU of immediate data or the last PDU of a sequence answering a R2T". I believe this is a typo and should be "the last PDU of _unsolicited_ data or the last PDU of a sequence answering an R2T". For consistency, it would seem that Command PDUs should also have an F bit defined to indicate the last PDU of unsolicited data if all of the unsolicited data is sent as immediate data, and DataIn PDUs should have an F bit to indicate the last PDU of data for the command. Granted, these two cases are easier to detect than the case already covered, since there can be at most one PDU of immediate data per command (so there's no overlap possible), and the Response PDU implies that there's no more DataIn coming. But having a single bit, preferably in the same position in all three PDU types, to indicate that "this data transfer is done" would simplify data transfer management. (Note that for DataOut and bi-directional operations there can be multiple data transfers per command.) From owner-ips@ece.cmu.edu Fri Jan 12 22:00:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA29546 for ; Fri, 12 Jan 2001 22:00:36 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0D1JkT15893 for ips-outgoing; Fri, 12 Jan 2001 20:19:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CNMal13269 for ; Fri, 12 Jan 2001 18:22:36 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0D0Xf029830; Fri, 12 Jan 2001 16:33:44 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Y P Cheng" , Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 12 Jan 2001 15:21:23 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <002c01c07ccb$0fa12c00$90c809c0@yp_portable.advansys.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Y.P., Hardware and software implementations will benefit from a direct use of FCP rather than those structures undergoing change to better support new environments which lead to FCIP and iFCP. An IP network will not permit a direct boot in the same manner as SCSI as there are no physically captured devices. A normal DHCP-MTFTP to obtain boot sectors will likely prevail in these environments as alternatives remain unstable. Emulating an existing system interface will benefit from a transport that adopts the same underlying structures and behavior. The strength of the argument may be found in simplicity with resulting conformity and stability. A stable scheme may be years away otherwise and stability represents strength. Doug > > As far as FCP stacks go, it is a truism that no OS > > (host) has a native FCP stack. Today. (Stay tuned, though; > > Blackcomb AS is going to be rather interesting) > > > > That's why the HBA rules. It's the entity performing > > all the translation between what the OSes see as a pure > > SCSI universe and the real (or virtual) device FC universe. > > Had we native motherboard FC physical interfaces and OS > > FCP stacks, we would not need FC HBAs. > > With native motherboard FC chips, the BIOS has the necessary code > to access > the chip. After booting, the OS transition from BIOS to the chip device > driver. There is no FCP stack. There is only the device driver code > supplied by the chip manufacturer. (In other words, different FC > chips have > different drivers.) > > > But, that's hosts. Consider the world of devices for > > a moment. No one can argue that there are not a > > plethora of FC devices today, all with FCP stacks by > > definition. Plus, many FC devices today are not just > > one FC port but multiple, with multiple WWPN and one WWNN > > to achieve higher RAS characteristics. > > You are confused between the device driver and TCP stack. Most RAID > manufacturers use SDK, Software Development Kit, from HBA manufacturers. > The SDK allows RTOS to build device drivers for the ICs. The ICs generate > FCP packets. Unlike TCP/IP, the Fibre Channel IC device drivers are not > aware of the format of the FCP packets. Therefore, they can not be called > the FCP stack. > > > So while your statement > > > iFCP as way to keep your investment in FCP stacks is a very > > > weak argument. > > is certainly true for hosts (initiators), it is certainly > > not true for devices (targets). > > No, Julo is correct. The argument is very weak. The statement is > especially weak if an iSCSI adapter uses the same implementation as an FCP > adapter. In the adapter implementation, the iSCSI header will be added by > the adapter. There won't be an iSCSI stack in the OS for adding the iSCSI > headers. Of course, a stack implementation of iSCSI in an OS is likely > although not efficiently. Most manufacturers with high > performance in mind > will choose to put the iSCSI stack inside the adapter, not in the OS. > > Y.P. Cheng, Connectcom Solutions. > > From owner-ips@ece.cmu.edu Fri Jan 12 22:03:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA29601 for ; Fri, 12 Jan 2001 22:03:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0D1Gh415797 for ips-outgoing; Fri, 12 Jan 2001 20:16:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0CCKva16632 for ; Fri, 12 Jan 2001 07:20:57 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA07322; Fri, 12 Jan 2001 07:20:56 -0500 (EST) Message-Id: <200101121220.HAA07322@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-iscsi-03.txt Date: Fri, 12 Jan 2001 07:20:55 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iSCSI Author(s) : J. Satran et al. Filename : draft-ietf-ips-iscsi-03.txt Pages : 94 Date : 11-Jan-01 The Small Computer Systems Interface (SCSI) is a popular family of protocols for communicating with I/O devices, especially storage devices. This memo describes a transport protocol for SCSI that operates on top of TCP. The iSCSI protocol aims to be fully compliant with the requirements laid out in the SCSI Architecture Model - 2 [SAM2] document A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-03.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-03.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-03.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20010111123950.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-iscsi-03.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-iscsi-03.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20010111123950.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Sat Jan 13 00:07:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA01856 for ; Sat, 13 Jan 2001 00:07:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0D3Kl018365 for ips-outgoing; Fri, 12 Jan 2001 22:20:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0D3Kfl18359 for ; Fri, 12 Jan 2001 22:20:41 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Fri, 12 Jan 2001 19:20:59 -0800 Message-ID: From: Charles Monia To: Y P Cheng Cc: ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Fri, 12 Jan 2001 19:20:07 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Y P: As I see it, the software issue is not only related to support for the low-level hardware and SCSI encapsulation. It's in the effort needed to transpose all the SAN value-add (naming, discovery, zoning, error recovery, management, etc) to a new network environment. In other words, all the higher level storage network and systems issues that must be dealt with by middleware and the upper reaches of the driver stack. Yes, as you say, it is possible through careful design to structure the driver stack around a common low-level shim that decouples it from the hardware ASICs and transport-specific SCSI encapsulation assists. And, indeed, this low-level software can be readily ported to a target environment with the appropriate tools. While such a shim provides a common interface to SCSI semantics, nonetheless, it cannot hide these fundamental differences in the network environment. It doesn't automatically get you a Symmetrix or a Shark running in a native IP environment. As an example, no shim can conceal the differences between a parallel SCSI bus, supporting 16 drives, from an FC environment, with addressability in the millions. Likewise, issues like booting, device naming and discovery have to be rethought and reimplemented in a new environment. A simple example is the third party copy functionality, which must be extended to incorporate the iSCSI naming model. So, rather than say that the iFCP preserves the existing driver stacks, we should go further and extend the scope to the totality of software needed to support the storage network environment. Charles > -----Original Message----- > From: Y P Cheng [mailto:ycheng@advansys.com] > Sent: Friday, January 12, 2001 11:09 AM > To: ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > > As far as FCP stacks go, it is a truism that no OS > > (host) has a native FCP stack. Today. (Stay tuned, though; > > Blackcomb AS is going to be rather interesting) > > > > That's why the HBA rules. It's the entity performing > > all the translation between what the OSes see as a pure > > SCSI universe and the real (or virtual) device FC universe. > > Had we native motherboard FC physical interfaces and OS > > FCP stacks, we would not need FC HBAs. > > With native motherboard FC chips, the BIOS has the necessary > code to access > the chip. After booting, the OS transition from BIOS to the > chip device > driver. There is no FCP stack. There is only the device driver code > supplied by the chip manufacturer. (In other words, different > FC chips have > different drivers.) > > > But, that's hosts. Consider the world of devices for > > a moment. No one can argue that there are not a > > plethora of FC devices today, all with FCP stacks by > > definition. Plus, many FC devices today are not just > > one FC port but multiple, with multiple WWPN and one WWNN > > to achieve higher RAS characteristics. > > You are confused between the device driver and TCP stack. Most RAID > manufacturers use SDK, Software Development Kit, from HBA > manufacturers. > The SDK allows RTOS to build device drivers for the ICs. The > ICs generate > FCP packets. Unlike TCP/IP, the Fibre Channel IC device > drivers are not > aware of the format of the FCP packets. Therefore, they can > not be called > the FCP stack. > > > So while your statement > > > iFCP as way to keep your investment in FCP stacks is a very > > > weak argument. > > is certainly true for hosts (initiators), it is certainly > > not true for devices (targets). > > No, Julo is correct. The argument is very weak. The statement is > especially weak if an iSCSI adapter uses the same > implementation as an FCP > adapter. In the adapter implementation, the iSCSI header > will be added by > the adapter. There won't be an iSCSI stack in the OS for > adding the iSCSI > headers. Of course, a stack implementation of iSCSI in an OS > is likely > although not efficiently. Most manufacturers with high > performance in mind > will choose to put the iSCSI stack inside the adapter, not in the OS. > > Y.P. Cheng, Connectcom Solutions. > From owner-ips@ece.cmu.edu Sat Jan 13 00:10:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA01882 for ; Sat, 13 Jan 2001 00:10:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0D4WnO19695 for ips-outgoing; Fri, 12 Jan 2001 23:32:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0D4Vml19677 for ; Fri, 12 Jan 2001 23:31:48 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id XAA16422 for ; Fri, 12 Jan 2001 23:25:02 -0500 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id VAA07278 for ; Fri, 12 Jan 2001 21:31:47 -0700 Importance: Normal Subject: iscsi boot draft To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.5 September 22, 2000 Message-ID: From: "Prasenjit Sarkar" Date: Fri, 12 Jan 2001 20:31:45 -0800 X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.5 |September 22, 2000) at 01/12/2001 08:31:47 PM MIME-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=882569D30018476C8f9e8a93df938690918c882569D30018476C" Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=882569D30018476C8f9e8a93df938690918c882569D30018476C Content-type: text/plain; charset=us-ascii Revision 2 of the iscsi boot draft has been submitted to IETF. (See attached file: draft-ietf-ips-iscsi-boot-01.txt) Change log: 1. Addition of a software stage to allow clients to load iscsi initiator software. (as suggested by Doug Otis). 2. Follow the procedures outlined in RFC 2939 to obtain a DHCP option for iSCSI boot service. Prasenjit --0__=882569D30018476C8f9e8a93df938690918c882569D30018476C Content-type: text/plain; name="draft-ietf-ips-iscsi-boot-01.txt" Content-Transfer-Encoding: base64 DQ0KDQ0KDQ0KDQ0KDQ0KDQ0KSVBTICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgUHJhc2Vuaml0IFNhcmthcg0NCkludGVybmV0IERyYWZ0ICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBJQk0NDQpEb2N1bWVu dDogZHJhZnQtaWV0Zi1pcHMtaXNjc2ktYm9vdC0wMS50eHQgICAgICAgICAgICAgRHVuY2FuIE1p c3NpbWVyDQ0KQ2F0ZWdvcnk6IFN0YW5kYXJkcyBUcmFjayAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBIUA0NCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIENvbnN0YW50aW4gU2FwdW50emFraXMNDQogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIENpc2NvDQ0K ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDEy IEphbnVhcnkgMjAwMQ0NCg0NCg0NCiAgICAgQSBTdGFuZGFyZCBmb3IgQm9vdFN0cmFwcGluZyBD bGllbnRzIHVzaW5nIHRoZSBpU0NTSSBQcm90b2NvbA0NCg0NClN0YXR1cyBvZiB0aGlzIE1lbW8N DQoNDQogICBUaGlzIGRvY3VtZW50IGlzIGFuIEludGVybmV0LURyYWZ0IGFuZCBpcyBpbiBmdWxs IGNvbmZvcm1hbmNlIHdpdGgNDQogICBhbGwgcHJvdmlzaW9ucyBvZiBTZWN0aW9uIDEwIG9mIFJG QzIwMjYgWzExXS4NDQoNDQogICBJbnRlcm5ldC1EcmFmdHMgYXJlIHdvcmtpbmcgZG9jdW1lbnRz IG9mIHRoZSBJbnRlcm5ldCBFbmdpbmVlcmluZw0NCiAgIFRhc2sgRm9yY2UgKElFVEYpLCBpdHMg YXJlYXMsIGFuZCBpdHMgd29ya2luZyBncm91cHMuIE5vdGUgdGhhdCBvdGhlcg0NCiAgIGdyb3Vw cyBtYXkgYWxzbyBkaXN0cmlidXRlIHdvcmtpbmcgZG9jdW1lbnRzIGFzIEludGVybmV0LURyYWZ0 cy4NDQogICBJbnRlcm5ldC1EcmFmdHMgYXJlIGRyYWZ0IGRvY3VtZW50cyB2YWxpZCBmb3IgYSBt YXhpbXVtIG9mIHNpeCBtb250aHMNDQogICBhbmQgbWF5IGJlIHVwZGF0ZWQsIHJlcGxhY2VkLCBv ciBtYWRlIG9ic29sZXRlIGJ5IG90aGVyIGRvY3VtZW50cyBhdA0NCiAgIGFueSB0aW1lLiBJdCBp cyBpbmFwcHJvcHJpYXRlIHRvIHVzZSBJbnRlcm5ldC0gRHJhZnRzIGFzIHJlZmVyZW5jZQ0NCiAg IG1hdGVyaWFsIG9yIHRvIGNpdGUgdGhlbSBvdGhlciB0aGFuIGFzICJ3b3JrIGluIHByb2dyZXNz LiIgIFRoZSBsaXN0DQ0KICAgb2YgY3VycmVudCBJbnRlcm5ldC1EcmFmdHMgY2FuIGJlIGFjY2Vz c2VkIGF0DQ0KICAgaHR0cDovL3d3dy5pZXRmLm9yZy9pZXRmLzFpZC1hYnN0cmFjdHMudHh0IFRo ZSBsaXN0IG9mIEludGVybmV0LURyYWZ0DQ0KICAgU2hhZG93IERpcmVjdG9yaWVzIGNhbiBiZSBh Y2Nlc3NlZCBhdA0NCiAgIGh0dHA6Ly93d3cuaWV0Zi5vcmcvc2hhZG93Lmh0bWwuDQ0KDQ0KQWJz dHJhY3QNDQoNDQogICBUaGUgU21hbGwgQ29tcHV0ZXIgU3lzdGVtcyBJbnRlcmZhY2UgKFNDU0kp IGlzIGEgcG9wdWxhciBmYW1pbHkgb2YNDQogICBwcm90b2NvbHMgZm9yIGNvbW11bmljYXRpbmcg d2l0aCBJL08gZGV2aWNlcywgZXNwZWNpYWxseSBzdG9yYWdlDQ0KICAgZGV2aWNlcy4gIGlTQ1NJ IGlzIGEgcHJvcG9zZWQgdHJhbnNwb3J0IHByb3RvY29sIGZvciBTQ1NJIHRoYXQNDQogICBvcGVy YXRlcyBvbiB0b3Agb2YgVENQWzEyXS4gIFRoaXMgbWVtbyBkZXNjcmliZXMgYSBzdGFuZGFyZCBt ZWNoYW5pc20NDQogICB0byBlbmFibGUgY2xpZW50cyB0byBib290c3RyYXAgdGhlbXNlbHZlcyB1 c2luZyB0aGUgaVNDU0kgcHJvdG9jb2wuDQ0KICAgVGhlIGdvYWwgb2YgdGhpcyBzdGFuZGFyZCBp cyB0byBlbmFibGUgY2xpZW50cyB0byBvYnRhaW4gdGhlDQ0KICAgaW5mb3JtYXRpb24gdG8gb3Bl biBhbiBpU0NTSSBzZXNzaW9uIHdpdGggdGhlIGlTQ1NJIGJvb3RzdHJwcGluZw0NCiAgIHNlcnZl ciwgYXNzdW1pbmcgdGhpcyBpbmZvcm1hdGlvbiBpcyBub3QgYXZhaWxhYmxlLg0NCg0NCjEuIFJl cXVpcmVtZW50cw0NCg0NCiAgIDEuIFRoZXJlIG11c3QgYmUgbm8gcmVzdHJpY3Rpb24gb2YgbmV0 d29yayB0b3BvbG9neSBiZXR3ZWVuIHRoZSBpU0NTSQ0NCiAgIGJvb3QgY2xpZW50IGFuZCB0aGUg Ym9vdCBzZXJ2ZXIuIENvbnNlcXVlbnRseSwgaXQgaXMgcG9zc2libGUgZm9yIGFuDQ0KICAgaVND U0kgYm9vdCBjbGllbnQgdG8gYm9vdCBmcm9tIGFuIGlTQ1NJIGJvb3Qgc2VydmVyIGJlaGluZA0N CiAgIGdhdGV3YXlzL2ZpcmV3YWxscy9ldGMgYXMgbG9uZyBhcyBpdCBpcyBwb3NzaWJsZSB0byBl c3RhYmxpc2ggYW4NDQogICBpU0NTSSBzZXNzaW9uIGJldHdlZW4gdGhlIGNsaWVudCBhbmQgdGhl IHNlcnZlci4NDQoNDQogICAyLiBUaGUgZm9sbG93aW5nIHJlcHJlc2VudHMgdGhlIG1pbmltdW0g aW5mb3JtYXRpb24gcmVxdWlyZWQgZm9yIGFuDQ0KDQ0KDQ0KDQ0KU2Fya2FyICAgICAgICAgICAg ICAgICAgICAgRXhwaXJlczogSnVseSAyMDAxICAgICAgICAgICAgICAgICAgIFtQYWdlIDFdDQ0K DQ0KDQ0KDQ0KDQ0KDQ0KU3RhbmRhcmRzLVRyYWNrICAgICAgICAgaVNDU0kgQm9vdFN0cmFwcGlu ZyBEcmFmdCAgICAgICAgMTIgSmFudWFyeSAyMDAxDQ0KDQ0KDQ0KICAgaVNDU0kgYm9vdCBjbGll bnQgdG8gY29udGFjdCBhbiBpU0NTSSBib290IHNlcnZlcjogKGEpIHRoZSBjbGllbnQncw0NCiAg IElQIGFkZHJlc3MgKElQdjYgb3IgSVB2NCk7IChiKSB0aGUgc2VydmVyJ3MgaVNDU0kgU2Vydmlj ZSBEZWxpdmVyeQ0NCiAgIFBvcnQgTmFtZTsgYW5kIChjKSBtYW5kYXRvcnkgaVNDU0kgaW5pdGlh dG9yIGNhcGFiaWxpdHkuDQ0KDQ0KICAgVGhlIGFib3ZlIGFzc3VtZXMgdGhhdCB0aGUgZGVmYXVs dCBMVU4gZm9yIHRoZSBib290IHByb2Nlc3MgaXMgMCBhbmQNDQogICB0aGUgZGVmYXVsdCBwb3J0 IGZvciB0aGUgaVNDU0kgYm9vdCBzZXJ2ZXIgaXMgdGhlIHdlbGwta25vd24gaVNDU0kNDQogICBw b3J0LiBIb3dldmVyLCBib3RoIG1heSBiZSBvdmVycmlkZGVuIGF0IHRoZSB0aW1lIG9mIGNvbmZp Z3VyYXRpb24uDQ0KDQ0KICAgQWRkaXRpb25hbCBpbmZvcm1hdGlvbiBtYXkgYmUgcmVxdWlyZWQg YXQgZWFjaCBzdGFnZSBvZiB0aGUgYm9vdA0NCiAgIHByb2Nlc3MuDQ0KDQ0KICAgMy4gSXQgaXMg cG9zc2libGUgZm9yIHRoZSBpU0NTSSBib290IGNsaWVudCB0byBoYXZlIG5vbmUgb2YgdGhlIGFi b3ZlDQ0KICAgaW5mb3JtYXRpb24gb3IgY2FwYWJpbGl0eSBvbiBzdGFydGluZy4NDQoNDQogICA0 LiBUaGUgY2xpZW50IHNob3VsZCBiZSBhYmxlIHRvIGNvbXBsZXRlIGJvb3Qgd2l0aG91dCB1c2Vy DQ0KICAgaW50ZXJ2ZW50aW9uIChmb3IgYm9vdHMgdGhhdCBvY2N1ciBkdXJpbmcgYW4gdW5hdHRl bmRlZCBwb3dlci11cCkuDQ0KICAgSG93ZXZlciwgdGhlcmUgc2hvdWxkIGJlIGEgbWVjaGFuaXNt IGZvciB0aGUgdXNlciB0byBpbnB1dCB2YWx1ZXMgc28NDQogICBhcyB0byBieXBhc3Mgc3RhZ2Vz IG9mIHRoZSBib290IHByb3RvY29sLg0NCg0NCiAgIDUuIEFkZGl0aW9uYWwgcHJvdG9jb2wgc29m dHdhcmUgKGZvciBleGFtcGxlLCBESENQKSBtYXkgYmUgbmVjZXNzYXJ5DQ0KICAgaWYgdGhlIG1p bmltdW0gaW5mb3JtYXRpb24gcmVxdWlyZWQgZm9yIGFuIGlTQ1NJIHNlc3Npb24gaXMgbm90DQ0K ICAgcHJvdmlkZWQuDQ0KDQ0KMi4gUmVsYXRlZCBXb3JrDQ0KDQ0KICAgVGhlIFJldmVyc2UgQWRk cmVzcyBSZXNvbHV0aW9uIFByb3RvY29sIChSQVJQKVs3XSh0aHJvdWdoIHRoZQ0NCiAgIGV4dGVu c2lvbnMgZGVmaW5lZCBpbiB0aGUgRHluYW1pYyBSQVJQIChEUkFSUCkpWzRdIGV4cGxpY2l0bHkN DQogICBhZGRyZXNzZXMgdGhlIHByb2JsZW0gb2YgbmV0d29yayBhZGRyZXNzIGRpc2NvdmVyeSwg YW5kIGluY2x1ZGVzIGFuDQ0KICAgYXV0b21hdGljIElQIGFkZHJlc3MgYXNzaWdubWVudCBtZWNo YW5pc20uICBUaGUgVHJpdmlhbCBGaWxlIFRyYW5zZmVyDQ0KICAgUHJvdG9jb2wgKFRGVFApWzld IHByb3ZpZGVzIGZvciB0cmFuc3BvcnQgb2YgYSBib290IGltYWdlIGZyb20gYSBib290DQ0KICAg c2VydmVyLiBCT09UUFs1LDgsMTBdIGlzIGEgdHJhbnNwb3J0IG1lY2hhbmlzbSBmb3IgYSBjb2xs ZWN0aW9uIG9mDQ0KICAgY29uZmlndXJhdGlvbiBpbmZvcm1hdGlvbi4gIEJPT1RQIGlzIGFsc28g ZXh0ZW5zaWJsZSwgYW5kIG9mZmljaWFsDQ0KICAgZXh0ZW5zaW9ucyBoYXZlIGJlZW4gZGVmaW5l ZCBmb3Igc2V2ZXJhbCBjb25maWd1cmF0aW9uIHBhcmFtZXRlcnMuDQ0KICAgREhDUHY0WzMsNl0g YW5kIERIQ1B2NlsxM10gYXJlIHN0YW5kYXJkcyBmb3IgaG9zdHMgdG8gYmUgZHluYW1pY2FsbHkN DQogICBjb25maWd1cmVkIGluIGFuIElQIG5ldHdvcmsuICBUaGUgU2VydmljZSBMb2NhdGlvbiBQ cm90b2NvbCBSTFANDQogICBwcm92aWRlcyBmb3IgbG9jYXRpb24gb2YgaGlnaGVyIGxldmVsIHNl cnZpY2VzWzEsMTVdLg0NCg0NCjMuIFNvZnR3YXJlIHN0YWdlDQ0KDQ0KICAgU29tZSBpU0NTSSBi b290IGNsaWVudHMgbWF5IGxhY2sgdGhlIHJlc291cmNlcyB0byBib290IHVwIHdpdGggdGhlDQ0K ICAgbWFuZGF0b3J5IGlTQ1NJIGluaXRpYXRvciBjYXBhYmlsaXR5LiBTdWNoIGJvb3QgY2xpZW50 cyBtYXkgY2hvb3NlIHRvDQ0KICAgb2J0YWluIGlTQ1NJIGluaXRpYXRvciBzb2Z0d2FyZSBmcm9t IGEgYm9vdCBzZXJ2ZXIuICBDdXJyZW50bHksIHRoZXJlDQ0KICAgYXJlIG1hbnkgZXN0YWJsaXNo ZWQgcHJvdG9jb2xzIHRoYXQgYWxsb3cgc3VjaCBhIHNlcnZpY2UgdG8gZW5hYmxlDQ0KICAgY2xp ZW50cyB0byBsb2FkIHNvZnR3YXJlIGltYWdlcy4gRm9yIGV4YW1wbGUsIEJPT1RQIGFuZCBESENQ IHNlcnZlcnMNDQogICBoYXZlIHRoZSBjYXBhYmlsaXR5IHRvIHByb3ZpZGUgc29mdHdhcmUgaW1h Z2VzIG9uIHJlcXVlc3RzIGZyb20gYm9vdA0NCiAgIGNsaWVudHMuIEEgcGFydGljdWxhciBpbXBs ZW1lbnRhdGlvbiBvZiB0aGlzIGFwcHJvYWNoIGlzIHRoZSBQWEUNDQogICBwcm90b2NvbFsxN10s IHdoaWNoIHVzZXMgREhDUCBleHRlbnNpb25zIGFuZCBNVEZUUCB0byBhbGxvdyBib290DQ0KICAg Y2xpZW50cyB0byBsb2FkIHNvZnR3YXJlIGltYWdlcy4NDQoNDQoNDQoNDQpTYXJrYXIgICAgICAg ICAgICAgICAgICAgICBFeHBpcmVzOiBKdWx5IDIwMDEgICAgICAgICAgICAgICAgICAgW1BhZ2Ug Ml0NDQoNDQoNDQoNDQoNDQoNDQpTdGFuZGFyZHMtVHJhY2sgICAgICAgICBpU0NTSSBCb290U3Ry YXBwaW5nIERyYWZ0ICAgICAgICAxMiBKYW51YXJ5IDIwMDENDQoNDQoNDQogICBJdCBpcyB0byBi ZSBub3RlZCB0aGF0IHRoaXMgZG9jdW1lbnQgZG9lcyBub3QgcmVjb21tZW5kIGFueSBvZiB0aGUN DQogICBhYm92ZSBwcm90b2NvbHMsIGFuZCB0aGUgZmluYWwgZGVjaXNpb24gb2Ygd2hpY2ggYm9v dCBwcm90b2NvbCBpcyB0bw0NCiAgIGJlIHVzZWQgdG8gbG9hZCBpU0NTSSBpbml0aWF0b3Igc29m dHdhcmUgaXMgbGVmdCB0byB0aGUgZGlzY3JldGlvbiBvZg0NCiAgIHRoZSBpbXBsZW1lbnRvci4N DQoNDQoNDQo0LiBESENQIHN0YWdlDQ0KDQ0KICAgSW4gb3JkZXIgdG8gdXNlIGFuIGlTQ1NJIGJv b3Qgc2VydmVyLCB0aGUgZm9sbG93aW5nIHBpZWNlcyBvZg0NCiAgIGluZm9ybWF0aW9uIGFyZSBy ZXF1aXJlZC4NDQoNDQogICAtIFRoZSBJUCBhZGRyZXNzIG9mIHRoZSBpU0NTSSBib290IGNsaWVu dCAoSVB2NCBvciBJUHY2KQ0NCg0NCiAgIC0gVGhlIElQIHRyYW5zcG9ydCBlbmRwb2ludCBmb3Ig dGhlIGlTQ1NJIHNlcnZpY2UgZGVsaXZlcnkgcG9ydCBmb3INDQogICB0aGUgaVNDU0kgYm9vdCBz ZXJ2ZXIuICBJZiB0aGUgdHJhbnNwb3J0IGlzIFRDUCwgZm9yIGV4YW1wbGUsIHRoaXMNDQogICBo YXMgdG8gcmVzb2x2ZSB0byBhbiBJUCBhZGRyZXNzIGFuZCBhIFRDUCBwb3J0IG51bWJlci4NDQoN DQogICAtIFRoZSBlaWdodC1ieXRlIExVTiBzdHJ1Y3R1cmUgaWRlbnRpZnlpbmcgdGhlIGRldmlj ZSB3aXRoaW4gdGhlDQ0KICAgaVNDU0kgYm9vdCBzZXJ2ZXIuDQ0KDQ0KICAgQXQgYm9vdCB0aW1l LCBhbGwgb3Igbm9uZSBvZiB0aGlzIGluZm9ybWF0aW9uIG1heSBiZSBzdG9yZWQgaW4gdGhlDQ0K ICAgZmlybXdhcmUgb2YgdGhlIGlTQ1NJIGJvb3QgY2xpZW50LiBUaGlzIHNlY3Rpb24gZGVzY3Jp YmVzIHRlY2huaXF1ZXMNDQogICBmb3Igb2J0YWluaW5nIHRoZSByZXF1aXJlZCBpbmZvcm1hdGlv bi4NDQoNDQogICBBbiBpU0NTSSBib290IGNsaWVudCB3aGljaCBkb2VzIG5vdCBrbm93IGl0cyBJ UCBhZGRyZXNzIGF0IHBvd2VyLW9uDQ0KICAgbWF5IGFjcXVpcmUgaXRzIElQIGFkZHJlc3Mgdmlh IERIQ1AuICBBbiBpU0NTSSBib290IGNsaWVudCB3aGljaCBpcw0NCiAgIGNhcGFibGUgb2YgdXNp bmcgYm90aCBESENQdjYgYW5kIERIQ1B2NCBzaG91bGQgZmlyc3QgYXR0ZW1wdCB0byB1c2UNDQog ICBESENQdjYgdG8gb2J0YWluIGl0cyBJUCBhZGRyZXNzLCBmYWxsaW5nIGJhY2sgb24gREhDUHY0 IGluIHRoZSBldmVudA0NCiAgIG9mIGZhaWx1cmUuDQ0KDQ0KICAgVW5sZXNzIG90aGVyd2lzZSBz cGVjaWZpZWQgaGVyZSwgREhDUCBmaWVsZHMgc3VjaCBhcyB0aGUgY2xpZW50IElEDQ0KICAgYW5k IGdhdGV3YXkgaW5mb3JtYXRpb24gYXJlIHVzZWQgaWRlbnRpY2FsbHkgd2l0aCBhcHBsaWNhdGlv bnMgb3RoZXINDQogICB0aGFuIGlTQ1NJLg0NCg0NCiAgIEEgREhDUCBzZXJ2ZXIgKHY0IG9yIHY2 KSBtYXkgaW5zdHJ1Y3QgYW4gaVNDU0kgY2xpZW50IGhvdyB0byByZWFjaA0NCiAgIGl0cyBib290 IGRldmljZS4gVGhpcyBpcyBkb25lIHVzaW5nIGEgdmFyaWFibGUgbGVuZ3RoIERIQ1Agb3B0aW9u DQ0KICAgZmllbGQga25vd24gYXMgdGhlIElTQ1NJIEJvb3QgU2VydmljZSBvcHRpb24uICBUaGUg b3B0aW9uIGlkZW50aWZpZXINDQogICBpcyB0byBiZSBhbGxvY2F0ZWQgYnkgdGhlIElFU0cgZHVy aW5nIHRoZSBhcHByb3ZhbCBwcm9jZXNzWzE5XS4NDQoNDQogICBUaGUgZmllbGQgY29uc2lzdHMg b2YgYW4gVVRGLThbMjBdIHN0cmluZyBhbmQgaGFzIHRoZSBmb2xsb3dpbmcNDQogICBjb21wb3Np dGlvbjoNDQoNDQogICAgICAgICAgIDxzZXJ2ZXJuYW1lPiAiOiIgPHBvcnQ+ICI6IiA8TFVOPiAi OiIgPHRhcmdldG5hbWU+DQ0KDQ0KICAgVGhlIGZpZWxkcyAicG9ydCIsICJMVU4iIGFuZCAidGFy Z2V0bmFtZSIgYXJlIG9wdGlvbmFsIGFuZCBzaG91bGQgYmUNDQogICBsZWZ0IGJsYW5rIGlmIHRo ZXJlIGFyZSBubyB2YWx1ZXMgY29ycmVzcG9uZGluZyB0byB0aGUgZmllbGRzLg0NCg0NCiAgIFRo ZSAic2VydmVybmFtZSIgaXMgdGhlIG5hbWUgb2YgaVNDU0kgc2VydmVyIGFuZCBjb250YWlucyBl aXRoZXIgYQ0NCg0NCg0NCg0NClNhcmthciAgICAgICAgICAgICAgICAgICAgIEV4cGlyZXM6IEp1 bHkgMjAwMSAgICAgICAgICAgICAgICAgICBbUGFnZSAzXQ0NCg0NCg0NCg0NCg0NCg0NClN0YW5k YXJkcy1UcmFjayAgICAgICAgIGlTQ1NJIEJvb3RTdHJhcHBpbmcgRHJhZnQgICAgICAgIDEyIEph bnVhcnkgMjAwMQ0NCg0NCg0NCiAgIHZhbGlkIGRvbWFpbiBuYW1lLCBhIGxpdGVyYWwgSVB2NCBh ZGRyZXNzLCBvciBhIGJyYWNrZXRlZCBsaXRlcmFsDQ0KICAgSVB2NiBhZGRyZXNzLiBJZiB0aGUg c2VydmVybmFtZSBmaWVsZCBjb250YWlucyBhIGxpdGVyYWwgSVB2NA0NCiAgIGFkZHJlc3MsIHRo ZSBJUHY0IGFkZHJlc3MgaXMgaW4gc3RhbmRhcmQgZG90dGVkIGRlY2ltYWwgbm90YXRpb24uIElm DQ0KICAgdGhlIHNlcnZlcm5hbWUgZmllbGQgY29udGFpbnMgYW4gSVB2NiBhZGRyZXNzLCB0aGUg YWRkcmVzcyBpcw0NCiAgIHJlcHJlc2VudGVkIGluIGJyYWNrZXRlZCBsaXRlcmFsIElQdjYgYWRk cmVzcyBmb3JtYXQuDQ0KDQ0KICAgSWYgdGhlICJzZXJ2ZXJuYW1lIiBpcyBhIGRvbWFpbiBuYW1l LCB0aGVuIHRoZSByZXBseSBmcm9tIHRoZSBob3N0DQ0KICAgY29uZmlndXJhdGlvbiBzZXJ2ZXIg bWF5IGNvbnRhaW4gdGhlIERvbWFpbiBOYW1lIFNlcnZlciBPcHRpb25bMl0uDQ0KDQ0KICAgVGhl ICJwb3J0IiBpcyB0aGUgZGVjaW1hbCByZXByZXNlbnRhdGlvbiBvZiB0aGUgcG9ydCBvbiB3aGlj aCB0aGUNDQogICBpU0NTSSBib290IHNlcnZlciBpcyBsaXN0ZW5pbmcuIElmIG5vdCBzcGVjaWZp ZWQsIHRoZSBwb3J0IGRlZmF1bHRzDQ0KICAgdG8gdGhlIHdlbGwta25vd24gaVNDU0kgcG9ydC4N DQoNDQogICBUaGUgIkxVTiIgZmllbGQgaXMgYSAxNiBieXRlIGhleGFkZWNpbWFsIHJlcHJlc2Vu dGF0aW9uIG9mIHRoZSA4LWJ5dGUNDQogICBMVSBudW1iZXIgaW4gaGV4LiBEaWdpdHMgYWJvdmUg OSBtYXkgYmUgZWl0aGVyIGxvd2VyIG9yIHVwcGVyIGNhc2UsDQ0KICAgYW5kIGFsbCAxNiBuaWJi bGVzIG11c3QgYmUgcHJlc2VudC4gSWYgdGhlIExVTiBmaWVsZCBpcyBibGFuaywgdGhlbg0NCiAg IExVTiAwIGlzIGFzc3VtZWQuDQ0KDQ0KICAgTm90ZSB0aGF0IFNDU0kgdGFyZ2V0cyBhcmUgYWxs b3dlZCB0byBwcmVzZW50IGRpZmZlcmVudCBMVSBudW1iZXJpbmdzDQ0KICAgZm9yIGRpZmZlcmVu dCBTQ1NJIGluaXRpYXRvcnMsIHNvIHRoYXQgdG8gb3VyIGtub3dsZWRnZSBub3RoaW5nDQ0KICAg cHJlY2x1ZGVzIGEgU0NTSSB0YXJnZXQgZnJvbSBleHBvcnRpbmcgc2V2ZXJhbCBkaWZmZXJlbnQg ZGV2aWNlcyB0bw0NCiAgIHNldmVyYWwgZGlmZmVyZW50IFNDU0kgaW5pdGlhdG9ycyBhcyB0aGVp ciByZXNwZWN0aXZlIExVIDBzLg0NCg0NCiAgIFRoZSAidGFyZ2V0bmFtZSIgZmllbGQgaXMgYSBz dHJpbmcgY29udGFpbmluZyB0aGUgbmFtZSBvZiB0aGUgaVNDU0kNDQogICB0YXJnZXQsIHRoZSBk ZXRhaWxzIG9mIHdoaWNoIGFyZSBzcGVjaWZpZWQgYnkgdGhlIGlTQ1NJIHN0YW5kYXJkWzEyXS4N DQogICBJZiB0aGUgdGFyZ2V0bmFtZSBpcyBwcm92aWRlZCwgdGhlIGlTQ1NJIGJvb3QgY2xpZW50 IG1heSB1c2UgdGhlDQ0KICAgdGFyZ2V0bmFtZSBhcyBtYW5kYXRlZCBieSB0aGUgaVNDU0kgc3Rh bmRhcmQuDQ0KDQ0KICAgVGhlIGFib3ZlIGFzc3VtZXMgdGhhdCB0aGUgZGVmYXVsdCBjb25uZWN0 aW9uIG1ldGhvZCB1c2VzIFRDUCBhcw0NCiAgIHN0YXRlZCBpbiB0aGUgaVNDU0kgc3RhbmRhcmQu IFNob3VsZCBTQ1RQWzE4XSBiZSBhbHNvIGFwcHJvdmVkIGFzIGENDQogICB0cmFuc3BvcnQgbWVj aGFuaXNtIGZvciBpU0NTSSwgdGhlbiB0aGUgZHJhZnQgd2lsbCBiZSBhbWVuZGVkIHRvDQ0KICAg cHJvdmlkZSBmb3IgYWx0ZXJuYXRlIHRyYW5zcG9ydCBwcm90b2NvbHMuDQ0KDQ0KNS4gRGlzY292 ZXJ5IFNlcnZpY2Ugc3RhZ2U6DQ0KDQ0KICAgVGhpcyBzdGFnZSBpcyByZXF1aXJlZCBpZiB0aGUg REhDUCBzZXJ2ZXIgKHY0IG9yIHY2KSBpcyB1bmF3YXJlIG9mDQ0KICAgdGhlIGlkZW50aXR5IG9m IHRoZSBpU0NTSSBib290IHNlcnZlci4NDQoNDQogICBUaGUgaVNDU0kgYm9vdCBjbGllbnQgdGhl biBtYXkgc3RhcnQgdGhlIGRpc2NvdmVyeSBwcm9jZXNzIGFjY29yZGluZw0NCiAgIHRvIHRoZSBz cGVjaWZpY2F0aW9ucyBzdGF0ZWQgaW4gdGhlIGlTQ1NJIE5hbWluZyBhbmQgRGlzY292ZXJ5DQ0K ICAgZG9jdW1lbnRbMTRdLiBUaGUgZGlzY292ZXJ5IHNlcnZpY2UgcHJvdmlkZXMgdGhlIGJvb3Qg Y2xpZW50IHdpdGggYQ0NCiAgIGxpc3Qgb2YgU0NTSSB0YXJnZXRzIHRoZSBjbGllbnQgaXMgYWxs b3dlZCB0byBhY2Nlc3MsIGFsb25nIHdpdGggdGhlDQ0KICAgYWNjZXNzIHBlcm1pc3Npb25zIGZv ciBlYWNoIG9mIHRoZSB0YXJnZXQuIFRoZSBuYXR1cmUgYW5kDQ0KICAgaW1wbGVtZW50aW9uIG9m IHRoZSBkaXNjb3Zlcnkgc2VydmljZSBpcyBvdXRzaWRlIHRoZSBzY29wZSBvZiB0aGlzDQ0KICAg ZG9jdW1lbnQuDQ0KDQ0KICAgVGhlIGlTQ1NJIGJvb3QgY2xpZW50IGdvZXMgdGhyb3VnaCB0aGUg bGlzdCBvZiBTQ1NJIHRhcmdldHMgYW5kIG11c3QNDQogICBzZWxlY3QgdGhlIGZpcnN0IFNDU0kg dGFyZ2V0IHdpdGggdGhlIGJvb3RhYmxlIGF0dHJpYnV0ZSBhcyB0aGUgaVNDU0kNDQoNDQoNDQoN DQpTYXJrYXIgICAgICAgICAgICAgICAgICAgICBFeHBpcmVzOiBKdWx5IDIwMDEgICAgICAgICAg ICAgICAgICAgW1BhZ2UgNF0NDQoNDQoNDQoNDQoNDQoNDQpTdGFuZGFyZHMtVHJhY2sgICAgICAg ICBpU0NTSSBCb290U3RyYXBwaW5nIERyYWZ0ICAgICAgICAxMiBKYW51YXJ5IDIwMDENDQoNDQoN DQogICBib290IHNlcnZlci4gSWYgc3VjaCBhbiBhdHRyaWJ1dGUgZG9lcyBub3QgZXhpc3QgaW4g YW55IG9mIHRoZSBTQ1NJDQ0KICAgdGFyZ2V0cywgdGhlIGJvb3QgY2xpZW50IG11c3Qgc2VsZWN0 IHRoZSBmaXJzdCBTQ1NJIHRhcmdldCBpbiB0aGUNDQogICBsaXN0IG9mIFNDU0kgdGFyZ2V0cyBh cyB0aGUgaVNDU0kgYm9vdCBzZXJ2ZXIuDQ0KDQ0KICAgSWYgdGhlIGxpc3Qgb2YgU0NTSSB0YXJn ZXRzIGlzIGVtcHR5LCBzdWJzZXF1ZW50IGFjdGlvbnMgYXJlIGxlZnQgdG8NDQogICB0aGUgZGlz Y3JldGlvbiBvZiB0aGUgaW1wbGVtZW50b3IuDQ0KDQ0KICAgVGhlIHBhY2tldHMgYW5kIHNvZnR3 YXJlIHJlcXVpcmVtZW50cyBhcmUgc3RhdGVkIGluIHRoZSBpU0NTSSBOYW1pbmcNDQogICBhbmQg RGlzY292ZXJ5IGRvY3VtZW50WzE0XS4NDQoNDQo2LiBCb290IFN0YWdlDQ0KDQ0KICAgT25jZSB0 aGUgaVNDU0kgYm9vdCBjbGllbnQgaGFzIG9idGFpbmVkIHRoZSBtaW5pbXVtIGluZm9ybWF0aW9u IHRvDQ0KICAgb3BlbiBhbiBpU0NTSSBzZXNzaW9uIHdpdGggdGhlIGlTQ1NJIGJvb3Qgc2VydmVy LCB0aGUgYWN0dWFsIGJvb3RpbmcNDQogICBwcm9jZXNzIGNhbiBzdGFydC4NDQoNDQogICBUaGUg YWN0dWFsIHNlcXVlbmNlIG9mIGlTQ1NJIGNvbW1hbmRzIG5lZWRlZCB0byBjb21wbGV0ZSB0aGUg Ym9vdA0NCiAgIHByb2Nlc3MgaXMgbGVmdCB0byB0aGUgaW1wbGVtZW50b3IuIFRoaXMgd2FzIGRv bmUgYmVjYXVzZSBvZiB2YXJ5aW5nDQ0KICAgcmVxdWlyZW1lbnRzIGZyb20gZGlmZmVyZW50IHZl bmRvcnMgYW5kIGVxdWlwbWVudHMsIG1ha2luZyBpdA0NCiAgIGRpZmZpY3VsdCB0byBzcGVjaWZ5 IGEgY29tbW9uIHN1YnNldCBvZiB0aGUgaVNDU0kgc3RhbmRhcmQgdGhhdCB3b3VsZA0NCiAgIGJl IGFjY2VwdGFibGUgdG8gZXZlcnlib2R5Lg0NCg0NCiAgIFRoZSBpU0NTSSBzZXNzaW9uIGVzdGFi bGlzaGVkIGZvciBib290IG1heSBiZSB0YWtlbiBvdmVyIHRoZSBib290ZWQNDQogICBzb2Z0d2Fy ZSBpbiB0aGUgYm9vc3RyYXBwaW5nIGNsaWVudCAtIHRoaXMgaXMgbGVmdCB0byB0aGUgZGlzY3Jl dGlvbg0NCiAgIG9mIHRoZSBpbXBsZW1lbnRvci4NDQoNDQo3LiBTZWN1cml0eQ0NCg0NCiAgIFNl Y3VyaW5nIHRoZSBob3N0IGNvbmZpZ3VyYXRpb24gcHJvdG9jb2wgaXMgYmV5b25kIHRoZSBzY29w ZSBvZiB0aGlzDQ0KICAgZG9jdW1lbnQuIEF1dGhlbnRpY2F0aW9uIG9mIERIQ1AgbWVzc2FnZXMg aXMgZGVzY3JpYmVkIGluIFsxNl0uDQ0KDQ0KICAgVGhlIGlTQ1NJIHN0YW5kYXJkIHN1cHBvcnQg dmFyaW91cyBtZXRob2RzIG9mIGF1dGhlbnRpY2F0ZWQgbG9naW4gYW5kDQ0KICAgZW5jcnlwdGVk IGFuZCBhdXRoZW50aWNhdGVkIGNvbm5lY3Rpb25zIGZvciBzZWN1cml0eS4gSG93IHRvDQ0KICAg Y29uZmlndXJlIHRoZSBzZWN1cml0eSBwYXJhbWV0ZXJzIG9mIGFuIGlTQ1NJIGJvb3QgY2xpZW50 IGlzIGJleW9uZA0NCiAgIHRoZSBzY29wZSBvZiB0aGlzIGRvY3VtZW50Lg0NCg0NCiAgIFRoZSBz ZWN1cml0eSBkaXNjdXNzaW9ucyBpbiB0aGUgaVNDU0kgc3RhbmRhcmRbMTJdIGFyZSBhcHBsaWNh YmxlIHRvDQ0KICAgdGhpcyBkb2N1bWVudC4NDQoNDQpBY2tub3dsZWRnbWVudHMNDQoNDQogICBX ZSB3aXNoIHRvIHRoYW5rIEpvaG4gSHVmZmVyZCBmb3IgdGFraW5nIHRoZSBpbml0aWF0aXZlIHRv IGZvcm0gdGhlDQ0KICAgaVNDU0kgYm9vdCB0ZWFtLiBXZSBhbHNvIHdpc2ggdG8gdGhhbmsgRG91 ZyBPdGlzIGFuZCBEYXZpZCBSb2JpbnNvbg0NCiAgIGZvciBoZWxwZnVsIHN1Z2dlc3Rpb25zIGFu ZCBwb2ludGVycyByZWdhcmRpbmcgdGhlIGRyYWZ0IGRvY3VtZW50Lg0NCg0NClJlZmVyZW5jZXMN DQoNDQogICBbMV0gR3V0dG1hbiwgRS4sIFBlcmtpbnMsIEMuLCBWZXJpemFkZXMsIEouLCBEYXks IE0uLCAiU2VydmljZQ0NCg0NCg0NCg0NClNhcmthciAgICAgICAgICAgICAgICAgICAgIEV4cGly ZXM6IEp1bHkgMjAwMSAgICAgICAgICAgICAgICAgICBbUGFnZSA1XQ0NCg0NCg0NCg0NCg0NCg0N ClN0YW5kYXJkcy1UcmFjayAgICAgICAgIGlTQ1NJIEJvb3RTdHJhcHBpbmcgRHJhZnQgICAgICAg IDEyIEphbnVhcnkgMjAwMQ0NCg0NCg0NCiAgIExvY2F0aW9uIFByb3RvY29sIHYyIiwgUkZDIDI2 MDgsIEp1bmUgMTk5OS4NDQoNDQogICBbMl0gQWxleGFuZGVyLCBTLiwgYW5kIFIuIERyb21zLCAi REhDUCBPcHRpb25zIGFuZCBCT09UUCBWZW5kb3INDQogICAgICAgICAgRXh0ZW5zaW9ucyIsIFJG QyAyMTMyLCBMYWNobWFuIFRlY2hub2xvZ3ksIEluYy4sIEJ1Y2tuZWxsDQ0KICAgICAgICAgIFVu aXZlcnNpdHksIE9jdG9iZXIgMTk5My4NDQoNDQogICBbM10gUi4gRHJvbXMsICJEeW5hbWljIEhv c3QgQ29uZmlndXJhdGlvbiBQcm90b2NvbCIsIFJGQyAyMTMxLA0NCiAgICAgICAgICBCdWNrbmVs bCBVbml2ZXJzaXR5LCBNYXJjaCAxOTk3Lg0NCg0NCiAgIFs0XSBCcm93bmVsbCwgRCwgIkR5bmFt aWMgUmV2ZXJzZSBBZGRyZXNzIFJlc29sdXRpb24gUHJvdG9jb2wNDQogICAgICAgICAgKERSQVJQ KSIsIFdvcmsgaW4gUHJvZ3Jlc3MuDQ0KDQ0KICAgWzVdIENyb2Z0LCBCLiwgYW5kIEouIEdpbG1v cmUsICJCb290c3RyYXAgUHJvdG9jb2wgKEJPT1RQKSIsIFJGQyA5NTEsDQ0KICAgICAgICAgIFN0 YW5mb3JkIGFuZCBTVU4gTWljcm9zeXN0ZW1zLCBTZXB0ZW1iZXIgMTk4NS4NDQoNDQogICBbNl0g RHJvbXMsIEQuLCAiSW50ZXJvcGVyYXRpb24gYmV0d2VlbiBESENQIGFuZCBCT09UUCIgUkZDIDE1 MzQsDQ0KICAgICAgICAgIEJ1Y2tuZWxsIFVuaXZlcnNpdHksIE9jdG9iZXIgMTk5My4NDQoNDQog ICBbN10gRmlubGF5c29uLCBSLiwgTWFubiwgVC4sIE1vZ3VsLCBKLiwgYW5kIE0uIFRoZWltZXIs ICJBIFJldmVyc2UNDQogICAgICAgICAgQWRkcmVzcyBSZXNvbHV0aW9uIFByb3RvY29sIiwgUkZD IDkwMywgU3RhbmZvcmQsIEp1bmUgMTk4NC4NDQoNDQogICBbOF0gUmV5bm9sZHMsIEouLCAiQk9P VFAgVmVuZG9yIEluZm9ybWF0aW9uIEV4dGVuc2lvbnMiLCBSRkMgMTQ5NywNDQogICAgICAgICAg VVNDL0luZm9ybWF0aW9uIFNjaWVuY2VzIEluc3RpdHV0ZSwgQXVndXN0IDE5OTMuDQ0KDQ0KICAg WzldIFNvbGxpbnMsIEsuLCAiVGhlIFRGVFAgUHJvdG9jb2wgKFJldmlzaW9uIDIpIiwgIFJGQyA3 ODMsIE5JQywNDQogICAgICAgICAgSnVuZSAxOTgxLg0NCg0NCiAgIFsxMF0gV2ltZXIsIFcuLCAi Q2xhcmlmaWNhdGlvbnMgYW5kIEV4dGVuc2lvbnMgZm9yIHRoZSBCb290c3RyYXANDQogICAgICAg ICAgUHJvdG9jb2wiLCBSRkMgMTUzMiwgQ2FybmVnaWUgTWVsbG9uIFVuaXZlcnNpdHksIE9jdG9i ZXIgMTk5My4NDQoNDQogICBbMTFdIEJyYWRuZXIsIFMuLCAiVGhlIEludGVybmV0IFN0YW5kYXJk cyBQcm9jZXNzIC0tDQ0KICAgICAgICAgUmV2aXNpb24gMyIsIFJGQyAyMDI2LCBPY3RvYmVyIDE5 OTYuDQ0KDQ0KICAgWzEyXSBTYXRyYW4sIEouLCAiaVNDU0kiLCBJbnRlcm5ldC1EcmFmdCwgTm92 ZW1iZXIgMjAwMC4NDQoNDQogICBbMTNdIEJvdW5kLCBKLiwgQ2FubmV5LCBNLiwgYW5kIFBlcmtp bnMsIEMuLCAiRHluYW1pYyBIb3N0DQ0KICAgQ29uZmlndXJhdGlvbg0NCiAgICAgICAgUHJvdG9j b2wgZm9yIElQdjYiLCBJbnRlcm5ldC1EcmFmdCwgTm92ZW1iZXIgMjAwMC4NDQoNDQogICBbMTRd IFZvcnVnYW50aSwgSy4gZXQgYWwuLCAiaVNDU0kgTmFtaW5nIGFuZCBEaXNjb3ZlcnkiLCBJbnRl cm5ldC0NDQogICBEcmFmdCwNDQogICAgICAgIE5vdmVtYmVyIDIwMDAuDQ0KDQ0KICAgWzE1XSBW ZWl6YWRlcywgSi4sIEd1dHRtYW4sIEUuLCBQZXJraW5zLCBDLiwgS2FwbGFuLCBTLiwgIlNlcnZp Y2UNDQogICBMb2NhdGlvbiBQcm90b2NvbCIsIFJGQyAyMTY1LCBKdW5lIDE5OTcuDQ0KDQ0KICAg WzE2XSBEcm9tcywgUi4sIEFyYmF1Z2gsIFcuLCAiQXV0aGVudGljYXRpb24gZm9yIERIQ1AgTWVz c2FnZXMiLA0NCiAgIEludGVybmV0LURyYWZ0LCBOb3ZlbWJlciAyMDAwLg0NCg0NCg0NCg0NClNh cmthciAgICAgICAgICAgICAgICAgICAgIEV4cGlyZXM6IEp1bHkgMjAwMSAgICAgICAgICAgICAg ICAgICBbUGFnZSA2XQ0NCg0NCg0NCg0NCg0NCg0NClN0YW5kYXJkcy1UcmFjayAgICAgICAgIGlT Q1NJIEJvb3RTdHJhcHBpbmcgRHJhZnQgICAgICAgIDEyIEphbnVhcnkgMjAwMQ0NCg0NCg0NCiAg IFsxN10gaHR0cDovL2RldmVsb3Blci5pbnRlbC5jb20vaWFsL1dmTS93Zm0yMC9kZXNpZ24vcHhl ZHQvaW5kZXguaHRtDQ0KDQ0KICAgWzE4XSBTdGV3YXJ0LCBSLiwgZXQgYWwuICJTdHJlYW0gQ29u dHJvbCBUcmFuc21pc3Npb24gUHJvdG9jb2wiLCBSRkMNDQogICAyOTYwLCBPY3RvYmVyIDIwMDAu DQ0KDQ0KICAgWzE5XSBEcm9tcywgUi4sICJQcm9jZWR1cmVzIGFuZCBJQU5BIEd1aWRlbGluZXMg Zm9yIEFwcHJvdmFsIG9mIE5ldw0NCiAgIERIQ1AgT3B0aW9ucyBhbmQgTWVzc2FnZSBUeXBlcyIs IFJGQyAyOTM5LCBTZXB0ZW1iZXIgMjAwMC4NDQoNDQogICBbMjBdIFllcmdlYXUsIEYuLCAiVVRG LTg6IEEgVHJhbnNmb3JtYXRpb24gRm9ybWF0IGZvciBJU08tMTA2NDYiLCBSRkMNDQogICAyMjc5 LCBKYW51YXJ5IDE5OTguDQ0KDQ0KQXV0aG9yJ3MgQWRkcmVzc2VzDQ0KDQ0KICAgUHJhc2Vuaml0 IFNhcmthcg0NCiAgIElCTSBBbG1hZGVuIFJlc2VhcmNoIENlbnRlcg0NCiAgIDY1MCBIYXJyeSBS b2FkDQ0KICAgU2FuIEpvc2UsIENBIDk1MTIwLCBVU0ENDQogICBQaG9uZTogKzEgNDA4IDkyNyAx NDE3DQ0KICAgRW1haWw6IHBzYXJrYXJAYWxtYWRlbi5pYm0uY29tDQ0KDQ0KICAgRHVuY2FuIE1p c3NpbWVyDQ0KICAgSGV3bGV0dC1QYWNrYXJkIENvbXBhbnkNDQogICAxOTQyMCBIb21lc3RlYWQg Um9hZCwgTS9TIDQzbG8NDQogICBDdXBlcnRpbm8sIENBIDk1MDE0LCBVU0ENDQogICBQaG9uZTog KzEgNDA4IDQ0NyA1MzkwDQ0KICAgRW1haWw6IGR1bmNhbl9taXNzaW1lckBocC5jb20NDQoNDQog ICBDb25zdGFudGluZSBTYXB1bnR6YWtpcw0NCiAgIENpc2NvIFN5c3RlbXMsIEluYy4NDQogICAx NzAgVy4gVGFzbWFuIERyaXZlDQ0KICAgU2FuIEpvc2UsIENBIDk1MTM0LCBVU0ENDQogICBQaG9u ZTogKzEgNjUwIDUyMCAwMjA1DQ0KICAgRW1haWw6IGNzYXB1bnR6QGNpc2NvLmNvbQ0NCg0NCg0N CkZ1bGwgQ29weXJpZ2h0IFN0YXRlbWVudA0NCg0NCiAgICJDb3B5cmlnaHQgKEMpIFRoZSBJbnRl cm5ldCBTb2NpZXR5IChkYXRlKS4gQWxsIFJpZ2h0cyBSZXNlcnZlZC4gVGhpcw0NCiAgIGRvY3Vt ZW50IGFuZCB0cmFuc2xhdGlvbnMgb2YgaXQgbWF5IGJlIGNvcGllZCBhbmQgZnVybmlzaGVkIHRv DQ0KICAgb3RoZXJzLCBhbmQgZGVyaXZhdGl2ZSB3b3JrcyB0aGF0IGNvbW1lbnQgb24gb3Igb3Ro ZXJ3aXNlIGV4cGxhaW4gaXQNDQogICBvciBhc3Npc3QgaW4gaXRzIGltcGxlbWVudGF0aW9uIG1h eSBiZSBwcmVwYXJlZCwgY29waWVkLCBwdWJsaXNoZWQNDQogICBhbmQgZGlzdHJpYnV0ZWQsIGlu IHdob2xlIG9yIGluIHBhcnQsIHdpdGhvdXQgcmVzdHJpY3Rpb24gb2YgYW55DQ0KICAga2luZCwg cHJvdmlkZWQgdGhhdCB0aGUgYWJvdmUgY29weXJpZ2h0IG5vdGljZSBhbmQgdGhpcyBwYXJhZ3Jh cGggYXJlDQ0KICAgaW5jbHVkZWQgb24gYWxsIHN1Y2ggY29waWVzIGFuZCBkZXJpdmF0aXZlIHdv cmtzLiBIb3dldmVyLCB0aGlzDQ0KICAgZG9jdW1lbnQgaXRzZWxmIG1heSBub3QgYmUgbW9kaWZp ZWQgaW4gYW55IHdheSwgc3VjaCBhcyBieSByZW1vdmluZw0NCiAgIHRoZSBjb3B5cmlnaHQgbm90 aWNlIG9yIHJlZmVyZW5jZXMgdG8gdGhlIEludGVybmV0IFNvY2lldHkgb3Igb3RoZXINDQogICBJ bnRlcm5ldCBvcmdhbml6YXRpb25zLCBleGNlcHQgYXMgbmVlZGVkIGZvciB0aGUgcHVycG9zZSBv Zg0NCiAgIGRldmVsb3BpbmcgSW50ZXJuZXQgc3RhbmRhcmRzIGluIHdoaWNoIGNhc2UgdGhlIHBy b2NlZHVyZXMgZm9yDQ0KDQ0KDQ0KDQ0KU2Fya2FyICAgICAgICAgICAgICAgICAgICAgRXhwaXJl czogSnVseSAyMDAxICAgICAgICAgICAgICAgICAgIFtQYWdlIDddDQ0KDQ0KDQ0KDQ0KDQ0KDQ0K U3RhbmRhcmRzLVRyYWNrICAgICAgICAgaVNDU0kgQm9vdFN0cmFwcGluZyBEcmFmdCAgICAgICAg MTIgSmFudWFyeSAyMDAxDQ0KDQ0KDQ0KICAgY29weXJpZ2h0cyBkZWZpbmVkIGluIHRoZSBJbnRl cm5ldCBTdGFuZGFyZHMgcHJvY2VzcyBtdXN0IGJlDQ0KICAgZm9sbG93ZWQsIG9yIGFzIHJlcXVp cmVkIHRvIHRyYW5zbGF0ZSBpdCBpbnRvIGxhbmd1YWdlcyBvdGhlciB0aGFuDQ0KICAgRW5nbGlz aC4NDQoNDQogICBUaGUgbGltaXRlZCBwZXJtaXNzaW9ucyBncmFudGVkIGFib3ZlIGFyZSBwZXJw ZXR1YWwgYW5kIHdpbGwgbm90IGJlDQ0KICAgcmV2b2tlZCBieSB0aGUgSW50ZXJuZXQgU29jaWV0 eSBvciBpdHMgc3VjY2Vzc29ycyBvciBhc3NpZ25zLg0NCg0NCiAgIFRoaXMgZG9jdW1lbnQgYW5k IHRoZSBpbmZvcm1hdGlvbiBjb250YWluZWQgaGVyZWluIGlzIHByb3ZpZGVkIG9uIGFuDQ0KICAg IkFTIElTIiBiYXNpcyBhbmQgVEhFIElOVEVSTkVUIFNPQ0lFVFkgQU5EIFRIRSBJTlRFUk5FVCBF TkdJTkVFUklORw0NCiAgIFRBU0sgRk9SQ0UgRElTQ0xBSU1TIEFMTCBXQVJSQU5USUVTLCBFWFBS RVNTIE9SIElNUExJRUQsIElOQ0xVRElORw0NCiAgIEJVVCBOT1QgTElNSVRFRCBUTyBBTlkgV0FS UkFOVFkgVEhBVCBUSEUgVVNFIE9GIFRIRSBJTkZPUk1BVElPTg0NCiAgIEhFUkVJTiBXSUxMIE5P VCBJTkZSSU5HRSBBTlkgUklHSFRTIE9SIEFOWSBJTVBMSUVEIFdBUlJBTlRJRVMgT0YNDQogICBN RVJDSEFOVEFCSUxJVFkgT1IgRklUTkVTUyBGT1IgQSBQQVJUSUNVTEFSIFBVUlBPU0UuIg0NCg0N Cg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0N Cg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NCg0NClNh cmthciAgICAgICAgICAgICAgICAgICAgIEV4cGlyZXM6IEp1bHkgMjAwMSAgICAgICAgICAgICAg ICAgICBbUGFnZSA4XQ0NCg0NCg0NCg== --0__=882569D30018476C8f9e8a93df938690918c882569D30018476C-- From owner-ips@ece.cmu.edu Sat Jan 13 00:10:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA01895 for ; Sat, 13 Jan 2001 00:10:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0D2ji917683 for ips-outgoing; Fri, 12 Jan 2001 21:45:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from shell11.ba.best.com (root@shell11.ba.best.com [206.184.139.142]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0D2jSl17677 for ; Fri, 12 Jan 2001 21:45:28 -0500 (EST) Received: (from bberg@localhost) by shell11.ba.best.com (8.9.3/8.9.2/best.sh) id SAA01929 for ips@ece.cmu.edu; Fri, 12 Jan 2001 18:45:01 -0800 (PST) Message-Id: <200101130245.SAA01929@shell11.ba.best.com> Subject: I-D ACTION:draft-ietf-ips-iscsi-03.txt (fwd) To: ips@ece.cmu.edu Date: Fri, 12 Jan 2001 18:45:01 -0800 (PST) Reply-To: bberg@bswd.com (Brian A. Berg) From: bberg@bswd.com (Brian A. Berg) Organization: Berg Software Design ( http://www.bswd.com/ ) X-Mailer: ELM [version 2.4ME+ PL38 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Please note that this new document is dated January 10, 2000, not 2001. ___________________________________________________________________ Brian A. Berg bberg@bswd.com Voice: 408.741.5010 Berg Software Design FAX: 408.741.5234 P.O. Box 3488 http://www.bswd.com/ 14500 Big Basin Way, Suite F Consulting: SCSI, storage, I/O Saratoga, CA 95070 USA ----- Forwarded message from Internet-Drafts@ietf.org ----- A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iSCSI Author(s) : J. Satran et al. Filename : draft-ietf-ips-iscsi-03.txt Pages : 94 Date : 11-Jan-01 The Small Computer Systems Interface (SCSI) is a popular family of protocols for communicating with I/O devices, especially storage devices. This memo describes a transport protocol for SCSI that operates on top of TCP. The iSCSI protocol aims to be fully compliant with the requirements laid out in the SCSI Architecture Model - 2 [SAM2] document A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-03.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-03.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-03.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. [message/External-body is not supported, skipping...] ----- End of forwarded message from Internet-Drafts@ietf.org ----- From owner-ips@ece.cmu.edu Sat Jan 13 00:15:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA01932 for ; Sat, 13 Jan 2001 00:15:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0D45nN19217 for ips-outgoing; Fri, 12 Jan 2001 23:05:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0D44nl19184 for ; Fri, 12 Jan 2001 23:04:49 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 396A3ABE; Fri, 12 Jan 2001 20:04:48 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id UAA08150; Fri, 12 Jan 2001 20:04:43 -0800 (PST) Message-ID: <3A5FD446.95E6714@cup.hp.com> Date: Fri, 12 Jan 2001 20:06:31 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: iSCSI : Abort Task Response violates SAM-2. References: Content-Type: multipart/mixed; boundary="------------AF897AD4AF70135B0A55F16B" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------AF897AD4AF70135B0A55F16B Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, SAM-2 Rev 15 Section 6.1 states that : "a response of FUNCTION COMPLETE shall indicate task was aborted or task was not in task set". The iSCSI draft is violating SAM-2 by returning "task not found" when the task was not in task set. Thanks, Santosh julian_satran@il.ibm.com wrote: > Well then Task does not exist will get all enough differentiation. > If you could not agree with reject (not enough differentiation) another use > can > have difficulty in accepting the other extreme - completed when a task is > really not found. > > Julo > > Pierre Labat on 10/01/2001 01:00:01 > > Please respond to Pierre Labat > > To: ips@ece.cmu.edu > cc: > Subject: iSCSI : Further review feedback on latest iSCSI-02.txt draft. > > Julian > > >>>>>>>>> > Section 2.6. SCSI Task Management Response > ================================== > The referenced Task Tag field can be removed since the > "No Task Found" response error is not valid. > (A Target should respond with an accept for an Abort Task > request specifying an invalid Initator Task Tag). > [js]Different people have different opinions on this - even with HP - > talk > to Pierre! [/js] > <<<<<<<<< > > We are in agreement, it is just a point of terminology. > > Some month ago, we had a discussion on the IPS about what to do in > case the target doesn't find the task to abort. > I requested that the Abort response returns "OK" > (something different than Function rejected). > You added the value "Task not found". > But "Function complete" can works too. > > Regards, > > Pierre --------------AF897AD4AF70135B0A55F16B Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------AF897AD4AF70135B0A55F16B-- From owner-ips@ece.cmu.edu Sat Jan 13 00:16:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA01948 for ; Sat, 13 Jan 2001 00:16:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0D4Dm919345 for ips-outgoing; Fri, 12 Jan 2001 23:13:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0D4DHl19329 for ; Fri, 12 Jan 2001 23:13:17 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 67F8FEA9; Fri, 12 Jan 2001 20:13:16 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id UAA08541; Fri, 12 Jan 2001 20:12:32 -0800 (PST) Message-ID: <3A5FD61B.B19F06CD@cup.hp.com> Date: Fri, 12 Jan 2001 20:14:20 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. References: Content-Type: multipart/mixed; boundary="------------0A3AC02CC244D7ECC77C7834" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------0A3AC02CC244D7ECC77C7834 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, I had 2 concerns on the format error handling issue : 1) This error recovery is applicable for the SCSI Command PDU. What is the error recovery for a format error on a Login, Logout, Text, NOP-OUT, NOP-IN, SCSI Task Management Command PDU ? These commands are either : not from the scsi layer and their responses do not get sent up the stack to the scsi layer. or in the case of scsi task mgmt cmds, the response does not contain sense data. How are the above to be handled ? 2) On the concept of initiators generating sense data on behalf of targets and faking a HARDWARE ERROR, this is not the approach parallel scsi and fibre channel take. In the case of fibre channel, a header format error results in Response Info being generated in FCP_RSP. (for scsi pdu's). However, I noticed that the latest version of the iSCSI draft has removed all references to response length and response data, and hence, it looks like this approach is not being considered (?) Generating sense data in an initiator is adding complexity and makes iSCSI HBA drivers SPC-2 aware in terms of the content and generation of sense data. Also, we will need to check some implementations of the upper scsi layer drivers to see if this can cause any different form of error recovery to be taken such as issuing a BDR to recover from HARDWARE ERRORs ? Regards, Santosh > Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. > Date: Fri, 12 Jan 2001 09:41:36 +0200 > From: julian_satran@il.ibm.com > To: ips@ece.cmu.edu > Santosh, > A machine producing a format error is DEFECTIVE and this a reported as > hardware error by SCSI (as SCSI does not distinguish between hardware and > firmware and rightfully so). --------------0A3AC02CC244D7ECC77C7834 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------0A3AC02CC244D7ECC77C7834-- From owner-ips@ece.cmu.edu Sat Jan 13 01:53:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA08059 for ; Sat, 13 Jan 2001 01:53:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0D4bm619781 for ips-outgoing; Fri, 12 Jan 2001 23:37:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0D4b8l19768 for ; Fri, 12 Jan 2001 23:37:08 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id XAA40206; Fri, 12 Jan 2001 23:30:22 -0500 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id VAA130936; Fri, 12 Jan 2001 21:37:07 -0700 Importance: Normal Subject: Re: iSCSI : Abort Task Response violates SAM-2. To: Santosh Rao Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.5 September 22, 2000 Message-ID: From: "Prasenjit Sarkar" Date: Fri, 12 Jan 2001 20:36:36 -0800 X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.5 |September 22, 2000) at 01/12/2001 08:37:07 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk The only constraint imposed is to ensure that a status for an aborted task should not follow a task management response for that task, so a response of FUNCTION COMPLETE for a task that is not found is perfectly valid. Sent by: owner-ips@ece.cmu.edu To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: iSCSI : Abort Task Response violates SAM-2. Julian, SAM-2 Rev 15 Section 6.1 states that : "a response of FUNCTION COMPLETE shall indicate task was aborted or task was not in task set". The iSCSI draft is violating SAM-2 by returning "task not found" when the task was not in task set. Thanks, Santosh julian_satran@il.ibm.com wrote: > Well then Task does not exist will get all enough differentiation. > If you could not agree with reject (not enough differentiation) another use > can > have difficulty in accepting the other extreme - completed when a task is > really not found. > > Julo > > Pierre Labat on 10/01/2001 01:00:01 > > Please respond to Pierre Labat > > To: ips@ece.cmu.edu > cc: > Subject: iSCSI : Further review feedback on latest iSCSI-02.txt draft. > > Julian > > >>>>>>>>> > Section 2.6. SCSI Task Management Response > ================================== > The referenced Task Tag field can be removed since the > "No Task Found" response error is not valid. > (A Target should respond with an accept for an Abort Task > request specifying an invalid Initator Task Tag). > [js]Different people have different opinions on this - even with HP - > talk > to Pierre! [/js] > <<<<<<<<< > > We are in agreement, it is just a point of terminology. > > Some month ago, we had a discussion on the IPS about what to do in > case the target doesn't find the task to abort. > I requested that the Abort response returns "OK" > (something different than Function rejected). > You added the value "Task not found". > But "Function complete" can works too. > > Regards, > > Pierre From owner-ips@ece.cmu.edu Sat Jan 13 12:28:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA18970 for ; Sat, 13 Jan 2001 12:28:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0DGE2d09816 for ips-outgoing; Sat, 13 Jan 2001 11:14:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0DGDVs09806 for ; Sat, 13 Jan 2001 11:13:31 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA92866 for ; Sat, 13 Jan 2001 17:13:24 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA240044 for ; Sat, 13 Jan 2001 17:13:25 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D3.00591D07 ; Sat, 13 Jan 2001 17:13:21 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 13 Jan 2001 18:09:04 +0200 Subject: Re: iSCSI: F bits in Command and DataIn PDUs, too? Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Stephen, I've corrected the statement and I was already considering moving the X bit to the command and have the P/F be always a Poll/Final bit (or 0)(not exactly what you wanted but close enough). I have however to check all the side effects... Thanks, Julo "Ferrari, Stephen" on 13/01/2001 02:27:30 Please respond to "Ferrari, Stephen" To: ips@ece.cmu.edu cc: Subject: iSCSI: F bits in Command and DataIn PDUs, too? According to the spec, the F bit in DataOut PDUs indicates "the last PDU of immediate data or the last PDU of a sequence answering a R2T". I believe this is a typo and should be "the last PDU of _unsolicited_ data or the last PDU of a sequence answering an R2T". For consistency, it would seem that Command PDUs should also have an F bit defined to indicate the last PDU of unsolicited data if all of the unsolicited data is sent as immediate data, and DataIn PDUs should have an F bit to indicate the last PDU of data for the command. Granted, these two cases are easier to detect than the case already covered, since there can be at most one PDU of immediate data per command (so there's no overlap possible), and the Response PDU implies that there's no more DataIn coming. But having a single bit, preferably in the same position in all three PDU types, to indicate that "this data transfer is done" would simplify data transfer management. (Note that for DataOut and bi-directional operations there can be multiple data transfers per command.) From owner-ips@ece.cmu.edu Sat Jan 13 12:31:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA19002 for ; Sat, 13 Jan 2001 12:31:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0DFPxv09007 for ips-outgoing; Sat, 13 Jan 2001 10:25:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0DFOxs08973 for ; Sat, 13 Jan 2001 10:24:59 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA165944 for ; Sat, 13 Jan 2001 16:24:53 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA208256 for ; Sat, 13 Jan 2001 16:24:53 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D3.0054AC37 ; Sat, 13 Jan 2001 16:24:51 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 13 Jan 2001 17:20:32 +0200 Subject: RE: iFCP as an IP Storage Work Item Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk There is no NT/2000 FCP support. Drivers are build on the "old" SCSI port driver. Very little if any FC is visible (even addressing is twisted). Julo Charles Monia on 13/01/2001 00:35:52 Please respond to Charles Monia To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Hi Julo: > iFCP as way to keep your investment in FCP stacks is a very > weak argument. > FCP stacks are not that stable neither that prevalent (there > is none in the > most widespread OS family - Windows). > I don't understand where you're coming from. Granted, you won't find many FC adapters on notebooks and desktops. However every enterprise-class operating system, including Windows NT/2000, provides robust Fibre Channel support. What's more HBAs, host driver stacks and middleware for management and failover have been deployed and in-service in mission-critical applications for some time. End users have an enormous investment in qualifying, deploying and supporting these systems. Anyone who's ever had to deal with such users in major accounts will appreciate what's at stake. From that perspective, I believe there's value in an approach to IP migration that protects that investment. Charles > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Thursday, January 11, 2001 9:23 PM > To: ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > > > Josh, > > iFCP as way to keep your investment in FCP stacks is a very > weak argument. > FCP stacks are not that stable neither that prevalent (there > is none in the > most widespread OS family - Windows). > > A gateway for a single device should be the exception rather > than the rule. > > I can support it as a work item ONLY if it plays a real > gateway role and > can coexist with FCIP is some synergistic fashion. > As a end-to-end proposal is has little value IMHO. > > Julo > > Joshua Tseng on 09/01/2001 07:39:18 > > Please respond to Joshua Tseng > > To: Venkat Rangan > cc: "Ips@Ece. Cmu. Edu" > Subject: RE: iFCP as an IP Storage Work Item > > > > > Venkat, > > > > Josh, > > > > Thanks for the clarification that iFCP is only presented as > a gateway > > protocol. The one comment we would make is that we have FC to > > SCSI gateways > > already in place, without the need for any standards body > > standardizing a > > new protocol. The function of the gateway is defined by the > > standards for > > the two protocols being "connected", and gateway details are left as > > implementation details. > > Actually, what I meant by gateway protocol is that it is a protocol > spoken by gateways. It doesn't specify anything about how to > implement the protocol, which is internal to the gateway device as you > point out. iFCP is the protocol that shows up on the IP network when > at least two iFCP gateways transport storage data between them. > > > > > On another note, it should be feasible to build a gateway > > that receives FCP > > frames from an N_Port or NL_Port of a SCSI Initiator and map > > the FCP frames > > into iSCSI frames. The frames are sent on an IP interface and > > routed by a > > normal IP network and another gateway reconverts the iSCSI > > PDUs back to FCP > > frames and sends them to the target. You will notice that > > this does not > > require any routing in the FC Plane and accomplishes the same > > end goals as > > iFCP. Also, this does not require any further standards work, > > besides the > > usual FCP, iSCSI and related naming protocols. This also > > provides the same > > scalability of number of nodes on the network, because the > > conversion from > > locally significant S_ID and D_ID to iSCSI IP addresses can > > be done, with > > help from a standardized naming effort such as iSNS. > > Yes, I agree. But this is a different gateway which has nothing > to do with an iFCP gateway. Definitely these types of gateways > will be needed as we transition to iSCSI. The new iSNS draft has > a diagram showing both iSCSI-FC gateways and iFCP gateways. They > have different roles. > > > > > Based on these, we believe the need for IP Storage working > > group to pick up > > iFCP as a work item is reduced. > > hmmm....you lost me here. An iFCP gateway has nothing to do with > iSCSI. They are completely separate. > > In order to use iSCSI, you need one of the following: > > 1) Two iSCSI devices > 2) One iSCSI device, one FC device, and one iSCSI-FC gateway > (which you described above) > > The situation of two FC devices and two iSCSI-FC gateways is clearly > not a design objective of iSCSI. Of course it can be done, but we > believe iFCP is clearly the best solution here. > > Fibre Channel has its issues, but one thing is certain: the > FCP driver > stacks are very stable and can provide excellent performance > for storage > applications. But the drawback is FC's networking capabilities leave > much to be desired. On the other hand, IP networking capabilities are > quite stable and work exceedingly well, but the iSCSI driver > stack does > not exist yet. > > So, iFCP's objective is to marry the best of both worlds--to take the > existing stable, high-performance driver stacks of Fibre Channel and > directly internetwork their individual N_PORTs using TCP/IP. > > Therefore, iFCP is an opportunity to leverage the existing and proven > Fibre Channel driver stacks and combine them with the > capabilities that > IP networking can provide. Until the day we have stable iSCSI driver > stacks everywhere, this may prove to be an attractive alternative for > many end users. Another comparison I liken it to is the need for > NAT until we have IPv6 everywhere. > > Josh > > > > > Regards, > > > > Venkat Rangan > > Rhapsody Networks Inc. > > http://www.rhapsodynetworks.com > > > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Joshua Tseng > > Sent: Thursday, January 04, 2001 10:54 AM > > To: Ips@Ece. Cmu. Edu > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > I don't want to stifle any creative technical discussion here, > > but I feel the need to remind everybody that iFCP is positioned > > as a gateway technology only. While the thought of "native" > > iFCP HBA's might be interesting, this discussion is > > completely irrelevant with regard to whether iFCP should > > or should not become an IPS work item. iFCP is being proposed > > as an IPS work item purely on its merits as a gateway technology. > > > > Regards, > > Josh > > > > > -----Original Message----- > > > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > > > Sent: Thursday, January 04, 2001 5:47 AM > > > To: 'ips@ece.cmu.edu' > > > Subject: FW: iFCP as an IP Storage Work Item > > > > > > > > > > > > > > > -----Original Message----- > > > From: Stephen Byan > > > Sent: Thursday, January 04, 2001 8:40 AM > > > To: 'Bill Terrell' > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > It's all the FC stuff that lets iFCP work over an unreliable > > > data transport > > > like UDP. It's redundant when running over TCP/IP. > > > > > > Regards, > > > -Steve > > > > > > > -----Original Message----- > > > > From: Bill Terrell [mailto:terrell@troikanetworks.com] > > > > Sent: Wednesday, January 03, 2001 6:10 PM > > > > To: 'Stephen Byan' > > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > > > > >The downside of this advantage is that native iFCP > > devices would be > > > > burdened > > > > >with greater complexity and cost. I therefor think iFCP > > > > should not be an IP > > > > >Storage work item. > > > > > > > > > >Regards, > > > > >-Steve > > > > > > > > How is a native iFCP endpoint (initiator or target) more > > > > complex or costly > > > > than an iSCSI native endpoint? What are the specific > > > > difficulties inherent > > > > to native iFCP devices versus native iSCSI devices? > > > > > > > > Bill > > > > > > > > > > > > From owner-ips@ece.cmu.edu Sat Jan 13 12:39:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA18969 for ; Sat, 13 Jan 2001 12:28:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0DFIxx08877 for ips-outgoing; Sat, 13 Jan 2001 10:18:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0DFIVs08870 for ; Sat, 13 Jan 2001 10:18:31 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA53348 for ; Sat, 13 Jan 2001 16:18:24 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA237106 for ; Sat, 13 Jan 2001 16:18:24 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D3.0054147C ; Sat, 13 Jan 2001 16:18:22 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 13 Jan 2001 17:11:39 +0200 Subject: Re: iSCSI : Concerns regarding DataRN in READ Data PDU. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Pierre, In a complex environment you have to leave to the initiator to decide when to ack. If data goes to another device (3rd party) the TCP ack is not enough. Data will be acked only after reaching the final place. Julo Pierre Labat on 12/01/2001 19:19:09 Please respond to Pierre Labat To: ips@ece.cmu.edu cc: Subject: Re: iSCSI : Concerns regarding DataRN in READ Data PDU. julian_satran@il.ibm.com wrote: > This feature was meant for long lasting operations (tons of data) that > could then recover > with ease. As for complexity... it is local counter. If that is not > available all long lasting commands will have to provide their own > checkpointing mechanisms or there will be no long-lasting commands. Julian, The target can use the TCP acks to know what data reached the initiator. From the TCP ack the target can find the corresponding data in the READ cmd that have been received by the initiator. And it can free the resources for this data and in case of connection failure it can restart from there. It implies that the initiator sends the TCP ack only after the data has been copied out of the adapter (in sever memory or alike). It's why the DATARN doesn't add value, it is because using TCP you can get the same benefit. Regards, Pierre > > > Julo > > "VOIGT,DOUG (HP-Boise,ex1)" on 11/01/2001 03:11:41 > > Please respond to "VOIGT,DOUG (HP-Boise,ex1)" > > To: IPS Reflector > cc: > Subject: RE: iSCSI : Concerns regarding DataRN in READ Data PDU. > > It seems to me that this feature is redundant with TCP's own delivery > guarantees, in an effort to minimize work loss in the event of TCP > connection failure. I too believe this level of complexity and potential > normal performance impact compared with accelerated recovery from > improbable > error events is not likely to yield a net gain. > > Is recoverable TCP connection loss more common than disk array controller > failure? Since different controllers in the same array are (quite > reasonably) viewed as different targets (pending SAM change), iSCSI level > failover won't compensate for controller failure. This example leads me to > suggest that the benefit we are shooting for has a high likelihood of being > overshadowed by other failure modes whose recovery is less graceful. > > Doug Voigt > > > -----Original Message----- > > From: Santosh Rao [mailto:santoshr@cup.hp.com] > > Sent: Monday, January 08, 2001 9:03 PM > > To: IPS Reflector > > Subject: iSCSI : Concerns regarding DataRN in READ Data PDU. > > > > > > Julian/All, > > > > Can anybody comment on the true benefit that the DataRN > > feature provides > > that justifies the complexity and performance issues it introduces to > > the protocol ? > > > > I see the following issues with DataRN : > > > > 1) It adds a performance penalty in that on every READ Data PDU > > targets will have to initialize one additional 32-bit word in the > > header. > > > > 2) Extra traffic on the outbound context from initiators sending > > NOP-OUTs to acknowledge the DataRNs. > > > > 3) Performance path checks for the "P" bit on every READ PDU. > > (Alternatively, checks for the DataNumber field from the Login > > Text Keys). > > > > 4) Implementing DataRNs per task is too short-lived to provide any > > real benefit. Implementing them per connection or session can result > > in quick use up of the 32-bit DataRN space (since each frame consumes > > one DataRN), adding additional complexity to the target code > > to use the > > "P" > > bit appropriately. > > > > 5) Further, "DataNumber", which is the size of the un-acknowledged > > DataRN window is negotiated at login time, whereas this is really > > desired to be a dynamic variable based on the target's > > current I/O load > > and the number of initiators speaking to it. Thus, DataRN can end up > > being > > under-utilized resulting in too much NOP-OUT traffic, or it can be > > over-utilized resulting in too much usage of the "P" bit in the READ > > PDUs, which, again, can cause too much NOP-OUT traffic. > > > > 6) Considering that the DataRN generation and acknowledgement are > > functions > > that would typically be in SCSI hardware assist engines and there is a > > need to > > keep these as simple as possible, adding such features in the > > spec means > > > > that these SCSI assist engines MUST implement DataRN generation > > and acknowledgement [since initiators have no way to turn it off if a > > target decides to use it] > > > > What is the true benefit of this feature ? Is it intended to > > be used as > > : > > > > a) Will allow targets to do partial freeing of their memory buffers as > > and when DataRNs are received ? > > > > b) Will allow targets to send back only the un-acknowledged data (as > > seen from the ExpDataRN received on the last NOP-OUT from the > > initiator) > > when the initiators retry commands with the "Retry" bit set ? > > > > If the benefit intended is (a), then, in order to derive this benefit, > > targets will have to advertise their DataNumber login key to be < the > > avg. number of frames transmitted per READ I/O. Having a DataNumber > > > the avg. number of frames per READ I/O implies the life of > > the task ends > > [and buffers for the task get released] before the initiator sends > > NOP-OUT, thereby, defeating benefit (a). > > > > If the benefit intended is (b) [and i do not see this > > explicitly stated > > in the draft], then, this is a dangerous assumption which can lead to > > data corruption issues. > > > > When commands are retried with the "Retry" bit set, are > > targets allowed > > to send back partial data based on the last ExpDataRN they > > received ? If > > so, this can have multiple issues like : > > > > o complexity in handling I/O underrun cases for the retried command > > which only saw partial data transfer from the target. > > > > o If the target only sent back partial data based > > on its last ExpDataRN received, then, it can open up > > possible data > > > > corruption windows. > > > > All in all, I believe that the DataRN is adding too much > > complexity and > > potential performance impact to iSCSI for the benefits it may provide. > > > > Moreover and most importantly, the draft does not give initiators any > > control over the usage of this feature and if a target decides to use > > this feature, initiators will have to support it. This exposes the > > initiators to all of the above issues without being able to turn off > > this feature. > > > > I would like to therefore ask that : > > > > either, > > a) DataRN support be removed from the iSCSI draft. > > > > or > > b) The support for DataRN be negotiated at login time giving initators > > control over this feature and allowing them to disable this feature. > > This will allow SCSI hardware assist engines to skip implementation of > > this functionality and their associated software can merely turn off > > DataNumber in the login negotiation. > > > > Thanks, > > Santosh > > > > > > > > From owner-ips@ece.cmu.edu Sat Jan 13 13:53:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA19397 for ; Sat, 13 Jan 2001 13:53:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0DGm3L10421 for ips-outgoing; Sat, 13 Jan 2001 11:48:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0DGlDs10401 for ; Sat, 13 Jan 2001 11:47:14 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA33584 for ; Sat, 13 Jan 2001 17:47:07 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA81506 for ; Sat, 13 Jan 2001 17:45:51 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D3.005C1395 ; Sat, 13 Jan 2001 17:45:43 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 13 Jan 2001 18:26:07 +0200 Subject: Re: iSCSI : Abort Task Response violates SAM-2. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, I would certainly not have used "violates the SAM-2" when what you probably meant that is that wording should be in line with SAM-2. I will change the wording to read as in SAM-2 although in my English they have exactly the same meaning. Julo Santosh Rao on 13/01/2001 06:06:31 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: iSCSI : Abort Task Response violates SAM-2. Julian, SAM-2 Rev 15 Section 6.1 states that : "a response of FUNCTION COMPLETE shall indicate task was aborted or task was not in task set". The iSCSI draft is violating SAM-2 by returning "task not found" when the task was not in task set. Thanks, Santosh julian_satran@il.ibm.com wrote: > Well then Task does not exist will get all enough differentiation. > If you could not agree with reject (not enough differentiation) another use > can > have difficulty in accepting the other extreme - completed when a task is > really not found. > > Julo > > Pierre Labat on 10/01/2001 01:00:01 > > Please respond to Pierre Labat > > To: ips@ece.cmu.edu > cc: > Subject: iSCSI : Further review feedback on latest iSCSI-02.txt draft. > > Julian > > >>>>>>>>> > Section 2.6. SCSI Task Management Response > ================================== > The referenced Task Tag field can be removed since the > "No Task Found" response error is not valid. > (A Target should respond with an accept for an Abort Task > request specifying an invalid Initator Task Tag). > [js]Different people have different opinions on this - even with HP - > talk > to Pierre! [/js] > <<<<<<<<< > > We are in agreement, it is just a point of terminology. > > Some month ago, we had a discussion on the IPS about what to do in > case the target doesn't find the task to abort. > I requested that the Abort response returns "OK" > (something different than Function rejected). > You added the value "Task not found". > But "Function complete" can works too. > > Regards, > > Pierre - santoshr.vcf From owner-ips@ece.cmu.edu Sat Jan 13 17:27:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA20505 for ; Sat, 13 Jan 2001 17:27:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0DKGB614439 for ips-outgoing; Sat, 13 Jan 2001 15:16:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0DKFYs14433 for ; Sat, 13 Jan 2001 15:15:34 -0500 (EST) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id f0DKFU014231; Sat, 13 Jan 2001 14:15:31 -0600 (CST) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA27978; Sat, 13 Jan 01 14:13:57 -0600 Message-Id: <10101132013.AA27978@candide.cs.uchicago.edu> To: ips@ece.cmu.edu, vfiroiu@nortelnetworks.com Subject: Re: Performance of iSCSI, FCIP and iFCP In-Reply-To: Message from julian_satran@il.ibm.com of "Fri, 12 Jan 2001 09:30:10 +0200." References: Date: Sat, 13 Jan 2001 14:15:13 -0600 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk Victor, > I have to admit that we did not have the model to support > numerically our contention that multiple links will behave better in > the presence of errors and performance will degrade more gracefully > even on congestion. The last time we had this discussion (multiple links between the same endpoints enabling higher performance) on this list, it was my opinion that doing this is not `TCP friendly', and so should not be allowed in the general, Internet context, which is one of the design points for iSCSI. Clearly, when slow starting, and recovering from congestion, n connections can effectively open the window n times as fast as 1 connection, but doing so will potentially be at the expense of other competing flows which don't use multiple connections. If ALL flows use multiple connections, you simply end up hitting the congestion wall harder, which, in an abstract sense, makes the system more likely to oscillate (or collapse?). I don't know if it WILL create oscillation or collapse (I'm just a hammer mechanic), but it seems to me that if hitting the congestion wall harder in this way were acceptable to overall network behavior, the single connection TCP congestion avoidance could be adjusted to create this behavior (and capture this benefit) without requiring multiple TCP connections. Your refereed, published work on this seems suggests that I am holding on to `myths' about the need to play fair with TCP. If so, I would greatly appreciate your debunking my myths here on this list, in the clearest way. Particularly, we need to know if the IETF (or IESG or IAB, or whomever it is....), will accept this type of behavior out of iSCSI. While I am opposed to iSCSI's multiconnection session, I do admit its benefits. My objections are that the feature will not be widely used, is difficult to implement correctly, AND the same benefits are already available through long-standing upper layer mechanisms. That aside, I accept the WG consensus on multiconnection sessions, but what the iSCSI spec still needs to do, in no uncertain terms, is indicate whether multiple connections per endpoint pair is something which implementations SHOULD or SHOULD NOT allow for use in a general, Internet context, or at all. Even if iSCSI specifies that multiple connections per endpoint pair SHOULD NOT be used in a general context, we know that implementations are not going to prohibit it (it's primarily a configuration decision), and that still allows the feature to be used in specifically engineered applications, such as those built with dedicated storage fabrics. Thanks, Steph From owner-ips@ece.cmu.edu Sat Jan 13 19:11:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA21093 for ; Sat, 13 Jan 2001 19:11:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0DM8GU16578 for ips-outgoing; Sat, 13 Jan 2001 17:08:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0DM7Ys16565 for ; Sat, 13 Jan 2001 17:07:35 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id E206AB07; Sat, 13 Jan 2001 14:07:33 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id OAA08973; Sat, 13 Jan 2001 14:07:06 -0800 (PST) From: Santosh Rao Message-Id: <200101132207.OAA08973@hpcuhe.cup.hp.com> Subject: Re: iSCSI : Abort Task Response violates SAM-2. To: julian_satran@il.ibm.com Date: Sat, 13 Jan 2001 14:07:06 -0800 (PST) Cc: ips@ece.cmu.edu In-Reply-To: from "julian_satran@il.ibm.com" at Jan 13, 2001 06:26:07 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Thanks for the clarification. The "Referenced Task Tag" field in the Abort Task response PDU also could be removed since it was provided to accompany the "No Task Found" response indicating the task tag. Thanks & Regards, Santosh > > > > Santosh, > > I would certainly not have used "violates the SAM-2" when what you probably > meant that is that wording should be in line with SAM-2. I will change the > wording to read as in SAM-2 although in my English they have exactly the > same meaning. > > Julo > > Santosh Rao on 13/01/2001 06:06:31 > > Please respond to Santosh Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: iSCSI : Abort Task Response violates SAM-2. > > > > > Julian, > > SAM-2 Rev 15 Section 6.1 states that : > > "a response of FUNCTION COMPLETE shall indicate task was aborted or task > was > not in task set". > > The iSCSI draft is violating SAM-2 by returning "task not found" when the > task > was not in task set. > > Thanks, > Santosh > > julian_satran@il.ibm.com wrote: > > > Well then Task does not exist will get all enough differentiation. > > If you could not agree with reject (not enough differentiation) another > use > > can > > have difficulty in accepting the other extreme - completed when a task is > > really not found. > > > > Julo > > > > Pierre Labat on 10/01/2001 01:00:01 > > > > Please respond to Pierre Labat > > > > To: ips@ece.cmu.edu > > cc: > > Subject: iSCSI : Further review feedback on latest iSCSI-02.txt draft. > > > > Julian > > > > >>>>>>>>> > > Section 2.6. SCSI Task Management Response > > ================================== > > The referenced Task Tag field can be removed since the > > "No Task Found" response error is not valid. > > (A Target should respond with an accept for an Abort Task > > request specifying an invalid Initator Task Tag). > > [js]Different people have different opinions on this - even with HP - > > talk > > to Pierre! [/js] > > <<<<<<<<< > > > > We are in agreement, it is just a point of terminology. > > > > Some month ago, we had a discussion on the IPS about what to do in > > case the target doesn't find the task to abort. > > I requested that the Abort response returns "OK" > > (something different than Function rejected). > > You added the value "Task not found". > > But "Function complete" can works too. > > > > Regards, > > > > Pierre > > - santoshr.vcf > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Sat Jan 13 19:13:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA21104 for ; Sat, 13 Jan 2001 19:13:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0DMHGD16750 for ips-outgoing; Sat, 13 Jan 2001 17:17:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0DMGes16745 for ; Sat, 13 Jan 2001 17:16:40 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id F4193D22; Sat, 13 Jan 2001 14:16:38 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id OAA09443; Sat, 13 Jan 2001 14:16:33 -0800 (PST) From: Santosh Rao Message-Id: <200101132216.OAA09443@hpcuhe.cup.hp.com> Subject: Re: iSCSI : Concerns regarding DataRN in READ Data PDU. To: julian_satran@il.ibm.com Date: Sat, 13 Jan 2001 14:16:33 -0800 (PST) Cc: ips@ece.cmu.edu In-Reply-To: from "julian_satran@il.ibm.com" at Jan 13, 2001 05:11:39 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, My original request on this issue was suggesting 2 alternatives. If a WG consensus cannot be achieved on eliminating the DataRN and all of its complexity, I would like to re-iterate the alternative proposal that was made, that initiators should be allowed to disable this feature during login negotiation. Regards, Santosh > > > -----Original Message----- > > > From: Santosh Rao [mailto:santoshr@cup.hp.com] > > > Sent: Monday, January 08, 2001 9:03 PM > > > To: IPS Reflector > > > Subject: iSCSI : Concerns regarding DataRN in READ Data PDU. > > > > > > I would like to therefore ask that : > > > > > > either, > > > a) DataRN support be removed from the iSCSI draft. > > > > > > or > > > b) The support for DataRN be negotiated at login time giving initators > > > control over this feature and allowing them to disable this feature. > > > This will allow SCSI hardware assist engines to skip implementation of > > > this functionality and their associated software can merely turn off > > > DataNumber in the login negotiation. > > > > > > Thanks, > > > Santosh > > > > > > > > > > > > > > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Sat Jan 13 19:15:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA21115 for ; Sat, 13 Jan 2001 19:15:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0DMXLg17064 for ips-outgoing; Sat, 13 Jan 2001 17:33:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0DMXJs17058 for ; Sat, 13 Jan 2001 17:33:19 -0500 (EST) Received: from yp_portable (slip-32-102-64-195.ca.us.prserv.net [32.102.64.195]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id C6R66Q5B; Sat, 13 Jan 2001 14:33:08 -0800 From: "Y P Cheng" To: Subject: RE: iFCP as an IP Storage Work Item Date: Sat, 13 Jan 2001 14:32:09 -0800 Message-ID: <000001c07db0$ab3d20c0$c3406620@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > As I see it, the software issue is not only related to support for the > low-level hardware and SCSI encapsulation. It's in the effort needed to > transpose all the SAN value-add (naming, discovery, zoning, error > recovery, management, etc) to a new network environment. > ......... ............. > So, rather than say that the iFCP preserves the existing driver stacks, we > should go further and extend the scope to the totality of > software needed to support the storage network environment. > > Charles Yes, we are now talking the right area. However, I don't see how iFCP will help preserve the software and hardware investment of SAN environment made by customers. Here is my understanding of the investment made by customers in SAN environment. (Experts of SAN can certainly correct me.) 1. Software from Oracle, Veritas, and Logato running in block address instead of file system address. I am pretty sure none of those software is specifically programmed for FCP devices. In fact, they treat everything like "hard disks or tapes" whether they are ATA, SCSI, FCP, or RAID. Shark and Symmetrix mentioned by you appear to the application software as reliable hard disk devices. 2. SAN management and configuration software. These are typically provided by storage device manufacturers. The software talks to the storage devices through a separate Ethernet connection. Most storage boxes use SNMP to send and get Management Information Blocks (MIBs). Again, iFCP is not particularly helpful in saving the investment in this area. 3. Hardware investment like routers and switches. The Name Server inside the switches provides target device discovery function. With Java support, the switches provide customers the comfort at home using a browser to log into the switches to check connectivity and do zone and domain management. In such case, FCIP is in a better candidate to preserve the existing investment. 4. Finally, the HBA's inside the servers and storage devices. I can assure you, every HBA manufacturer will have device driver to do FCP, iSCSI, iFCP, or FCIP stack processing, whatever protocol chosen by the customers. The protocol stack processing are done inside the IC by microcode, not the device driver. Even MicroSoft Winsock Direct and TCP Remote DMA will be implemented in the microcode of the IC to offload the host protocol stack processing. No software investment is made by customers. (The drivers are free with the HBA.) In all, if I may summarize, the strongest argument iFCP has is the OSPF routing and scalability. But, FCIP gets OSPF routing in the IP plane too. For the scalability, storage-device attachment to servers is somewhat static and confined, unlike the HTTP which requires huge amount of connectivity and IPv6. No Internet or dot com companies would need 16 million block-device addresses allowed by the 24-bit D_ID of FCIP at one time. :-) Y.P. Cheng, ConnectCom Solutions. From owner-ips@ece.cmu.edu Sat Jan 13 19:15:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA21126 for ; Sat, 13 Jan 2001 19:15:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0DN4NY17663 for ips-outgoing; Sat, 13 Jan 2001 18:04:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from exchange-1.xiotech.com (ftp.xiotech.com [209.46.118.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0DN4Gs17654 for ; Sat, 13 Jan 2001 18:04:16 -0500 (EST) Message-ID: From: "Peglar, Robert" To: "'julian_satran@il.ibm.com'" , ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Sat, 13 Jan 2001 17:03:26 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Some HBA vendors offer a full-blown NT port driver now, instead of the (prior) NT mini-port which indeed interfaced with the standard NT SCSIPORT.SYS driver. Viz. Qlogic with their QLDirect driver. It bypasses SCSIPORT.SYS completely. Rob > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Saturday, January 13, 2001 9:21 AM > To: ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > > > There is no NT/2000 FCP support. Drivers are build on the > "old" SCSI port > driver. Very little if any FC is visible (even addressing is twisted). > > Julo > > Charles Monia on 13/01/2001 00:35:52 > > Please respond to Charles Monia > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: RE: iFCP as an IP Storage Work Item > > > > > Hi Julo: > > > iFCP as way to keep your investment in FCP stacks is a very > > weak argument. > > FCP stacks are not that stable neither that prevalent (there > > is none in the > > most widespread OS family - Windows). > > > > I don't understand where you're coming from. > > Granted, you won't find many FC adapters on notebooks and desktops. > However > every enterprise-class operating system, including Windows NT/2000, > provides > robust Fibre Channel support. What's more HBAs, host driver > stacks and > middleware for management and failover have been deployed and > in-service in > mission-critical applications for some time. > > End users have an enormous investment in qualifying, deploying and > supporting these systems. Anyone who's ever had to deal with > such users in > major accounts will appreciate what's at stake. From that > perspective, I > believe there's value in an approach to IP migration that > protects that > investment. > > Charles > > -----Original Message----- > > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > > Sent: Thursday, January 11, 2001 9:23 PM > > To: ips@ece.cmu.edu > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > > Josh, > > > > iFCP as way to keep your investment in FCP stacks is a very > > weak argument. > > FCP stacks are not that stable neither that prevalent (there > > is none in the > > most widespread OS family - Windows). > > > > A gateway for a single device should be the exception rather > > than the rule. > > > > I can support it as a work item ONLY if it plays a real > > gateway role and > > can coexist with FCIP is some synergistic fashion. > > As a end-to-end proposal is has little value IMHO. > > > > Julo > > > > Joshua Tseng on 09/01/2001 07:39:18 > > > > Please respond to Joshua Tseng > > > > To: Venkat Rangan > > cc: "Ips@Ece. Cmu. Edu" > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > > Venkat, > > > > > > Josh, > > > > > > Thanks for the clarification that iFCP is only presented as > > a gateway > > > protocol. The one comment we would make is that we have FC to > > > SCSI gateways > > > already in place, without the need for any standards body > > > standardizing a > > > new protocol. The function of the gateway is defined by the > > > standards for > > > the two protocols being "connected", and gateway details > are left as > > > implementation details. > > > > Actually, what I meant by gateway protocol is that it is a protocol > > spoken by gateways. It doesn't specify anything about how to > > implement the protocol, which is internal to the gateway > device as you > > point out. iFCP is the protocol that shows up on the IP > network when > > at least two iFCP gateways transport storage data between them. > > > > > > > > On another note, it should be feasible to build a gateway > > > that receives FCP > > > frames from an N_Port or NL_Port of a SCSI Initiator and map > > > the FCP frames > > > into iSCSI frames. The frames are sent on an IP interface and > > > routed by a > > > normal IP network and another gateway reconverts the iSCSI > > > PDUs back to FCP > > > frames and sends them to the target. You will notice that > > > this does not > > > require any routing in the FC Plane and accomplishes the same > > > end goals as > > > iFCP. Also, this does not require any further standards work, > > > besides the > > > usual FCP, iSCSI and related naming protocols. This also > > > provides the same > > > scalability of number of nodes on the network, because the > > > conversion from > > > locally significant S_ID and D_ID to iSCSI IP addresses can > > > be done, with > > > help from a standardized naming effort such as iSNS. > > > > Yes, I agree. But this is a different gateway which has nothing > > to do with an iFCP gateway. Definitely these types of gateways > > will be needed as we transition to iSCSI. The new iSNS draft has > > a diagram showing both iSCSI-FC gateways and iFCP gateways. They > > have different roles. > > > > > > > > Based on these, we believe the need for IP Storage working > > > group to pick up > > > iFCP as a work item is reduced. > > > > hmmm....you lost me here. An iFCP gateway has nothing to do with > > iSCSI. They are completely separate. > > > > In order to use iSCSI, you need one of the following: > > > > 1) Two iSCSI devices > > 2) One iSCSI device, one FC device, and one iSCSI-FC gateway > > (which you described above) > > > > The situation of two FC devices and two iSCSI-FC gateways is clearly > > not a design objective of iSCSI. Of course it can be done, but we > > believe iFCP is clearly the best solution here. > > > > Fibre Channel has its issues, but one thing is certain: the > > FCP driver > > stacks are very stable and can provide excellent performance > > for storage > > applications. But the drawback is FC's networking > capabilities leave > > much to be desired. On the other hand, IP networking > capabilities are > > quite stable and work exceedingly well, but the iSCSI driver > > stack does > > not exist yet. > > > > So, iFCP's objective is to marry the best of both > worlds--to take the > > existing stable, high-performance driver stacks of Fibre Channel and > > directly internetwork their individual N_PORTs using TCP/IP. > > > > Therefore, iFCP is an opportunity to leverage the existing > and proven > > Fibre Channel driver stacks and combine them with the > > capabilities that > > IP networking can provide. Until the day we have stable > iSCSI driver > > stacks everywhere, this may prove to be an attractive > alternative for > > many end users. Another comparison I liken it to is the need for > > NAT until we have IPv6 everywhere. > > > > Josh > > > > > > > > Regards, > > > > > > Venkat Rangan > > > Rhapsody Networks Inc. > > > http://www.rhapsodynetworks.com > > > > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Joshua Tseng > > > Sent: Thursday, January 04, 2001 10:54 AM > > > To: Ips@Ece. Cmu. Edu > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > I don't want to stifle any creative technical discussion here, > > > but I feel the need to remind everybody that iFCP is positioned > > > as a gateway technology only. While the thought of "native" > > > iFCP HBA's might be interesting, this discussion is > > > completely irrelevant with regard to whether iFCP should > > > or should not become an IPS work item. iFCP is being proposed > > > as an IPS work item purely on its merits as a gateway technology. > > > > > > Regards, > > > Josh > > > > > > > -----Original Message----- > > > > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > > > > Sent: Thursday, January 04, 2001 5:47 AM > > > > To: 'ips@ece.cmu.edu' > > > > Subject: FW: iFCP as an IP Storage Work Item > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Stephen Byan > > > > Sent: Thursday, January 04, 2001 8:40 AM > > > > To: 'Bill Terrell' > > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > > > > It's all the FC stuff that lets iFCP work over an unreliable > > > > data transport > > > > like UDP. It's redundant when running over TCP/IP. > > > > > > > > Regards, > > > > -Steve > > > > > > > > > -----Original Message----- > > > > > From: Bill Terrell [mailto:terrell@troikanetworks.com] > > > > > Sent: Wednesday, January 03, 2001 6:10 PM > > > > > To: 'Stephen Byan' > > > > > Subject: RE: iFCP as an IP Storage Work Item > > > > > > > > > > > > > > > >The downside of this advantage is that native iFCP > > > devices would be > > > > > burdened > > > > > >with greater complexity and cost. I therefor think iFCP > > > > > should not be an IP > > > > > >Storage work item. > > > > > > > > > > > >Regards, > > > > > >-Steve > > > > > > > > > > How is a native iFCP endpoint (initiator or target) more > > > > > complex or costly > > > > > than an iSCSI native endpoint? What are the specific > > > > > difficulties inherent > > > > > to native iFCP devices versus native iSCSI devices? > > > > > > > > > > Bill > > > > > > > > > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Sun Jan 14 03:36:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA04448 for ; Sun, 14 Jan 2001 03:36:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0E69cT25345 for ips-outgoing; Sun, 14 Jan 2001 01:09:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0E69Vs25341 for ; Sun, 14 Jan 2001 01:09:31 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Sat, 13 Jan 2001 22:10:02 -0800 Message-ID: From: Joshua Tseng To: Y P Cheng Cc: ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Sat, 13 Jan 2001 22:09:00 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk YP, I think we are losing sight of the original topic. The objective is not to preserve ALL of the existing investment in the SAN environment. Indeed, if everything in Fibre Channel worked great and met everyone's needs, then there would be no need for the IPS WG, iSCSI, and iFCP. Rather, the FC storage transport has unresolved issues that can be addressed with IP, and so we are here to figure out how to transport block storage data over TCP/IP. IMO, iFCP is very consistent with the IPS WG's objectives in this regard. One day, iSCSI will solve all of our problems by integrating all components of block-level storage networking natively on to TCP/IP. But until that day, iFCP is an attractive solution that fixes the "squeaky wheel" (i.e., Fibre Channel networking) by replacing it with a working wheel (i.e., IP networking). And it does it in a way without the need to buy a brand-new car (i.e., iSCSI). And even after iSCSI becomes a reality, there will still be a huge installed base of FC devices that will benefit by utilizing iFCP. See below for responses to your comments: > Yes, we are now talking the right area. However, I don't see > how iFCP will > help preserve the software and hardware investment of SAN > environment made > by customers. Here is my understanding of the investment > made by customers > in SAN environment. (Experts of SAN can certainly correct me.) > > 1. Software from Oracle, Veritas, and Logato running in block address > instead of file system address. I am pretty sure none of > those software is > specifically programmed for FCP devices. In fact, they treat > everything > like "hard disks or tapes" whether they are ATA, SCSI, FCP, > or RAID. Shark > and Symmetrix mentioned by you appear to the application software as > reliable hard disk devices. > > 2. SAN management and configuration software. These are > typically provided > by storage device manufacturers. The software talks to the > storage devices > through a separate Ethernet connection. Most storage boxes > use SNMP to send > and get Management Information Blocks (MIBs). Again, iFCP is not > particularly helpful in saving the investment in this area. > > 3. Hardware investment like routers and switches. The Name > Server inside > the switches provides target device discovery function. With > Java support, > the switches provide customers the comfort at home using a > browser to log > into the switches to check connectivity and do zone and > domain management. > In such case, FCIP is in a better candidate to preserve the existing > investment. Preserving FC switches is not the objective of iFCP. Fibre Channel's limited networking capabilities is THE issue that needs to be resolved. > > 4. Finally, the HBA's inside the servers and storage devices. > I can assure > you, every HBA manufacturer will have device driver to do > FCP, iSCSI, iFCP, > or FCIP stack processing, whatever protocol chosen by the > customers. The > protocol stack processing are done inside the IC by microcode, not the > device driver. Even MicroSoft Winsock Direct and TCP Remote > DMA will be > implemented in the microcode of the IC to offload the host > protocol stack > processing. No software investment is made by customers. > (The drivers are > free with the HBA.) > > In all, if I may summarize, the strongest argument iFCP has > is the OSPF > routing and scalability. But, FCIP gets OSPF routing in the > IP plane too. iFCP and FCIP have nothing to do with OSPF. I think your statements show that you do not understand iFCP and its objectives. I would have hoped that those critical of iFCP would have at least a minimal understanding of it. > For the scalability, storage-device attachment to servers is > somewhat static > and confined, unlike the HTTP which requires huge amount of > connectivity and > IPv6. No Internet or dot com companies would need 16 million > block-device > addresses allowed by the 24-bit D_ID of FCIP at one time. :-) In case you missed Charles' e-mail, he already answered your address space issue. However, there are other ways in which iFCP facilitates the scalability of a storage network. For example, SSP's need to be able to manage storage traffic flow between many different independent autonomous network systems, as they will need to separate their own network from their many customers' networks, and their customers' networks from each other. IP has this capability through the combination of OSPF and BGP routing; Fibre Channel does not have the equivalent to BGP. Therefore, using a Fibre Channel fabric and FCIP alone, the SSP will need to set up a large number of small, independent, and non-interoperable SANs. IMO, this will be difficult to manage and scale. Furthermore, as Wayland pointed out earlier, a single large FC fabric stretched over multiple WAN links by FCIP is vulnerable to temporary disruptions in the IP network which may trigger reconfigurations in the FC fabric, such as reallocations of DOMAIN_ID's. iFCP addresses this issue by assigning N_PORT ID's locally, eliminating the dependence on a central addressing authority and the need for any "Class F" traffic. The stability and scalability of a large storage network is thus improved. Josh > > Y.P. Cheng, ConnectCom Solutions. > From owner-ips@ece.cmu.edu Sun Jan 14 05:04:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA07512 for ; Sun, 14 Jan 2001 05:04:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0E89kp27267 for ips-outgoing; Sun, 14 Jan 2001 03:09:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp4.mail.yahoo.com (smtp4.mail.yahoo.com [128.11.69.101]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f0E890s27249 for ; Sun, 14 Jan 2001 03:09:00 -0500 (EST) Received: from unknown (HELO SG2351) (206.112.104.207) by smtp.mail.vip.suc.yahoo.com with SMTP; 14 Jan 2001 08:08:58 -0000 X-Apparently-From: Reply-To: From: "Somesh Gupta" To: "Victor Firoiu" , "IPS Reflector" Cc: "Franco Travostino" Subject: RE: Performance of iSCSI, FCIP and iFCP Date: Sun, 14 Jan 2001 00:06:10 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: <3A5BB1BF.27CEED03@nortelnetworks.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Franco, Thanks for getting us to exercise the under-utilized math part of our brains :-) I wanted to echo the comments that Stephen made. What I think you are saying is that If a single packet drop occurs no more frequently than the time to recover (the rate of the connection which existed before the error occured), then a quarter of the bandwidth during the recovery period is lost. In the simplistic case where an error occurs with just the correct frequency, you loose a quarter of the available bandwidth. If you have N connections sharing the same bandwidth, then the time to recover is reduced to 1/N and since the rate on each connection is (1/N), the lost rate is proportional to (1/N)^2. A minor quibble with the math. I assume (considering this low rate) that the error is likely due to transmission errors which is proportional to the number of bytes sent, the (1/n)^2 is not quite true as the number of bytes sent will be larger so errors will occor somewhat more frequently as a function of time. The really interesting equation to me is to look at the lost oportunity in isolation of the number of connections (i.e. you can scale the numbers any which way) D = C/8*I/4 = C^2*RTT^2/(256*M) As long as RTTs are small, and/or data rate (C) is not too large, we don't loose too large a percentage of the available bandwidth. However, if both become large which is what we face, and keep becoming larger, then we keep getting closer and closer to the 1/4 loss rate, unless we keep increasing the number of connections to compensate. In a perfect world, you would be able to convince the IETF to use the product of rate and RTT as a factor to adjust the A and the M part in AIMD algorithm to achieve the same result. It seems a little bit of waste of energy to get around the math by using multiple connections (BTW - are there any published papers on experiences with multiple connections for a single data stream in a high performance environment - would love to learn from someone else's mistakes). Somesh -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Victor Firoiu Sent: Tuesday, January 09, 2001 4:50 PM To: IPS Reflector Cc: Franco Travostino Subject: Performance of iSCSI, FCIP and iFCP Hi, we would like to add to the recent discussion on IP storage protocols a performance point of view showing significant differences between the protocols discussed. Any comment would be highly appreciated. Briefly, transporting a set of storage connections between two storage enclaves using multiple TCP sessions (iSCSI and iFCP) provides significantly higher aggregate average throughput than transporting the same set of storage connections over a single TCP session (FCIP), the difference being proportional to the square of the number of TCP sessions. This is due to the way TCP congestion control reacts to packet losses. Here we have two questions: why are we talking about packet losses and why is this difference in throughput. There are several reasons for packet losses: network congestion, link errors and network errors. Network congestion is pervasive in current IP networks, where the only way to control congestion is through dropping packets. Traffic engineering, admission control and bandwidth reservation are currently in early stages of definition. DiffServ supporting QoS infrastructure will not be widely available in the near future (especially when we realize that it is not a simple matter of asserting the EF PHB bit as if it were the old IP ToS; it rather needs network services and supporting SLA negotiation). The DiffServ EF PHB RFC is currently under redefinition. On the other hand, if such supporting QoS infrastructure were indeed available and pervasive, today, why would we need TCP to begin with? Even in a perfectly engineered network, link errors occur. If we take the Fibre Channel objective of 10^-12 Bit Error Rate, for a 10Gb/s link, this is one error every 100 seconds. Network errors also occur with significant frequency in IP networks. Jonathan Stone and Craig Partridge recently reported in Sigcomm 2000 that network errors caught by TCP checksum occur with significant frequency (between one packet in 1100 and 1 in 32000) and without link CRC catching it. For the second question, TCP throughput is impacted by each packet loss. Following TCP's congestion control algorithm existent in all major implementations (Tahoe, Reno, New-Reno, SACK) each packet loss results in the TCP sender's congestion window being reduced to half of its current value, and therefore (assuming constant Round Trip Time), TCP's throughput is halved. After that, the window increases by roughly one packet every two Round Trip Times (assuming the popular Delayed-Acknowledgement algorithm). The temporary decrease in TCP's rate translates into an amount of data missing transmission opportunity. As we show later, for N storage connections sharing an IP "pipe" of rate E, the amount of data missing the opportunity to be transmitted due to a packet loss is D(N) = E^2/(N^2)*RTT^2/(256*M) in the case of iSCSI and iFCP, and D(1) = E^2*RTT^2/(256*M) = D(N)*N^2 in the case of FCIP where RTT = Round Trip Time M = packet size For example, for a set of N=100 connections totaling E=10Gb/s, RTT=10ms, M=1500B, the data not transmitted in time due to a packet loss for iSCSI or iFCP is D(N)=2.6MB. For the same set transported over one TCP session as in FCIP, the data not sent in time is D(1)= 26GB, a 10,000 fold increase. The time interval for TCP to recover its sending rate to its initial value after a packet loss is I(N)= 0.833seconds in the case of iSCSI and iFCP, and I(1)=83.3seconds in the case of FCIP. Observe that in the case of FCIP, the time to recover its rate, I(1)=83.3s, is of the same order of magnitude as the time between two packet losses due exclusively to the link Bit Error Rate. In other words, a packet losse occurs almost immediately after TCP has recovered its rate. This means that FCIP delivers on average about 3/4 of the required 10Gb/s rate, since 1/4 of rate is lost during the time TCP rate increases linearly from 1/2 to full rate. (More precisely, the effective rate is 8.27Gb/s because 1/4 of rate is lost during 83.3s, and the time between two errors is now 120.825s due to decreased sending rate). By comparison, iSCSI or iFCP deliver approximately 9.99979Gb/s (i.e., lost 1/4 of one TCP full rate of 100Mb/s during 0.833s out of a 100s interval). In conclusion, from a performance point of view, transporting storage connections (SCSI or Fibre Channel) on multiple TCP sessions is much more effective than tunneling through a single TCP session, and the difference is proportional to the square of the number of TCP sessions. The math. For a TCP session to sustain a rate of C bits/second, the TCP's congestion window W (measured in number packets) has to be W=RTT*C/(8*M) where RTT = Round Trip Time in seconds M = packet size in Bytes The time needed by the TCP sender to recover from a single packet loss and have its sending rate reach the previous C value is I = 2*RTT*W/2 = RTT*W = RTT^2*C/(8*M) The total amount of data (in Bytes) missing the opportunity to be transmitted in this time interval I is D = C/8*I/4 = C^2*RTT^2/(256*M) If we consider a set of N storage connections sharing an IP "pipe" of rate E, they can be transported in N TCP sessions, as in iSCSI or iFCP. Assuming all connections equal, each TCP session sends at a rate of E/N. One packet loss impacts only one TCP session, and thus, the total amount of data missing the opportunity to be transmitted due to a packet loss is D(N) = E^2/(N^2)*RTT^2/(256*M) On the other hand, if the same set of N storage connections is transported in one TCP session, as in FCIP, the total amount of data losing the opportunity to be transmitted due to a packet loss is D(1) = E^2*RTT^2/(256*M) = D(N)*N^2. For more details on TCP performance see for example: "Modeling TCP Reno Performance: A Simple Model and its Empirical Validation." J. Padhye, V. Firoiu, D. Towsley and J. Kurose, IEEE/ACM Transactions on Networking, April 2000. Franco Travostino, Victor Firoiu ----- Content Internetworking Lab, Technology Center Nortel Networks, Inc. 600 Technology Park Billerica, MA 01821 USA _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Sun Jan 14 05:05:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA07524 for ; Sun, 14 Jan 2001 05:05:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0E7cfx26779 for ips-outgoing; Sun, 14 Jan 2001 02:38:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0E7c6s26767 for ; Sun, 14 Jan 2001 02:38:06 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA23934 for ; Sun, 14 Jan 2001 08:38:00 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA274456 for ; Sun, 14 Jan 2001 08:37:59 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D4.0029ED11 ; Sun, 14 Jan 2001 08:37:56 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 14 Jan 2001 09:33:38 +0200 Subject: Re: iSCSI : Abort Task Response violates SAM-2. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk The referenced task tag should stay as it "clarifies" the response. Julo Santosh Rao on 14/01/2001 00:07:06 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI : Abort Task Response violates SAM-2. Julian, Thanks for the clarification. The "Referenced Task Tag" field in the Abort Task response PDU also could be removed since it was provided to accompany the "No Task Found" response indicating the task tag. Thanks & Regards, Santosh > > > > Santosh, > > I would certainly not have used "violates the SAM-2" when what you probably > meant that is that wording should be in line with SAM-2. I will change the > wording to read as in SAM-2 although in my English they have exactly the > same meaning. > > Julo > > Santosh Rao on 13/01/2001 06:06:31 > > Please respond to Santosh Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: iSCSI : Abort Task Response violates SAM-2. > > > > > Julian, > > SAM-2 Rev 15 Section 6.1 states that : > > "a response of FUNCTION COMPLETE shall indicate task was aborted or task > was > not in task set". > > The iSCSI draft is violating SAM-2 by returning "task not found" when the > task > was not in task set. > > Thanks, > Santosh > > julian_satran@il.ibm.com wrote: > > > Well then Task does not exist will get all enough differentiation. > > If you could not agree with reject (not enough differentiation) another > use > > can > > have difficulty in accepting the other extreme - completed when a task is > > really not found. > > > > Julo > > > > Pierre Labat on 10/01/2001 01:00:01 > > > > Please respond to Pierre Labat > > > > To: ips@ece.cmu.edu > > cc: > > Subject: iSCSI : Further review feedback on latest iSCSI-02.txt draft. > > > > Julian > > > > >>>>>>>>> > > Section 2.6. SCSI Task Management Response > > ================================== > > The referenced Task Tag field can be removed since the > > "No Task Found" response error is not valid. > > (A Target should respond with an accept for an Abort Task > > request specifying an invalid Initator Task Tag). > > [js]Different people have different opinions on this - even with HP - > > talk > > to Pierre! [/js] > > <<<<<<<<< > > > > We are in agreement, it is just a point of terminology. > > > > Some month ago, we had a discussion on the IPS about what to do in > > case the target doesn't find the task to abort. > > I requested that the Abort response returns "OK" > > (something different than Function rejected). > > You added the value "Task not found". > > But "Function complete" can works too. > > > > Regards, > > > > Pierre > > - santoshr.vcf > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Sun Jan 14 11:12:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA09126 for ; Sun, 14 Jan 2001 11:12:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0EE6vs11597 for ips-outgoing; Sun, 14 Jan 2001 09:06:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0EE65s11585 for ; Sun, 14 Jan 2001 09:06:05 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id PAA146802 for ; Sun, 14 Jan 2001 15:05:59 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id PAA277744 for ; Sun, 14 Jan 2001 15:05:59 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D4.004D6A05 ; Sun, 14 Jan 2001 15:05:34 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 14 Jan 2001 14:23:30 +0200 Subject: Re: iSCSI : Concerns regarding DataRN in READ Data PDU. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, The general assumption was that a target is resource constrained. If a target wants acks an initiator should be able to give those to the target. If a target gives a large number in the DataNumber it practically says that it can do without and will request ACK explicitly or not at all. Letting an initiator control this might negatively impact performance for unrelated initiators as the target will be bound to keep resources for the lazy initiator. I hope this clarifies the issue. Julo Santosh Rao on 14/01/2001 00:16:33 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI : Concerns regarding DataRN in READ Data PDU. Julian, My original request on this issue was suggesting 2 alternatives. If a WG consensus cannot be achieved on eliminating the DataRN and all of its complexity, I would like to re-iterate the alternative proposal that was made, that initiators should be allowed to disable this feature during login negotiation. Regards, Santosh > > > -----Original Message----- > > > From: Santosh Rao [mailto:santoshr@cup.hp.com] > > > Sent: Monday, January 08, 2001 9:03 PM > > > To: IPS Reflector > > > Subject: iSCSI : Concerns regarding DataRN in READ Data PDU. > > > > > > I would like to therefore ask that : > > > > > > either, > > > a) DataRN support be removed from the iSCSI draft. > > > > > > or > > > b) The support for DataRN be negotiated at login time giving initators > > > control over this feature and allowing them to disable this feature. > > > This will allow SCSI hardware assist engines to skip implementation of > > > this functionality and their associated software can merely turn off > > > DataNumber in the login negotiation. > > > > > > Thanks, > > > Santosh > > > > > > > > > > > > > > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Sun Jan 14 11:13:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA09140 for ; Sun, 14 Jan 2001 11:13:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0EE6xA11600 for ips-outgoing; Sun, 14 Jan 2001 09:06:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0EE65s11584 for ; Sun, 14 Jan 2001 09:06:05 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id PAA29232 for ; Sun, 14 Jan 2001 15:05:58 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id PAA277742 for ; Sun, 14 Jan 2001 15:05:59 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D4.004D6A4C ; Sun, 14 Jan 2001 15:05:34 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 14 Jan 2001 15:50:07 +0200 Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, I understand your concerns. The previous versions had the response field in the response PDU as a summary of several format errors. The format was selected to be close to FCP. FCP cumulates there the task management responses (for which we have a separate PDU) and the format errors. Surfacing the Service Response can be accomplished either through a command response or through a command reject and we decided that with the format we have we can give more details than with the response field. However we could not find any mapping of the a service response neither in SAM nor in FCP nor in CAM. We could leave this mapping to be implementation dependent (and that is a perfectly valid choice as it has to be supported by the OS) or we could map it into a Status and Sense and we though that this is closer to what we would like an initiator to do in order to express its unhappiness with a specific target implementation (regardless of the sacrosanct layering). Nevertheless we are open to suggestions. Regards, Julo Santosh Rao on 13/01/2001 06:14:20 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. Julian, I had 2 concerns on the format error handling issue : 1) This error recovery is applicable for the SCSI Command PDU. What is the error recovery for a format error on a Login, Logout, Text, NOP-OUT, NOP-IN, SCSI Task Management Command PDU ? These commands are either : not from the scsi layer and their responses do not get sent up the stack to the scsi layer. or in the case of scsi task mgmt cmds, the response does not contain sense data. How are the above to be handled ? 2) On the concept of initiators generating sense data on behalf of targets and faking a HARDWARE ERROR, this is not the approach parallel scsi and fibre channel take. In the case of fibre channel, a header format error results in Response Info being generated in FCP_RSP. (for scsi pdu's). However, I noticed that the latest version of the iSCSI draft has removed all references to response length and response data, and hence, it looks like this approach is not being considered (?) Generating sense data in an initiator is adding complexity and makes iSCSI HBA drivers SPC-2 aware in terms of the content and generation of sense data. Also, we will need to check some implementations of the upper scsi layer drivers to see if this can cause any different form of error recovery to be taken such as issuing a BDR to recover from HARDWARE ERRORs ? Regards, Santosh > Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. > Date: Fri, 12 Jan 2001 09:41:36 +0200 > From: julian_satran@il.ibm.com > To: ips@ece.cmu.edu > Santosh, > A machine producing a format error is DEFECTIVE and this a reported as > hardware error by SCSI (as SCSI does not distinguish between hardware and > firmware and rightfully so). - santoshr.vcf From owner-ips@ece.cmu.edu Sun Jan 14 12:14:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA09385 for ; Sun, 14 Jan 2001 12:14:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0EFBxo13075 for ips-outgoing; Sun, 14 Jan 2001 10:11:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0EFBDs13064 for ; Sun, 14 Jan 2001 10:11:13 -0500 (EST) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id f0EFBD029347 for ; Sun, 14 Jan 2001 09:11:13 -0600 (CST) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA28927; Sun, 14 Jan 01 09:09:39 -0600 Message-Id: <10101141509.AA28927@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. In-Reply-To: Message from julian_satran@il.ibm.com of "Fri, 12 Jan 2001 09:41:36 +0200." References: Date: Sun, 14 Jan 2001 09:10:56 -0600 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, This seems like the zillionth time aired this same disagreement. I think we should try to reach a WG consensus on whether iSCSI should use SCSI status as a means for reporting protocol-related errors, and kill it once and for all. I'm strongly against it. > And an error in the iSCSI layer gets reported by the next layer - that is > the regular layering technique (and BTW I am getting a bit uneasy about all > this preaching on layering when it is not obvious that you understand all > the implications of the point). Santosh seems to understand 100%. I agree with Santosh. SAM defines three pieces of status returned by Execute Command() (which is, in turn implemented by each SCSI protocol, e.g. iSCSI): 1) Service Response: task complete, linked command complete, service delivery or target failure. 2) SCSI status byte 3) SCSI sense data SAM clearly suggest that a protocol's means for signalling protocol-detected errors is the service response status when it says: The actual protocol events corresponding to a response of TASK COMPLETE, LINKED COMMAND COMPLETE or SERVICE DELIVERY OR TARGET FAILURE shall be specified in each protocol standard. Note that defining protocol error events in terms of TC, LCC, SDF and TF, remains an abstraction. An actual implementation can chose to signal these events by whatever means. All SCSI implementations I've seen do have a status return component which exactly corresponds to SAM's service response status, but has more than just these four alternatives. Typically, that includes things like success, command timeout, addressing failure (selection timeout in ||SCSI, bad AL_PA in FCAL, etc.), command aborted, bus parity error, etc.. What iSCSI does need to do is clearly define its error events AS protocol events, which is what describing them in terms of the SAM specified set does. ||SCSI, FCP and the other SCSI protocol standards do this. Operationally, this means that in iSCSI: o protocol-specific errors for a task detected by the target without a CLEARLY corresponding SCSI error return should be signalled using the iSCSI response mechanism. The initiator will handle these errors by recording them in some appropriate way, and selecting an appropriate service response status value. o errors for a task detected directly by the initiator are handled by recording them in some appropriate way, and selecting an appropriate service response status value. I don't know that there's a single sentence or section in SAM which says this, but it clearly implies that the components of SCSI status (status byte and sense) are data which are CARRIED (not created) by SCSI protocols, for the use of the protocol-independent components. For example, a disk peripheral driver reacts to SCSI status and sense generated by disks for SCSI operations that it starts. SCSI status should be equivalent to the SCSI status returned by the logical unit. SCSI protocols are not citizens of the SCSI status space, which means that the real citizens (the command standards, and SAM), may define semantics of SCSI error codes which conflict with iSCSI's selections. The `hardware error' sense key might be defined to mean something very specific within a particular SCSI peripheral command set, and your choice of synthesizing SCSI status within the protocol could cause this behavior to misfire. > [js} the error was generated by a faulty controller and I did not find > any other SCSI sense fit for it[/js] That's because there IS no SCSI sense fit for it. It is a protocol-detected, protocol-unique error. Therefore it should be signalled using the service response status. Steph From owner-ips@ece.cmu.edu Sun Jan 14 14:11:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA10053 for ; Sun, 14 Jan 2001 14:11:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0EGm2N14745 for ips-outgoing; Sun, 14 Jan 2001 11:48:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0EGlKs14731 for ; Sun, 14 Jan 2001 11:47:20 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA162324 for ; Sun, 14 Jan 2001 17:47:08 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA62566 for ; Sun, 14 Jan 2001 17:47:09 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D4.005C3370 ; Sun, 14 Jan 2001 17:47:04 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 14 Jan 2001 18:42:50 +0200 Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk It would be simple if things where that clear-cut. What is a format error coming from a target? IMHO it is a target error and not a protocol failure. Should target errors be reported in the service-response? Except for task management that is common to all protocols I did not see any other thing popping up in any SCSI driver. Preaching layering won't make the issue disappear. Julo Stephen Bailey on 14/01/2001 17:10:56 Please respond to Stephen Bailey To: ips@ece.cmu.edu cc: Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. Julian, This seems like the zillionth time aired this same disagreement. I think we should try to reach a WG consensus on whether iSCSI should use SCSI status as a means for reporting protocol-related errors, and kill it once and for all. I'm strongly against it. > And an error in the iSCSI layer gets reported by the next layer - that is > the regular layering technique (and BTW I am getting a bit uneasy about all > this preaching on layering when it is not obvious that you understand all > the implications of the point). Santosh seems to understand 100%. I agree with Santosh. SAM defines three pieces of status returned by Execute Command() (which is, in turn implemented by each SCSI protocol, e.g. iSCSI): 1) Service Response: task complete, linked command complete, service delivery or target failure. 2) SCSI status byte 3) SCSI sense data SAM clearly suggest that a protocol's means for signalling protocol-detected errors is the service response status when it says: The actual protocol events corresponding to a response of TASK COMPLETE, LINKED COMMAND COMPLETE or SERVICE DELIVERY OR TARGET FAILURE shall be specified in each protocol standard. Note that defining protocol error events in terms of TC, LCC, SDF and TF, remains an abstraction. An actual implementation can chose to signal these events by whatever means. All SCSI implementations I've seen do have a status return component which exactly corresponds to SAM's service response status, but has more than just these four alternatives. Typically, that includes things like success, command timeout, addressing failure (selection timeout in ||SCSI, bad AL_PA in FCAL, etc.), command aborted, bus parity error, etc.. What iSCSI does need to do is clearly define its error events AS protocol events, which is what describing them in terms of the SAM specified set does. ||SCSI, FCP and the other SCSI protocol standards do this. Operationally, this means that in iSCSI: o protocol-specific errors for a task detected by the target without a CLEARLY corresponding SCSI error return should be signalled using the iSCSI response mechanism. The initiator will handle these errors by recording them in some appropriate way, and selecting an appropriate service response status value. o errors for a task detected directly by the initiator are handled by recording them in some appropriate way, and selecting an appropriate service response status value. I don't know that there's a single sentence or section in SAM which says this, but it clearly implies that the components of SCSI status (status byte and sense) are data which are CARRIED (not created) by SCSI protocols, for the use of the protocol-independent components. For example, a disk peripheral driver reacts to SCSI status and sense generated by disks for SCSI operations that it starts. SCSI status should be equivalent to the SCSI status returned by the logical unit. SCSI protocols are not citizens of the SCSI status space, which means that the real citizens (the command standards, and SAM), may define semantics of SCSI error codes which conflict with iSCSI's selections. The `hardware error' sense key might be defined to mean something very specific within a particular SCSI peripheral command set, and your choice of synthesizing SCSI status within the protocol could cause this behavior to misfire. > [js} the error was generated by a faulty controller and I did not find > any other SCSI sense fit for it[/js] That's because there IS no SCSI sense fit for it. It is a protocol-detected, protocol-unique error. Therefore it should be signalled using the service response status. Steph From owner-ips@ece.cmu.edu Sun Jan 14 19:49:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA11520 for ; Sun, 14 Jan 2001 19:49:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0EN3Ga22090 for ips-outgoing; Sun, 14 Jan 2001 18:03:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0EN3As22085 for ; Sun, 14 Jan 2001 18:03:10 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 04D7679F; Sun, 14 Jan 2001 15:03:09 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id PAA08385; Sun, 14 Jan 2001 15:03:04 -0800 (PST) From: Santosh Rao Message-Id: <200101142303.PAA08385@hpcuhe.cup.hp.com> Subject: Re: iSCSI : Abort Task Response violates SAM-2. To: julian_satran@il.ibm.com Date: Sun, 14 Jan 2001 15:03:04 -0800 (PST) Cc: ips@ece.cmu.edu In-Reply-To: from "julian_satran@il.ibm.com" at Jan 14, 2001 09:33:38 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, I'm confused by your response on this. What does the "Referenced Task Tag" in the SCSI Task Management Response PDU "clarify" ? Its description in the draft states : "Initiator Task Tag of the task not found". If the "No Task Found" response is being removed from this section, of what use is the "Referenced Task Tag" ? If the LUN and "Referenced Task Tag" fields are used in the task mgmt response pdu to give LUN or task context to the task mgmt response, then, this information is already present at the initiator and can be looked up based on the Initiator Task Tag from the Task Management Response PDU. Regards, Santosh > > > > The referenced task tag should stay as it "clarifies" the response. > > Julo > > Santosh Rao on 14/01/2001 00:07:06 > > Please respond to Santosh Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: iSCSI : Abort Task Response violates SAM-2. > > > > > Julian, > > Thanks for the clarification. The "Referenced Task Tag" field in the Abort > Task response PDU also could be removed since it was provided to accompany > the "No Task Found" response indicating the task tag. > > Thanks & Regards, > Santosh > > > > > > > > > Santosh, > > > > I would certainly not have used "violates the SAM-2" when what you > probably > > meant that is that wording should be in line with SAM-2. I will change > the > > wording to read as in SAM-2 although in my English they have exactly the > > same meaning. > > > > Julo > > > > Santosh Rao on 13/01/2001 06:06:31 > > > > Please respond to Santosh Rao > > > > To: Julian Satran/Haifa/IBM@IBMIL > > cc: ips@ece.cmu.edu > > Subject: iSCSI : Abort Task Response violates SAM-2. > > > > > > > > > > Julian, > > > > SAM-2 Rev 15 Section 6.1 states that : > > > > "a response of FUNCTION COMPLETE shall indicate task was aborted or task > > was > > not in task set". > > > > The iSCSI draft is violating SAM-2 by returning "task not found" when the > > task > > was not in task set. > > > > Thanks, > > Santosh > > > > julian_satran@il.ibm.com wrote: > > > > > Well then Task does not exist will get all enough differentiation. > > > If you could not agree with reject (not enough differentiation) another > > use > > > can > > > have difficulty in accepting the other extreme - completed when a task > is > > > really not found. > > > > > > Julo > > > > > > Pierre Labat on 10/01/2001 01:00:01 > > > > > > Please respond to Pierre Labat > > > > > > To: ips@ece.cmu.edu > > > cc: > > > Subject: iSCSI : Further review feedback on latest iSCSI-02.txt draft. > > > > > > Julian > > > > > > >>>>>>>>> > > > Section 2.6. SCSI Task Management Response > > > ================================== > > > The referenced Task Tag field can be removed since the > > > "No Task Found" response error is not valid. > > > (A Target should respond with an accept for an Abort Task > > > request specifying an invalid Initator Task Tag). > > > [js]Different people have different opinions on this - even with HP - > > > talk > > > to Pierre! [/js] > > > <<<<<<<<< > > > > > > We are in agreement, it is just a point of terminology. > > > > > > Some month ago, we had a discussion on the IPS about what to do in > > > case the target doesn't find the task to abort. > > > I requested that the Abort response returns "OK" > > > (something different than Function rejected). > > > You added the value "Task not found". > > > But "Function complete" can works too. > > > > > > Regards, > > > > > > Pierre > > > > - santoshr.vcf > > > > > > > > > > > -- > ################################# > Santosh Rao > Software Design Engineer, > HP, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################# > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Sun Jan 14 19:51:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA11562 for ; Sun, 14 Jan 2001 19:51:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0EMdFf21572 for ips-outgoing; Sun, 14 Jan 2001 17:39:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0EMcLs21554 for ; Sun, 14 Jan 2001 17:38:21 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 5963CE08; Sun, 14 Jan 2001 14:38:20 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id OAA06727; Sun, 14 Jan 2001 14:34:36 -0800 (PST) From: Santosh Rao Message-Id: <200101142234.OAA06727@hpcuhe.cup.hp.com> Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. To: julian_satran@il.ibm.com Date: Sun, 14 Jan 2001 14:34:36 -0800 (PST) Cc: ips@ece.cmu.edu In-Reply-To: from "julian_satran@il.ibm.com" at Jan 14, 2001 06:42:50 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > It would be simple if things where that clear-cut. > > What is a format error coming from a target? IMHO it is a target error and > not a protocol failure. Julian, A SCSI Protocol(i.e. iSCSI) header format error constitutes a "Service Delivery or Target Failure" service response, since the failure occurs in the service delivery sub-system (SCSI Protocol). SAM-2 explains this service response as "The command has ended due to a service delivery failure or target device mal-function", which should be the service response to be returned on a header format error. Further, SAM-2 Section 5 also states that the application client (i.e. the SCSI upper layer driver) shall treat the SCSI Status to be un-defined if the command ends with a service response of "Service Delivery or Target Failure". This implies that the SCSI Upper Layer driver shall ignore the CHECK CONDITION scsi status that iSCSI initiators are going to fake to their upper layers on a header format error, due to a service response of "Service Delivery or Target Mal-function". So, is iSCSI proposing to return a service response of "Task Complete" to its upper layers on a header format error ? If not and it is the intent that a service response of "Service Delivery or Target Mal-function" should be returned, then, the upper layer scsi driver is going to ignore the CHECK CONDITION that iSCSI intends to fake. It would be interesting to hear what T10 has to say on this approach that iSCSI is currently proposing to adopt on a header format error. I see the following problems with the current approach being advocated : ------------------------------------------------------------------------ 1) It does NOT solve the problem for commands other than the SCSI Command PDU. I am yet to see a response to my questions on how header format errors for Login, Logout, Text, NOP-OUT, NOP-IN and SCSI Task Management Command PDU are handled by the current proposal described in section 5.4 of the 03 iSCSI draft. 2) It is not in line with [violates ?] Section 5 of SAM-2. 3) It can cause upper layers to initiate Auto Contingent Allegiance recovery such as CLEAR ACA due to the CHECK CONDITION returned. 4) It can cause upper layers to over-react on seeing a HARDWARE ERROR by resorting to error recovery such as BDR to recover from a HARDWARE ERROR. 5) It adds extra complexity to the iSCSI initiator drivers by making them SPC-2 aware and having to generate sense data on behalf of a target, not something that has been required in other SCSI Protocols such as parallel scsi and fibre channel. 6) It causes a mis-leading error log of a HARDWARE ERROR, where, a more specific error log of the header format error that occurred based on the response data would have been more useful in quick fault isolation. 7) Last, but not the least, it is a violation of layering between the ULP and LLP, wherein, an LLP Service Delivery Failure is being treated as a ULP SCSI error returned by the LUN. I would like to propose the following solution : ------------------------------------------------ 1) On discovery of header format error, a target MUST convey the specific type of format error that was discovered through use of mechanisms like the Response Data in a SCSI Response PDU, Response Field in a SCSI Task Management Response PDU and the Reason Code field in the REJECT PDU. 2) On a header format error discovered at the initiator, a service response of "Service Delivery or Target Failure" MUST be returned to their upper layers [and the initiator may log the specific header format error that was discovered.]. The draft need not attempt to elaborate on these service response definitions since these are defined by the SCSI Stack of each O.S. as a set of return values exchanged b/n the LLP and the ULP. Benefits of the proposed solution : ----------------------------------- 1) It addresses the problem for header format errors on ALL types of PDUs. 2) It is in line with [complies with ?] SAM-2 semantics of "Execute Command". 3) It provides more value-added error logging based on the specific header format error that occurred rather than a more general HARDWARE ERROR, allowing for quicker isolation and root cause of the problem. 4) It avoids ULPs resorting to inappropriate error recovery such as CLEAR ACA or a BDR on seeing CHECK CONDITION with HARDWARE ERROR. 5) It retains layering semantics and differentiates between LLP service delivery errors and ULP SCSI errors. I'd be happy to learn what implications of this issue I've missed. Thanks & Regards, Santosh Rao > Should target errors be reported in the service-response? Except for task > management that is common to all protocols I did not see any other thing > popping up in any SCSI driver. > > Preaching layering won't make the issue disappear. > > Julo > > Stephen Bailey on 14/01/2001 17:10:56 > > Please respond to Stephen Bailey > > To: ips@ece.cmu.edu > cc: > Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. > > > > > Julian, > > This seems like the zillionth time aired this same disagreement. > > I think we should try to reach a WG consensus on whether iSCSI should > use SCSI status as a means for reporting protocol-related errors, and > kill it once and for all. > > I'm strongly against it. > > > And an error in the iSCSI layer gets reported by the next layer - that is > > the regular layering technique (and BTW I am getting a bit uneasy about > all > > this preaching on layering when it is not obvious that you understand all > > the implications of the point). > > Santosh seems to understand 100%. I agree with Santosh. > > SAM defines three pieces of status returned by Execute Command() > (which is, in turn implemented by each SCSI protocol, e.g. iSCSI): > 1) Service Response: task complete, linked command complete, service > delivery or target failure. > 2) SCSI status byte > 3) SCSI sense data > > SAM clearly suggest that a protocol's means for signalling > protocol-detected errors is the service response status when it says: > > The actual protocol events corresponding to a response of TASK > COMPLETE, LINKED COMMAND COMPLETE or SERVICE DELIVERY OR TARGET > FAILURE shall be specified in each protocol standard. > > Note that defining protocol error events in terms of TC, LCC, SDF and > TF, remains an abstraction. An actual implementation can chose to > signal these events by whatever means. All SCSI implementations I've > seen do have a status return component which exactly corresponds to > SAM's service response status, but has more than just these four > alternatives. Typically, that includes things like success, command > timeout, addressing failure (selection timeout in ||SCSI, bad AL_PA in > FCAL, etc.), command aborted, bus parity error, etc.. What iSCSI does > need to do is clearly define its error events AS protocol events, > which is what describing them in terms of the SAM specified set does. > > ||SCSI, FCP and the other SCSI protocol standards do this. > > Operationally, this means that in iSCSI: > > o protocol-specific errors for a task detected by the target without > a CLEARLY corresponding SCSI error return should be signalled > using the iSCSI response mechanism. > > The initiator will handle these errors by recording them in some > appropriate way, and selecting an appropriate service response > status value. > > o errors for a task detected directly by the initiator are handled > by recording them in some appropriate way, and selecting an > appropriate service response status value. > > I don't know that there's a single sentence or section in SAM which > says this, but it clearly implies that the components of SCSI status > (status byte and sense) are data which are CARRIED (not created) by > SCSI protocols, for the use of the protocol-independent components. > For example, a disk peripheral driver reacts to SCSI status and sense > generated by disks for SCSI operations that it starts. > > SCSI status should be equivalent to the SCSI status returned by the > logical unit. > > SCSI protocols are not citizens of the SCSI status space, which means > that the real citizens (the command standards, and SAM), may define > semantics of SCSI error codes which conflict with iSCSI's selections. > The `hardware error' sense key might be defined to mean something very > specific within a particular SCSI peripheral command set, and your > choice of synthesizing SCSI status within the protocol could cause > this behavior to misfire. > > > [js} the error was generated by a faulty controller and I did not find > > any other SCSI sense fit for it[/js] > > That's because there IS no SCSI sense fit for it. It is a > protocol-detected, protocol-unique error. Therefore it should be > signalled using the service response status. > > Steph > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Sun Jan 14 20:41:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA11907 for ; Sun, 14 Jan 2001 20:41:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0ENdL422880 for ips-outgoing; Sun, 14 Jan 2001 18:39:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0ENcas22866 for ; Sun, 14 Jan 2001 18:38:36 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id C1AEE49F for ; Sun, 14 Jan 2001 15:38:35 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id PAA10359; Sun, 14 Jan 2001 15:38:30 -0800 (PST) From: Santosh Rao Message-Id: <200101142338.PAA10359@hpcuhe.cup.hp.com> Subject: iSCSI : LOGO originated by target. (fwd) To: ips@ece.cmu.edu (ips) Date: Sun, 14 Jan 2001 15:38:30 -0800 (PST) Cc: santoshr@hpcuhe.cup.hp.com (Santosh Rao) X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, I did not see any response on this issue and hence, I am re-sending this again. Issue : ======= iSCSI requires targets to use AEN to request a logout, rather than allowing targets to originate a Logout Command PDU. Problem : ========= 1) The AEN feature is turned off by default (RAERP, UAAERP and EAERP bits in control mode page set to 0) thereby turning off AEN on the target side. 2) AEN usage requires host SCSI Stacks to be modified since they may not be turning on RAERP, UAAERP or EAERP in the control mode page. 3) Targets that need to logout an initiator cannot depend on AEN since the host may have disabled AEN, and hence, need to be allowed to originate a Logout. Proposed Solution : =================== iSCSI should allow targets to originate logouts. Thanks, Santosh > > > Section 2.17 Asynchronous Event > > ========================= > > 1) "Target requests Logout on this connection" iSCSI event > > indicator. Why not just allow the target to originate a Logout > > as is done in Fibre Channel ? > > > Besides, this AEN does not meet the requirement. > > How does the target convey the CID of the connection to be > > logged out ? > > [js] CID an timeout appear now in parameters [/js] > > The simplest approach and the most expeditious error recovery is to > allow the target to originate a Logout. To send an AEN and then wait for > the initiator to log out is opening up the target to dependencies on the > initiator honoring and acting on that AEN, which is un-necessary. > > This should not be a violation of SAM since SAM does not restrict > unsolicited traffic for the interconnect specific commands. (Fibre > Chanel does this.) > > Thanks, > Santosh > > > I do not think SAM-2 lays any guidelines for how the SCSI > > interconnect protocol handle its native PDUs. (i.e. non-scsi > > operations). > > Fibre Channel allows unsolicited inbound traffic for ELS and > > LS commands and allowing Logout and NOP-OUT > > erhaps better thought of as a NOP command) commands to > > be originated by targets would be desirable and would get rid of this > > particular AEN iSCSI event indicator. > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Sun Jan 14 21:31:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA12263 for ; Sun, 14 Jan 2001 21:31:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0F0KK023722 for ips-outgoing; Sun, 14 Jan 2001 19:20:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0F0JVs23688 for ; Sun, 14 Jan 2001 19:19:31 -0500 (EST) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id f0F0JV008034 for ; Sun, 14 Jan 2001 18:19:31 -0600 (CST) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA29400; Sun, 14 Jan 01 18:17:57 -0600 Message-Id: <10101150017.AA29400@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. In-Reply-To: Message from julian_satran@il.ibm.com of "Sun, 14 Jan 2001 18:42:50 +0200." References: Date: Sun, 14 Jan 2001 18:19:14 -0600 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > It would be simple if things where that clear-cut. It is that simple. It is clear-cut. > What is a format error coming from a target? IMHO it is a target > error and not a protocol failure. Should target errors be reported > in the service-response? TARGET ERROR clearly IS one of the service response codes defined by SAM. Therefore, yep, go ahead and define iSCSI errors in terms of TARGET FAILURE and SERVICE DELIVERY FAILURE. SCSI implementations define their own set of service response status values and whatever iSCSI specifies, a protocol driver will make its own mapping of protocol conditions to service response status values. For that reason, the exact specification of TF or SDF is less relevant than that you identify the condition as a service response reported failure. > Except for task management that is common to all protocols I did not > see any other thing popping up in any SCSI driver. ??? Are you saying that you don't see where the service response status code is concretely instantiated in ANY SCSI driver? Take CAM. The CAM status is the service response status code. It includes values like: request completed without error request aborted by host invalid request target selection timeout command timeout data overrun unexpected bus free etc.. Solaris and IRIX are similar (don't have the code on hand). I can't imagine that ANY other SCSI driver does not have this component, but I clearly have not seen all other SCSI drivers. Did you mean SCSI protocol rather than SCSI driver? FCP certainly mentions service response. > Preaching layering won't make the issue disappear. Perhaps we could hear a resounding chorus of other participants that think I'm just `preaching layering' to try to make some issue disappear. If so, I'll agree to say no more on this topic. Presently, it seems to be two (Santosh and me) against one (you). It's clear how to specify and how implementations will handle, and have historically handled these types of errors. Where's the conflict? What's hard about this? Steph From owner-ips@ece.cmu.edu Sun Jan 14 23:25:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA14620 for ; Sun, 14 Jan 2001 23:25:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0F1xSY25658 for ips-outgoing; Sun, 14 Jan 2001 20:59:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0F1wps25644 for ; Sun, 14 Jan 2001 20:58:51 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 99E6DA1A; Sun, 14 Jan 2001 17:58:50 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id RAA17892; Sun, 14 Jan 2001 17:58:06 -0800 (PST) From: Santosh Rao Message-Id: <200101150158.RAA17892@hpcuhe.cup.hp.com> Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. To: steph@cs.uchicago.edu (Stephen Bailey) Date: Sun, 14 Jan 2001 17:58:05 -0800 (PST) Cc: ips@ece.cmu.edu In-Reply-To: <10101150017.AA29400@candide.cs.uchicago.edu> from "Stephen Bailey" at Jan 14, 2001 06:19:14 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, I've enclosed below the mapping of service response status from the LLP to the ULP for HP-UX, Solaris, SVR4 (MP-RAS) and NT. As you can see, all O.S SCSI stacks should have some form of a service response status from their LLP (HBA drivers, mini-port drivers, LLDs, interface drivers) to their ULP (target drivers, class drivers, HLDs, device drivers). In SAM-2 (Section 4.13) terminology : A SCSI Protocol (Parallel SCSI, SCSI-FCP, iSCSI, etc) MUST NOT specify the service response in the "Protocol Service Confirmation" as this is common for all SCSI LLPs with their ULPs for each O.S. scsi stack. (see the list below). SCSI Protocols MUST specify the service response in the protocol specific PDU Responses that are exchanged between target and initiator, as this is to be defined for target-initiator communication for each SCSI protocol. Ex : Response Info in FCP_RSP PDU of Fibre Channel as defined in SCSI-FCP-2. Given the above, I would like to propose : ========================================== o Removal of the 3rd paragraph of Section 5.4 of the 03 rev of the iSCSI draft which states : "When a session is active whenever an initiator receives an iSCSI PDU with a format error......." o Re-introduction and specification of Response Data in the SCSI Response PDU along the lines of the Response Info defined for FCP_RSP in FCP-2. Even a single Response byte along the lines of the current Respnse field in SCSI Task Management Response PDU should be sufficient. o Specification of a more detailed Response field in the SCSI Task Management Response PDU that should include the following : 1) "No such LUN" (Abort Task Set, CLear Task Set or LUN Reset is attempted on an invalid LUN.) 2) "Logical Busy" (A 2nd Task Management Command is attempted while a prior conflicting task is in progress. Ex : attempting a LUN Reset while a Target Reset is in progress, etc.) 3) "Function Not Supported" (To indicate no support for certain task management commands like Target Reset.) Thanks & Regards, Santosh ================================================================== O.S. Service Response Service Delivery or Target Failure ================================================================== HP-UX scb->cdb_status SCTL_INCOMPLETE, SCTL_SELECT_TIMEOUT. Solaris scsi_pkt->pkt_reason CMD_INCOMPLETE, CMD_TRAN_ERROR, CMD_ABORTED, CMD_TIMEOUT scsi_pkt->pkt_statistics STAT_ABORTED STAT_TIMEOUT SVR4 MP-RAS Scsi_Op->OP_Status (Had something similar. Can't recollect off the top of my head.) NT srb->SrbStatus SRB_STATUS_ABORTED, SRB_STATUS_ERROR, SRB_STATUS_TIMEOUT, and many more. ================================================================== > > Except for task management that is common to all protocols I did not > > see any other thing popping up in any SCSI driver. > > ??? Are you saying that you don't see where the service response > status code is concretely instantiated in ANY SCSI driver? > > Take CAM. The CAM status is the service response status code. It includes > values like: > request completed without error > request aborted by host > invalid request > target selection timeout > command timeout > data overrun > unexpected bus free > > etc.. > > Solaris and IRIX are similar (don't have the code on hand). I can't > imagine that ANY other SCSI driver does not have this component, but I > clearly have not seen all other SCSI drivers. > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Mon Jan 15 05:05:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA00116 for ; Mon, 15 Jan 2001 05:05:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0F7gWf02129 for ips-outgoing; Mon, 15 Jan 2001 02:42:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0F7fqs02114 for ; Mon, 15 Jan 2001 02:41:52 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id DBE9B765; Sun, 14 Jan 2001 23:41:51 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id XAA07434; Sun, 14 Jan 2001 23:41:46 -0800 (PST) From: Santosh Rao Message-Id: <200101150741.XAA07434@hpcuhe.cup.hp.com> Subject: Re: iSCSI: Abort task To: julian_satran@il.ibm.com Date: Sun, 14 Jan 2001 23:41:46 -0800 (PST) Cc: ips@ece.cmu.edu In-Reply-To: from "julian_satran@il.ibm.com" at Jan 12, 2001 07:38:09 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Another reason for enforcing "connection allegiance" of Abort Task and its associated task : In the case of multi-connection sessions, each connection can be using a different NIC. The SCSI command state information is maintained in SCSI hardware assists (SCSI Exchange State Tables) on the adapter and such structures need to be invalidated while aborting an I/O. Handling the Abort Task and the task on different connections implies having to then share state information across NIC instances, something that iSCSI has been trying to avoid. Regards, Santosh > > > > Pierre, > > I am not sure that I follow completely (and accurately) your line of > thought but if you have a multiple connection session the command > numbering is meant to offer a total ordering. Any of the scenarios you > mention can't happen unless the target decides to deliver commands to the > "SCSI layer" out of order. > And even if it does deliver command out of order (there might be good > reasons for it) it should build a "sync barrier" for those commands that > mandate ordering. This type of problem exist also in FCP (on a single > connection!) and I assume that the thinking there is to use the per LU > command ordering to avoid task management PDUs (like clear a queue) > arriving before the commands they are referring to. > > I hope this clarifies the issue. > > Julo > > Pierre Labat on 09/01/2001 11:39:53 > > Please respond to Pierre Labat > > To: ips@ece.cmu.edu > cc: > Subject: iSCSI: Abort task > > > > > Julian, > > > It seems that we need to specify more the "Abort task" to avoid > some problems. Below are 3 points we should add in the spec: > > > 1) An "Abort task" must be sent on the TCP connection that was used > to send the command. This does not apply if the connection used > for the command has been "logged out". In this case of course, the > "Abort task" can be send on another connection and it will not hurt. > > It is to avoid that a command that was delayed in the network, finally > make it > after the "Abort task" as been processed. > > 2) When the initiator receives the "Abort task" response, if the > ExpCmdRN returned in the "Abort task" response is > <= CmdRN of the aborted task, the initiator must re-use this > CmdRN for another task (any). > > 3) The target when receiving an "Abort task" for a task whose the > CmdRN (retreived from the initiator task tag) is >= ExpCmdRN > must not bridge this CmdRN. I mean by bridge: allow ExpCmdRN to > pass the CmdRN value of the aborted task. It will be up to the > initiator > to send another command with the same CmdRN in order to allow > the target to move ExpCmdRN over the CmdRN value of the > aborted task. > > > > "3)" Is to prevent to following scenario: > > Two TCP connections 1 and 2 in the session > ExpCmdRN = 2 > > a) the command with CmdRN = 3 is sent over connection1 > > b) the command with CmdRN = 4 is sent over connection 2 > > c) the command CmdRN = 4 reaches the target > > d) the initiator aborts the command 4 and bridges CmdRN=4 > > e) the target receives the command 3, and as CmdRN=4 is bridged, > advances ExpCmdRN to 5 > > f) the initiator sends another command (not related with the aborted > one) with CmdRN=4 > > g) the target drops this command because CmdRN is less than ExpCmdRN. > > > > "2)" needs to exist because if the initiator doesn't re-use the CmdRN > the target > will deadlock because ExpCmdRN will be stuck behind the CmdRN of > the aborted command. > > > > An alternative to the points 2) and 3) would be the initiator not > re-using > the CmdRN of the aborted commands. But this requires the target to > bridge > all the CmdRNs of the Aborted tasks (need extra states) and anyway there > > is a scenario where it doesn't work. Hence this is a bad alternative. > > Scenario where it doesn't work: > > Session with 2 connections 1 and 2. > ExpCmdRN=1 > > a) Command with CmdRN=4 is sent over connection1 > > b) Connection 1 drops and the command 4 will never make it to the > target. > > c) A logout for the connection 1 is sent over connection 2 > > d) The initiator sends an "Abort task" (for the task CmdRN=4) over > the connection 2 > > e) The target, from the referenced initiator task tag cannot find the > CmdRN of the aborted task because the initiator task tag > corresponds > to nothing because the command has never been received. > Hence the target is not able to bridge CmdRN=4 > > f) As the initiator does not re-use CmdRN=4 the target will deadlock > when ExpCmdRN will be 4. > > > Regards, > > Pierre > > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Mon Jan 15 07:25:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA02069 for ; Mon, 15 Jan 2001 07:25:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0F9ra104250 for ips-outgoing; Mon, 15 Jan 2001 04:53:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0F9rDs04241 for ; Mon, 15 Jan 2001 04:53:13 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id BCC771047 for ; Mon, 15 Jan 2001 01:53:12 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id BAA14636; Mon, 15 Jan 2001 01:53:08 -0800 (PST) From: Santosh Rao Message-Id: <200101150953.BAA14636@hpcuhe.cup.hp.com> Subject: iSCSI : FirstBurst, I.T.T value of 0. To: ips@ece.cmu.edu (ips) Date: Mon, 15 Jan 2001 01:53:07 -0800 (PST) X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, 3 quick comments on the 03 rev of the iSCSI draft : 1) Section 3.1.3 refers to a non-existent login key InitialBurstLength. Is this supposed to be FirstBurst ? 2) Section 2.7 continues to use Target Task Tag and Target Transfer Tag field names in different places in the figures and text. Only one of the 2 names should be used and all references to the other removed. 3) Section 2.12 and 2.13 (NOP-OUT and NOP-IN) continue to use a value of 0 as the reserved usage of Initiator Task Tag. This should be changed to 0xFFFFFFFF. Regards, Santosh -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Mon Jan 15 12:50:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA12211 for ; Mon, 15 Jan 2001 12:50:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FFukV20803 for ips-outgoing; Mon, 15 Jan 2001 10:56:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx10.quantum.com (mx10.quantum.com [204.212.103.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FFuDs20785 for ; Mon, 15 Jan 2001 10:56:13 -0500 (EST) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx10.quantum.com (8.9.3+Sun/8.9.3) with ESMTP id HAA16529 for ; Mon, 15 Jan 2001 07:54:52 -0800 (PST) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Mon, 15 Jan 2001 07:56:07 -0800 Message-ID: <8133266FE373D11190CD00805FA768BF055BD2E3@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: ips@ece.cmu.edu Subject: RE: iSCSI : Initiators expected to fake CHECK CONDITIONS. Date: Mon, 15 Jan 2001 07:56:02 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I completely concur with Stephen Bailey. Protocol errors belong to a different name-space than SCSI status. iSCSI cannot hijack SCSI status codes for protocol errors. iSCSI could union it's protocol-dependent error field with the SCSI status field, as long as protocol-dependent errors are defined as returning a service response of SERVICE DELIVERY OR TARGET FAILURE. (SAM clearly states that the SCSI status value is invalid if the service response is SERVICE DELIVERY OR TARGET FAILURE.) If the protocol error field is unioned with the SCSI status field, it should be given a different member name - similar to an anonymous union in C - in order to prevent confusion regarding its semantics. In particular, there is no allegiance created for protocol errors. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ece.cmu.edu Mon Jan 15 12:50:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA12228 for ; Mon, 15 Jan 2001 12:50:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FFJmL19518 for ips-outgoing; Mon, 15 Jan 2001 10:19:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FBibs14680 for ; Mon, 15 Jan 2001 06:44:37 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA01004; Mon, 15 Jan 2001 06:44:36 -0500 (EST) Message-Id: <200101151144.GAA01004@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-iscsi-boot-01.txt Date: Mon, 15 Jan 2001 06:44:36 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : A Standard for BootStrapping Clients using the iSCSI Protocol Author(s) : P. Sarkar, D. Missimer, C. Sapuntzakis Filename : draft-ietf-ips-iscsi-boot-01.txt Pages : 8 Date : 12-Jan-01 The Small Computer Systems Interface (SCSI) is a popular family of protocols for communicating with I/O devices, especially storage devices. iSCSI is a proposed transport protocol for SCSI that operates on top of TCP[12]. This memo describes a standard mechanism to enable clients to bootstrap themselves using the iSCSI protocol. The goal of this standard is to enable clients to obtain the information to open an iSCSI session with the iSCSI bootstrpping server, assuming this information is not available. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-boot-01.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-boot-01.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-boot-01.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20010112140343.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-iscsi-boot-01.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-iscsi-boot-01.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20010112140343.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Mon Jan 15 12:53:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA12281 for ; Mon, 15 Jan 2001 12:53:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FFJlm19515 for ips-outgoing; Mon, 15 Jan 2001 10:19:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FBhxs14670 for ; Mon, 15 Jan 2001 06:43:59 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA00827; Mon, 15 Jan 2001 06:43:58 -0500 (EST) Message-Id: <200101151143.GAA00827@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-iscsi-boot-01.txt Date: Mon, 15 Jan 2001 06:43:57 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : A Standard for BootStrapping Clients using the iSCSI Protocol Author(s) : P. Sarkar, D. Missimer, C. Sapuntzakis Filename : draft-ietf-ips-iscsi-boot-01.txt Pages : 8 Date : 12-Jan-01 The Small Computer Systems Interface (SCSI) is a popular family of protocols for communicating with I/O devices, especially storage devices. iSCSI is a proposed transport protocol for SCSI that operates on top of TCP[12]. This memo describes a standard mechanism to enable clients to bootstrap themselves using the iSCSI protocol. The goal of this standard is to enable clients to obtain the information to open an iSCSI session with the iSCSI bootstrpping server, assuming this information is not available. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-boot-01.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-boot-01.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-boot-01.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20010112121935.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-iscsi-boot-01.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-iscsi-boot-01.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20010112121935.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Mon Jan 15 14:04:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA13707 for ; Mon, 15 Jan 2001 14:04:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FGgps22523 for ips-outgoing; Mon, 15 Jan 2001 11:42:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FGfns22479 for ; Mon, 15 Jan 2001 11:41:50 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id C6R66RYP; Mon, 15 Jan 2001 08:41:39 -0800 From: "Y P Cheng" To: Subject: iSCSI & iFCP Overlapping (Was iFCP as an IP Storage Work Item) Date: Mon, 15 Jan 2001 08:40:41 -0800 Message-ID: <000201c07f11$e5ed9c60$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > iFCP and FCIP have nothing to do with OSPF. I think your statements > show that you do not understand iFCP and its objectives. I would > have hoped that those critical of iFCP would have at least a minimal > understanding of it. Josh, Had I stayed with the topic of "saving the customers' investment in FCP stack" I would have done better and not invited the above statement. :-) I have no doubt that you are a very good and wonderful network engineer. But, I am afraid, iFCP simply is a better mouse trap solving the same problem as that by iSCSI. After reading the iFCP draft carefully, I can't help but conclude that by connecting N-Port to IP directly iFCP usurps the fibre channel Extended Link Services and replaces class F traffic by all the wonderful stuff like EGP, BGP, OSPF, etc., offered by the Big Internet. I found the iFCP effort overlaps with iSCSI and it provides no additional benefit. As you have indicated that iFCP provides a transition into the iSCSI world of the future. We must agree that initially even iSCSI storage devices will have FCP or SCSI devices "inside the box". This is because the storage industry will take some time to change its infrastructure to produce iSCSI drives. On the other hand, the iSCSI HBAs will be available quickly -- I hope you do give me more credit in knowing the HBA world. :-) Therefore, it is very easy for the Network Storage Industry to have an iSCSI target adapter inside its box to communicate with the iSCSI hosts and clients. If iFCP is a viable solution, the HBA industry can produce iFCP target and host adapters just as quickly as the iSCSI adapters. Trust me, if we can implement the Fibre Channel and InfiniBand specifications, we can do iFCP. I do hope you see my logic now. Therefore, while iFCP may be a better mouse trap than FCIP, IMHO, it is providing the same solution as that of iSCSI. By the way, lets hear from other folks on this topic, please! Y.P. Cheng, ConnectCom Solutions. From owner-ips@ece.cmu.edu Mon Jan 15 15:55:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA15567 for ; Mon, 15 Jan 2001 15:55:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FJOq927975 for ips-outgoing; Mon, 15 Jan 2001 14:24:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FJOfs27970 for ; Mon, 15 Jan 2001 14:24:42 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0FKZ7036144; Mon, 15 Jan 2001 12:35:08 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Y P Cheng" , Subject: RE: iSCSI & iFCP Overlapping (Was iFCP as an IP Storage Work Item) Date: Mon, 15 Jan 2001 11:22:56 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <000201c07f11$e5ed9c60$90c809c0@yp_portable.advansys.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Y.P. A company's success with ST-506 leveraged off an inappropriately simple interface, floppy, and has now monopolized drive interfaces where SCSI enjoyed a 10 year lead. iFCP has potential in that it attempts to make fewer changes to a complex but stable interface (some may argue not stable enough). I encourage those working on both FCIP and iFCP to share a common encapsulation and define node handling in a separate proposal. There is significant differences between iSCSI and iFCP. Problems in iSCSI proposals have already been solved with FCP so unless there is a problem, what is being fixed? What limitations with FCP merit iSCSI? Let the marketplace decide as SCTP may obsolete choices made in iSCSI. The IPS WG should attempt to minimize variants of a common theme. In that respect, iFCP seems to be more in line than does iSCSI but, there are many wishing to make a career out of iSCSI. One wonders if they will have time should iFCP provide a stable solution first. Doug > > iFCP and FCIP have nothing to do with OSPF. I think your statements > > show that you do not understand iFCP and its objectives. I would > > have hoped that those critical of iFCP would have at least a minimal > > understanding of it. > > Josh, > > Had I stayed with the topic of "saving the customers' investment in FCP > stack" I would have done better and not invited the above > statement. :-) I > have no doubt that you are a very good and wonderful network > engineer. But, > I am afraid, iFCP simply is a better mouse trap solving the same > problem as > that by iSCSI. > > After reading the iFCP draft carefully, I can't help but conclude that by > connecting N-Port to IP directly iFCP usurps the fibre channel > Extended Link > Services and replaces class F traffic by all the wonderful stuff like EGP, > BGP, OSPF, etc., offered by the Big Internet. I found the iFCP effort > overlaps with iSCSI and it provides no additional benefit. As you have > indicated that iFCP provides a transition into the iSCSI world of the > future. We must agree that initially even iSCSI storage devices will have > FCP or SCSI devices "inside the box". This is because the > storage industry > will take some time to change its infrastructure to produce iSCSI drives. > On the other hand, the iSCSI HBAs will be available quickly -- I > hope you do > give me more credit in knowing the HBA world. :-) Therefore, it is very > easy for the Network Storage Industry to have an iSCSI target > adapter inside > its box to communicate with the iSCSI hosts and clients. If iFCP is a > viable solution, the HBA industry can produce iFCP target and > host adapters > just as quickly as the iSCSI adapters. Trust me, if we can implement the > Fibre Channel and InfiniBand specifications, we can do iFCP. I > do hope you > see my logic now. > > Therefore, while iFCP may be a better mouse trap than FCIP, IMHO, it is > providing the same solution as that of iSCSI. > > By the way, lets hear from other folks on this topic, please! > > Y.P. Cheng, ConnectCom Solutions. > > > From owner-ips@ece.cmu.edu Mon Jan 15 15:56:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA15595 for ; Mon, 15 Jan 2001 15:56:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FImvD26709 for ips-outgoing; Mon, 15 Jan 2001 13:48:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FIm1s26683 for ; Mon, 15 Jan 2001 13:48:01 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id TAA25570 for ; Mon, 15 Jan 2001 19:47:55 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id TAA267878 for ; Mon, 15 Jan 2001 19:46:37 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D5.006721D9 ; Mon, 15 Jan 2001 19:46:28 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 15 Jan 2001 14:05:27 +0200 Subject: Re: iSCSI : Abort Task Response violates SAM-2. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, I did not that "task not found" will be removed. I said that the wording will match closer the SAM wording. Julo Santosh Rao on 15/01/2001 01:03:04 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI : Abort Task Response violates SAM-2. Julian, I'm confused by your response on this. What does the "Referenced Task Tag" in the SCSI Task Management Response PDU "clarify" ? Its description in the draft states : "Initiator Task Tag of the task not found". If the "No Task Found" response is being removed from this section, of what use is the "Referenced Task Tag" ? If the LUN and "Referenced Task Tag" fields are used in the task mgmt response pdu to give LUN or task context to the task mgmt response, then, this information is already present at the initiator and can be looked up based on the Initiator Task Tag from the Task Management Response PDU. Regards, Santosh > > > > The referenced task tag should stay as it "clarifies" the response. > > Julo > > Santosh Rao on 14/01/2001 00:07:06 > > Please respond to Santosh Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: iSCSI : Abort Task Response violates SAM-2. > > > > > Julian, > > Thanks for the clarification. The "Referenced Task Tag" field in the Abort > Task response PDU also could be removed since it was provided to accompany > the "No Task Found" response indicating the task tag. > > Thanks & Regards, > Santosh > > > > > > > > > Santosh, > > > > I would certainly not have used "violates the SAM-2" when what you > probably > > meant that is that wording should be in line with SAM-2. I will change > the > > wording to read as in SAM-2 although in my English they have exactly the > > same meaning. > > > > Julo > > > > Santosh Rao on 13/01/2001 06:06:31 > > > > Please respond to Santosh Rao > > > > To: Julian Satran/Haifa/IBM@IBMIL > > cc: ips@ece.cmu.edu > > Subject: iSCSI : Abort Task Response violates SAM-2. > > > > > > > > > > Julian, > > > > SAM-2 Rev 15 Section 6.1 states that : > > > > "a response of FUNCTION COMPLETE shall indicate task was aborted or task > > was > > not in task set". > > > > The iSCSI draft is violating SAM-2 by returning "task not found" when the > > task > > was not in task set. > > > > Thanks, > > Santosh > > > > julian_satran@il.ibm.com wrote: > > > > > Well then Task does not exist will get all enough differentiation. > > > If you could not agree with reject (not enough differentiation) another > > use > > > can > > > have difficulty in accepting the other extreme - completed when a task > is > > > really not found. > > > > > > Julo > > > > > > Pierre Labat on 10/01/2001 01:00:01 > > > > > > Please respond to Pierre Labat > > > > > > To: ips@ece.cmu.edu > > > cc: > > > Subject: iSCSI : Further review feedback on latest iSCSI-02.txt draft. > > > > > > Julian > > > > > > >>>>>>>>> > > > Section 2.6. SCSI Task Management Response > > > ================================== > > > The referenced Task Tag field can be removed since the > > > "No Task Found" response error is not valid. > > > (A Target should respond with an accept for an Abort Task > > > request specifying an invalid Initator Task Tag). > > > [js]Different people have different opinions on this - even with HP - > > > talk > > > to Pierre! [/js] > > > <<<<<<<<< > > > > > > We are in agreement, it is just a point of terminology. > > > > > > Some month ago, we had a discussion on the IPS about what to do in > > > case the target doesn't find the task to abort. > > > I requested that the Abort response returns "OK" > > > (something different than Function rejected). > > > You added the value "Task not found". > > > But "Function complete" can works too. > > > > > > Regards, > > > > > > Pierre > > > > - santoshr.vcf > > > > > > > > > > > -- > ################################# > Santosh Rao > Software Design Engineer, > HP, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################# > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Mon Jan 15 15:58:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA15626 for ; Mon, 15 Jan 2001 15:58:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FItDE26974 for ips-outgoing; Mon, 15 Jan 2001 13:55:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from falcon.vixel.com (mail.vixel.com [207.115.190.210]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FIn5s26717 for ; Mon, 15 Jan 2001 13:49:10 -0500 (EST) Received: from vixel.com ([192.168.1.207]) by falcon.vixel.com (Netscape Messaging Server 4.15) with ESMTP id G77W9R00.FJJ; Mon, 15 Jan 2001 10:49:03 -0800 Message-ID: <3A634614.60199BCF@vixel.com> Date: Mon, 15 Jan 2001 10:48:52 -0800 From: "Ken Hirata" Organization: Vixel Corporation X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en,pdf MIME-Version: 1.0 To: Joshua Tseng , ips@ece.cmu.edu CC: Y P Cheng Subject: Re: iFCP as an IP Storage Work Item References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Just a nit, see embedded comment. Ken Joshua Tseng wrote: > Furthermore, as Wayland pointed out earlier, a single large FC fabric > stretched over multiple WAN links by FCIP is vulnerable to temporary > disruptions in the IP network which may trigger reconfigurations in > the FC fabric, such as reallocations of DOMAIN_ID's. Reallocations of Domain IDs will not occur without administrative intervention if the switches in the Fibre Channel Fabric comply with the Methodologies for Interconnects (FC-MI) specification. > iFCP addresses > this issue by assigning N_PORT ID's locally, eliminating the dependence > on a central addressing authority and the need for any "Class F" traffic. > The stability and scalability of a large storage network is thus improved. > > Josh -- Kenneth Hirata Vixel Corporation Irvine, CA 92618 Phone: (949) 450-6100 Email: khirata@vixel.com From owner-ips@ece.cmu.edu Mon Jan 15 15:58:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA15625 for ; Mon, 15 Jan 2001 15:58:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FIZ0B26271 for ips-outgoing; Mon, 15 Jan 2001 13:35:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcars04f.ca.nortel.com (h57s242a129n47.user.nortelnetworks.com [47.129.242.57]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FIXns26238 for ; Mon, 15 Jan 2001 13:33:49 -0500 (EST) Received: from zbl6c012.corpeast.baynetworks.com by zcars04f.ca.nortel.com; Mon, 15 Jan 2001 13:31:20 -0500 Received: from zbl6c002.corpeast.baynetworks.com ([132.245.205.52]) by zbl6c012.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id ZLK0BJ0C; Mon, 15 Jan 2001 13:31:22 -0500 Received: from nortelnetworks.com (VFIROIU3 [47.16.4.28]) by zbl6c002.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id CN99DHYR; Mon, 15 Jan 2001 13:31:19 -0500 Message-ID: <3A6340DC.637128E3@nortelnetworks.com> Date: Mon, 15 Jan 2001 13:26:36 -0500 X-Sybari-Space: 00000000 00000000 00000000 From: "Victor Firoiu" Organization: Nortel Networks X-Mailer: Mozilla 4.76 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Stephen Bailey CC: ips@ece.cmu.edu, "Franco Travostino" Subject: Re: Performance of iSCSI, FCIP and iFCP References: <10101132013.AA27978@candide.cs.uchicago.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Orig: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Steph, thanks for raising this point, which is clearly central to the fairness discussion in a more general context, not restricted to IP Storage. There are two cases to consider: whether the storage traffic is isolated or is multiplexed with other traffic in the IP network. My original message on this subject of performance was assuming the first case, where all available bandwidth (in my example 10Gb/s) was available for storage traffic, and showed that in some scenarios, about 1/4 of bandwidth was lost unnecessarily with the one-TCP solution. Definitely, no fairness issue was involved. Now let's consider the second case, where storage traffic traverses a non-exclusive IP network, such as "the Internet", where fairness is an issue. Unfortunately, as far as I know, despite the TCP friendly mandate expressed or implied in the IETF community, there is no clear definition of what is considered to be fair use of network bandwidth (TCP friendly) and what is unfair. RFC 2914, citing RFC 2309: ".. the issue of the appropriate granularity of a "flow", where we define a `flow' as the level of granularity appropriate for the application of both fairness and congestion control. From RFC 2309: "There are a few `natural' answers: 1) a TCP or UDP connection (source address/port, destination address/port); 2) a source/destination host pair; 3) a given source host or a given destination host. We would guess that the source/destination host pair gives the most appropriate granularity in many circumstances. The granularity of flows for congestion management is, at least in part, a policy question that needs to be addressed in the wider IETF community." In the context of IP storage, the IPS gateways are different than the source and destination hosts for a TCP flow in a regular IP network. For example, in the ECM (Endpoint Congestion Manager) WG, we had a discussion on this subject. The ECM proposal attempts to regulate the sending rate for a pair of endpoints to be limited to roughly one TCP connection between those two hosts, given the current network conditions (loss probability, round trip delay). I raised a few questions. For example, if an endpoint is a multi-user system, and N users from this system simultaneously connect to the same web server, under ECM they get 1/N of the bandwidth they would otherwise get if using N PCs, which is not fair. Some of us agreed that the fair unit of bandwidth should be per user and not per system, but was not put on paper. This opinion is underscored by the following from RFC 2616, also cited by RFC 2914: "A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy." In the same spirit, I would argue that the storage traffic would be entitled to one share of (TCP-friendly) bandwidth per storage connection, and not per pair of gateways, since one storage connection more closely emulates one user activity (or maybe even more that that). In other words, mandating one TCP connection per pair of IP storage gateways would be like mandating one TCP per pair of IP routers. Of course, this is my subjective opinion with a bit of common sense, in the absence of a clear definition of fair network usage. The paper I referred to in my other message does not contain any consideration to fairness, only an analytical model for TCP throughput as a function of delay and packet loss. A side implication is that different TCP implementations are not fair (friendly) to each other, but that's a long known fact. Surely, you can modify TCP congestion control algorithm to get as much bandwidth as N other TCPs, but that creates a new, "super-TCP" protocol that has to be designed, verified and approved. (Observe that super-TCP would have a different way to recover from losses that N TCPs.) Moreover, even if super-TCP would be used, the IP storage gateways would still need to be aware of the identity and number of storage connections. Your comments, as always, are highly appreciated. Victor Stephen Bailey wrote: > > Victor, > > > I have to admit that we did not have the model to support > > numerically our contention that multiple links will behave better in > > the presence of errors and performance will degrade more gracefully > > even on congestion. > > The last time we had this discussion (multiple links between the same > endpoints enabling higher performance) on this list, it was my opinion > that doing this is not `TCP friendly', and so should not be allowed in > the general, Internet context, which is one of the design points for > iSCSI. > > Clearly, when slow starting, and recovering from congestion, n > connections can effectively open the window n times as fast as 1 > connection, but doing so will potentially be at the expense of other > competing flows which don't use multiple connections. If ALL flows > use multiple connections, you simply end up hitting the congestion > wall harder, which, in an abstract sense, makes the system more likely > to oscillate (or collapse?). > > I don't know if it WILL create oscillation or collapse (I'm just a > hammer mechanic), but it seems to me that if hitting the congestion > wall harder in this way were acceptable to overall network behavior, > the single connection TCP congestion avoidance could be adjusted to > create this behavior (and capture this benefit) without requiring > multiple TCP connections. > > Your refereed, published work on this seems suggests that I am holding > on to `myths' about the need to play fair with TCP. If so, I would > greatly appreciate your debunking my myths here on this list, in the > clearest way. Particularly, we need to know if the IETF (or IESG or > IAB, or whomever it is....), will accept this type of behavior out of > iSCSI. > > While I am opposed to iSCSI's multiconnection session, I do admit its > benefits. My objections are that the feature will not be widely used, > is difficult to implement correctly, AND the same benefits are already > available through long-standing upper layer mechanisms. That aside, I > accept the WG consensus on multiconnection sessions, but what the > iSCSI spec still needs to do, in no uncertain terms, is indicate > whether multiple connections per endpoint pair is something which > implementations SHOULD or SHOULD NOT allow for use in a general, > Internet context, or at all. > > Even if iSCSI specifies that multiple connections per endpoint pair > SHOULD NOT be used in a general context, we know that implementations > are not going to prohibit it (it's primarily a configuration > decision), and that still allows the feature to be used in > specifically engineered applications, such as those built with > dedicated storage fabrics. > > Thanks, > Steph From owner-ips@ece.cmu.edu Mon Jan 15 15:59:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA15647 for ; Mon, 15 Jan 2001 15:58:56 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FIlp826677 for ips-outgoing; Mon, 15 Jan 2001 13:47:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FIkus26651 for ; Mon, 15 Jan 2001 13:46:56 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id TAA38924 for ; Mon, 15 Jan 2001 19:46:49 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id TAA116848 for ; Mon, 15 Jan 2001 19:46:49 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D5.00672784 ; Mon, 15 Jan 2001 19:46:43 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 15 Jan 2001 14:48:39 +0200 Subject: Re: iSCSI: Abort task Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, That is an interesting point. Not so much for abort task but for linked commands. For abort task we could point out that it might we advisable to keep connection allegiance but we want a way to abort tasks beyond the connection (as your argument holds for clear task set too). On the other hand we did not require connection allegiance beyond individual commands in a linked command set. Do your exchange state go beyond individual commands in a linked set? Julo Santosh Rao on 15/01/2001 09:41:46 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI: Abort task Julian, Another reason for enforcing "connection allegiance" of Abort Task and its associated task : In the case of multi-connection sessions, each connection can be using a different NIC. The SCSI command state information is maintained in SCSI hardware assists (SCSI Exchange State Tables) on the adapter and such structures need to be invalidated while aborting an I/O. Handling the Abort Task and the task on different connections implies having to then share state information across NIC instances, something that iSCSI has been trying to avoid. Regards, Santosh > > > > Pierre, > > I am not sure that I follow completely (and accurately) your line of > thought but if you have a multiple connection session the command > numbering is meant to offer a total ordering. Any of the scenarios you > mention can't happen unless the target decides to deliver commands to the > "SCSI layer" out of order. > And even if it does deliver command out of order (there might be good > reasons for it) it should build a "sync barrier" for those commands that > mandate ordering. This type of problem exist also in FCP (on a single > connection!) and I assume that the thinking there is to use the per LU > command ordering to avoid task management PDUs (like clear a queue) > arriving before the commands they are referring to. > > I hope this clarifies the issue. > > Julo > > Pierre Labat on 09/01/2001 11:39:53 > > Please respond to Pierre Labat > > To: ips@ece.cmu.edu > cc: > Subject: iSCSI: Abort task > > > > > Julian, > > > It seems that we need to specify more the "Abort task" to avoid > some problems. Below are 3 points we should add in the spec: > > > 1) An "Abort task" must be sent on the TCP connection that was used > to send the command. This does not apply if the connection used > for the command has been "logged out". In this case of course, the > "Abort task" can be send on another connection and it will not hurt. > > It is to avoid that a command that was delayed in the network, finally > make it > after the "Abort task" as been processed. > > 2) When the initiator receives the "Abort task" response, if the > ExpCmdRN returned in the "Abort task" response is > <= CmdRN of the aborted task, the initiator must re-use this > CmdRN for another task (any). > > 3) The target when receiving an "Abort task" for a task whose the > CmdRN (retreived from the initiator task tag) is >= ExpCmdRN > must not bridge this CmdRN. I mean by bridge: allow ExpCmdRN to > pass the CmdRN value of the aborted task. It will be up to the > initiator > to send another command with the same CmdRN in order to allow > the target to move ExpCmdRN over the CmdRN value of the > aborted task. > > > > "3)" Is to prevent to following scenario: > > Two TCP connections 1 and 2 in the session > ExpCmdRN = 2 > > a) the command with CmdRN = 3 is sent over connection1 > > b) the command with CmdRN = 4 is sent over connection 2 > > c) the command CmdRN = 4 reaches the target > > d) the initiator aborts the command 4 and bridges CmdRN=4 > > e) the target receives the command 3, and as CmdRN=4 is bridged, > advances ExpCmdRN to 5 > > f) the initiator sends another command (not related with the aborted > one) with CmdRN=4 > > g) the target drops this command because CmdRN is less than ExpCmdRN. > > > > "2)" needs to exist because if the initiator doesn't re-use the CmdRN > the target > will deadlock because ExpCmdRN will be stuck behind the CmdRN of > the aborted command. > > > > An alternative to the points 2) and 3) would be the initiator not > re-using > the CmdRN of the aborted commands. But this requires the target to > bridge > all the CmdRNs of the Aborted tasks (need extra states) and anyway there > > is a scenario where it doesn't work. Hence this is a bad alternative. > > Scenario where it doesn't work: > > Session with 2 connections 1 and 2. > ExpCmdRN=1 > > a) Command with CmdRN=4 is sent over connection1 > > b) Connection 1 drops and the command 4 will never make it to the > target. > > c) A logout for the connection 1 is sent over connection 2 > > d) The initiator sends an "Abort task" (for the task CmdRN=4) over > the connection 2 > > e) The target, from the referenced initiator task tag cannot find the > CmdRN of the aborted task because the initiator task tag > corresponds > to nothing because the command has never been received. > Hence the target is not able to bridge CmdRN=4 > > f) As the initiator does not re-use CmdRN=4 the target will deadlock > when ExpCmdRN will be 4. > > > Regards, > > Pierre > > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Mon Jan 15 15:59:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA15661 for ; Mon, 15 Jan 2001 15:59:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FImxG26715 for ips-outgoing; Mon, 15 Jan 2001 13:48:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FIlos26674 for ; Mon, 15 Jan 2001 13:47:50 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id TAA24880 for ; Mon, 15 Jan 2001 19:47:44 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id TAA40484 for ; Mon, 15 Jan 2001 19:46:27 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D5.00671F34 ; Mon, 15 Jan 2001 19:46:21 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 15 Jan 2001 13:08:26 +0200 Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, OK - you have convinced me (and I've found the CAM3 list of statuses). I will specify in 04 the relevant values for the service response in line with CAM3. Julo Santosh Rao on 15/01/2001 00:34:36 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. > It would be simple if things where that clear-cut. > > What is a format error coming from a target? IMHO it is a target error and > not a protocol failure. Julian, A SCSI Protocol(i.e. iSCSI) header format error constitutes a "Service Delivery or Target Failure" service response, since the failure occurs in the service delivery sub-system (SCSI Protocol). SAM-2 explains this service response as "The command has ended due to a service delivery failure or target device mal-function", which should be the service response to be returned on a header format error. Further, SAM-2 Section 5 also states that the application client (i.e. the SCSI upper layer driver) shall treat the SCSI Status to be un-defined if the command ends with a service response of "Service Delivery or Target Failure". This implies that the SCSI Upper Layer driver shall ignore the CHECK CONDITION scsi status that iSCSI initiators are going to fake to their upper layers on a header format error, due to a service response of "Service Delivery or Target Mal-function". So, is iSCSI proposing to return a service response of "Task Complete" to its upper layers on a header format error ? If not and it is the intent that a service response of "Service Delivery or Target Mal-function" should be returned, then, the upper layer scsi driver is going to ignore the CHECK CONDITION that iSCSI intends to fake. It would be interesting to hear what T10 has to say on this approach that iSCSI is currently proposing to adopt on a header format error. I see the following problems with the current approach being advocated : ------------------------------------------------------------------------ 1) It does NOT solve the problem for commands other than the SCSI Command PDU. I am yet to see a response to my questions on how header format errors for Login, Logout, Text, NOP-OUT, NOP-IN and SCSI Task Management Command PDU are handled by the current proposal described in section 5.4 of the 03 iSCSI draft. 2) It is not in line with [violates ?] Section 5 of SAM-2. 3) It can cause upper layers to initiate Auto Contingent Allegiance recovery such as CLEAR ACA due to the CHECK CONDITION returned. 4) It can cause upper layers to over-react on seeing a HARDWARE ERROR by resorting to error recovery such as BDR to recover from a HARDWARE ERROR. 5) It adds extra complexity to the iSCSI initiator drivers by making them SPC-2 aware and having to generate sense data on behalf of a target, not something that has been required in other SCSI Protocols such as parallel scsi and fibre channel. 6) It causes a mis-leading error log of a HARDWARE ERROR, where, a more specific error log of the header format error that occurred based on the response data would have been more useful in quick fault isolation. 7) Last, but not the least, it is a violation of layering between the ULP and LLP, wherein, an LLP Service Delivery Failure is being treated as a ULP SCSI error returned by the LUN. I would like to propose the following solution : ------------------------------------------------ 1) On discovery of header format error, a target MUST convey the specific type of format error that was discovered through use of mechanisms like the Response Data in a SCSI Response PDU, Response Field in a SCSI Task Management Response PDU and the Reason Code field in the REJECT PDU. 2) On a header format error discovered at the initiator, a service response of "Service Delivery or Target Failure" MUST be returned to their upper layers [and the initiator may log the specific header format error that was discovered.]. The draft need not attempt to elaborate on these service response definitions since these are defined by the SCSI Stack of each O.S. as a set of return values exchanged b/n the LLP and the ULP. Benefits of the proposed solution : ----------------------------------- 1) It addresses the problem for header format errors on ALL types of PDUs. 2) It is in line with [complies with ?] SAM-2 semantics of "Execute Command". 3) It provides more value-added error logging based on the specific header format error that occurred rather than a more general HARDWARE ERROR, allowing for quicker isolation and root cause of the problem. 4) It avoids ULPs resorting to inappropriate error recovery such as CLEAR ACA or a BDR on seeing CHECK CONDITION with HARDWARE ERROR. 5) It retains layering semantics and differentiates between LLP service delivery errors and ULP SCSI errors. I'd be happy to learn what implications of this issue I've missed. Thanks & Regards, Santosh Rao > Should target errors be reported in the service-response? Except for task > management that is common to all protocols I did not see any other thing > popping up in any SCSI driver. > > Preaching layering won't make the issue disappear. > > Julo > > Stephen Bailey on 14/01/2001 17:10:56 > > Please respond to Stephen Bailey > > To: ips@ece.cmu.edu > cc: > Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. > > > > > Julian, > > This seems like the zillionth time aired this same disagreement. > > I think we should try to reach a WG consensus on whether iSCSI should > use SCSI status as a means for reporting protocol-related errors, and > kill it once and for all. > > I'm strongly against it. > > > And an error in the iSCSI layer gets reported by the next layer - that is > > the regular layering technique (and BTW I am getting a bit uneasy about > all > > this preaching on layering when it is not obvious that you understand all > > the implications of the point). > > Santosh seems to understand 100%. I agree with Santosh. > > SAM defines three pieces of status returned by Execute Command() > (which is, in turn implemented by each SCSI protocol, e.g. iSCSI): > 1) Service Response: task complete, linked command complete, service > delivery or target failure. > 2) SCSI status byte > 3) SCSI sense data > > SAM clearly suggest that a protocol's means for signalling > protocol-detected errors is the service response status when it says: > > The actual protocol events corresponding to a response of TASK > COMPLETE, LINKED COMMAND COMPLETE or SERVICE DELIVERY OR TARGET > FAILURE shall be specified in each protocol standard. > > Note that defining protocol error events in terms of TC, LCC, SDF and > TF, remains an abstraction. An actual implementation can chose to > signal these events by whatever means. All SCSI implementations I've > seen do have a status return component which exactly corresponds to > SAM's service response status, but has more than just these four > alternatives. Typically, that includes things like success, command > timeout, addressing failure (selection timeout in ||SCSI, bad AL_PA in > FCAL, etc.), command aborted, bus parity error, etc.. What iSCSI does > need to do is clearly define its error events AS protocol events, > which is what describing them in terms of the SAM specified set does. > > ||SCSI, FCP and the other SCSI protocol standards do this. > > Operationally, this means that in iSCSI: > > o protocol-specific errors for a task detected by the target without > a CLEARLY corresponding SCSI error return should be signalled > using the iSCSI response mechanism. > > The initiator will handle these errors by recording them in some > appropriate way, and selecting an appropriate service response > status value. > > o errors for a task detected directly by the initiator are handled > by recording them in some appropriate way, and selecting an > appropriate service response status value. > > I don't know that there's a single sentence or section in SAM which > says this, but it clearly implies that the components of SCSI status > (status byte and sense) are data which are CARRIED (not created) by > SCSI protocols, for the use of the protocol-independent components. > For example, a disk peripheral driver reacts to SCSI status and sense > generated by disks for SCSI operations that it starts. > > SCSI status should be equivalent to the SCSI status returned by the > logical unit. > > SCSI protocols are not citizens of the SCSI status space, which means > that the real citizens (the command standards, and SAM), may define > semantics of SCSI error codes which conflict with iSCSI's selections. > The `hardware error' sense key might be defined to mean something very > specific within a particular SCSI peripheral command set, and your > choice of synthesizing SCSI status within the protocol could cause > this behavior to misfire. > > > [js} the error was generated by a faulty controller and I did not find > > any other SCSI sense fit for it[/js] > > That's because there IS no SCSI sense fit for it. It is a > protocol-detected, protocol-unique error. Therefore it should be > signalled using the service response status. > > Steph > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Mon Jan 15 16:42:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA16245 for ; Mon, 15 Jan 2001 16:42:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FJmtx28826 for ips-outgoing; Mon, 15 Jan 2001 14:48:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FJm5s28794 for ; Mon, 15 Jan 2001 14:48:05 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 15 Jan 2001 14:47:59 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101422@corpmx9.isus.emc.com> From: Black_David@emc.com To: psarkar@almaden.ibm.com, ips@ece.cmu.edu Subject: Posting Drafts to the list Date: Mon, 15 Jan 2001 14:47:48 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk For future reference: DO NOT POST internet drafts to the mailing list. Send them to internet-drafts@ietf.org . If time is of the essence, post the draft to a web site and send the URL to the mailing list. Those who occasionally (or not so occasionally) have to access email through a modem will appreciate your concern. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Jan 15 16:48:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA16313 for ; Mon, 15 Jan 2001 16:48:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FJqA228962 for ips-outgoing; Mon, 15 Jan 2001 14:52:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FJpGs28928 for ; Mon, 15 Jan 2001 14:51:16 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0FL2I036166 for ; Mon, 15 Jan 2001 13:02:18 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: Subject: RE:draft-ietf-ips-iscsi-boot-01.txt Date: Mon, 15 Jan 2001 11:50:06 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <200101151143.GAA00827@ietf.org> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit All, The field consists of an UTF-8[20] string and has the following composition: ":" ":" ":" Would it be possible to adopt language explaining that servername and targetname could be dotted decimal expansions of the IP address in place of the name. DNS code for a bootstrap adds complexity. It would make it easier if perhaps some symbols could be used to indicate such an expansion has been made such as .IP6.INT or .IP4.INT suffix where the bytes are in reverse order with no embedded spaces. Doug > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-boot-01.txt From owner-ips@ece.cmu.edu Mon Jan 15 17:40:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA17001 for ; Mon, 15 Jan 2001 17:40:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FKNrh00096 for ips-outgoing; Mon, 15 Jan 2001 15:23:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FKMqs00069 for ; Mon, 15 Jan 2001 15:22:53 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id VAA178748 for ; Mon, 15 Jan 2001 21:22:48 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id VAA175936 for ; Mon, 15 Jan 2001 21:22:48 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D5.006FEF90 ; Mon, 15 Jan 2001 21:22:38 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu cc: Dafna_Sheinwald@il.ibm.com Message-ID: Date: Mon, 15 Jan 2001 22:18:08 +0200 Subject: A memo on some checksums Mime-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=gotZChCsQuE1Iy8cO0Ka46Mhcah9aNMOVQuJiR1OCrJOudBknnZPMSoF" Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=gotZChCsQuE1Iy8cO0Ka46Mhcah9aNMOVQuJiR1OCrJOudBknnZPMSoF Content-type: text/plain; charset=us-ascii Content-Disposition: inline As several people from this list have asked us to look at various alternatives to the TCP checksum and or cheaper alternatives for the CRC digests - Dr. Dafna Sheinwald from the IBM Haifa Research Lab. took a look at 2 that gained some popularity lately and here is a preliminary report. (See attached file: Dafna-Sheinwald-codes.wri) As for alternative CRCs two classes of checks have to be performed on them before we can really consider them: test of goodnes (there several well known rules (look at Andrew Tannenbaum - Computer Network, or Richard Wells - Applied Coding and Information Theory for Engineers or one of the books that appear in Dafna's list - e.g., the classical Peterson & Weldon) a computer scan to show how "distant" good generated codes are As up to 32 bit we have several good polynomials and the "nesting" does not reduce their effectiveness I am not going to look for something else but we should be ready to accept any proposal that comes with the right credentials. On 64 bit we are going to spend some time as there is no agreed and tested polynomial available or at least I was not able to get to one. Quantum advertises using one for the DLT tapes but I have no clue what it is. If somebody on this list is aware of anything relevant in this area (even patent protected - at least we should know what to avoid if we can -:) please speak up. Testing a new polynomial is not a trivial effort. Julo --0__=gotZChCsQuE1Iy8cO0Ka46Mhcah9aNMOVQuJiR1OCrJOudBknnZPMSoF Content-type: application/msword; name="Dafna-Sheinwald-codes.wri" Content-Disposition: attachment; filename="Dafna-Sheinwald-codes.wri" Content-Description: Word 6.0 Windows/Mac Content-Transfer-Encoding: base64 0M8R4KGxGuEAAAAAAAAAAAAAAAAAAAAAPgADAP7/CQAGAAAAAAAAAAAAAAABAAAAAQAAAAAAAAAA EAAAAgAAAAEAAAD+////AAAAAAAAAAD///////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// ///////////////////////////////////////////////////////////////////////////9 /////v////7///8EAAAABQAAAAYAAAAHAAAACAAAAAkAAAAKAAAA/v///wwAAAANAAAADgAAAA8A AAAQAAAAEQAAABIAAAATAAAAAwAAAP7///////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// /////////////////////////////////////////////////////////////////////////1IA bwBvAHQAIABFAG4AdAByAHkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAWAAUA//////////8BAAAAAAkCAAAAAADAAAAAAAAARgAAAAAAAAAAAAAAAMBR9hwrf8AB FAAAAIAAAAAAAAAAVwBvAHIAZABEAG8AYwB1AG0AZQBuAHQAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAABoAAgECAAAA//////////8AAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAALAAAABiEAAAAAAAABAEMAbwBtAHAATwBiAGoAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEgACAP///////////////wAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABuAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//////// ////////AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAA AP7///////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////9wIDEx LTEzLCBKYW51YXJ5IDE5OTQuDVszXSBSRkMxMTQ2OiBUQ1AgQWx0ZXJuYXRlIENoZWNrc3VtIE9w dGlvbnMsICBhdCAgICBodHRwOi8vcmZjLm5ldC9yZmMxMTQ2Lmh0bWwNWzRdIFJGQzExNTA6IFpM SUIgQ29tcHJlc3NlZCBEYXRhIEZvcm1hdCBTcGVjaWZpY2F0aW9uIHZlcnNpb24gMy4zLCAgYXQg ICAgaHR0cDovL3JmYy5uZXQvcmZjMTE1MC5odG1sDVs1XSAiQ3ljbGljIFJlZHVuZGFuY3kgQ2hl Y2tpbmcgZm9yIEV0aGVybmV0IiAgYXQgaHR0cDovL2xldi55dWRhbGV2aWNoLnRyaXBvZC5jb20v RUNDL2NyYy5odG1sDVs2XSAiRmFzdCBDUkMzMiBpbiBTb2Z0d2FyZSIgIGJ5IFJpY2hhcmQgQmxh Y2ssIDE5OTQsIGF0ICB3d3cuY2wuY2FtLmFjLnVrL1Jlc2VhcmNoL1NSRy9ibHVlYm9vay8yMS9j cmMvY3JjLmh0bWwgDVs3XSAiTmFzYSBGSVRTIGRvY3VtZW50cyIgIGF0ICBodHRwOi8vaGVhc2Fy Yy5nc2ZjLm5hc2EuZ292L2RvY3MvaGVhc2FyYy9vZndnL2RvY3MvZ2VuZXJhbC9jaGVja3N1bS9u b2RlMjYuaHRtbA1bOF0gIkluZm9ybWF0aW9uIG9uIFByaW1pdGl2ZSBhbmQgSXJyZWR1Y2libGUg UG9seW5vbWlhbHMiIGF0IGh0dHA6Ly93d3cudGhlb3J5LmNzYy51dmljLmNhL35jb3MvaW5mL25l Y2svUG9seUluZm8uaHRtbA0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/AIAACoD AAArAwAALAMAAHcDAAB4AwAAGgQAABsEAABxBAAAcgQAAJ4EAAChBAAA8gQAAPMEAABJBQAASgUA AEsFAACDBQAAhAUAAKIGAACnBgAAdAcAAHkHAADCCAAAwwgAAH0JAAB+CQAAfwkAALoJAAC7CQAA qgoAAKsKAACsCgAAyAoAAMkKAAArCwAA+fPt5+Hd2dXRzcjEwLy4s66ppaCcl5OPi4WCf3x4cmxm YFwAAAAAAAddAgBxAgIAC1UBXQIAXgFxAgIAC1UBXQIAXgFxAgIAC1UBXQIAXgFxAgIAC1UBXQIA XgFxAgIAB10CAHECAgAFXQMAXgEFXQMAXgEFXQMAXgELVQFdAgBeAXECAgAHXQIAcQICAAddAgBx AgIAB10CAHECAgAJVQFdAgBxAgIAB10CAHECAgAJVQFdAgBxAgIAB10CAHECAgAJXQIAXgFxAgIA CV0CAF4BcQICAAldAgBeAXECAgAHXQIAcQICAAddAgBxAgIAB10CAHECAgAHXQIAcQICAAlVAV0C AHECAgAHXQIAcQICAAddAgBxAgIAB10CAHECAgAHXQIAcQICAAddAgBxAgIAC1UBXQIAXgFxAgIA C1UBXQIAXgFxAgIAC1UBXQIAXgFxAgIAC1UBXQIAXgFxAgIAC1UBXQIAXgFxAgIAACMrCwAALAsA AGoMAABrDAAAiwwAAJEMAABYDQAAWQ0AAOMNAADkDQAA5Q0AAPMNAAD0DQAACw4AABAOAABYDgAA WQ4AAMMOAADEDgAAxQ4AALAQAACxEAAAshAAABkRAAAaEQAAGxEAACcRAAAoEQAALBEAAG8RAABw EQAAdBEAABYSAAAXEgAAGxIAAGYSAABnEgAA+fXx7ejk4NzW0szGwr25tbGtqaWhnZmVkYyHgn56 dXFtaGRgAAAAAAAAAAAHXQIAcQICAAddAgBxAgIACVUBXQIAcQICAAddAgBxAgIAB10CAHECAgAJ VQFdAgBxAgIAB10CAHECAgAHXQIAcQICAAlVAV0CAHECAgAJVQFdAgBxAgIACVUBXQIAcQICAAdd AgBxAgIAB10CAHECAgAHXQIAcQICAAddAgBxAgIAB10CAHECAgAHXQIAcQICAAddAgBxAgIAB10C AHECAgAHXQIAcQICAAddAgBxAgIAB10CAHECAgAJVQFdAgBxAgIAB10CAHECAgALVQFdAgBeAXEC AgALVQFdAgBeAXECAgAHXQIAcQICAAtVAV0CAF4BcQICAAddAgBxAgIAB10CAHECAgAHXQIAcQIC AAlVAV0CAHECAgAHXQIAcQICAAddAgBxAgIAB10CAHECAgALVQFdAgBeAXECAgAAJGcSAABrEgAA chIAAHQSAAB4EgAAeRIAAM0SAADOEgAA0hIAAC0TAAAuEwAAMhMAAGgTAACfEwAAoBMAAKMTAAAP FAAAEBQAABMUAAAVFAAAThQAAIcUAACIFAAA/Pr39fLw7Ofj39rW0czHw7+6trSwrAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB10CAHEC AgAHXQIAcQICAANdAwAHXQIAcQICAAlVAV0CAHECAgAHXQIAcQICAAddAgBxAgIACVUBXQIAcQIC AAldAwBiAXMCAQAJXQMAYgFzAgEAB10CAHECAgAJVQFdAgBxAgIAB10CAHECAgAHXQIAcQICAAlV AV0CAHECAgAHXQIAcQICAANdAwAFVQFdAwADXQMABVUBXQMAA10DAAVVAV0DAAAW/AIAACsDAAAs AwAAeAMAABsEAAByBAAA8wQAAEoFAABLBQAAhAUAAMMIAAB+CQAAfwkAALsJAACrCgAArAoAAMkK AAAsCwAAawwAAFkNAADkDQAA5Q0AAPQNAABZDgAAxA4AAPkAAAAAAAD0AAAAAAAA7wAAAAAAAOoA AAAAAADlAAAAAAAA4AAAAAAAANsAAAAAAADWAAAAAAAA0QAAAAAAAMwAAAAAAADHAAAAAAAAwgAA AAAAAL0AAAAAAAC4AAAAAAAAswAAAAAAAK4AAAAAAACpAAAAAAAApAAAAAAAAJ8AAAAAAACaAAAA AAAAlQAAAAAAAJAAAAAAAACLAAAAAAAAhgAAAAAAAAQAABTwAAEAAAAEAAAU8AABAAAABAAAFPAA AQAAAAQAABTwAAEAAAAEAAAU8AABAAAABAAAFPAAAQAAAAQAABTwAAEAAAAEAAAU8AABAAAABAAA FPAAAQAAAAQAABTwAAEAAAAEAAAU8AABAAAABAAAFPAAAQAAAAQAABTwAAEAAAAEAAAU8AABAAAA BAAAFPAAAQAAAAQAABTwAAEAAAAEAAAU8AABAAAABAAAFPAAAQAAAAQAABTwAAEAAAAEAAAU8AAB AAAABAAAFPAAAQAAAAQAABTwAAEAAAAEAAAU8AABAAAABQAABQEU8AABAAAAABjEDgAAxQ4AALEQ AACyEAAAGhEAABsRAAAoEQAAcBEAABcSAABnEgAAzhIAAC4TAACgEwAAEBQAAIgUAAD6AAAAAAAA 9QAAAAAAAPAAAAAAAADrAAAAAAAA5gAAAAAAAOEAAAAAAADcAAAAAAAA1wAAAAAAANIAAAAAAADN AAAAAAAAyAAAAAAAAMMAAAAAAAC+AAAAAAAAuQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAABAAAFPAAAQAAAAQAABTwAAEAAAAEAAAU8AABAAAABAAAFPAAAQAA AAQAABTwAAEAAAAEAAAU8AABAAAABAAAFPAAAQAAAAQAABTwAAEAAAAEAAAU8AABAAAABAAAFPAA AQAAAAQAABTwAAEAAAAEAAAU8AABAAAABAAAFPAAAQAAAAQAABTwAAEAAAAADvwCAACIFAAACwD8 AgAAiBQAAA4AAAAAAIwRAAAAAP////8AAP////8sHQAADgAPAAgAAQBLAA8AAAAAABoAAEDx/wIA GgAGTm9ybWFsAAIAAAADAGEJBAAAAAAAAAAAAAAAAAAAAAAAAAAiAEFA8v+hACIAFkRlZmF1bHQg UGFyYWdyYXBoIEZvbnQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAANACAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AP9AFAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA3KVoACPACQQAAAAA AAAAAAAAAAAAAAAA/AIAAIgUAAAGIQAAAAAAAAAAAAAAAAAAAAAAAIwRAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAACwgAABsAAAALCAAAGwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAABQgAAAUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAgAAAKAAAACiAAAAoAAAAAAAAAAAAAAKoCAABSAAAAKCAAAAQAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPAgAAACAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAmCAAAFgAAADyIAAAFAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAsADgADAAIAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFIAEw54AAByTVMgU2Fu cyBTZXJpZgAPHngAAnJNVCBTeW1ib2wAFR54AAByVGltZXMgTmV3IFJvbWFuABUeeAAAclRpbWVz IE5ldyBSb21hbgBPbiBGbGV0Y2hlciBhbmQgQWRsZXIgY29kZXMsIGFuZCBjbGFzc2ljIENSQy1z DQ1GbGV0Y2hlciBDb2RlIFszXSBhbmQgQWRsZXIgQ29kZSBbNF0gZGV0ZWN0IG11Y2ggbGVzcyB0 aGFuIENSQy0zMiBjb2RlcyBbNV0NRmxldGNoZXIgYW5kIEFkbGVyIGNvZGUgc2VlbSB0byBsYWNr IGEgdGhlb3JldGljYWwgZ3JvdW5kLCBhbmQgdGhlaXIgZXJyb3IgZGV0ZWN0aW9uIGNhcGFiaWxp dHkgaXMgaW5mZXJpb3IgdG8gdGhhdCBhdHRhaW5hYmxlIGJ5IENSQyBvZiB0aGUgc2FtZSBudW1i ZXIgb2YgYml0cy4gDUNSQy0zMiwgQWRsZXItMzIsIGFuZCBGbGV0Y2hlci0zMiAoMioxNiksIGFs bCBhcHBlbmQgMzIgYml0cyB0byB0aGUgaW5mb3JtYXRpb24gZGF0YS4gDVdpdGggdGhpcyBudW1i ZXIgb2YgYml0cywgQ1JDLTMyIGNhbiBkZXRlY3QgYW55IHNpbmdsZSBidXJzdCBvZiB1cCB0byAz MiBiaXRzLCBvbiBhIHN0cmVhbSBvZiBkYXRhIG9mIGFueSBsZW5ndGggWzFdLCBbMl0sWzVdLiAg DUhlcmUgYXJlIGV4YW1wbGVzIGZvciByZWxhdGl2ZWx5IHNob3J0IGJ1cnN0cyB0aGF0IEFkbGVy IGFuZCBGbGV0Y2hlciBjYW4gbm90IGRldGVjdC4gDQ1BIDI0LWJpdCBidXJzdCBlcnJvciB3aGlj aCBBZGxlci0zMiBjb2RlIGNhbiBub3QgZGV0ZWN0Og1SZWNhbGwgdGhhdCBBZGxlci0zMiBydW5z IHR3byBzdW1zOiBzMSBhbmQgczIgWzRdLiBTdXBwb3NlIHRoYXQgYXQgc29tZSBwb2ludCBvbiB0 aGUgZGF0YSBzdHJlYW0gczE9YSBhbmQgczI9Yi4gU3VwcG9zZSB0aGF0IGJvdGggYSBhbmQgYiBh cmUgd2F5IHNtYWxsZXIgdGhhbiA2NTUyMSwgdGhlIHZhbHVlIGNhbGxlZCAiQkFTRSIgYXQgWzRd LiBBbHNvIHN1cHBvc2UgdGhhdCBhdCB0aGF0IHBvaW50IHRoZSBvcmlnaW5hbCBkYXRhIHN0cmVh bSBjb250aW51ZXMgd2l0aCBieXRlcyBvZiB2YWx1ZXMgNCwyLDEuIEF0IHRoZSBlbmQgb2YgdGhl c2UgdGhyZWUgYnl0ZXMsIHRoZSB2YWx1ZXMgb2YgczEgZ3Jvd3MgdG8gYSs0KzIrMSA9IGErNywg d2hlcmVhcyBzMiBncm93cyB0byBiK2ErNCthKzYrYSs3ID0gYiszYSsxNy4gIE5vdywgc3VwcG9z ZSB0aGF0IGEgMjQgYnVyc3Qgb2NjdXJyZWQgb24gdGhlc2UgdGhyZWUgYnl0ZXMsIHdoaWNoIG1v ZGlmaWVkIHRoZW0gdG8gNSwwLDIuIE5vdywgd2hlbiBkb2luZyB0aGUgZGV0ZWN0aW9uLCBzMT1h IGFuZCBzMj1iIGp1c3QgYmVmb3JlIHRoZXNlIHRocmVlIGJ5dGVzLCBhbmQgb24gY29tcGxldGlv biBvZiB0aGVpciBwcm9jZXNzaW5nLCBzMT1hKzUrMCsyID0gYSs3LCAgYW5kIHMyPWIrYSs1K2Er NSthKzc9YiszYSsxNy4gRGV0ZWN0aW5nIGZyb20gdGhhdCBwb2ludCBhbmQgb24gdG8gdGhlIGVu ZCBvZiB0aGUgZGF0YSBzdHJlYW0sIHMxIGFuZCBzMiB3aWxsIHRyYWNlIHRoZSB2ZXJ5IHNhbWUg dmFsdWVzIHRoZXkgaGFkIG9uIGVuY29kaW5nLCBhbmQgdGh1cyB0aGUgYnVyc3QgaXMgbm90IGRl dGVjdGVkLg1UaGlzIG9jY3VycyBmb3IgZXZlcnkgdGhyZWUgY29uc2VjdXRpdmUgYnl0ZXMgeCx5 LHogd2hpY2ggYXJlIG1vZGlmaWVkIHRvIHgnLHknLCBhbmQgeicsIHN1Y2ggdGhhdCAyeCt5ID0g MngnK3knLCBhbmQgeiBhbmQgeicgYXJlIGFueSB0d28gbnVtYmVycyBzdWNoIHRoYXQgeicteiA9 IHgreS14Jy15Jy4gVG9vIGxpa2VseS4NDUEgMTYtYml0IGJ1cnN0IGVycm9yIHdoaWNoIEZsZXRj aGVyLTMyIGNvZGUgY2FuIG5vdCBkZXRlY3Q6DUJlY2F1c2UgRmxldGNoZXIgZG9lcyAxJ3MgY29t cGxlbWVudCBjYWxjdWxhdGlvbnMsIHRoZSBhZGRpdGlvbiBvZiAweDAwMDAgdG8gYW55IG51bWJl ciBvdGhlciB0aGFuIDAgeWllbGRzIHRoZSBzYW1lIHJlc3VsdCBhcyBhZGRpbmcgb2YgMHhGRkZG LiBUaHVzLCBGbGV0Y2hlciBjb2RlIGNhbiBub3QgZGV0ZWN0IHR3byBjb25zZWN1dGl2ZSBieXRl cyB3aGljaCB0dXJuZWQgYm90aCBmcm9tIDB4MDAgdG8gMHhGRiAuDQ1GYXN0IFNvZnR3YXJlIGlt cGxlbWVudGF0aW9uDVRoZSBjZWxlYnJhdGVkIHByb3BlcnR5IG9mIGJvdGggRmxldGNoZXIgYW5k IEFkbGVyIGlzIHRoZSBzcGVlZCBvZiB0aGVpciBzb2Z0d2FyZSBpbXBsZW1lbnRhdGlvbnMuDU5l dmVydGhlbGVzcywgdGhlcmUgYXJlIGtub3duIHRlY2huaXF1ZXMgZm9yIHByb2dyYW1pbmcgQ1JD LTMyIFs2XSwgc3VjaCB0aGF0IHRoZSBjb2RpbmcgKGFzIHdlbGwgYXMgdGhlIGRlY29kaW5nKSBv ZiBlYWNoIGJ5dGUgY2FsbHMgZm9yIG9uZSB0YWJsZSBsb29rIHVwLCBpbiBhIHRhYmxlIG9mIDI1 NiAzMi1iaXQgd29yZHMsIG9uZSBBTkQgb3BlcmF0aW9uICh3aGljaCBjYW4gYmUgYXZvaWRlZCBp biBtYWNoaW5lIGNvZGUgLS0gdGFraW5nIEFMIGZyb20gdGhlIHdob2xlIG9mIEVBWCwgc2F5KSwg b25lIFhPUiwgYW5kIG9uZSA4LWJpdCBzaGlmdC4gIA1BZGxlciBbNF0gZGVtYW5kcyB0aGUgZXhw ZW5zaXZlIG1vZHVsbyBvcGVyYXRpb24gd2l0aCBhIHZlcnkgbGFyZSBwcmltZSwgZm9yIHRoZSBw cm9jZXNzaW5nIG9mIGVhY2ggYnl0ZSwgb3Igc29tZSB0ZXN0IHRvIGluZGljYXRlIHRoZSBuZWNl c3NpdHkgaW4gdGhhdCBvcGVyYXRpb24uIEZsZXRjaGVyIFs3XSBvbmx5IGRlbWFuZHMgYW4gYWRk aXRpb24sIGFuZCBjYW4gd29yayBvbiAxNi1iaXQgYXQgYSB0aW1lLiANQ1JDLTMyIHRvbyBjYW4g YmUgd29ya2VkIDE2LWJpdCBhdCBhIHRpbWUsIGJ1dCB0aGlzIHdvdWxkIG1lYW4gdXNpbmcgYSB0 YWJsZSBvZiAyXjE2PTY0SyBlbnRyaWVzIG9mIDMyLWJpdCBlYWNoLCB3aGljaCBtaWdodCBodXJ0 IGNhY2hpbmcuDQ1Hb29kIENSQyBjb2Rlcw1FdmVyeSBDUkMgY29kZSBkZXRlY3RzIGV2ZXJ5IG9u ZSBidXJzdCBlcnJvciB0aGF0IGlzIG5vdCBsb25nZXIgdGhhbiB0aGUgbnVtYmVyIG9mIGJpdHMg b2YgdGhlIENSQy4gDVdpdGggcHJvYmFiaWxpdHkgMl4tciAod2hlcmUgciBpcyB0aGUgbnVtYmVy IG9mIGJpdHMgb3IgdGhlIENSQykgaXQgZmFpbHMgdG8gZGV0ZWN0IGEgYnVyc3QgbG9uZ2VyIHRo YW4gci4NDVdvbGYgWzJdIHByZXNlbnRzIGFuIGludGVyZXN0aW5nIGRlZmluaXRpb24gb2YgYSBi dXJzdCBlcnJvciwgd2hlcmUgdGhlIGJ1cnN0IGlzIHBhcmFtZXRlcml6ZWQgYnkgYm90aCAtIGl0 cyB3aWR0aCBhbmQgdGhlIHByb2JhYmlsaXR5IG9mIGludmVydGluZyBhIGJpdCB3aXRoaW4gdGhh dCB3aWR0aC4gV2l0aCB0aGlzIGRlZmluaXRpb24sIHRoZSBwcm9iYWJpbGl0eSBvZiBkZXRlY3Rp bmcgYSBidXJzdCBlcnJvciB0aGF0IGlzIHdpZGVyIHRoYW4gdGhlIG51bWJlciBvZiBDUkMgYml0 cyBkZXBlbmRzIG9uIHRoZSBzcGVjaWZpYyBnZW5lcmF0b3IgcG9seW5vbWlhbCBwaWNrZWQsIGFu ZCBub3Qgb25seSBvbiBpdHMgZGVncmVlICg9bnVtYmVyIG9mIENSQyBiaXRzKS4gV29sZiBmb3Vu ZCB0aGF0IGdvb2QgZ2VuZXJhdGluZyBwb2x5bm9taWFscyBhcmUgb2YgdGhlIGZvcm0gKDEreClw KHgpIHdoZXJlIHAoeCkgaXMgYSBwcmltaXRpdmUgcG9seW5vbWlhbC4gDQ1Mb29raW5nIGF0IGdv b2QgQ1JDLTY0IGNvZGVzLCB3ZSB3b3VsZCB0aHVzIHdhbnQgdG8gaW52ZXN0aWdhdGUgcHJpbWl0 aXZlIHBvbHlub21pYWxzIG9mIGRlZ3JlZSA2MyBbOF0uDQ1CaWJsaW9ncmFwaHkNWzFdIEVycm9y IENvcnJlY3RpbmcgQ29kZXMgIGJ5IFBldGVyc29uIGFuZCBXZWxkb24sIHRoZSBNSVQgUHJlc3Ms IDE5NjENWzJdICJUaGUgc2luZ2xlIGJ1cnN0IGVycm9yIGRldGVjdGlvbiBwZXJmb3JtYW5jZSBv ZiBiaW5hcnkgY3ljbGljIGNvZGVzIiBieSBKYWNrIFdvbGYgYW5kIERleHRlciBDaHVuLCBJRUVF IFRyYW5zLiBvbiBDb21tdW5pY2F0aW9ucywgVm9sIDQyLCBwAQD+/wMKAAD/////AAkCAAAAAADA AAAAAAAARhwAAABNaWNyb3NvZnQgV29yZCA2LjAgRG9jdW1lbnQACgAAAE1TV29yZERvYwAQAAAA V29yZC5Eb2N1bWVudC42APQ5snEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= --0__=gotZChCsQuE1Iy8cO0Ka46Mhcah9aNMOVQuJiR1OCrJOudBknnZPMSoF-- From owner-ips@ece.cmu.edu Mon Jan 15 18:22:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA17436 for ; Mon, 15 Jan 2001 18:22:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FKssR01176 for ips-outgoing; Mon, 15 Jan 2001 15:54:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FKsWs01156 for ; Mon, 15 Jan 2001 15:54:34 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 15 Jan 2001 15:54:03 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101426@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: Draft San Diego minutes Date: Mon, 15 Jan 2001 15:54:02 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Here's the initial draft of the San Diego minutes. Comments need to be made in the next few days, as the final version for the proceedings will probably be sent in on Friday. Thanks, --David ---------- IPS Meeting Minutes, Monday December 11, 2000 EMC will be sending out an IPR notice regarding a patent related to iSCSI and FCIP. David will be sending the information to the IPS mailing list this week. Interim meeting being scheduled for week of January 15, to coincide with T10 in Orlando - Grosvenor resort. -- Framework document - Mark Carlson. Describes environments for IP Storage. Includes terms, background on various protocols. This is a living document. Currently more of a survey. This document will coordinate with Naming and Discovery. Looking for more co-authors, please contact Mark if you are interested. -- Framing discussion -- Randy Haagens and Allyn Romanow - Allyn and Randy were asked to compose this presentation by the ADs. Purpose was to try to clarify the problem and present a range of solutions. - Framing is a common challenge with for both iSCSI, FCIP as well as non IPS documents. While framing is not explicitly required, a solution for a more effective iSCSI specification is highly desirable. The focus of the presentation was understanding the requirements of framing (i.e. the problem). Reaching consensus on a solution was not one the goals of the presentation. Allyn started the presentation by pointing out that this topic will also be discussed on Monday night in the TSVWG. - The problem: TCP reassembly can be costly, and in some instances not feasible. Also, there is limited host memory and host bus bandwidth, so we want to avoid manipulating the data more than once. Best would be one use of the bus and memory - zero copy. Note: This is not the same as TCP zero copy. In TCP, typically wait for all the data to arrive, then copy data to host. - In outbound direction, data can be transferred directly from memory to the protocol controller and out onto the wire. In the inbound direction, when received out of order, requires data to be put in reassembly buffer until all data is received. - One solution: Direct Memory Placement (Payload steering; data steering; RDMA) -- In order to conserve host memory bandwidth, CPU cycles and reduce on-board memory requirements, it is desirable to deliver iSCSI data directly to host buffers, avoiding the overhead of TCP reassembly buffers. The TCP reassembly buffer can be 250MB for a 10Gbps link with 200ms round-trip time. At 1Gbps, reassembly is possible but very costly. But when get to 10Gbps speeds or above, reassembly is no longer feasible. So, the goal is to get rid of a separate TCP reassembly buffer. Can decode ULP (iSCSI) headers and place payload directly in host memory without intermediate buffers. This would not be a conventional NIC card; instead it would be very iSCSI aware, but it would not necessarily process the iSCSI headers, but just use them to determine where to place the data. As in TCP, the iSCSI stream is presented to the iSCSI protocol processor in-order. - In this solution, must address loss of ULP sync - when a segment containing a ULP header is dropped or delayed, ULP sync is lost. Direct data placement cannot continue; data must be diverted to a reassembly buffer. Goal is to recover ULP sync at the next ULP header. There are both TCP aware and TCP unaware solutions to recovering ULP sync. - TCP unaware approaches: a) SCTP - issues include not widely deployed b) Special Characters - requires byte by bytes processing c) Fixed length ULP messages - Inefficient for short ULP messages d) Periodic Marker - Best solution for this class of approaches Sublayer of a framing protocol. Managable; relatively easy to implement in hardware Marker 4 byte field number of ULP bytes remaining in current PDU. Marker inserted and removed by framing protocol; e.g. iSCSI. After loss of sync, locate next mearker; use to locate the next ULP PDU. Markers are transmitted twice in a row; ensures markers cannot be split by stream fragmentation/segmentation. - TCP aware Approaches a) URGent pointer - disallowed b) PSH bit - disallowed - Another TCP aware approach can be considered by the TSV working group. Allyn Romanow presented what the TSV working group works on. The working group works on small items in the transport area that do not need a full working group as well as TCP/UDP transport issues. - Allyn Romanow presented a technique for demarcating message boundaries using TCP option. This consists of using one of the reserved bits in the TCP header to extend TCP to support this type of framing. Then can add up to 40 bytes before the TCP payload. Problem is that these reserved bits are a scarce resource; need to evaluate the need for the change. Also any time a change to TCP is proposed, there is tension, e.g. tension between the need to update TCP and stability of TCP. - Procedure for standardizing a TCP option consists of a) The IESG has to approve new work items for the TSV wg. b) Ask the Transport Services (TSV) working group to adopt this as a WG item c) Pros-and cons will be discussed on the TSV wg mailing list. If it supported, hopefully the spec will be wrapped at the next IETF (roughly 3 month time frame). If no support, it's dead. The advantage of the TSV wg is that transport experts will be able to contribute feedback. d) If supported, will be adopted at next IETF meeting. Advantage is that people who are experts in transport will be able to contribute, and that this will not be an iSCSI specific solution. IPS should follow this process and contribute. Make sure that the solution (since not iSCSI specific) meets the needs of this group. This is a very common problem, that is worthy of consideration at the transport layer. Addresses areas beyond IPS Allison pointed out that TCP option is not the only approach. TCP header bits could potentially be used for framing. The flag approach may send many packets that are less than MSS. This is potentially a risky change to TCP. - Message Boundary Option Two approaches. Not drafts, very introductory. Flag approach -- Costa has written up; will post as draft. The flag approach may send many packets that are less than MSS. This is potentially a risky change to TCP. ULP header is aligned with first byte of TCP payload. Offset Approach -- 4 bytes. 2 byte offset indicates offset into TCP payload of first ULP header in the segment. Write-up forthcoming. Discussion - Lead by Steve Bellovin Steve Requested the group concentrate on Requirements Somesh Gupta (HP) proposed another option -- periodic alignment instead of periodic marker. There could be a requirement in iSCSI that an upper-layer header appear every n kbytes in the TCP stream. Padding could be used to make sure this happens. Requires API change to TCP/IP stack. Ed Cox indicated that this issue has originated multiple times in past. It needs to be a general case, not IPS specific. Randy/Allyn argues that this is the general case, in that ULP usually have own PDU size info. Somebody thought that you would need to encode multiple message boundaries in any TCP options or otherwise use just one upper layer PDU per TCP segment. This would either imply small packets or lots of overhead. The reply was that we don't need to identify every boundary. We can use the length fields in the ULP frames to find the next ULP header in the packet. Steph Bailey (Genroco) asked how do you handle losing the first part of the packet; don't you get into the same situation trying to avoid? Pointed out that the message boundary proposals don't address the issue of the message being very long. Can't have unbounded ULP, since if we do, and lose the ULP header, have to be prepared to buffer an unlimited amount of data in anonymous buffers . With max ULP, must buffer up to max length of ULP, and this must be a reasonable size. Julian Satran said that the mechanism should be generic enough that other ULPs can use it easily. Venkat seconded this suggestion, saying that we should also treat RDMA, VI, etc. in the proposal. Want to have something that would have wider application, so to reduce HW dev costs. Someone from Sun asked when when is the TCP option examined - only when there is a loss on the receive side, in order to recover. Question asked how does this relate to RDMA? Allyn response -- RDMA different but related. RDMA proposal either implicitly or explicitly addresses framing. May make sense to do a generalized RDMA protocol that would make use of this framing mechanism. There was consensus at the end of the discussion - that framing is best done at the transport layer and should be done generically. Modifying TCP vs not : Luciano Dalle Ore pointed out that deployment of a general solution, will take much longer to get a solution. Will not be mandatory. May run into interoperability problems. Options are truly options some will be able to support; others will not. If inband, can spec it now, possibly require it. Question asked based on past experience w/options in TCP, how long will this take to propagate? Allyn responded once defined, based on previous experience, how long to get procedure defined - a couple of months (by March, 2001); work over mailing list. Will not hold up iSCSI development effort. Deployment 1-2 years. Allison pointed out that there is motivation to get this done/deployed, so deployment could occur much quicker. When SACK done, was no motivation to get adopted. Question asked if this framing would be mandatory. David indicated it would probably not be. Asked about necessity of attending tonight's meeting. Allyn responded that the proposal will be discussed tonight, but much more discussion on reflector. Recommended IPS participants sign up for TSV reflector. Somebody asked: What if we're unlucky and lose multiple iSCSI headers in a TCP window? Well, you have to buffer proportional to the number of headers that you lose. Also, the sending rate decreases quite a bit. There was some discussion of whether this enhanced framing would force TCP to deliver out-of-order. The answer is no: this architecture does enhanced data placement. TCP semantics need to be observed by any implementation. This is a difference between data placement and data delivery. Data delivery is still done in-order, according to the rules of TCP. The ULP is not aware that out of order data has arrived. Correct implementations will not deliver data out of order. Note: The memory the NIC is placing the data in is owned by the NIC. Mark Bakke said that this would be a good time to treat data integrity as well as framing. The protocols that want data integrity are CIFS and NFS; these are the same that want greater reliability from a CRC. Recommended having a SCSI data level CRC; customers will be looking for this from a file level as well. May be opportunity to put in a HW implemented CRC. David Black suggested that Mark Bakke send out a draft on this matter. There was some confusion as to why the TCP header is not sufficient. It was pointed out that multiple simultaneous SCSI transaction are placed on a single TCP connection so headers and data are mixed on a single TCP connection and sequence numbers do not a-priori indicate what is data and what is header. Buffer offset question - the iSCSI protocol packet classifier (or filter) is placing the data, not TCP. Steve Bellovin asked for a hum of the room on whether to solve the "framing problem" in an iSCSI-specific way or whether to pursue a mechanism to add to TCP. The hum in the room was to do it in TCP. ISCSI document review - presented by Julian Satran. - Rough consensus has been reached on the session model - Symetric with optional multiple connections. - Login Session context - good understanding. - Login Security context - more work needed. - Commands, messages, tasks, and tags almost complete. Items open - coding, some layout. - Response numbering scheme is well understood; complete. - The data numbering scheme has received no consensus. It may be removed. Julian's personal opinion is that it's optional and low cost with advantages. - For recovery, command restart and status well understood. No consensus on data recovery. Digest not well understood; needs to be readdressed. - Text commands - negotiation mechanisms done. - Mapping moved to T10 (aliasing). Dropped from iSCSI. - RDMA/Sync, Security/Authentication - all are still open issues. - Authentication - login phase must provide authentication. This was the consensus at the last meeting. Every iSCSI PDU must provide data integrity and authentication. - A mechanism should enable optional end2end data protection/authentication. Would like to use TCP recovery in presence of error. Digests can be activated at a higher level. Need a mechanism that can be activated on demand, ideally at login. - The current digest scheme needs to be changed. Julian suggested using IPSec for data integrity, since all the above mechanisms are provided by IPSec, it is a best fit for what is needed and very cheap if use only what is needed. Can insert own policies, including policies that will verify integrity verses provide security but use same mechanisms. Policies will be addressed in next two weeks. - David: IPSec does negotiation securely. What is currently in the draft is most likely vulnerable to man-in-the-middle attack. - Steve Bellovin indicated that the IPSec WG would be extremely opposed to any insecure non-cryptographic algorithm being defined for IPSec. Silicon must support SHA-1 or MD5 in order to do key negotiation. There are active discussions/proposals on how to do high speed encryption/negotiation. Early in process; drafts not yet standards, but worth looking at this. - Mark Bakke really wants to maintain the separate iSCSI header/iSCSI payload digests. This separation is lost by moving to IPSec. Gained data integrity is only as good as the group is willing to pay. Good integration with encryption. - Can use IPSec in transport mode, which will provide end2end protection. Integrity is required end2end, but security may not be. Security may need to be removed at the firewall/gateway, but need to still be able to verify integrity at the endpoints. Can have multiple layers of IPSec if needed. Comment from audience - not recommended. - David Peterson of Cisco asked whether ACA will be mandated by the draft. The consensus, after the discussion, is that iSCSI must support ACA but that a device need not support ACA (Ralph Weber pointed out that few initiator use ACA today). There was some grumbling because ACA is needed for reliable pipelining of ordered commands in the face of errors. - There was a question on whether asynchronous event notification (AEN) was mandatory to implement in iSCSI. Again, iSCSI transports must support asynchronous events but iSCSI devices need not. Somebody pointed out that SCSI mode pages can be used to regulate whether a device generate AENs. - Ralph Weber of T10 praised iSCSI for trying to advance the state of the art in SCSI. -- iSCSI requirements --- presented by Marjorie Krueger Doug Otis asked whether the T10 work on authorization was going to be integrated into iSCSI. David Black said that the documents won't be integrated into a single text. SCSI provides authorization, try to leave to T10. Randy Haagens pointed out that SCSI/T10 is not quite there on privacy, authorization and authentication so we have to do our own mechanisms. Also, since iSCSI introduces the authentication problem (by running SCSI over IP networks), iSCSI is the appropriate place to fix it. T10 work will be referenced where applicable. It was noted that the point of iSCSI authentication and authorization was to control who was able to get to a target. -- Bootstrapping -- presented by Prasenjit Sarkar This document contains guidelines for how iSCSI boot clients connect to iSCSI boot server. Included description of how to use existing techniques. iSCSI boot clients need IP address, iSCSI boot server service delivery port name, default; LUN = 0; iSCSI initiator software. Boot process steps: Client software stage Use PXE or related bootp/tftp protocol to get iSCSI initiator software DHCP stage Use DHCP to configure client IP address Use new DHCP option to configure iSCSI boot server service delivery port name Discovery server stage Use "to be defined" iSCSI delivery service to get iSCSI There was a question on whether the boot client had to have IPsec, in light of the integrity proposal by Julian and security proposals by others. Prasenjit answered that it was not required; you just need bootp. Mark Carlson noted that he didn't see any requirements for security in the boot process. He pointed out that booting from disk is a security-critical operation in many environments. Prasenjit countered that the boot stuff doesn't disallow security. There was some question on what to do with the iSCSI session once a bootstrap program was done with it. It was noted that it was probably simplest to close it and have the loaded program establish a new iSCSI session. -- MIB presentation - Mark Bakke A group forming to work on iSCSI MIB. An initial stab, via SNMP, taken. Manage iSCSI portion - iSCSI only, not SCSI 'stuff'. If needed, separate SCSI MIB, if does not already exist, needs to be addressed separately. Original MIB structure not adequate, being redone. Also reflects older version of iSCSI draft. Kevin (Nishan) - Has the MIB group looked into zoned environment support, similar to FC? Mark indicated that he had not looked at this. Where does zoning fit into iSCSI architecture, if at all? Where is MIB running? Could be anything running iSCSI including initiator, target, gateway. FC HBA API available from SNIA, might be of interest to this group. It has a complete list of things management tools want to be able to see out of an initiator. ----- Tuesday, December 12, 2000 -- Naming and Discovery Requirements - Mark Bakke, Cisco Mark said that the naming and discovery would specify target discovery but it would leave LUN discovery to SCSI mechanisms, such as REPORT LUNs. There was a bit of debate on this; why not go all the way and support LUN discovery in the naming system? Some people countered with a layering argument: "Leave unto SCSI what is SCSI's". Scaling requirements include both small and large environments. Find targets by querying SNS. Small environments do not require SNS. Hierarchical format, with Naming Authority. World Wide Unique Identifier Address composed of IP addr+TCP port+Target Name, URL like. Plan to apply for well known port for TCP. In such a case, an address w/o TCP specified would default to this well known port. Format includes info on naming authority, including support for 'local' naming authority. Character set to be allowed? Unicode? Recommend UI schemes for naming authority. Need to look at security issues. T10 issues - reservations, reset, LUN naming Target reset discussion. Noted that T10 is thinking of making target reset optional. Is breaking of a connection in iSCSI equivalent to a target reset? Consensus is no: the end of a session was equivalent to a target reset and would also cause any persistent reservations to be released. Naming scheme will allow multiple port and multiple initiator/target discovery. Will give list of targets + all paths to that target. Draft currently an individual submission - concensus (hum) taken, to be adopted as working group document. No opposition hums. -- iSNS document presented by Josh Tseng, Nishan ISNS describes a scalable information facility for registration, discovery and manament of networked facilities. ISNS follows a client/server architecture. If client registers with name server, allows itself to be managed by the name server. Why needed? Simplifies storage management implementations. Allows greater scalability over broadcast/multicase discovery methods. Supports zoning. Next step - incorporate requirements/suggestions from IPS working group. Extend document for FCIP Access control - what is name server role? Targets upload public key to name server. Enforced at the end node/target. Supports both soft and hard zoning. How does it fit into discovery. Naming and discovery team will look at this to see how well it fits. Should this be maintained as a separate document vs incorporated into naming/discovery team. In reading the draft, reliance on WWN. What do you do about devices which do not have WWN/don't want WWNs. Work done prior to n&d requirements document. This draft would need to be redone to support WWUI of n&d requirements. Direction is one in which naming and discovery team approves of? Yes, close. Is there working group concensus as a base document; working w/ NDT group to produce a revised document, aligned with N&D, which would then be adopted as an official wg document. Rough concencus - next revised version of document will become an official working group document. Not unanimous. -- FCIP - Status and progress of FCIP. - Raj Bhagwat Current status - difference from previous presentation Solution for bridging remote FC SAN islands. From FC point of view, appears tobe entirely an FC network. Initially did not have congestion management (last presentation). Draft overhauled to incorporate TCP as transport in order to address congestion management and recovery mechanisms. In rev -00, PSH flag incorporated. Based on feedback from mailing list, this was eliminated and in -01, a new frame boundary mechanism introduced. Topics under discussion -- QOS, security, MTU/MSS, Framing/synchronization, order of delivery, discovery, error recovery. Alignment with new project in T11 - FC-BB2. FC-BB2 focused on issues outside the scope of the IETF, including link level issues. Target date for completion - June 2001. David Robinson complained that much FC/IP work is done on conference calls. He asked that these conference calls be made public so as to allow broader participation. Conference calls are design team calls open to design team members and authors. Public review on mailing lists. What is an FCIP device - a gateway between an FC SAN and IP network. Discovery of FCIP gateway (device) of other FCIP gateways. How do gateways discover each other? Currently in spec is static configuration. Dynamic configuration support is envisioned, perhaps using iSNS. David to work with authors offline on QoS text. -- iFCP - presented by Charles Monia, Nishan What is the difference between iFCP and FCIP? FCIP is a tunneling model between FC SANs. A conduit for FC frames to flow transparently to FC network over IP backbone. IFCP network model extends up to the FC storage device itself. Uses a session model. Consolidates FC storage switching and routing functions in the IP fabric. Reduce total cost of ownership, unify network and storage management domains and exploit IP technology investment. Extend SAN over lan/man/wan distances. Next step -- complete the n_port session model. Encapsulation changes for additional end-to-end error detection. The authors of iFCP would like to see it considered for adoption as a work group item. Adoption of iFCP as a work group item requires modification to the WG charter. David requested input on this be set to the WG chairs. Revising of the charter requires consultation of the area directors and working group chairs. After the presentation, Julian Satran suggested that iFCP and FC/IP should merge since they are so similar. Others agreed with Julian. Charles Monia countered that they would be difficult to merge because they take different approaches. iFCP works by intercepting FC logins (connection requests) and modifying FC frames. In addition, it doesn't run FC routing protocols between FC SANs. Clarification of FCIP and iFCP - the latter is for FCP protocol mapping only, whereas FCIP can transport any FC upper level protocol. FC/IP works at a lower level than iFCP. It doesn't modify FC frames. FC/IP requires running FC routing/switching protocols between FC domains. Some thought that iFCP was a superset of FC/IP. Somebody was concerned that the iFCP gateway would need to run IP routing protocols. It was eventually decided the iFCP gateway was just an IP host and didn't have to run IP routing protocols. Other comments need to sent to mailing list or chairs directly. -- Adaptation Layer presentation -- Randall Stewart, Cisco Randall Stewart's presentation introduced how the IPS protocols could be architected an adaptation layer independent of the underlying transport (i.e. at least both SCTP and TCP). To do this, a uniform API boundary between the ULP and transport would need to be defined. This would require many changes to all existing drafts. APIs would need to be a message oriented type of mechanism. Critical path would need to be done so that they would be protocol agnostic. Transport interface would need to provide methods for passing buffers to/from control of transport, e.g. for zero copy. Adaption layer would need to worry about Framing Zero copy Parallel paths Message retrieval Notifications Must be very careful that this API would not make assumptions about the transport being used. In adaptation model, would need to figure out how to overcome the issues. Julian Satran though this was a good way to proceed and would like to see Randall write up a section on this for the draft. Randall would be more than glad to help by contributing both advice and/or drafts. Randy Haagens thought that the adaptation layer would add too many layers between iSCSI and TCP and that separate protocol should be done for SCTP. Steph Bailey suggested that the CAM may be an inspiration for the adaptation layer. Others responded that the CAM is at the wrong layer, above iSCSI. From owner-ips@ece.cmu.edu Mon Jan 15 19:43:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA18159 for ; Mon, 15 Jan 2001 19:43:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FMw2S05224 for ips-outgoing; Mon, 15 Jan 2001 17:58:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FMv8s05193 for ; Mon, 15 Jan 2001 17:57:08 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id RAA39362 for ; Mon, 15 Jan 2001 17:50:17 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id PAA109216 for ; Mon, 15 Jan 2001 15:57:04 -0700 X-Priority: 1 (High) Importance: Normal Subject: iFCP as an IP Storage Work Item, Reset To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Mon, 15 Jan 2001 12:14:14 -0800 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 01/15/2001 03:57:04 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I think the messages on iFCP have become somewhat decisive and unfocused. And quite frankly it has been permitted to happen by both supporters and detractors. I think this has occurred because engineers tend to solve things, whether or not the solution is useful or if it has a business need requiring a fix. Therefore, having said that, we need to restate what the playing field looks like. (The reader is cautioned that this is a lengthy note.) The iFCP draft says that it is intended to be a Gateway to Gateway protocol. Therefore supporters of iFCP should stop responding to the infinite possibilities to hallucinate various other purposes for this protocol. The protocol is NOT an end to end protocol, and responses that do not end the speculation waist our joint time, and polarizes iSCSI end-to-end folks against iFCP. I believe, there are a number of things that maybe missing for a robust end-to-end protocol with iFCP, but the point is it does not matter, the protocol is structured, proposed and analyzed as a Gateway to Gateway protocol, and nothing else is proposed or analyzed as part of iFCP. The head of the company that brought us iFCP, has stated that he only wants this as a gateway to gateway protocol, and that they are committed to iSCSI also. I blame the iFCP supporters for not just shutting down these variant thoughts. The iSCSI supporters would be expected to worry and be paranoid about anything they can hallucinate that might threaten iSCSI. Therefore, because the iFCP authors did not stop these thoughts dead in their tracks (although Josh tried a couple of time), things got completely out of hand, and we moved from a protocol, that in some ways was redundant to FCIP (or visa versa), to something that also seemed to threaten iSCSI. I strongly suggest that no real interests are served by this wild speculation (Note: I am not completely pure here either, since I also let paranoia force me off the reservation when I worried about mFCP and a related organization.) So I want to strongly suggest that we look at iFCP for only what it is proposed to be, and NOT stray into other areas. To get my head straight, I had to refocus on the environments where FC, FCIP, iFCP and iSCSI are going to play to see where the effective redundancy and conflict might exist. Clearly if I thought that iFCP gave a serious problem to iSCSI and to FCIP, and was redundant to both, I would have a problem. To analyze this, I had to go through some "market think". I know that we are suppose to be in an engineering environment, but it is our views of the market that get us polarized. So the following is the analysis I went though and I hope it is helpful to others. Working with iFCP defined as a Gateway to Gateway protocol, lets look at the various environments where this technology could be deployed. Also, even though there are two types of Gateways that could be built with this protocol, a Router, and a Switching Router (Switch with Router capabilities) I will assume that a Switching Router is the subject product. A Switching Router has several ports that can either be plugged with FC cable or Gigabit Ethernet cable, in various combinations. If only FC cables are connected, it is a FC switch, when only Ethernet cables are connected it is a IP Switch, with various combinations of FC and Ethernet, it is a iFCP Routing Switch, called hereafter an iFCP Switch or an iFCP Gateway. We should also note that the iFCP Switch Vendor(s) are also claiming (as I said above) that they will have, over time, iSCSI support also. I will submit that those environments which have iFCP (with latent iSCSI capability) solutions will also help the deployment of iSCSI solutions. This is because, the Gateways to FC Host and FC Storage will have already have been put in place. Here are the environments I examined: 1. Existing Single SAN Fibre Channel environments with a number of FC Switches. If the FC SAN already exists, the assumption here is that the customer might want to replace his current FC Switches and Hubs with an iFCP set of switches and LAN type interconnects. However, I submit this is not really very fertile ground for iFCP switches. (By the way it is not very fertile for iSCSI or FCIP solutions, in the short term, either.) This set of customers will be very hesitant to change since they have already had a lot of investment, and they probably think they are on the down hill slop of the compatibility and support issues. They may wish that the cost were lower, but any change will be only lots of additional cost to them. Continuing with FC will be their probably action. 2. Existing Single FC environment, that uses FC to connect mostly point to point and not invested in many, if any, switches. This type of environment may use Sharks or Symmetrix type Storage Controllers directly. And some time they will need to expand with new storage controllers, Host etc. This gives a possible reason to install a FC Switch with iFCP capability, (for future expansion). But this will be a head to head fight with existing FC Switch vendors, and therefore such iFCP Gateways will not have a strong market here. (By the way, the iSCSI message is hard in this environment, unless the customer has visions of important future expansion with iSCSI storage controllers, or needs to bring Desktops and Laptops into the common storage pool.) Likewise there is not a FCIP message here either.) 3 a. Existing Multiple FC environments (locations) on a single campus. FC with normal Switch to Switch E-Port connections via long FC cables will be able to easily bring interconnection to this environment. However, the bringing together of various FC environments is also a strong play for iFCP Gateways. That is, each environment can add a iFCP capable Gateway and thereby interconnect the environments, and permit the sharing of various storage facilities. Each individual iFCP interconnected FC environment will be able to change its addressing etc., without needing to coordinate with any other local site (a Big Plus). The FCIP solution is also viable in this environment, however, when you get to more then two environments, the number of (single in -- single out) Edge Connect FCIP routers, go up dramatically as the number of locations increase. The physical space issue can be mitigated by a multi-in, multi-out FCIP capable router box. That is, a box that puts the electronics for multiple connections on the FC side -- one for each location to location interconnect. Note, this also requires a FC switch with a E-Port that can be connected to an E-Port at a specific target location. Also the FCIP has a disadvantage in each of the locations that might look like environment 2 above (since they have few or no FC Switches). iSCSI will find this a difficult environment in general because of the existing FC investment. (Note: proposed combinations sets of "FC-to-iSCSI boxes which route to iSCSI-to-FC boxes" can not be assumed to work since iSCSI has not defined any protocol to handle FC ELS, as has iFCP.) Therefore, iSCSI will not be a popular solution here. The higher the number of location the stronger the iFCP appears (cost wise) against FCIP. Note: if a iFCP Gateway gets installed, and if the unit also supports iSCSI (as the primary vendor has committed), the inroad for iSCSI, in general, will be greatly simplified. iSCSI folks, therefore, should be careful in their dismissing of iFCP, because when it is teamed up with iSCSI in this manor, it becomes be a great ally. 3 b. Existing Multiple FC environments across a Wide Area. This case is the same as 3a above, except normal FC techniques do not apply. However, both iFCP and FCIP approaches apply. Again the advantage of iFCP vrs FCIP have to do with the number of sites that are interconnected, and the site independence that iFCP offers. Also, if the iFCP Gateway has iSCSI capabilities, it should permit strong and quick inroads for iSCSI into this environment. 4. SCSI, and ATA based Host environments that need to grow into a network attachment. The non server host, Desktops and Laptops, and servers that have incidental remote Storage Access will not generally consider FC as an approprate option. Generally this environment is approprate for the iSCSI initiators (with normal TCP/IP NICs, and iSCSI SW Drivers). Without FC the FCIP will not apply to this environment. iSCSI to FC Gateways will exist at the remote environments and possibly iSCSI native Storage Controllers. iFCP itself is not approprate but if the iFCP box also has iSCSI capability, then that can be as valuable as any iSCSI to FC gateway. 5. Single SCSI, based Server environments with needs to grow. These environments will have high access requirements. When iSCSI HBAs are available (1H2001), and with the availability of iSCSI to FC Routers (1H2001), this is a fertile market for an iSCSI sale. When iSCSI Storage Controllers are available, this environment will be a significant growth area for iSCSI. It has a head to head message with FC SAN. However, the existing knowledge of IP networks and IP Switch cost will be the swing consideration to iSCSI. FCIP has no evolvement in this environment. iFCP in this environment would require FC HBAs, and at lest two iFCP Switching Routers. If the customer wants to add iSCSI Storage Controllers, the iFCP Switches will not be useful (until it supports iSCSI also). In this environment where the target storage is FC based, iSCSI requires (in addition to iSCSI HBAs, & IP Switches), one or more iSCSI to FC Gateways. The winner here, between iFCP and iSCSI will be the solution with the least cost. That is, the total cost of the HBAs (initially this maybe a push between FC and iSCSI), the IP Switches, and the iSCSI to FC Gateways vs the Cost of FC adapters, IP Switches and iFCP Gateways. If the resultant cost, is close, then iSCSI will probably be the winner here, because of industry direction. Having said that, this is one of the environments where iSCSI folks think that the iFCP message is diversionary. If the customer is not polarized against FC, and the customers looks at having to buy FC adapters for the iFCP deployment, they are probably going to look at a complete FC SAN as a more straight forward deployment. Therefore, the competitive play will very often be between FC and iSCSI the possibly diversionary iFCP messages will not be significant and maybe helpful selling the IP Storage message. 6 a. Multiple SCSI, based Servers environments (locations) on a single campus. Like 5. above, these environments will have high access requirements, and when iSCSI HBAs are available (1H2001), and the iSCSI to FC Routers (Gateways) are available (1H2001), this becomes a fertile market for an iSCSI sale. iSCSI will then have a head to head competitive message in this environment with FC SANs, and very probably a head to head competitive message with the FC and FCIP combo. (So lets not deceive ourselves that iSCSI doesn't have a market conflict with FCIP.) Since I believe that in this environment, the customer probably had a reason for not already jumping on FC, and because since I also think they are probably well versed in TCP/IP Networks, the iSCSI message, will be well received. The iFCP message may also have a play in this environment, they can play to the same TCP/IP knowledge as iSCSI does, and suggest to the customer that iFCP is a better choice because there are more HBAs and Storage Devices based on FC then on iSCSI, etc. Therefore I think that this is a competitive play between everyone, FC SAN, FC and FCIP, iFCP, and iSCSI. However. the more iSCSI based storage controllers are shipped, the more the play tips toward iSCSI in this environment. Some iSCSI folks look at iFCP having a diversionary message in this case also, however it looks to me as a general free for all amongst all the options that are valid in this environment and cost will be the driving force. By the way, the iFCP IP Storage message will probably do more to help iSCSI, then it does to distract from iSCSI. Also, in this environment, the trade offs between FCIP and iFCP, mentioned in 3a. above, apply here as well. 6 b. Multiple SCSI, based Servers environments (locations) across a wide area. This environment fits the same solutions as 6 a. above, except a FC only solution is not an option. That is, an iFCP or FC&FCIP solution is approprate as is the iSCSI option, just as in 6a above, but not FC only. 7. Multi Hosting environments with Storage Service Providers (SSPs). These environments will be primarily networked environments which when built today, are Fibre Channel, but here a iFCP Gateway would be a reasonable consideration. In this environment today, the Hosts will have a FC adapters so in either case, the question is -- will the cost be less using iFCP Gateways (where the FC Fibre leave the "Cage", and then goes through IP Switches, and then terminates at a iFCP Gateway just before the FC based Storage Units)? Since the SSPs will be competent in both FC and IP networks, the trade off will be primarily cost. If the costs are close, the edge will probably go to iFCP since it can also combine multiple sites at less cost then multiple FCIP Gateways (see discussion in point 3 above). Having said all that, if iSCSI HBAs, and iSCSI Gateways are available (1H2001), then, the SSP will be able, to apply additional security features (if required by the customer), and they only need low cost IP Switches all the way into the Storage Area. This will probably the most flexible and lowest cost. Since iSCSI will also be able to glue multiple sites together it will be the most probable configuration in this environment, over time. Until that time, iFCP has a strong chance of being heavily used in a SSP environment, perhaps supplanting FC networks in this environment. An iFCP & iSCSI Gateway, would seem to be a very high seller, and permit graceful migration to an all iSCSI network. -------------------------------------------------------------- The iFCP messages, in all but environment #7, will be dwarfed by the number of vendors shipping iSCSI, and the iSCSI message will be the only important IP Storage message that will come through. But since the number of SSPs is small they are easy to market too, and will be fully aware of the approprate maturity of iFCP and iSCSI. However, iFCP will probably only be a stepping stone to iSCSI in an SSP environment. Further, if you take the primary iFCP vendor at their word, they will combine their iFCP product by including iSCSI, and therefore the total message will be very supportive of iSCSI. Let me build a matrix that can summarize the above environments (Cases) with probability of success. Here are the cases again: 1. Existing Single SAN Fibre Channel environments with a number of FC Switches. 2. Existing Single FC environment, that uses FC to connect mostly point to point and not invested in many, if any, switches. 3 a. Existing Multiple FC environments (locations) on a single campus. 3 b. Existing Multiple FC environments across a Wide Area. 4. SCSI, and ATA based Host environments that need to grow into a network attachment. 5. Single SCSI, based Server environments with needs to grow. 6 a. Multiple SCSI, based Servers environments (locations) on a single campus. 6 b. Multiple SCSI, based Servers environments (locations) across a wide area 7. Multi Hosting environments with Storage Service Providers (SSPs). +----+-------+-------+-----+------+----------+--------+ |Case|FC only|FC&FCIP|iFCP | iSCSI|iFCP&iSCSI|iFCP Msg| +----+-------+-------+-----+------+----------+--------+ | 1 | High | NA | Low | Low | Med |Helpful | +----+-------+-------+-----+------+----------+--------+ | 2 | High | NA | Low | Low | Med |Helpful | +------------+-------+-----+------+----------+--------+ | 3a | High | High |High+| Low |Very High |Helpful | +----+-------+-------+-----+------+----------+--------+ | 3b | NA | High |High+| Low |Very High |Helpful | +----+-------+-------+-----+------+----------+--------+ | 4 | Low | NA | NA | High | Med |NA | +----+-------+-------+-----+------+----------+--------+ | 5 | Low+ | NA |Low+ | High | Med |Min Help| +----+-------+-------+-----+------+----------+--------+ | 6a | Med | Med |Med+ | High |Very High |Min Help| +----+-------+-------+-----+------+----------+--------+ | 6b | NA | Med |Med+ | High |Very High |Min Help| +----+-------+-------+-----+------+----------+--------+ | 7 |Hi->Med| Med |High |Med>Hi|Very High |Helpful | +----+-------+-------+-----+------+----------+--------+ This table says that in the existing FC environment Case 1-3b, iSCSI, will have tough play. On the other hand, in multiple location environments, iFCP will have a strong play, against FC & FCIP. When the iFCP product is combined with iSCSI support its probability of success becomes Very High. In other environments (4-7), iSCSI has a high probability of success. And when the iFCP product gets iSCSI support it will greatly improve its success position. The marketing messages from iFCP are Helpful, or Minimally Helpful. Hence the only important conflict is between FCIP and iFCP, and this is mostly a "push" except for Multiple Sites, where the Cost might favor iFCP (and that is not completely obvious). Many of us wanted the FCIP and the iFCP combined so this conflict does not exist, however, this looks like both sides have refused to cooperate, so a product shoot out will occur. It looks to me that both will take a piece of the market and continue to exist. The table indicates that when iFCP is combined with iSCSI, the results offers a better product in several areas then either by themselves. Therefore, I believe that iFCP has important potential value to our customers, and should be part of the IP Storage effort within the IETF. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403 Internet address: hufferd@us.ibm.com From owner-ips@ece.cmu.edu Mon Jan 15 19:45:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA18172 for ; Mon, 15 Jan 2001 19:45:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FMDvx03841 for ips-outgoing; Mon, 15 Jan 2001 17:13:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FMDms03832 for ; Mon, 15 Jan 2001 17:13:48 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 15 Jan 2001 17:14:00 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101429@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: FCIP QoS text Date: Mon, 15 Jan 2001 17:13:37 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk At the last meeting I promised to provide some advice about the DSCP and QoS text in the FCIP draft. This email is sent in my role as one of the authors of the Diffserv header and architecture RFCs (2474 and 2475), rather than as a WG co-chair. Section 5.3 of the FCIP draft says: DSCP (6 bits): The recommended DSCP field setting is 101110 corresponding to the EF service. As I would recommend replacing this text with text describing the properties of the forwarding service that are needed/desired (e.g., high bandwidth, low delay, low variation in delay, essentially no loss), and then citing EF as an example of a PHB that can be used to implement such a service (overprovisioned AF will likely also work). An actual diffserv service is a combination of PHB and edge traffic conditioning (for EF to behave as intended, there has to be a bandwidth limit on the inbound traffic at the edge of the domain). An important thing to understand about diffserv is that it's a toolkit. Neither EF nor AF have to be implemented in a diffserv- compliant network - the only required PHBs are defined/described in RFC 2474 (Default, and the PHBs mapped to the class selectors). The approach I'm recommending provides guidance to the user of the toolkit about what the required result should be (e.g., tighten that nut) rather than overspecifying the tool to be used (e.g., requiring a crescent wrench when a socket will also work). The QoS discussion in Section 9.1 should also be revised to reflect this toolkit philosophy. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Jan 15 19:49:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA18224 for ; Mon, 15 Jan 2001 19:48:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FNC2K05650 for ips-outgoing; Mon, 15 Jan 2001 18:12:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FNAxs05619 for ; Mon, 15 Jan 2001 18:10:59 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id SAA39670; Mon, 15 Jan 2001 18:04:08 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id QAA18104; Mon, 15 Jan 2001 16:10:56 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI & iFCP Overlapping (Was iFCP as an IP Storage Work Item) To: "Y P Cheng" Cc: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Mon, 15 Jan 2001 15:10:28 -0800 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 01/15/2001 04:10:56 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk With my TC hat on: Please reframe from speculation about iFCP as an End to End protocal. iFCP has not been proposed by its authors to be End to End, and is not up for discussion without an approprate draft. The authors and the iFCP Draft clearly define iFCP as an Gateway to Gateway protocol. We are currently having enough problems just working on the Gateway to Gateway issues, and do not need to explore areas where the Draft did not go. Please lets limit further iFCP discussions to iFCP as a Gateway to Gateway protocol. TC hat off. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403 Internet address: hufferd@us.ibm.com "Y P Cheng" @ece.cmu.edu on 01/15/2001 08:40:41 AM Sent by: owner-ips@ece.cmu.edu To: cc: Subject: iSCSI & iFCP Overlapping (Was iFCP as an IP Storage Work Item) > iFCP and FCIP have nothing to do with OSPF. I think your statements > show that you do not understand iFCP and its objectives. I would > have hoped that those critical of iFCP would have at least a minimal > understanding of it. Josh, Had I stayed with the topic of "saving the customers' investment in FCP stack" I would have done better and not invited the above statement. :-) I have no doubt that you are a very good and wonderful network engineer. But, I am afraid, iFCP simply is a better mouse trap solving the same problem as that by iSCSI. After reading the iFCP draft carefully, I can't help but conclude that by connecting N-Port to IP directly iFCP usurps the fibre channel Extended Link Services and replaces class F traffic by all the wonderful stuff like EGP, BGP, OSPF, etc., offered by the Big Internet. I found the iFCP effort overlaps with iSCSI and it provides no additional benefit. As you have indicated that iFCP provides a transition into the iSCSI world of the future. We must agree that initially even iSCSI storage devices will have FCP or SCSI devices "inside the box". This is because the storage industry will take some time to change its infrastructure to produce iSCSI drives. On the other hand, the iSCSI HBAs will be available quickly -- I hope you do give me more credit in knowing the HBA world. :-) Therefore, it is very easy for the Network Storage Industry to have an iSCSI target adapter inside its box to communicate with the iSCSI hosts and clients. If iFCP is a viable solution, the HBA industry can produce iFCP target and host adapters just as quickly as the iSCSI adapters. Trust me, if we can implement the Fibre Channel and InfiniBand specifications, we can do iFCP. I do hope you see my logic now. Therefore, while iFCP may be a better mouse trap than FCIP, IMHO, it is providing the same solution as that of iSCSI. By the way, lets hear from other folks on this topic, please! Y.P. Cheng, ConnectCom Solutions. From owner-ips@ece.cmu.edu Mon Jan 15 20:35:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA18543 for ; Mon, 15 Jan 2001 20:35:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0FNX3G06385 for ips-outgoing; Mon, 15 Jan 2001 18:33:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0FNWRs06369 for ; Mon, 15 Jan 2001 18:32:27 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0G0h6036335; Mon, 15 Jan 2001 16:43:11 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: , Subject: RE: Draft San Diego minutes Date: Mon, 15 Jan 2001 15:30:54 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704101426@corpmx9.isus.emc.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > Somesh Gupta (HP) proposed another option -- periodic alignment instead of > periodic marker. There could be a requirement in iSCSI that an upper-layer header > appear every n kbytes in the TCP stream. Padding could be used to make sure this > happens. Requires API change to TCP/IP stack. Why would placing a PDU at a fixed number of bytes impact the TCP stack? This can be done above TCP within the application. Making the alignment interval a power of two keeps the modulo business simple. Doug From owner-ips@ece.cmu.edu Mon Jan 15 22:13:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA20384 for ; Mon, 15 Jan 2001 22:13:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0G0s5K08466 for ips-outgoing; Mon, 15 Jan 2001 19:54:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0G0rhs08458 for ; Mon, 15 Jan 2001 19:53:43 -0500 (EST) Received: from sponge.cisco.com (sponge.cisco.com [171.71.61.25]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id QAA12486; Mon, 15 Jan 2001 16:53:47 -0800 (PST) Received: from DAPW2K (rtp-dial-2-62.cisco.com [10.83.96.62]) by sponge.cisco.com (Mirapoint) with SMTP id AFG06493; Mon, 15 Jan 2001 18:53:34 -0600 (CST) From: "David Peterson" To: "Santosh Rao" , "ips" Subject: RE: iSCSI : SPC-2 Protocol Identifiers Date: Mon, 15 Jan 2001 18:52:11 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <200101160037.QAA09033@hpcuhe.cup.hp.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Yes, will be proposing to add iSCSI to table 58, table 172, table C.5, and table C.6. Dave > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Santosh Rao > Sent: Monday, January 15, 2001 6:37 PM > To: ips > Subject: iSCSI : SPC-2 Protocol Identifiers > > > It looks like the introduction of a new SCSI transport protocol > (i.e. iSCSI) requires a corresponding addition to be made to > the Protocol Identifiers Table in SPC-2. (Table 172. SPC-2 Rev 18). > > Is this being co-ordinated with T10 ? > > Regards, > Santosh > > -- > ################################# > Santosh Rao > Software Design Engineer, > HP, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################# From owner-ips@ece.cmu.edu Mon Jan 15 22:17:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA20427 for ; Mon, 15 Jan 2001 22:17:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0G0c4p08119 for ips-outgoing; Mon, 15 Jan 2001 19:38:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0G0bSs08111 for ; Mon, 15 Jan 2001 19:37:28 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id B89A876D for ; Mon, 15 Jan 2001 16:37:27 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id QAA09033 for ips@ece.cmu.edu; Mon, 15 Jan 2001 16:37:22 -0800 (PST) From: Santosh Rao Message-Id: <200101160037.QAA09033@hpcuhe.cup.hp.com> Subject: iSCSI : SPC-2 Protocol Identifiers To: ips@ece.cmu.edu (ips) Date: Mon, 15 Jan 2001 16:37:21 -0800 (PST) X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit It looks like the introduction of a new SCSI transport protocol (i.e. iSCSI) requires a corresponding addition to be made to the Protocol Identifiers Table in SPC-2. (Table 172. SPC-2 Rev 18). Is this being co-ordinated with T10 ? Regards, Santosh -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Mon Jan 15 22:59:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA21764 for ; Mon, 15 Jan 2001 22:59:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0G1bBn09508 for ips-outgoing; Mon, 15 Jan 2001 20:37:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0G1ass09499 for ; Mon, 15 Jan 2001 20:36:55 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0G2lr036424; Mon, 15 Jan 2001 18:47:53 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "John Hufferd" , Subject: RE: iFCP as an IP Storage Work Item, Reset Date: Mon, 15 Jan 2001 17:35:41 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit John, Are you concerned iFCP is too competitive with iSCSI? Although you could suggest iFCP is just a gateway to gateway solution, this does not constrain a gateway if it contains but a single device nor does it require hallucinations to arrive at that conclusion. In that case, the differences between iSCSI and iFCP hinge upon reliance of existing FCP protocol merged with a lightweight transport. The motivation of all proposals is to incorporate IP managed networks rather than rely upon dedicated fiber for SAN and, in that case, all proposals are in conflict. Disruption in the existing equipment and software will play a significant role in acceptance of any solution. Which protocol your company supports may be a marketing decision and not a technical one. I am not sure I understand your concern about potential applications unless you wish to thwart alternatives by prohibiting efforts that may be competitive. As iSCSI has developed support for multiple connections, error handling and retry mechanisms, flow control, and unique security requirements which may hinder or help eventual acceptance of iSCSI compared to iFCP. A technical perspective of the differences would be whether these features are useful or not and if they are easier to implement in simpler configurations. Your concern about potentially competitive solutions is troubling, but at least you do see value offered by the services in iFCP. I also see simplicity another benefit and hope for merger of the encapsulation between iFCP and FCIP with an eye toward SCTP in the future. Any arm twisting should be in this direction and would hope marketing is not the WG main interest. Doug > > Here are the cases again: > 1. Existing Single SAN Fibre Channel environments with a number of FC > Switches. > 2. Existing Single FC environment, that uses FC to connect mostly point to > point and not invested in many, if any, switches. > 3 a. Existing Multiple FC environments (locations) on a single campus. > 3 b. Existing Multiple FC environments across a Wide Area. > 4. SCSI, and ATA based Host environments that need to grow into a network > attachment. > 5. Single SCSI, based Server environments with needs to grow. > 6 a. Multiple SCSI, based Servers environments (locations) on a single > campus. > 6 b. Multiple SCSI, based Servers environments (locations) across a wide > area > 7. Multi Hosting environments with Storage Service Providers (SSPs). > > +----+-------+-------+-----+------+----------+--------+ > |Case|FC only|FC&FCIP|iFCP | iSCSI|iFCP&iSCSI|iFCP Msg| > +----+-------+-------+-----+------+----------+--------+ > | 1 | High | NA | Low | Low | Med |Helpful | > +----+-------+-------+-----+------+----------+--------+ > | 2 | High | NA | Low | Low | Med |Helpful | > +------------+-------+-----+------+----------+--------+ > | 3a | High | High |High+| Low |Very High |Helpful | > +----+-------+-------+-----+------+----------+--------+ > | 3b | NA | High |High+| Low |Very High |Helpful | > +----+-------+-------+-----+------+----------+--------+ > | 4 | Low | NA | NA | High | Med |NA | > +----+-------+-------+-----+------+----------+--------+ > | 5 | Low+ | NA |Low+ | High | Med |Min Help| > +----+-------+-------+-----+------+----------+--------+ > | 6a | Med | Med |Med+ | High |Very High |Min Help| > +----+-------+-------+-----+------+----------+--------+ > | 6b | NA | Med |Med+ | High |Very High |Min Help| > +----+-------+-------+-----+------+----------+--------+ > | 7 |Hi->Med| Med |High |Med>Hi|Very High |Helpful | > +----+-------+-------+-----+------+----------+--------+ > > This table says that in the existing FC environment Case 1-3b, iSCSI, > will have tough play. On the other hand, in multiple location > environments, iFCP will have a strong play, against FC & FCIP. When the > iFCP product is combined with iSCSI support its probability of success > becomes Very High. In other environments (4-7), iSCSI has a high > probability of success. And when the iFCP product gets iSCSI support it > will greatly improve its success position. The marketing messages from > iFCP are Helpful, or Minimally Helpful. > > Hence the only important conflict is between FCIP and iFCP, and this is > mostly a "push" except for Multiple Sites, where the Cost might favor iFCP > (and that is not completely obvious). > > Many of us wanted the FCIP and the iFCP combined so this conflict does not > exist, however, this looks like both sides have refused to cooperate, so a > product shoot out will occur. It looks to me that both will take a piece > of the market and continue to exist. > > The table indicates that when iFCP is combined with iSCSI, the results > offers a better product in several areas then either by themselves. > > Therefore, I believe that iFCP has important potential value to our > customers, and should be part of the IP Storage effort within the IETF. > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > (408) 256-0403, Tie: 276-0403 > Internet address: hufferd@us.ibm.com > > From owner-ips@ece.cmu.edu Mon Jan 15 22:59:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA21766 for ; Mon, 15 Jan 2001 22:59:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0G1Y6409415 for ips-outgoing; Mon, 15 Jan 2001 20:34:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0G1X5s09389 for ; Mon, 15 Jan 2001 20:33:06 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id CAA112008 for ; Tue, 16 Jan 2001 02:33:01 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id CAA248986 for ; Tue, 16 Jan 2001 02:33:01 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D6.00088072 ; Tue, 16 Jan 2001 02:32:51 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 15 Jan 2001 22:41:18 +0200 Subject: Re: iSCSI : FirstBurst, I.T.T value of 0. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Fixe all - Thanks, Julo Santosh Rao on 15/01/2001 11:53:07 Please respond to Santosh Rao To: ips@ece.cmu.edu (ips) cc: Subject: iSCSI : FirstBurst, I.T.T value of 0. Julian, 3 quick comments on the 03 rev of the iSCSI draft : 1) Section 3.1.3 refers to a non-existent login key InitialBurstLength. Is this supposed to be FirstBurst ? 2) Section 2.7 continues to use Target Task Tag and Target Transfer Tag field names in different places in the figures and text. Only one of the 2 names should be used and all references to the other removed. 3) Section 2.12 and 2.13 (NOP-OUT and NOP-IN) continue to use a value of 0 as the reserved usage of Initiator Task Tag. This should be changed to 0xFFFFFFFF. Regards, Santosh -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Mon Jan 15 23:51:26 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA22453 for ; Mon, 15 Jan 2001 23:51:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0G2JGX10618 for ips-outgoing; Mon, 15 Jan 2001 21:19:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0G2ICs10600 for ; Mon, 15 Jan 2001 21:18:13 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 1F1FA70F; Mon, 15 Jan 2001 18:18:12 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id SAA15235; Mon, 15 Jan 2001 18:18:07 -0800 (PST) From: Santosh Rao Message-Id: <200101160218.SAA15235@hpcuhe.cup.hp.com> Subject: Re: iSCSI: Abort task To: julian_satran@il.ibm.com Date: Mon, 15 Jan 2001 18:18:07 -0800 (PST) Cc: ips@ece.cmu.edu In-Reply-To: from "julian_satran@il.ibm.com" at Jan 15, 2001 02:48:39 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Exchange state should not go beyond each CDB in the linked command. The only requirement is that the same Initiator Task Tag be used for all the CDBs in the linked command [and thereby, the task tag cannot be re-used for another task, until the linked command is complete]. So, I don't think this is an issue for Linked Commands. However, the Abort Task/Task_Set, Clear Task Set, LUN Reset and Target Reset : a) Involve device drivers co-ordinating individual I/O aborts across different NIC instances to invalidate the SCSI H/W Assist structures that are in use by each NIC for the I/Os being aborted on their connections. b) Could cause out-of-order issues with commands arriving after completion of the task management command. While it is true that command ordering should handle any out-of-order issues when notifying the SCSI ULP at the target end, it may not be sufficient for the following reasons : o The Task Management Commands may be sent with a CmdRN of 0 (immediate delivery). o The Abort process at the target end would typically involve invalidation of SCSI Hardware Assist structures at the iSCSI layer and this act is not subject to the ordering enforced by CmdRN. BTW, SCSI-FCP implementations are not likely to be prone to this out-of-ordering syndrome, because, in a private loop, order is maintained and all fabric initiator implementations are required to request in-order delivery in their FLOGI with the fabric. (FC-FLA Table 3). Enforcing "Connection Allegiance" can solve the out-of-ordering and multiple network adapter connection issue for the Abort Task. For Abort Task Set, Clear Task Set, LUN Reset and Target Reset, out-of-ordering and dependence across multiple NIC instances seem to be issues. Regards, Santosh > > Santosh, > > That is an interesting point. Not so much for abort task but for linked > commands. > For abort task we could point out that it might we advisable to keep > connection allegiance but > we want a way to abort tasks beyond the connection (as your argument holds > for clear task set too). > On the other hand we did not require connection allegiance beyond > individual commands in > a linked command set. Do your exchange state go beyond individual commands > in a linked set? > > Julo > > > > Santosh Rao on 15/01/2001 09:41:46 > > Please respond to Santosh Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: iSCSI: Abort task > > > > > Julian, > > Another reason for enforcing "connection allegiance" of Abort Task and its > associated task : > > In the case of multi-connection sessions, each connection can be using a > different NIC. The SCSI command state information is maintained in SCSI > hardware assists (SCSI Exchange State Tables) on the adapter and such > structures need to be invalidated while aborting an I/O. > > Handling the Abort Task and the task on different connections implies > having to then share state information across NIC instances, something > that iSCSI has been trying to avoid. > > Regards, > Santosh > > > > > > > > > > Pierre, > > > > I am not sure that I follow completely (and accurately) your line of > > thought but if you have a multiple connection session the command > > numbering is meant to offer a total ordering. Any of the scenarios you > > mention can't happen unless the target decides to deliver commands to the > > "SCSI layer" out of order. > > And even if it does deliver command out of order (there might be good > > reasons for it) it should build a "sync barrier" for those commands that > > mandate ordering. This type of problem exist also in FCP (on a single > > connection!) and I assume that the thinking there is to use the per LU > > command ordering to avoid task management PDUs (like clear a queue) > > arriving before the commands they are referring to. > > > > I hope this clarifies the issue. > > > > Julo > > > > Pierre Labat on 09/01/2001 11:39:53 > > > > Please respond to Pierre Labat > > > > To: ips@ece.cmu.edu > > cc: > > Subject: iSCSI: Abort task > > > > > > > > > > Julian, > > > > > > It seems that we need to specify more the "Abort task" to avoid > > some problems. Below are 3 points we should add in the spec: > > > > > > 1) An "Abort task" must be sent on the TCP connection that was used > > to send the command. This does not apply if the connection used > > for the command has been "logged out". In this case of course, the > > "Abort task" can be send on another connection and it will not hurt. > > > > It is to avoid that a command that was delayed in the network, finally > > make it > > after the "Abort task" as been processed. > > > > 2) When the initiator receives the "Abort task" response, if the > > ExpCmdRN returned in the "Abort task" response is > > <= CmdRN of the aborted task, the initiator must re-use this > > CmdRN for another task (any). > > > > 3) The target when receiving an "Abort task" for a task whose the > > CmdRN (retreived from the initiator task tag) is >= ExpCmdRN > > must not bridge this CmdRN. I mean by bridge: allow ExpCmdRN to > > pass the CmdRN value of the aborted task. It will be up to the > > initiator > > to send another command with the same CmdRN in order to allow > > the target to move ExpCmdRN over the CmdRN value of the > > aborted task. > > > > > > > > "3)" Is to prevent to following scenario: > > > > Two TCP connections 1 and 2 in the session > > ExpCmdRN = 2 > > > > a) the command with CmdRN = 3 is sent over connection1 > > > > b) the command with CmdRN = 4 is sent over connection 2 > > > > c) the command CmdRN = 4 reaches the target > > > > d) the initiator aborts the command 4 and bridges CmdRN=4 > > > > e) the target receives the command 3, and as CmdRN=4 is bridged, > > advances ExpCmdRN to 5 > > > > f) the initiator sends another command (not related with the aborted > > one) with CmdRN=4 > > > > g) the target drops this command because CmdRN is less than ExpCmdRN. > > > > > > > > "2)" needs to exist because if the initiator doesn't re-use the CmdRN > > the target > > will deadlock because ExpCmdRN will be stuck behind the CmdRN of > > the aborted command. > > > > > > > > An alternative to the points 2) and 3) would be the initiator not > > re-using > > the CmdRN of the aborted commands. But this requires the target to > > bridge > > all the CmdRNs of the Aborted tasks (need extra states) and anyway there > > > > is a scenario where it doesn't work. Hence this is a bad alternative. > > > > Scenario where it doesn't work: > > > > Session with 2 connections 1 and 2. > > ExpCmdRN=1 > > > > a) Command with CmdRN=4 is sent over connection1 > > > > b) Connection 1 drops and the command 4 will never make it to the > > target. > > > > c) A logout for the connection 1 is sent over connection 2 > > > > d) The initiator sends an "Abort task" (for the task CmdRN=4) over > > the connection 2 > > > > e) The target, from the referenced initiator task tag cannot find the > > CmdRN of the aborted task because the initiator task tag > > corresponds > > to nothing because the command has never been received. > > Hence the target is not able to bridge CmdRN=4 > > > > f) As the initiator does not re-use CmdRN=4 the target will deadlock > > when ExpCmdRN will be 4. > > > > > > Regards, > > > > Pierre > > > > > > > > > > > > > -- > ################################# > Santosh Rao > Software Design Engineer, > HP, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################# > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Mon Jan 15 23:53:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA22477 for ; Mon, 15 Jan 2001 23:53:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0G2n8Z11396 for ips-outgoing; Mon, 15 Jan 2001 21:49:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0G2mHs11376 for ; Mon, 15 Jan 2001 21:48:22 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id VAA32820; Mon, 15 Jan 2001 21:41:27 -0500 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id TAA149428; Mon, 15 Jan 2001 19:48:15 -0700 Importance: Normal Subject: RE:draft-ietf-ips-iscsi-boot-01.txt To: "Douglas Otis" Cc: X-Mailer: Lotus Notes Release 5.0.5 September 22, 2000 Message-ID: From: "Prasenjit Sarkar" Date: Mon, 15 Jan 2001 18:48:04 -0800 X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.5 |September 22, 2000) at 01/15/2001 06:48:15 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I thought that the following paragraph in draft-01 allowed servername to be represented in both IPv4 and IPv6 formats. The "servername" is the name of iSCSI server and contains either a valid domain name, a literal IPv4 address, or a bracketed literal IPv6 address. If the servername field contains a literal IPv4 address, the IPv4 address is in standard dotted decimal notation. If the servername field contains an IPv6 address, the address is represented in bracketed literal IPv6 address format. The main iSCSI standard defines targetname, we will adhere to whatever is accepted there. -Prasenjit ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ All, The field consists of an UTF-8[20] string and has the following composition: ":" ":" ":" Would it be possible to adopt language explaining that servername and targetname could be dotted decimal expansions of the IP address in place of the name. DNS code for a bootstrap adds complexity. It would make it easier if perhaps some symbols could be used to indicate such an expansion has been made such as .IP6.INT or .IP4.INT suffix where the bytes are in reverse order with no embedded spaces. Doug > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-boot-01.txt From owner-ips@ece.cmu.edu Tue Jan 16 00:57:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA23132 for ; Tue, 16 Jan 2001 00:57:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0G35Ao11816 for ips-outgoing; Mon, 15 Jan 2001 22:05:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0G349s11790 for ; Mon, 15 Jan 2001 22:04:09 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id C6R66SRZ; Mon, 15 Jan 2001 19:04:07 -0800 From: "Y P Cheng" To: "John Hufferd" Cc: Subject: RE: iSCSI & iFCP Overlapping (Was iFCP as an IP Storage Work Item) Date: Mon, 15 Jan 2001 19:02:50 -0800 Message-ID: <003e01c07f68$cf7d5e00$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > Please reframe from speculation about iFCP as an End to End > protocal. iFCP has not been proposed by its authors to be End to End, > and is not up for discussion without an approprate draft. > The authors and the iFCP Draft clearly define iFCP as an Gateway to > Gateway protocol. We are currently having enough problems just working > on the Gateway to Gateway issues, and do not need to explore areas where > the Draft did not go. Please lets limit further iFCP discussions to > iFCP as a Gateway to Gateway protocol. John, Thank you for taking time to write a long essay about iFCP in another posting. It is not my intent to compare iFCP to an end-to-end solution like iSCSI. Let me respond by staying within the realm of treating iFCP as a Gateway protocol. I want to look at the whole issue from another angle. Most of us would agree the market driver for iSCSI, iFCP and FCIP is the need to have block-address storage devices on Internet. The number of new SSP companies tells the story. In fact, we can divide the world into two: clients and SSPs, even when many data centers only serve internal customers. As a client, if he has made no investment to fibre channel before, iSCSI is the nature choice provided his SSP supports iSCSI. If his SSP has FC or the client has already invested in FC, he needs to add a gateway of iFCP or FCIP whichever is required by his SSP. Of course, he can choose iSCSI if it is supported by his SSP too. His decision is based on cost, confidence, availability and performance. In other words, iSCSI, iFCP, and FCIP are choices of a client. Most likely, he is unconcerned whether it is end-to-end or a gateway until he writes the check. As an SSP, for some time to come, most will be built around FC and SCSI devices. (Some SSP may even wish to build around ATA drives. Checks with 3ware.) However, an SSP may choose to "export" its disk drives with iSCSI, iFCP, or FCIP support. Notice, what devices an SSP has vs. what it chooses to export can be unrelated. It can do so simply with the target HBAs that support the iSCSI, iFCP or FCIP protocols while the storage devices are configured as "virtual" drives. An SSP can invest switches supporting iSCSI, iFCP and FCIP for export as well, even iFCP is a gateway and iSCSI is an end-to-end protocol. You have mentioned more than once that the iFCP gateway supplier would support iSCSI in the future. The exporting protocol of the devices is only important to the clients as long as they don't have to throw away the old investment or spend a lot of money buying new equipment. Most companies making RAID or large storage boxes is quite familiar with this virtual device game. For example, you can choose your own FC or SCSI connection for a RAID box. It is in this context that I believe iSCSI and iFCP may have overlapping utilities because they both take full advantage of IP routing. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ece.cmu.edu Tue Jan 16 02:59:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA06966 for ; Tue, 16 Jan 2001 02:59:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0G5RCn15327 for ips-outgoing; Tue, 16 Jan 2001 00:27:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0G5QPs15320 for ; Tue, 16 Jan 2001 00:26:30 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Mon, 15 Jan 2001 21:27:18 -0800 Message-ID: From: Joshua Tseng To: Ken Hirata , ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item Date: Mon, 15 Jan 2001 21:25:51 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Yes, you are correct. A disruption in the IP network at an inopportune moment can lead to the FC fabric deciding the tunneled ISL is no good, leaving it isolated indefinitely. Human intervention will be necessary (unless the FCIP gateway does it) to reconfigure the fabric (and trigger the reallocation of DOMAIN_ID's) so that connectivity across the tunnel can be restored. BTW, this isn't a small issue either, as disruptions in the IP network are a common and expected event. Thanks, Josh > -----Original Message----- > From: Ken Hirata [mailto:Ken.Hirata@Vixel.com] > Sent: Monday, January 15, 2001 10:49 AM > To: Joshua Tseng; ips@ece.cmu.edu > Cc: Y P Cheng > Subject: Re: iFCP as an IP Storage Work Item > > > Just a nit, see embedded comment. Ken > > Joshua Tseng wrote: > > > Furthermore, as Wayland pointed out earlier, a single large > FC fabric > > stretched over multiple WAN links by FCIP is vulnerable to temporary > > disruptions in the IP network which may trigger reconfigurations in > > the FC fabric, such as reallocations of DOMAIN_ID's. > > Reallocations of Domain IDs will not occur without administrative > intervention if the switches in the Fibre Channel Fabric comply with > the Methodologies for Interconnects (FC-MI) specification. > > > iFCP addresses > > this issue by assigning N_PORT ID's locally, eliminating > the dependence > > on a central addressing authority and the need for any > "Class F" traffic. > > The stability and scalability of a large storage network is > thus improved. > > > > Josh > > -- > Kenneth Hirata > Vixel Corporation > Irvine, CA 92618 > Phone: (949) 450-6100 > Email: khirata@vixel.com > > From owner-ips@ece.cmu.edu Tue Jan 16 03:55:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA07218 for ; Tue, 16 Jan 2001 03:55:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0G6aFl16614 for ips-outgoing; Tue, 16 Jan 2001 01:36:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0G6Zks16608 for ; Tue, 16 Jan 2001 01:35:46 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id BAA29888; Tue, 16 Jan 2001 01:28:56 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id XAA50680; Mon, 15 Jan 2001 23:35:44 -0700 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI & iFCP Overlapping (Was iFCP as an IP Storage Work Item) To: "Y P Cheng" Cc: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Mon, 15 Jan 2001 22:25:52 -0800 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 01/15/2001 11:35:44 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk YP, Please refrain from discussing iFCP in any context except what the Draft supports. We need to reach consensus on real proposals. If you want to author another draft we can perhaps discuss that. The authors of the current draft believe your comments are removing the focus they need on their iFCP Draft. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403 Internet address: hufferd@us.ibm.com "Y P Cheng" @ece.cmu.edu on 01/15/2001 07:02:50 PM Sent by: owner-ips@ece.cmu.edu To: John Hufferd/San Jose/IBM@IBMUS cc: Subject: RE: iSCSI & iFCP Overlapping (Was iFCP as an IP Storage Work Item) > Please reframe from speculation about iFCP as an End to End > protocal. iFCP has not been proposed by its authors to be End to End, > and is not up for discussion without an approprate draft. > The authors and the iFCP Draft clearly define iFCP as an Gateway to > Gateway protocol. We are currently having enough problems just working > on the Gateway to Gateway issues, and do not need to explore areas where > the Draft did not go. Please lets limit further iFCP discussions to > iFCP as a Gateway to Gateway protocol. John, Thank you for taking time to write a long essay about iFCP in another posting. It is not my intent to compare iFCP to an end-to-end solution like iSCSI. Let me respond by staying within the realm of treating iFCP as a Gateway protocol. I want to look at the whole issue from another angle. Most of us would agree the market driver for iSCSI, iFCP and FCIP is the need to have block-address storage devices on Internet. The number of new SSP companies tells the story. In fact, we can divide the world into two: clients and SSPs, even when many data centers only serve internal customers. As a client, if he has made no investment to fibre channel before, iSCSI is the nature choice provided his SSP supports iSCSI. If his SSP has FC or the client has already invested in FC, he needs to add a gateway of iFCP or FCIP whichever is required by his SSP. Of course, he can choose iSCSI if it is supported by his SSP too. His decision is based on cost, confidence, availability and performance. In other words, iSCSI, iFCP, and FCIP are choices of a client. Most likely, he is unconcerned whether it is end-to-end or a gateway until he writes the check. As an SSP, for some time to come, most will be built around FC and SCSI devices. (Some SSP may even wish to build around ATA drives. Checks with 3ware.) However, an SSP may choose to "export" its disk drives with iSCSI, iFCP, or FCIP support. Notice, what devices an SSP has vs. what it chooses to export can be unrelated. It can do so simply with the target HBAs that support the iSCSI, iFCP or FCIP protocols while the storage devices are configured as "virtual" drives. An SSP can invest switches supporting iSCSI, iFCP and FCIP for export as well, even iFCP is a gateway and iSCSI is an end-to-end protocol. You have mentioned more than once that the iFCP gateway supplier would support iSCSI in the future. The exporting protocol of the devices is only important to the clients as long as they don't have to throw away the old investment or spend a lot of money buying new equipment. Most companies making RAID or large storage boxes is quite familiar with this virtual device game. For example, you can choose your own FC or SCSI connection for a RAID box. It is in this context that I believe iSCSI and iFCP may have overlapping utilities because they both take full advantage of IP routing. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ece.cmu.edu Tue Jan 16 05:12:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA07632 for ; Tue, 16 Jan 2001 05:12:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0G7oHU17994 for ips-outgoing; Tue, 16 Jan 2001 02:50:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0G7o1s17984 for ; Tue, 16 Jan 2001 02:50:01 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 12156FD2; Mon, 15 Jan 2001 23:50:00 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id XAA07163; Mon, 15 Jan 2001 23:49:29 -0800 (PST) From: Santosh Rao Message-Id: <200101160749.XAA07163@hpcuhe.cup.hp.com> Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. To: julian_satran@il.ibm.com Date: Mon, 15 Jan 2001 23:49:28 -0800 (PST) Cc: ips@ece.cmu.edu In-Reply-To: from "julian_satran@il.ibm.com" at Jan 15, 2001 01:08:26 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Thanks for accepting this change into the draft. I hope I understood your resolution correctly as follows : The service response exchanged between the LLP and ULP is specified by each O.S' SCSI Stack and the draft is not required to elaborate on this. The service response exchanged b/n the target and initiator in PDU responses must be specified by the draft. (Response field in SCSI Task Mgmt Response, Response Data in SCSI Response PDU, Reason field in Reject PDU). Targets and Initiators will indicate a service response of "Service Delivery or Target Failure" in case of header format errors. Thanks & Regards, Santosh > > > > Santosh, > > OK - you have convinced me (and I've found the CAM3 list of statuses). I > will specify in 04 the relevant values for the service response in line > with CAM3. > > Julo > > Santosh Rao on 15/01/2001 00:34:36 > > Please respond to Santosh Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. > > > > > > It would be simple if things where that clear-cut. > > > > What is a format error coming from a target? IMHO it is a target error > and > > not a protocol failure. > > Julian, > > A SCSI Protocol(i.e. iSCSI) header format error constitutes a > "Service Delivery or Target Failure" service response, since the failure > occurs in the service delivery sub-system (SCSI Protocol). SAM-2 explains > this service response as "The command has ended due to a service delivery > failure or target device mal-function", which should be the > service response to be returned on a header format error. > > Further, SAM-2 Section 5 also states that the application client > (i.e. the SCSI upper layer driver) shall treat the SCSI Status to be > un-defined if the command ends with a service response of > "Service Delivery or Target Failure". > This implies that the SCSI Upper Layer driver shall ignore the > CHECK CONDITION scsi status that iSCSI initiators are going to fake to > their upper layers on a header format error, due to a service response of > "Service Delivery or Target Mal-function". > > So, is iSCSI proposing to return a service response of "Task Complete" to > its upper layers on a header format error ? If not and it is the intent > that a service response of "Service Delivery or Target Mal-function" > should be returned, then, the upper layer scsi driver is going to ignore > the CHECK CONDITION that iSCSI intends to fake. > > It would be interesting to hear what T10 has to say on this approach that > iSCSI is currently proposing to adopt on a header format error. > > I see the following problems with the current approach being advocated : > ------------------------------------------------------------------------ > > 1) It does NOT solve the problem for commands other than the SCSI Command > PDU. I am yet to see a response to my questions on how header format > errors for Login, Logout, Text, NOP-OUT, NOP-IN and SCSI Task Management > Command PDU are handled by the current proposal described in section 5.4 > of the 03 iSCSI draft. > > 2) It is not in line with [violates ?] Section 5 of SAM-2. > > 3) It can cause upper layers to initiate Auto Contingent Allegiance > recovery such as CLEAR ACA due to the CHECK CONDITION returned. > > 4) It can cause upper layers to over-react on seeing a HARDWARE ERROR by > resorting to error recovery such as BDR to recover from a HARDWARE ERROR. > > 5) It adds extra complexity to the iSCSI initiator drivers by making them > SPC-2 aware and having to generate sense data on behalf of a target, > not something that has been required in other SCSI Protocols such as > parallel scsi and fibre channel. > > 6) It causes a mis-leading error log of a > HARDWARE ERROR, where, a more specific error log of the > header format error that occurred based on the response data would > have been more useful in quick fault isolation. > > 7) Last, but not the least, it is a violation of layering between the > ULP and LLP, wherein, an LLP Service Delivery Failure is being treated as > a ULP SCSI error returned by the LUN. > > I would like to propose the following solution : > ------------------------------------------------ > 1) On discovery of header format error, a target MUST convey the > specific type of format error that was discovered through use of > mechanisms like the Response Data in a SCSI Response PDU, > Response Field in a SCSI Task Management Response PDU > and the Reason Code field in the REJECT PDU. > > 2) On a header format error discovered at the initiator, > a service response of "Service Delivery or Target Failure" MUST be > returned to their upper layers [and the initiator may log the > specific header format error that was discovered.]. The draft need not > attempt to elaborate on these service response definitions since these are > defined by the SCSI Stack of each O.S. as a set of return values exchanged > b/n the LLP and the ULP. > > Benefits of the proposed solution : > ----------------------------------- > 1) It addresses the problem for header format errors on ALL types of PDUs. > > 2) It is in line with [complies with ?] SAM-2 semantics of "Execute > Command". > > 3) It provides more value-added error logging based on the specific header > format error that occurred rather than a more general HARDWARE ERROR, > allowing for quicker isolation and root cause of the problem. > > 4) It avoids ULPs resorting to inappropriate error recovery such as CLEAR > ACA or a BDR on seeing CHECK CONDITION with HARDWARE ERROR. > > 5) It retains layering semantics and differentiates between LLP service > delivery errors and ULP SCSI errors. > > I'd be happy to learn what implications of this issue I've missed. > > Thanks & Regards, > Santosh Rao > > > > > > > > Should target errors be reported in the service-response? Except for > task > > management that is common to all protocols I did not see any other thing > > popping up in any SCSI driver. > > > > Preaching layering won't make the issue disappear. > > > > Julo > > > > Stephen Bailey on 14/01/2001 17:10:56 > > > > Please respond to Stephen Bailey > > > > To: ips@ece.cmu.edu > > cc: > > Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. > > > > > > > > > > Julian, > > > > This seems like the zillionth time aired this same disagreement. > > > > I think we should try to reach a WG consensus on whether iSCSI should > > use SCSI status as a means for reporting protocol-related errors, and > > kill it once and for all. > > > > I'm strongly against it. > > > > > And an error in the iSCSI layer gets reported by the next layer - that > is > > > the regular layering technique (and BTW I am getting a bit uneasy about > > all > > > this preaching on layering when it is not obvious that you understand > all > > > the implications of the point). > > > > Santosh seems to understand 100%. I agree with Santosh. > > > > SAM defines three pieces of status returned by Execute Command() > > (which is, in turn implemented by each SCSI protocol, e.g. iSCSI): > > 1) Service Response: task complete, linked command complete, service > > delivery or target failure. > > 2) SCSI status byte > > 3) SCSI sense data > > > > SAM clearly suggest that a protocol's means for signalling > > protocol-detected errors is the service response status when it says: > > > > The actual protocol events corresponding to a response of TASK > > COMPLETE, LINKED COMMAND COMPLETE or SERVICE DELIVERY OR TARGET > > FAILURE shall be specified in each protocol standard. > > > > Note that defining protocol error events in terms of TC, LCC, SDF and > > TF, remains an abstraction. An actual implementation can chose to > > signal these events by whatever means. All SCSI implementations I've > > seen do have a status return component which exactly corresponds to > > SAM's service response status, but has more than just these four > > alternatives. Typically, that includes things like success, command > > timeout, addressing failure (selection timeout in ||SCSI, bad AL_PA in > > FCAL, etc.), command aborted, bus parity error, etc.. What iSCSI does > > need to do is clearly define its error events AS protocol events, > > which is what describing them in terms of the SAM specified set does. > > > > ||SCSI, FCP and the other SCSI protocol standards do this. > > > > Operationally, this means that in iSCSI: > > > > o protocol-specific errors for a task detected by the target without > > a CLEARLY corresponding SCSI error return should be signalled > > using the iSCSI response mechanism. > > > > The initiator will handle these errors by recording them in some > > appropriate way, and selecting an appropriate service response > > status value. > > > > o errors for a task detected directly by the initiator are handled > > by recording them in some appropriate way, and selecting an > > appropriate service response status value. > > > > I don't know that there's a single sentence or section in SAM which > > says this, but it clearly implies that the components of SCSI status > > (status byte and sense) are data which are CARRIED (not created) by > > SCSI protocols, for the use of the protocol-independent components. > > For example, a disk peripheral driver reacts to SCSI status and sense > > generated by disks for SCSI operations that it starts. > > > > SCSI status should be equivalent to the SCSI status returned by the > > logical unit. > > > > SCSI protocols are not citizens of the SCSI status space, which means > > that the real citizens (the command standards, and SAM), may define > > semantics of SCSI error codes which conflict with iSCSI's selections. > > The `hardware error' sense key might be defined to mean something very > > specific within a particular SCSI peripheral command set, and your > > choice of synthesizing SCSI status within the protocol could cause > > this behavior to misfire. > > > > > [js} the error was generated by a faulty controller and I did not find > > > any other SCSI sense fit for it[/js] > > > > That's because there IS no SCSI sense fit for it. It is a > > protocol-detected, protocol-unique error. Therefore it should be > > signalled using the service response status. > > > > Steph > > > > > > > > > > > -- > ################################# > Santosh Rao > Software Design Engineer, > HP, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################# > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Tue Jan 16 08:39:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id IAA11163 for ; Tue, 16 Jan 2001 08:39:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0GBsSZ00880 for ips-outgoing; Tue, 16 Jan 2001 06:54:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0GBrts00865 for ; Tue, 16 Jan 2001 06:53:55 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id B37713ED for ; Tue, 16 Jan 2001 04:53:54 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 02EB8AC for ; Tue, 16 Jan 2001 06:53:53 -0500 (EST) Received: from agilent.com (cos1nai130070.cos.agilent.com [141.184.130.70]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id DAA04384 for ; Tue, 16 Jan 2001 03:53:49 -0800 (PST) Message-ID: <3A63D118.A5F7DD5B@agilent.com> Date: Mon, 15 Jan 2001 20:42:00 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : On the subject of R2T and Task Tags. References: <200101090846.OAA29278@divyaroot.India.Sun.COM> <3A5B95B6.29771A0B@cup.hp.com> <3A5BA1FA.652B86CD@agilent.com> <3A5BD6AB.FB7F5590@cup.hp.com> <3A5CCB08.74D95ED9@agilent.com> <3A5D1968.9227D23A@hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Pierre Labat wrote: > To have differents tag or (tag + subtag) one for each R2T allows the target to find > directly the buffer address. For example using the tag as an index in a table. If > not, the target has > to go through a list of R2T control structures and for each one compare the offset > with > with the interval recorded in the R2T structure, in order to find the right buffer. > As it doesn't hurt on the initiator side, why not to do it? > > In fact an R2T and the data related are as an exchange. Hence it is natural > to have one identifier/exchange. > Of course if the target want to use the same it could. You already have everything you need. The target_task_tag references the context, and the Offset field references where in the buffer(s) to place the data. -Matt From owner-ips@ece.cmu.edu Tue Jan 16 08:43:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id IAA11415 for ; Tue, 16 Jan 2001 08:43:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0GBsPB00874 for ips-outgoing; Tue, 16 Jan 2001 06:54:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0GBrns00863 for ; Tue, 16 Jan 2001 06:53:49 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id DADBF3BA for ; Tue, 16 Jan 2001 04:53:48 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 089C983 for ; Tue, 16 Jan 2001 06:53:48 -0500 (EST) Received: from agilent.com (cos1nai130070.cos.agilent.com [141.184.130.70]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id DAA04380 for ; Tue, 16 Jan 2001 03:53:45 -0800 (PST) Message-ID: <3A63D068.82931D2A@agilent.com> Date: Mon, 15 Jan 2001 20:39:04 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : On the subject of R2T and Task Tags. References: <200101110720.XAA26173@hpcuhe.cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > The advantage of defining the fields explicitly in the header as opposed > to allowing targets to vertically encode a portion of the tag field for > tracking the R2T is that these standard fields are interpret-able > by iSCSI Analyzers and will make debugging and tracking the > sequence of the I/O easier than implementation specific > vertical encoding into a single field. Uh, now that any kind of "out of band" framing is gone, it will be very difficult to make an analyzer that is capable of syncing up with an iSCSI session. -Matt From owner-ips@ece.cmu.edu Tue Jan 16 09:43:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id JAA13270 for ; Tue, 16 Jan 2001 09:43:36 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0GBsUo00883 for ips-outgoing; Tue, 16 Jan 2001 06:54:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0GBrls00861 for ; Tue, 16 Jan 2001 06:53:47 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 8444B41D for ; Tue, 16 Jan 2001 04:53:46 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 43A6B14F for ; Tue, 16 Jan 2001 06:53:43 -0500 (EST) Received: from agilent.com (cos1nai130070.cos.agilent.com [141.184.130.70]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id DAA04376 for ; Tue, 16 Jan 2001 03:53:39 -0800 (PST) Message-ID: <3A63CB93.2BFD8351@agilent.com> Date: Mon, 15 Jan 2001 20:18:27 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI: header digest error at initiator References: <3A5B5090.14444240@hp.com> <3A5BD336.F983F9F8@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > > If this is the intention of the recommended error recovery, it is the > result of not allowing score-boarding. By score-boarding an initiator > would detect an underrun and would just error the affected I/O back. Two comments here. First, in your example, the initiator is inventing an error that really didn't occur in the target. The target completed the I/O successfully, it was the transport that experienced an error, but you're treating it like a target error. Second, you say that initiators and targets routinely perform scoreboarding. How is this done today? Buffer(s) are provided to an I/O chip. The I/O chip writes the data into the buffers. It does not have the memory to determine that each and every byte has been written to. So how is the initiator/target supposed to be absolutely sure that every byte was written to? > this case, all outstanding tasks on that connection are being affected. > > Both the Format Error and Digest Error handling seem too extreme. A > format error or digest error recovery should only involve the affected > task and none others. > > Thanks, > Santosh -Matt Wakeley From owner-ips@ece.cmu.edu Tue Jan 16 09:50:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id JAA13391 for ; Tue, 16 Jan 2001 09:50:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0GAwNS29889 for ips-outgoing; Tue, 16 Jan 2001 05:58:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0GAwDs29882 for ; Tue, 16 Jan 2001 05:58:13 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id LAA112014 for ; Tue, 16 Jan 2001 11:58:06 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id LAA94314 for ; Tue, 16 Jan 2001 11:58:07 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D6.003C3C73 ; Tue, 16 Jan 2001 11:57:56 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 16 Jan 2001 12:52:55 +0200 Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk That is a good approximation although I am not sure about all of them. Julo Santosh Rao on 16/01/2001 09:49:28 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. Julian, Thanks for accepting this change into the draft. I hope I understood your resolution correctly as follows : The service response exchanged between the LLP and ULP is specified by each O.S' SCSI Stack and the draft is not required to elaborate on this. The service response exchanged b/n the target and initiator in PDU responses must be specified by the draft. (Response field in SCSI Task Mgmt Response, Response Data in SCSI Response PDU, Reason field in Reject PDU). Targets and Initiators will indicate a service response of "Service Delivery or Target Failure" in case of header format errors. Thanks & Regards, Santosh > > > > Santosh, > > OK - you have convinced me (and I've found the CAM3 list of statuses). I > will specify in 04 the relevant values for the service response in line > with CAM3. > > Julo > > Santosh Rao on 15/01/2001 00:34:36 > > Please respond to Santosh Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. > > > > > > It would be simple if things where that clear-cut. > > > > What is a format error coming from a target? IMHO it is a target error > and > > not a protocol failure. > > Julian, > > A SCSI Protocol(i.e. iSCSI) header format error constitutes a > "Service Delivery or Target Failure" service response, since the failure > occurs in the service delivery sub-system (SCSI Protocol). SAM-2 explains > this service response as "The command has ended due to a service delivery > failure or target device mal-function", which should be the > service response to be returned on a header format error. > > Further, SAM-2 Section 5 also states that the application client > (i.e. the SCSI upper layer driver) shall treat the SCSI Status to be > un-defined if the command ends with a service response of > "Service Delivery or Target Failure". > This implies that the SCSI Upper Layer driver shall ignore the > CHECK CONDITION scsi status that iSCSI initiators are going to fake to > their upper layers on a header format error, due to a service response of > "Service Delivery or Target Mal-function". > > So, is iSCSI proposing to return a service response of "Task Complete" to > its upper layers on a header format error ? If not and it is the intent > that a service response of "Service Delivery or Target Mal-function" > should be returned, then, the upper layer scsi driver is going to ignore > the CHECK CONDITION that iSCSI intends to fake. > > It would be interesting to hear what T10 has to say on this approach that > iSCSI is currently proposing to adopt on a header format error. > > I see the following problems with the current approach being advocated : > ------------------------------------------------------------------------ > > 1) It does NOT solve the problem for commands other than the SCSI Command > PDU. I am yet to see a response to my questions on how header format > errors for Login, Logout, Text, NOP-OUT, NOP-IN and SCSI Task Management > Command PDU are handled by the current proposal described in section 5.4 > of the 03 iSCSI draft. > > 2) It is not in line with [violates ?] Section 5 of SAM-2. > > 3) It can cause upper layers to initiate Auto Contingent Allegiance > recovery such as CLEAR ACA due to the CHECK CONDITION returned. > > 4) It can cause upper layers to over-react on seeing a HARDWARE ERROR by > resorting to error recovery such as BDR to recover from a HARDWARE ERROR. > > 5) It adds extra complexity to the iSCSI initiator drivers by making them > SPC-2 aware and having to generate sense data on behalf of a target, > not something that has been required in other SCSI Protocols such as > parallel scsi and fibre channel. > > 6) It causes a mis-leading error log of a > HARDWARE ERROR, where, a more specific error log of the > header format error that occurred based on the response data would > have been more useful in quick fault isolation. > > 7) Last, but not the least, it is a violation of layering between the > ULP and LLP, wherein, an LLP Service Delivery Failure is being treated as > a ULP SCSI error returned by the LUN. > > I would like to propose the following solution : > ------------------------------------------------ > 1) On discovery of header format error, a target MUST convey the > specific type of format error that was discovered through use of > mechanisms like the Response Data in a SCSI Response PDU, > Response Field in a SCSI Task Management Response PDU > and the Reason Code field in the REJECT PDU. > > 2) On a header format error discovered at the initiator, > a service response of "Service Delivery or Target Failure" MUST be > returned to their upper layers [and the initiator may log the > specific header format error that was discovered.]. The draft need not > attempt to elaborate on these service response definitions since these are > defined by the SCSI Stack of each O.S. as a set of return values exchanged > b/n the LLP and the ULP. > > Benefits of the proposed solution : > ----------------------------------- > 1) It addresses the problem for header format errors on ALL types of PDUs. > > 2) It is in line with [complies with ?] SAM-2 semantics of "Execute > Command". > > 3) It provides more value-added error logging based on the specific header > format error that occurred rather than a more general HARDWARE ERROR, > allowing for quicker isolation and root cause of the problem. > > 4) It avoids ULPs resorting to inappropriate error recovery such as CLEAR > ACA or a BDR on seeing CHECK CONDITION with HARDWARE ERROR. > > 5) It retains layering semantics and differentiates between LLP service > delivery errors and ULP SCSI errors. > > I'd be happy to learn what implications of this issue I've missed. > > Thanks & Regards, > Santosh Rao > > > > > > > > Should target errors be reported in the service-response? Except for > task > > management that is common to all protocols I did not see any other thing > > popping up in any SCSI driver. > > > > Preaching layering won't make the issue disappear. > > > > Julo > > > > Stephen Bailey on 14/01/2001 17:10:56 > > > > Please respond to Stephen Bailey > > > > To: ips@ece.cmu.edu > > cc: > > Subject: Re: iSCSI : Initiators expected to fake CHECK CONDITIONS. > > > > > > > > > > Julian, > > > > This seems like the zillionth time aired this same disagreement. > > > > I think we should try to reach a WG consensus on whether iSCSI should > > use SCSI status as a means for reporting protocol-related errors, and > > kill it once and for all. > > > > I'm strongly against it. > > > > > And an error in the iSCSI layer gets reported by the next layer - that > is > > > the regular layering technique (and BTW I am getting a bit uneasy about > > all > > > this preaching on layering when it is not obvious that you understand > all > > > the implications of the point). > > > > Santosh seems to understand 100%. I agree with Santosh. > > > > SAM defines three pieces of status returned by Execute Command() > > (which is, in turn implemented by each SCSI protocol, e.g. iSCSI): > > 1) Service Response: task complete, linked command complete, service > > delivery or target failure. > > 2) SCSI status byte > > 3) SCSI sense data > > > > SAM clearly suggest that a protocol's means for signalling > > protocol-detected errors is the service response status when it says: > > > > The actual protocol events corresponding to a response of TASK > > COMPLETE, LINKED COMMAND COMPLETE or SERVICE DELIVERY OR TARGET > > FAILURE shall be specified in each protocol standard. > > > > Note that defining protocol error events in terms of TC, LCC, SDF and > > TF, remains an abstraction. An actual implementation can chose to > > signal these events by whatever means. All SCSI implementations I've > > seen do have a status return component which exactly corresponds to > > SAM's service response status, but has more than just these four > > alternatives. Typically, that includes things like success, command > > timeout, addressing failure (selection timeout in ||SCSI, bad AL_PA in > > FCAL, etc.), command aborted, bus parity error, etc.. What iSCSI does > > need to do is clearly define its error events AS protocol events, > > which is what describing them in terms of the SAM specified set does. > > > > ||SCSI, FCP and the other SCSI protocol standards do this. > > > > Operationally, this means that in iSCSI: > > > > o protocol-specific errors for a task detected by the target without > > a CLEARLY corresponding SCSI error return should be signalled > > using the iSCSI response mechanism. > > > > The initiator will handle these errors by recording them in some > > appropriate way, and selecting an appropriate service response > > status value. > > > > o errors for a task detected directly by the initiator are handled > > by recording them in some appropriate way, and selecting an > > appropriate service response status value. > > > > I don't know that there's a single sentence or section in SAM which > > says this, but it clearly implies that the components of SCSI status > > (status byte and sense) are data which are CARRIED (not created) by > > SCSI protocols, for the use of the protocol-independent components. > > For example, a disk peripheral driver reacts to SCSI status and sense > > generated by disks for SCSI operations that it starts. > > > > SCSI status should be equivalent to the SCSI status returned by the > > logical unit. > > > > SCSI protocols are not citizens of the SCSI status space, which means > > that the real citizens (the command standards, and SAM), may define > > semantics of SCSI error codes which conflict with iSCSI's selections. > > The `hardware error' sense key might be defined to mean something very > > specific within a particular SCSI peripheral command set, and your > > choice of synthesizing SCSI status within the protocol could cause > > this behavior to misfire. > > > > > [js} the error was generated by a faulty controller and I did not find > > > any other SCSI sense fit for it[/js] > > > > That's because there IS no SCSI sense fit for it. It is a > > protocol-detected, protocol-unique error. Therefore it should be > > signalled using the service response status. > > > > Steph > > > > > > > > > > > -- > ################################# > Santosh Rao > Software Design Engineer, > HP, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################# > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Tue Jan 16 13:33:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA18758 for ; Tue, 16 Jan 2001 13:33:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0GGLop07885 for ips-outgoing; Tue, 16 Jan 2001 11:21:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcars04f.ca.nortel.com (h57s242a129n47.user.nortelnetworks.com [47.129.242.57]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0GGKvs07857 for ; Tue, 16 Jan 2001 11:20:57 -0500 (EST) Received: from zbl6c012.corpeast.baynetworks.com by zcars04f.ca.nortel.com; Tue, 16 Jan 2001 10:57:00 -0500 Received: from zbl6c002.corpeast.baynetworks.com ([132.245.205.52]) by zbl6c012.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id ZLK0BLGP; Tue, 16 Jan 2001 10:53:22 -0500 Received: from nortelnetworks.com (dhcp223-140.engeast.baynetworks.com [192.32.223.140]) by zbl6c002.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id CN99DJD5; Tue, 16 Jan 2001 10:53:22 -0500 Message-ID: <3A646E8E.D32FFE3A@nortelnetworks.com> Date: Tue, 16 Jan 2001 10:53:50 -0500 X-Sybari-Space: 00000000 00000000 00000000 From: "Victor Firoiu" Organization: Nortel Networks X-Mailer: Mozilla 4.76 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: somesh_gupta CC: Stephen Bailey , ips , "Franco Travostino" Subject: Re: Performance of iSCSI, FCIP and iFCP References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Orig: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Somesh, my answers between your lines. Victor Somesh Gupta wrote: > > Victor, > > First I would like to address the second case where - based on the > need to be fair - you raise the point that multiple connections per > session is closer to emulating multiple users on a server than a > single connection. That being a valuable point, is probably better > addressed at the upper layers of the host software, where multiple > independent sessions can be created to the same target. The argument > made more often in favor of multiple connections is that the likelyhood > of some connections escaping RED - or other disasters - when you have > multiple connections is good and therefore the overall performance > will be better. > Not sure what you mean by "multiple connections per session." Maybe I was not clear, I was arguing for one TCP connection for one storage connection, which can give multiple TCP connectons for one pair of IP storage gateways. The performance considerations that I presented in my first message regard any possible packet loss, including RED. > However, as Steph pointed out, that skirts the issue of fairness. > > My point about the modifications to congestion control algorithm was > to point out that your work makes an interesting observation(or my > paraphrasing of it) - that it becomes less and less possible to > acheive the maximum possible throughput (in LFNs) as the rate and > distances get larger and larger - and that issue should be addressed > directly. This is true even when a single "data flow" or "application" > might have the entire TCP bandwidth at its disposal. A modification > hopefully will not create super TCPs i.e. would be a reasonable approx > of current equation (flow slow long links or fast short links), but > would provide a better choice (which should propagate to all TCPs) > for 10G and beyond at long distances. > As I was saying in some other message, modifying TCP is very problematic for many reasons, and it is definetly outside the scope of this WG. > There is a cost of synchronizing across multiple connections - both > on the target and the initiator. Someone has to do the work, be it > the host or the adapter. On the initiator side, multiple > connections/adapters (actually their completion routine) will be updating > a single variable MaxCmdRn and the software posting the commands will > have to ensure that MaxCmdRn is not exceeded and has to post the commands > to the appropriate queues (how does it know which). And that is when > there is no stall. Assuming this informations is written/read from > multiple processors (with the pinging of the cache lines), the cost > will escalate rapidly with the number of processors involved. > > There is similar situation on the target. It probably > will prevent different controller boards on the target from sharing > connections in a session (whenever one of the controller decides on a > new increased value of MaxCmdRn, it should be propagated to the other > controller also ??) - maybe this is not so. > > That is why I am very interested in any pointers you or anyone else > could be provide on high-speed implementation of a "single data-flow" > across multiple connections. I am sure this has been tried in the > past so hopefully there are some pointers to results. > > Somesh > I think that what you are saying is somewhat orthogonal to my points. I was arguing for one TCP connection for one storage connection, which is different, simpler and more natural than bundling or anonymizing all storage connections in one big flow and then striping a "single data-flow" across multiple connections. I am not sure about other literature, I'll be looking for it myself. Victor From owner-ips@ece.cmu.edu Tue Jan 16 13:34:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA18786 for ; Tue, 16 Jan 2001 13:34:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0GG3aY07228 for ips-outgoing; Tue, 16 Jan 2001 11:03:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mtiwmhc27.worldnet.att.net (mtiwmhc27.worldnet.att.net [204.127.131.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0GD1gs02185 for ; Tue, 16 Jan 2001 08:01:43 -0500 (EST) Received: from SG2351 ([32.102.202.72]) by mtiwmhc27.worldnet.att.net (InterMail vM.4.01.03.10 201-229-121-110) with SMTP id <20010116130135.YIKC2234.mtiwmhc27.worldnet.att.net@SG2351>; Tue, 16 Jan 2001 13:01:35 +0000 Reply-To: From: "Somesh Gupta" To: "Victor Firoiu" , "Stephen Bailey" Cc: , "Franco Travostino" Subject: RE: Performance of iSCSI, FCIP and iFCP Date: Tue, 16 Jan 2001 04:58:47 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <3A6340DC.637128E3@nortelnetworks.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Victor, First I would like to address the second case where - based on the need to be fair - you raise the point that multiple connections per session is closer to emulating multiple users on a server than a single connection. That being a valuable point, is probably better addressed at the upper layers of the host software, where multiple independent sessions can be created to the same target. The argument made more often in favor of multiple connections is that the likelyhood of some connections escaping RED - or other disasters - when you have multiple connections is good and therefore the overall performance will be better. However, as Steph pointed out, that skirts the issue of fairness. My point about the modifications to congestion control algorithm was to point out that your work makes an interesting observation(or my paraphrasing of it) - that it becomes less and less possible to acheive the maximum possible throughput (in LFNs) as the rate and distances get larger and larger - and that issue should be addressed directly. This is true even when a single "data flow" or "application" might have the entire TCP bandwidth at its disposal. A modification hopefully will not create super TCPs i.e. would be a reasonable approx of current equation (flow slow long links or fast short links), but would provide a better choice (which should propagate to all TCPs) for 10G and beyond at long distances. There is a cost of synchronizing across multiple connections - both on the target and the initiator. Someone has to do the work, be it the host or the adapter. On the initiator side, multiple connections/adapters (actually their completion routine) will be updating a single variable MaxCmdRn and the software posting the commands will have to ensure that MaxCmdRn is not exceeded and has to post the commands to the appropriate queues (how does it know which). And that is when there is no stall. Assuming this informations is written/read from multiple processors (with the pinging of the cache lines), the cost will escalate rapidly with the number of processors involved. There is similar situation on the target. It probably will prevent different controller boards on the target from sharing connections in a session (whenever one of the controller decides on a new increased value of MaxCmdRn, it should be propagated to the other controller also ??) - maybe this is not so. That is why I am very interested in any pointers you or anyone else could be provide on high-speed implementation of a "single data-flow" across multiple connections. I am sure this has been tried in the past so hopefully there are some pointers to results. Somesh -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Victor Firoiu Sent: Monday, January 15, 2001 10:27 AM To: Stephen Bailey Cc: ips@ece.cmu.edu; Franco Travostino Subject: Re: Performance of iSCSI, FCIP and iFCP Steph, thanks for raising this point, which is clearly central to the fairness discussion in a more general context, not restricted to IP Storage. There are two cases to consider: whether the storage traffic is isolated or is multiplexed with other traffic in the IP network. My original message on this subject of performance was assuming the first case, where all available bandwidth (in my example 10Gb/s) was available for storage traffic, and showed that in some scenarios, about 1/4 of bandwidth was lost unnecessarily with the one-TCP solution. Definitely, no fairness issue was involved. Now let's consider the second case, where storage traffic traverses a non-exclusive IP network, such as "the Internet", where fairness is an issue. Unfortunately, as far as I know, despite the TCP friendly mandate expressed or implied in the IETF community, there is no clear definition of what is considered to be fair use of network bandwidth (TCP friendly) and what is unfair. RFC 2914, citing RFC 2309: ".. the issue of the appropriate granularity of a "flow", where we define a `flow' as the level of granularity appropriate for the application of both fairness and congestion control. From RFC 2309: "There are a few `natural' answers: 1) a TCP or UDP connection (source address/port, destination address/port); 2) a source/destination host pair; 3) a given source host or a given destination host. We would guess that the source/destination host pair gives the most appropriate granularity in many circumstances. The granularity of flows for congestion management is, at least in part, a policy question that needs to be addressed in the wider IETF community." In the context of IP storage, the IPS gateways are different than the source and destination hosts for a TCP flow in a regular IP network. For example, in the ECM (Endpoint Congestion Manager) WG, we had a discussion on this subject. The ECM proposal attempts to regulate the sending rate for a pair of endpoints to be limited to roughly one TCP connection between those two hosts, given the current network conditions (loss probability, round trip delay). I raised a few questions. For example, if an endpoint is a multi-user system, and N users from this system simultaneously connect to the same web server, under ECM they get 1/N of the bandwidth they would otherwise get if using N PCs, which is not fair. Some of us agreed that the fair unit of bandwidth should be per user and not per system, but was not put on paper. This opinion is underscored by the following from RFC 2616, also cited by RFC 2914: "A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy." In the same spirit, I would argue that the storage traffic would be entitled to one share of (TCP-friendly) bandwidth per storage connection, and not per pair of gateways, since one storage connection more closely emulates one user activity (or maybe even more that that). In other words, mandating one TCP connection per pair of IP storage gateways would be like mandating one TCP per pair of IP routers. Of course, this is my subjective opinion with a bit of common sense, in the absence of a clear definition of fair network usage. The paper I referred to in my other message does not contain any consideration to fairness, only an analytical model for TCP throughput as a function of delay and packet loss. A side implication is that different TCP implementations are not fair (friendly) to each other, but that's a long known fact. Surely, you can modify TCP congestion control algorithm to get as much bandwidth as N other TCPs, but that creates a new, "super-TCP" protocol that has to be designed, verified and approved. (Observe that super-TCP would have a different way to recover from losses that N TCPs.) Moreover, even if super-TCP would be used, the IP storage gateways would still need to be aware of the identity and number of storage connections. Your comments, as always, are highly appreciated. Victor Stephen Bailey wrote: > > Victor, > > > I have to admit that we did not have the model to support > > numerically our contention that multiple links will behave better in > > the presence of errors and performance will degrade more gracefully > > even on congestion. > > The last time we had this discussion (multiple links between the same > endpoints enabling higher performance) on this list, it was my opinion > that doing this is not `TCP friendly', and so should not be allowed in > the general, Internet context, which is one of the design points for > iSCSI. > > Clearly, when slow starting, and recovering from congestion, n > connections can effectively open the window n times as fast as 1 > connection, but doing so will potentially be at the expense of other > competing flows which don't use multiple connections. If ALL flows > use multiple connections, you simply end up hitting the congestion > wall harder, which, in an abstract sense, makes the system more likely > to oscillate (or collapse?). > > I don't know if it WILL create oscillation or collapse (I'm just a > hammer mechanic), but it seems to me that if hitting the congestion > wall harder in this way were acceptable to overall network behavior, > the single connection TCP congestion avoidance could be adjusted to > create this behavior (and capture this benefit) without requiring > multiple TCP connections. > > Your refereed, published work on this seems suggests that I am holding > on to `myths' about the need to play fair with TCP. If so, I would > greatly appreciate your debunking my myths here on this list, in the > clearest way. Particularly, we need to know if the IETF (or IESG or > IAB, or whomever it is....), will accept this type of behavior out of > iSCSI. > > While I am opposed to iSCSI's multiconnection session, I do admit its > benefits. My objections are that the feature will not be widely used, > is difficult to implement correctly, AND the same benefits are already > available through long-standing upper layer mechanisms. That aside, I > accept the WG consensus on multiconnection sessions, but what the > iSCSI spec still needs to do, in no uncertain terms, is indicate > whether multiple connections per endpoint pair is something which > implementations SHOULD or SHOULD NOT allow for use in a general, > Internet context, or at all. > > Even if iSCSI specifies that multiple connections per endpoint pair > SHOULD NOT be used in a general context, we know that implementations > are not going to prohibit it (it's primarily a configuration > decision), and that still allows the feature to be used in > specifically engineered applications, such as those built with > dedicated storage fabrics. > > Thanks, > Steph From owner-ips@ece.cmu.edu Tue Jan 16 13:36:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA18820 for ; Tue, 16 Jan 2001 13:36:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0GFbWF06349 for ips-outgoing; Tue, 16 Jan 2001 10:37:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcars04e.ca.nortel.com (h56s242a129n47.user.nortelnetworks.com [47.129.242.56]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0GFaqs06330 for ; Tue, 16 Jan 2001 10:36:52 -0500 (EST) Received: from zbl6c012.corpeast.baynetworks.com by zcars04e.ca.nortel.com; Tue, 16 Jan 2001 10:32:53 -0500 Received: from zbl6c002.corpeast.baynetworks.com ([132.245.205.52]) by zbl6c012.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id ZLK0BL12; Tue, 16 Jan 2001 10:32:53 -0500 Received: from nortelnetworks.com (dhcp223-140.engeast.baynetworks.com [192.32.223.140]) by zbl6c002.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id CN99DJBP; Tue, 16 Jan 2001 10:32:53 -0500 Message-ID: <3A6469C1.34E403B6@nortelnetworks.com> Date: Tue, 16 Jan 2001 10:33:21 -0500 X-Sybari-Space: 00000000 00000000 00000000 From: "Victor Firoiu" Organization: Nortel Networks X-Mailer: Mozilla 4.76 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: someshg CC: IPS Reflector , "Franco Travostino" Subject: Re: Performance of iSCSI, FCIP and iFCP References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Orig: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Somesh, our answer is beween your lines below. Victor Somesh Gupta wrote: > > Franco, > > Thanks for getting us to exercise the under-utilized math part of > our brains :-) > > I wanted to echo the comments that Stephen made. What I think > you are saying is that > > If a single packet drop occurs no more frequently than the time > to recover (the rate of the connection which existed before the > error occured), then a quarter of the bandwidth during the > recovery period is lost. In the simplistic case where an error > occurs with just the correct frequency, you loose a quarter of > the available bandwidth. > > If you have N connections sharing the same bandwidth, then the > time to recover is reduced to 1/N and since the rate on each > connection is (1/N), the lost rate is proportional to (1/N)^2. > > A minor quibble with the math. I assume (considering this low > rate) that the error is likely due to transmission errors which > is proportional to the number of bytes sent, the (1/n)^2 is not > quite true as the number of bytes sent will be larger so errors > will occor somewhat more frequently as a function of time. > You are right. That is why in our numerical example, the one-TCP case has an effective rate of 8.27Gb/s out of 10Gb/s available and not 7.5Gb/s. In general, the ratio between the amount of data losing transmission opportunity per loss event in one-TCP case versus the N-TCP case is proportional to N^2. > The really interesting equation to me is to look at the lost > oportunity in isolation of the number of connections (i.e. > you can scale the numbers any which way) > > D = C/8*I/4 = C^2*RTT^2/(256*M) > > As long as RTTs are small, and/or data rate (C) is not too large, > we don't loose too large a percentage of the available bandwidth. > > However, if both become large which is what we face, and keep > becoming larger, then we keep getting closer and closer to the > 1/4 loss rate, unless we keep increasing the number of connections > to compensate. > > In a perfect world, you would be able to convince the IETF to > use the product of rate and RTT as a factor to adjust the A and > the M part in AIMD algorithm to achieve the same result. It > seems a little bit of waste of energy to get around the math > by using multiple connections (BTW - are there any published > papers on experiences with multiple connections for a single > data stream in a high performance environment - would love to > learn from someone else's mistakes). > > Somesh > I agree that we can mitigate bandwidth loss through another congestion control algorithm. But we have no control whatsoever of TCP, and thus we are committed to using whatever a vanilla TCP looks like. Even if super-TCP would be used, the FCIP storage gateways would still need to be aware of the identity and number of storage connections, and thus their processing overhead would be close to the TCP-connection-for-storage-connection solution as in iFCP or iSCSI. Therefore, the solution with one TCP connection per storage connection seems to be more appropriate than one TCP connection per PHY. Victor From owner-ips@ece.cmu.edu Tue Jan 16 16:01:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA21567 for ; Tue, 16 Jan 2001 16:01:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0GIgaC13049 for ips-outgoing; Tue, 16 Jan 2001 13:42:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gate.internaut.com ([64.38.134.108]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0GIgOs13043 for ; Tue, 16 Jan 2001 13:42:24 -0500 (EST) Received: from e1kj2 (kidneybean [64.38.134.109]) by gate.internaut.com (8.10.2/8.10.2) with SMTP id f0GIcbR00468; Tue, 16 Jan 2001 10:38:38 -0800 From: "Bernard Aboba" To: "Stephen Bailey" , , Subject: RE: Performance of iSCSI, FCIP and iFCP Date: Tue, 16 Jan 2001 10:42:15 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-Mimeole: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <10101132013.AA27978@candide.cs.uchicago.edu> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit >I don't know if it WILL create oscillation or collapse (I'm just a >hammer mechanic) The result will be synchronization and oscillation, resulting in under-utilization of the link, unless RED is enabled. With RED turned on, the TCP connections will not see packet drops at the same time, so synchronizatoin will not occur and overall utilization will be improved. From owner-ips@ece.cmu.edu Tue Jan 16 16:02:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA21604 for ; Tue, 16 Jan 2001 16:02:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0GHoYC11159 for ips-outgoing; Tue, 16 Jan 2001 12:50:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0GHo1s11116 for ; Tue, 16 Jan 2001 12:50:01 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0GJ0o037546; Tue, 16 Jan 2001 11:00:59 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Prasenjit Sarkar" Cc: Subject: RE: draft-ietf-ips-iscsi-boot-01.txt Date: Tue, 16 Jan 2001 09:48:39 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Prasenjit, I find this language vague. If you wish to adopt standards of iSCSI, then include language that indicates how this expansion is done. Are bytes reverse order (LSB first); can you include embedded spaces? Could you also perhaps show examples of these expansions using a non-routable IP as an example. As the expansion replaces a DNS lookup, the standards used in DNS would add thrift as the same parsing code could be employed if both DNS and literal expansions are supported. These are just ideas, the important aspect is a clear definition. Doug > I thought that the following paragraph in draft-01 allowed > servername to be > represented in both IPv4 and IPv6 formats. > > > The "servername" is the name of iSCSI server and contains either a > valid domain name, a literal IPv4 address, or a bracketed literal > IPv6 address. If the servername field contains a literal IPv4 > address, the IPv4 address is in standard dotted decimal notation. If > the servername field contains an IPv6 address, the address is > represented in bracketed literal IPv6 address format. > > The main iSCSI standard defines targetname, we will adhere to whatever is > accepted there. > > -Prasenjit > > > ------------------------------------------------------------------ > ------------------------------------------------------------------ > ------------------------------------ > > All, > > The field consists of an UTF-8[20] string and has the following > composition: > > ":" ":" ":" > > Would it be possible to adopt language explaining that servername and > targetname could be dotted decimal expansions of the IP address > in place of > the name. DNS code for a bootstrap adds complexity. It would make it > easier if perhaps some symbols could be used to indicate such an expansion > has been made such as .IP6.INT or .IP4.INT suffix where the bytes are in > reverse order with no embedded spaces. > > Doug > > > A URL for this Internet-Draft is: > > http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-boot-01.txt > > > > > > From owner-ips@ece.cmu.edu Tue Jan 16 16:05:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA21639 for ; Tue, 16 Jan 2001 16:05:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0GINam12304 for ips-outgoing; Tue, 16 Jan 2001 13:23:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0GIMhs12269 for ; Tue, 16 Jan 2001 13:22:43 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id CF6B14F4 for ; Tue, 16 Jan 2001 11:22:40 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 53550F4 for ; Tue, 16 Jan 2001 13:22:39 -0500 (EST) Received: from agilent.com (cos1nai130117.cos.agilent.com [141.184.130.117]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id KAA18106 for ; Tue, 16 Jan 2001 10:22:34 -0800 (PST) Message-ID: <3A646E4B.2DC21364@agilent.com> Date: Tue, 16 Jan 2001 07:52:43 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iFCP as an IP Storage Work Item, Reset References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit John Hufferd wrote: > The iFCP draft says that it is intended to be a Gateway to Gateway > protocol. Just for clarification, that's certainly not the picture that was presented in San Diego. -Matt From owner-ips@ece.cmu.edu Tue Jan 16 16:05:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA21652 for ; Tue, 16 Jan 2001 16:05:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0GHlam11023 for ips-outgoing; Tue, 16 Jan 2001 12:47:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0GHlEs11004 for ; Tue, 16 Jan 2001 12:47:14 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id C6R66TJ3; Tue, 16 Jan 2001 09:47:10 -0800 From: "Y P Cheng" To: "John Hufferd" Cc: Subject: RE: iSCSI & iFCP Overlapping (Was iFCP as an IP Storage Work Item) Date: Tue, 16 Jan 2001 09:46:08 -0800 Message-ID: <001001c07fe4$35365c80$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > YP, > Please refrain from discussing iFCP in any context except what the Draft > supports. > We need to reach consensus on real proposals. > If you want to author another draft we can perhaps discuss that. > The authors of the current draft believe your comments are removing the > focus they need on their iFCP Draft. > . > John L. Hufferd Before focusing on the iFCP draft, I found myself in agreement with Doug in his statement: "Although you could suggest iFCP is just a gateway to gateway solution, this does not constrain a gateway if it contains but a single device nor does it require hallucinations to arrive at that conclusion. In that case, the differences between iSCSI and iFCP hinge upon reliance of existing FCP protocol merged with a lightweight transport. The motivation of all proposals is to incorporate IP managed networks rather than rely upon dedicated fiber for SAN and, in that case, all proposals are in conflict." Focusing on the draft, iFCP, like the FCIP proposal, does have a distinguish merit of leveraging off the working FCP headers that most FC adapters and devices on the market today have no trouble understanding. If we assume a SSP is built from parallel SCSI, ATA, and fibre channel devices to support iFCP with "virtual devices", then, preserving the ELS of fibre channel in iFCP is not essential. For example, there is no need for Class 2 parameters when iFCP has its own ACKs from TCP. The most important thing for iFCP is to pass the FCP packets to the other side. I suspect most ELS parameters will become much less useful in a IP network. For example, using iSNS services, FLOGI and PLOGI might be done locally. What does EE credit mean when FCP has its own window size? What does E_D_TOV mean when TCP has its own time out? The R_A_TOV has a totally different meaning in a IP network. Besides focusing on passing FCP packets, just like iSCSI, iFCP needs to focus on device naming convention and discovery, TCP session creation and destroy, text parameters for log-in, multiple connections, IPSec and user authentication, large TCP window for high throughput on network with long latency, TCP flow control and recovery on lossy connections, frame boundary and Remote DMA, frame reassembly, etc., etc., ... In summary, if we do iFCP, lets focus on passing SCSI commands in FCP format. If we simply wish to connect multiple SAN islands, lets use FCIP. Y.P. Cheng, ConnectCom Solutions. From owner-ips@ece.cmu.edu Tue Jan 16 18:01:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA23057 for ; Tue, 16 Jan 2001 18:01:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0GK8cq16558 for ips-outgoing; Tue, 16 Jan 2001 15:08:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0GK8Hs16541 for ; Tue, 16 Jan 2001 15:08:17 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id CC44837B for ; Tue, 16 Jan 2001 12:08:15 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id MAA28904 for ; Tue, 16 Jan 2001 12:08:07 -0800 (PST) Message-ID: <3A64AA9D.FA30F662@cup.hp.com> Date: Tue, 16 Jan 2001 12:10:05 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Abort Task Response violates SAM-2. Content-Type: multipart/mixed; boundary="------------62AFD5A80E40E961C2A3745D" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------62AFD5A80E40E961C2A3745D Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, I will hold off on this till I see the changes in 04. However, I see no value add in retaining the "No Task Found" [and its corresponding "Referenced Task Tag"] when SAM-2 states that "Function Complete" shall indicate that a task was not found or the abort was completed. Regards, Santosh julian_satran@il.ibm.com wrote: > Santosh, > > I did not that "task not found" will be removed. I said that the wording > will match closer the SAM wording. > > Julo > > Santosh Rao on 15/01/2001 01:03:04 > > Please respond to Santosh Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: iSCSI : Abort Task Response violates SAM-2. > > Julian, > > I'm confused by your response on this. What does the "Referenced Task Tag" > in the SCSI Task Management Response PDU "clarify" ? > > Its description in the draft states : > "Initiator Task Tag of the task not found". > > If the "No Task Found" response is being removed from this section, of > what use is the "Referenced Task Tag" ? > > If the LUN and "Referenced Task Tag" fields are used in the task mgmt > response pdu to give LUN or task context to the task mgmt response, > then, this information is already present at the initiator and can > be looked up based on the Initiator Task Tag from the Task > Management Response PDU. > > Regards, --------------62AFD5A80E40E961C2A3745D Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------62AFD5A80E40E961C2A3745D-- From owner-ips@ece.cmu.edu Tue Jan 16 18:01:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA23071 for ; Tue, 16 Jan 2001 18:01:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0GKffj17668 for ips-outgoing; Tue, 16 Jan 2001 15:41:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0GKevs17644 for ; Tue, 16 Jan 2001 15:40:58 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 5EB92452 for ; Tue, 16 Jan 2001 12:40:56 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id MAA01777 for ; Tue, 16 Jan 2001 12:40:48 -0800 (PST) Message-ID: <3A64B246.267E2CE3@cup.hp.com> Date: Tue, 16 Jan 2001 12:42:47 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI: header digest error at initiator References: <3A5B5090.14444240@hp.com> <3A5BD336.F983F9F8@cup.hp.com> <3A63CB93.2BFD8351@agilent.com> Content-Type: multipart/mixed; boundary="------------688FE917370DBFD19D3DB8AE" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------688FE917370DBFD19D3DB8AE Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Matt, My replies inline. Regards, Santosh Matt Wakeley wrote: > Santosh Rao wrote: > > > > If this is the intention of the recommended error recovery, it is the > > result of not allowing score-boarding. By score-boarding an initiator > > would detect an underrun and would just error the affected I/O back. > > Two comments here. First, in your example, the initiator is inventing an > error that really didn't occur in the target. The target completed the I/O > successfully, it was the transport that experienced an error, but you're > treating it like a target error. The LLP (initiator) would return an error to ULP indicating a transport error (service response of "service delivery or target failure") occurred. [due to the data underrun.] > Second, you say that initiators and targets routinely perform scoreboarding. > How is this done today? Buffer(s) are provided to an I/O chip. The I/O chip > writes the data into the buffers. It does not have the memory to determine > that each and every byte has been written to. So how is the initiator/target > supposed to be absolutely sure that every byte was written to? The initiator would use a check along the following lines : (total bytes xfer'ed as indicated by the chip ) = (no. of bytes of data xfer specified by ULP) - (resid reported by the target in SCSI Response PDU). to verify that all the data the target sent is accounted for at the initiator end. --------------688FE917370DBFD19D3DB8AE Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------688FE917370DBFD19D3DB8AE-- From owner-ips@ece.cmu.edu Tue Jan 16 19:56:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA24255 for ; Tue, 16 Jan 2001 19:56:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0GMHgl20967 for ips-outgoing; Tue, 16 Jan 2001 17:17:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0GMHas20961 for ; Tue, 16 Jan 2001 17:17:36 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id C9C0835D for ; Tue, 16 Jan 2001 15:17:35 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id A728B189 for ; Tue, 16 Jan 2001 17:17:34 -0500 (EST) Received: from agilent.com (cos1nai130116.cos.agilent.com [141.184.130.116]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id OAA05067 for ; Tue, 16 Jan 2001 14:17:27 -0800 (PST) Message-ID: <3A64C897.5FA46CBE@agilent.com> Date: Tue, 16 Jan 2001 14:17:59 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI: header digest error at initiator References: <3A5B5090.14444240@hp.com> <3A5BD336.F983F9F8@cup.hp.com> <3A63CB93.2BFD8351@agilent.com> <3A64B246.267E2CE3@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > > Second, you say that initiators and targets routinely perform scoreboarding. > > How is this done today? Buffer(s) are provided to an I/O chip. The I/O chip > > writes the data into the buffers. It does not have the memory to determine > > that each and every byte has been written to. So how is the initiator/target > > supposed to be absolutely sure that every byte was written to? > > The initiator would use a check along the following lines : > > (total bytes xfer'ed as indicated by the chip ) = > (no. of bytes of data xfer specified by ULP) - (resid reported by the target in > SCSI Response PDU). > > to verify that all the data the target sent is accounted for at the initiator > end. I think you need to use a different word than score boarding. Score boarding means that you have a big "map" that describes every byte of data you want to receive. As each byte is received, you "check off" the indicator for that byte that indicates that it was received. When all the indicators are checked off, you have all the data. Score boarding is not a mathematical calculation. -Matt From owner-ips@ece.cmu.edu Tue Jan 16 20:00:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA24340 for ; Tue, 16 Jan 2001 20:00:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0GN0js22373 for ips-outgoing; Tue, 16 Jan 2001 18:00:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0GN0Gs22324 for ; Tue, 16 Jan 2001 18:00:16 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0H07C037759; Tue, 16 Jan 2001 16:07:15 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Santosh Rao" , "IPS Reflector" Subject: RE: iSCSI: header digest error at initiator Date: Tue, 16 Jan 2001 14:55:02 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <3A64B246.267E2CE3@cup.hp.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh, Do not just use byte counting to validate delivery as there may be overlap. The word SHOULD and a real world of not all devices sending data according to this SHOULD... requires that you MUST not assume data is consecutive and not overlapping. The non-consecutive nature may cause problems if you simply overlay data digests rather than stripping it off before placement. The problem with the overlap is obvious if you expect a byte count total to have meaning. From revision 3: "An R2T MAY be answered with one or more iSCSI Data-out PDU with a matching Target Transfer Tag. If an R2T is answered with a single Data PDU the Buffer Offset in the Data PDU MUST be the same as the one specified by the R2T and the data length of the Data PDU must not exceed the Desired Data Length specified in R2T. If the R2T is answered with a sequence of Data PDUs the Buffer Offset and Length must be within the range of those specified by R2T, the last PDU should have the F bit set to 1, the Buffer Offsets and Lengths for consecutive PDUs SHOULD form a continuous non-overlapping range and the PDUs should be sent in increasing offset order." Doug > Matt, > > My replies inline. > > Regards, > Santosh > > Matt Wakeley wrote: > > > Santosh Rao wrote: > > > > > > If this is the intention of the recommended error recovery, it is the > > > result of not allowing score-boarding. By score-boarding an initiator > > > would detect an underrun and would just error the affected I/O back. > > > > Two comments here. First, in your example, the initiator is > inventing an > > error that really didn't occur in the target. The target > completed the I/O > > successfully, it was the transport that experienced an error, but you're > > treating it like a target error. > > The LLP (initiator) would return an error to ULP indicating a > transport error > (service response of "service delivery or target failure") > occurred. [due to the > data underrun.] > > > > Second, you say that initiators and targets routinely perform > scoreboarding. > > How is this done today? Buffer(s) are provided to an I/O chip. > The I/O chip > > writes the data into the buffers. It does not have the memory > to determine > > that each and every byte has been written to. So how is the > initiator/target > > supposed to be absolutely sure that every byte was written to? > > The initiator would use a check along the following lines : > > (total bytes xfer'ed as indicated by the chip ) = > (no. of bytes of data xfer specified by ULP) - (resid reported by > the target in > SCSI Response PDU). > > to verify that all the data the target sent is accounted for at > the initiator > end. > > > > > From owner-ips@ece.cmu.edu Tue Jan 16 23:21:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA29572 for ; Tue, 16 Jan 2001 23:21:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0H1Y2e26092 for ips-outgoing; Tue, 16 Jan 2001 20:34:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0H1Xas26083 for ; Tue, 16 Jan 2001 20:33:37 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Tue, 16 Jan 2001 17:34:38 -0800 Message-ID: From: Joshua Tseng To: ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item, Reset Date: Tue, 16 Jan 2001 17:33:02 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, Charles asked me to send the following message, as he is at the IPS WG meeting and does not have access to e-mail. He also wants everyone to know he will respond to e-mails this weekend when he gets back. -------------------------------------- Hi John: Thanks for helping to refocus the discussion. I want to emphasize once again that iFCP is a gateway to gateway protocol. Other discussions to the contrary are simply not relevant or productive. As much as I dislike playing the role of thought policemen, I feel it's important to move beyond these debates and focus on matters of technical substance. That said, I'm not sure whose job it is to declare these issues off limits. I simply hope folks will cooperate and let the matter rest. Charles Monia > -----Original Message----- > From: John Hufferd [mailto:hufferd@us.ibm.com] > Sent: Monday, January 15, 2001 12:14 PM > To: ips@ece.cmu.edu > Subject: iFCP as an IP Storage Work Item, Reset > > > I think the messages on iFCP have become somewhat decisive > and unfocused. > And quite frankly it has been permitted to happen by both > supporters and > detractors. I think this has occurred because engineers tend to solve > things, whether or not the solution is useful or if it has a > business need > requiring a fix. Therefore, having said that, we need to > restate what the > playing field looks like. (The reader is cautioned that this > is a lengthy > note.) > > The iFCP draft says that it is intended to be a Gateway to Gateway > protocol. Therefore supporters of iFCP should stop responding to the > infinite possibilities to hallucinate various other purposes for this > protocol. The protocol is NOT an end to end protocol, and > responses that > do not end the speculation waist our joint time, and polarizes iSCSI > end-to-end folks against iFCP. I believe, there are a > number of things > that maybe missing for a robust end-to-end protocol with iFCP, but the > point is it does not matter, the protocol is structured, proposed and > analyzed as a Gateway to Gateway protocol, and nothing else > is proposed or > analyzed as part of iFCP. The head of the company that > brought us iFCP, > has stated that he only wants this as a gateway to gateway > protocol, and > that they are committed to iSCSI also. > > I blame the iFCP supporters for not just shutting down these variant > thoughts. The iSCSI supporters would be expected to worry > and be paranoid > about anything they can hallucinate that might threaten > iSCSI. Therefore, > because the iFCP authors did not stop these thoughts dead in > their tracks > (although Josh tried a couple of time), things got completely > out of hand, > and we moved from a protocol, that in some ways was redundant > to FCIP (or > visa versa), to something that also seemed to threaten iSCSI. > I strongly > suggest that no real interests are served by this wild speculation > (Note: I am not completely pure here either, since I also let paranoia > force me off the reservation when I worried about mFCP and a related > organization.) So I want to strongly suggest that we look at > iFCP for only > what it is proposed to be, and NOT stray into other areas. > > To get my head straight, I had to refocus on the environments > where FC, > FCIP, iFCP and iSCSI are going to play to see where the effective > redundancy and conflict might exist. Clearly if I thought > that iFCP gave a > serious problem to iSCSI and to FCIP, and was redundant to > both, I would > have a problem. To analyze this, I had to go through some > "market think". > I know that we are suppose to be in an engineering > environment, but it is > our views of the market that get us polarized. So the > following is the > analysis I went though and I hope it is helpful to others. > > Working with iFCP defined as a Gateway to Gateway protocol, > lets look at > the various environments where this technology could be > deployed. Also, > even though there are two types of Gateways that could be > built with this > protocol, a Router, and a Switching Router (Switch with Router > capabilities) I will assume that a Switching Router is the > subject product. > A Switching Router has several ports that can either be > plugged with FC > cable or Gigabit Ethernet cable, in various combinations. If only FC > cables are connected, it is a FC switch, when only Ethernet cables are > connected it is a IP Switch, with various combinations of FC > and Ethernet, > it is a iFCP Routing Switch, called hereafter an iFCP Switch > or an iFCP > Gateway. > > We should also note that the iFCP Switch Vendor(s) are also > claiming (as I > said above) that they will have, over time, iSCSI support > also. I will > submit that those environments which have iFCP (with latent iSCSI > capability) solutions will also help the deployment of iSCSI > solutions. > This is because, the Gateways to FC Host and FC Storage will > have already > have been put in place. > > Here are the environments I examined: > > 1. Existing Single SAN Fibre Channel environments with a number of FC > Switches. If the FC SAN already exists, the assumption here > is that the > customer might want to replace his current FC Switches and > Hubs with an > iFCP set of switches and LAN type interconnects. However, I > submit this is > not really very fertile ground for iFCP switches. (By the > way it is not > very fertile for iSCSI or FCIP solutions, in the short term, > either.) This > set of customers will be very hesitant to change since they > have already > had a lot of investment, and they probably think they are on > the down hill > slop of the compatibility and support issues. They may wish > that the cost > were lower, but any change will be only lots of additional > cost to them. > Continuing with FC will be their probably action. > > 2. Existing Single FC environment, that uses FC to connect > mostly point to > point and not invested in many, if any, switches. This type > of environment > may use Sharks or Symmetrix type Storage Controllers > directly. And some > time they will need to expand with new storage controllers, > Host etc. This > gives a possible reason to install a FC Switch with iFCP > capability, (for > future expansion). But this will be a head to head fight > with existing FC > Switch vendors, and therefore such iFCP Gateways will not > have a strong > market here. (By the way, the iSCSI message is hard in this > environment, > unless the customer has visions of important future expansion > with iSCSI > storage controllers, or needs to bring Desktops and Laptops > into the common > storage pool.) Likewise there is not a FCIP message here either.) > > 3 a. Existing Multiple FC environments (locations) on a > single campus. FC > with normal Switch to Switch E-Port connections via long FC > cables will be > able to easily bring interconnection to this environment. > However, the > bringing together of various FC environments is also a strong > play for iFCP > Gateways. That is, each environment can add a iFCP capable > Gateway and > thereby interconnect the environments, and permit the sharing > of various > storage facilities. Each individual iFCP interconnected FC > environment > will be able to change its addressing etc., without needing > to coordinate > with any other local site (a Big Plus). The FCIP solution > is also viable > in this environment, however, when you get to more then two > environments, > the number of (single in -- single out) Edge Connect FCIP > routers, go up > dramatically as the number of locations increase. The > physical space issue > can be mitigated by a multi-in, multi-out FCIP capable router > box. That > is, a box that puts the electronics for multiple connections > on the FC side > -- one for each location to location interconnect. Note, this also > requires a FC switch with a E-Port that can be connected to > an E-Port at a > specific target location. Also the FCIP has a disadvantage > in each of the > locations that might look like environment 2 above (since > they have few or > no FC Switches). iSCSI will find this a difficult > environment in general > because of the existing FC investment. (Note: proposed > combinations sets > of "FC-to-iSCSI boxes which route to iSCSI-to-FC boxes" can > not be assumed > to work since iSCSI has not defined any protocol to handle FC > ELS, as has > iFCP.) Therefore, iSCSI will not be a popular solution here. > The higher > the number of location the stronger the iFCP appears (cost > wise) against > FCIP. Note: if a iFCP Gateway gets installed, and if the unit also > supports iSCSI (as the primary vendor has committed), the inroad for > iSCSI, in general, will be greatly simplified. iSCSI folks, > therefore, > should be careful in their dismissing of iFCP, because when > it is teamed up > with iSCSI in this manor, it becomes be a great ally. > > 3 b. Existing Multiple FC environments across a Wide Area. > This case is > the same as 3a above, except normal FC techniques do not > apply. However, > both iFCP and FCIP approaches apply. Again the advantage of > iFCP vrs FCIP > have to do with the number of sites that are interconnected, > and the site > independence that iFCP offers. Also, if the iFCP Gateway has iSCSI > capabilities, it should permit strong and quick inroads for > iSCSI into this > environment. > > 4. SCSI, and ATA based Host environments that need to grow > into a network > attachment. The non server host, Desktops and Laptops, and > servers that > have incidental remote Storage Access will not generally > consider FC as an > approprate option. Generally this environment is approprate > for the iSCSI > initiators (with normal TCP/IP NICs, and iSCSI SW Drivers). > Without FC > the FCIP will not apply to this environment. iSCSI to FC > Gateways will > exist at the remote environments and possibly iSCSI native Storage > Controllers. iFCP itself is not approprate but if the iFCP > box also has > iSCSI capability, then that can be as valuable as any iSCSI > to FC gateway. > > 5. Single SCSI, based Server environments with needs to grow. These > environments will have high access requirements. When iSCSI HBAs are > available (1H2001), and with the availability of iSCSI to FC Routers > (1H2001), this is a fertile market for an iSCSI sale. When > iSCSI Storage > Controllers are available, this environment will be a > significant growth > area for iSCSI. It has a head to head message with FC SAN. > However, the > existing knowledge of IP networks and IP Switch cost will be the swing > consideration to iSCSI. FCIP has no evolvement in this > environment. iFCP > in this environment would require FC HBAs, and at lest two > iFCP Switching > Routers. If the customer wants to add iSCSI Storage > Controllers, the iFCP > Switches will not be useful (until it supports iSCSI also). In this > environment where the target storage is FC based, iSCSI requires (in > addition to iSCSI HBAs, & IP Switches), one or more iSCSI to > FC Gateways. > The winner here, between iFCP and iSCSI will be the solution > with the least > cost. That is, the total cost of the HBAs (initially this maybe a push > between FC and iSCSI), the IP Switches, and the iSCSI to FC > Gateways vs the > Cost of FC adapters, IP Switches and iFCP Gateways. If the > resultant cost, > is close, then iSCSI will probably be the winner here, > because of industry > direction. Having said that, this is one of the environments > where iSCSI > folks think that the iFCP message is diversionary. If the > customer is not > polarized against FC, and the customers looks at having to > buy FC adapters > for the iFCP deployment, they are probably going to look at a > complete FC > SAN as a more straight forward deployment. Therefore, the > competitive play > will very often be between FC and iSCSI the possibly diversionary iFCP > messages will not be significant and maybe helpful selling > the IP Storage > message. > > 6 a. Multiple SCSI, based Servers environments (locations) > on a single > campus. Like 5. above, these environments will have high access > requirements, and when iSCSI HBAs are available (1H2001), and > the iSCSI to > FC Routers (Gateways) are available (1H2001), this becomes a > fertile market > for an iSCSI sale. iSCSI will then have a head to head > competitive message > in this environment with FC SANs, and very probably a head to head > competitive message with the FC and FCIP combo. (So lets not deceive > ourselves that iSCSI doesn't have a market conflict with > FCIP.) Since I > believe that in this environment, the customer probably had a > reason for > not already jumping on FC, and because since I also think > they are probably > well versed in TCP/IP Networks, the iSCSI message, will be > well received. > The iFCP message may also have a play in this environment, > they can play to > the same TCP/IP knowledge as iSCSI does, and suggest to the > customer that > iFCP is a better choice because there are more HBAs and > Storage Devices > based on FC then on iSCSI, etc. Therefore I think that this is a > competitive play between everyone, FC SAN, FC and FCIP, iFCP, > and iSCSI. > However. the more iSCSI based storage controllers are > shipped, the more the > play tips toward iSCSI in this environment. Some iSCSI folks > look at iFCP > having a diversionary message in this case also, however it > looks to me as > a general free for all amongst all the options that are valid in this > environment and cost will be the driving force. By the way, > the iFCP IP > Storage message will probably do more to help iSCSI, then it does to > distract from iSCSI. Also, in this environment, the trade > offs between > FCIP and iFCP, mentioned in 3a. above, apply here as well. > > 6 b. Multiple SCSI, based Servers environments (locations) > across a wide > area. This environment fits the same solutions as 6 a. above, > except a FC > only solution is not an option. That is, an iFCP or FC&FCIP > solution is > approprate as is the iSCSI option, just as in 6a above, but > not FC only. > > 7. Multi Hosting environments with Storage Service Providers > (SSPs). These > environments will be primarily networked environments which when built > today, are Fibre Channel, but here a iFCP Gateway would be a > reasonable > consideration. In this environment today, the Hosts will have a FC > adapters so in either case, the question is -- will the cost > be less using > iFCP Gateways (where the FC Fibre leave the "Cage", and then > goes through > IP Switches, and then terminates at a iFCP Gateway just before the FC > based Storage Units)? Since the SSPs will be competent in > both FC and IP > networks, the trade off will be primarily cost. If the costs > are close, > the edge will probably go to iFCP since it can also combine > multiple sites > at less cost then multiple FCIP Gateways (see discussion in > point 3 above). > Having said all that, if iSCSI HBAs, and iSCSI Gateways are available > (1H2001), then, the SSP will be able, to apply additional security > features (if required by the customer), and they only need low cost IP > Switches all the way into the Storage Area. This will > probably the most > flexible and lowest cost. Since iSCSI will also be able to > glue multiple > sites together it will be the most probable configuration in this > environment, over time. Until that time, iFCP has a strong chance of > being heavily used in a SSP environment, perhaps supplanting > FC networks > in this environment. An iFCP & iSCSI Gateway, would seem to > be a very > high seller, and permit graceful migration to an all iSCSI network. > > -------------------------------------------------------------- > > The iFCP messages, in all but environment #7, will be dwarfed > by the number > of vendors shipping iSCSI, and the iSCSI message will be the > only important > IP Storage message that will come through. But since the > number of SSPs is > small they are easy to market too, and will be fully aware of the > approprate maturity of iFCP and iSCSI. However, iFCP will > probably only be > a stepping stone to iSCSI in an SSP environment. Further, if > you take the > primary iFCP vendor at their word, they will combine their > iFCP product by > including iSCSI, and therefore the total message will be very > supportive of > iSCSI. > > Let me build a matrix that can summarize the above > environments (Cases) > with probability of success. > > Here are the cases again: > 1. Existing Single SAN Fibre Channel environments with a number of FC > Switches. > 2. Existing Single FC environment, that uses FC to connect > mostly point to > point and not invested in many, if any, switches. > 3 a. Existing Multiple FC environments (locations) on a single campus. > 3 b. Existing Multiple FC environments across a Wide Area. > 4. SCSI, and ATA based Host environments that need to grow > into a network > attachment. > 5. Single SCSI, based Server environments with needs to grow. > 6 a. Multiple SCSI, based Servers environments (locations) > on a single > campus. > 6 b. Multiple SCSI, based Servers environments (locations) > across a wide > area > 7. Multi Hosting environments with Storage Service Providers (SSPs). > > +----+-------+-------+-----+------+----------+--------+ > |Case|FC only|FC&FCIP|iFCP | iSCSI|iFCP&iSCSI|iFCP Msg| > +----+-------+-------+-----+------+----------+--------+ > | 1 | High | NA | Low | Low | Med |Helpful | > +----+-------+-------+-----+------+----------+--------+ > | 2 | High | NA | Low | Low | Med |Helpful | > +------------+-------+-----+------+----------+--------+ > | 3a | High | High |High+| Low |Very High |Helpful | > +----+-------+-------+-----+------+----------+--------+ > | 3b | NA | High |High+| Low |Very High |Helpful | > +----+-------+-------+-----+------+----------+--------+ > | 4 | Low | NA | NA | High | Med |NA | > +----+-------+-------+-----+------+----------+--------+ > | 5 | Low+ | NA |Low+ | High | Med |Min Help| > +----+-------+-------+-----+------+----------+--------+ > | 6a | Med | Med |Med+ | High |Very High |Min Help| > +----+-------+-------+-----+------+----------+--------+ > | 6b | NA | Med |Med+ | High |Very High |Min Help| > +----+-------+-------+-----+------+----------+--------+ > | 7 |Hi->Med| Med |High |Med>Hi|Very High |Helpful | > +----+-------+-------+-----+------+----------+--------+ > > This table says that in the existing FC environment Case 1-3b, iSCSI, > will have tough play. On the other hand, in multiple location > environments, iFCP will have a strong play, against FC & > FCIP. When the > iFCP product is combined with iSCSI support its probability > of success > becomes Very High. In other environments (4-7), iSCSI has a high > probability of success. And when the iFCP product gets iSCSI > support it > will greatly improve its success position. The marketing > messages from > iFCP are Helpful, or Minimally Helpful. > > Hence the only important conflict is between FCIP and iFCP, > and this is > mostly a "push" except for Multiple Sites, where the Cost > might favor iFCP > (and that is not completely obvious). > > Many of us wanted the FCIP and the iFCP combined so this > conflict does not > exist, however, this looks like both sides have refused to > cooperate, so a > product shoot out will occur. It looks to me that both will > take a piece > of the market and continue to exist. > > The table indicates that when iFCP is combined with iSCSI, the results > offers a better product in several areas then either by themselves. > > Therefore, I believe that iFCP has important potential value to our > customers, and should be part of the IP Storage effort within > the IETF. > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > (408) 256-0403, Tie: 276-0403 > Internet address: hufferd@us.ibm.com > From owner-ips@ece.cmu.edu Wed Jan 17 02:11:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA14248 for ; Wed, 17 Jan 2001 02:11:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0H4nYf00421 for ips-outgoing; Tue, 16 Jan 2001 23:49:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mtiwmhc24.worldnet.att.net (mtiwmhc24.worldnet.att.net [204.127.131.49]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0H4k3s00347 for ; Tue, 16 Jan 2001 23:46:03 -0500 (EST) Received: from SG2351 ([32.102.233.99]) by mtiwmhc24.worldnet.att.net (InterMail vM.4.01.03.10 201-229-121-110) with SMTP id <20010117044556.SYOV1480.mtiwmhc24.worldnet.att.net@SG2351>; Wed, 17 Jan 2001 04:45:56 +0000 Reply-To: From: "Somesh Gupta" To: "Victor Firoiu" Cc: "Stephen Bailey" , "ips" , "Franco Travostino" Subject: RE: Performance of iSCSI, FCIP and iFCP Date: Tue, 16 Jan 2001 20:43:09 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <3A646E8E.D32FFE3A@nortelnetworks.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Victor, I just assumed that you were familiar with the iSCSI specification of multiple connections per session - which is essentially as you describe in the second last paragraph - there is one big "flow" between the initiator and the target (called an iSCSI session) which is then striped across multiple TCP connections - and I think people assumed that your statements were in support of such a proposal. Having said that, the math still holds. And also the fact that fundamental limits to the throughput of a single TCP connection really do require a change to TCP congestion avoidance (esp when mistaking link error for congestion) formulae. If the link error rate is such that errors occur faster than time to recover (or of that order), TCP will have to accomodate that somehow. I don't think the TCP gurus are inflexible in this area - PAWS, window scaling, and SACK are all example of TCP accomodating the characteristics of the underlying network. Somesh -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Victor Firoiu Sent: Tuesday, January 16, 2001 7:54 AM To: somesh_gupta Cc: Stephen Bailey; ips; Franco Travostino Subject: Re: Performance of iSCSI, FCIP and iFCP Somesh, my answers between your lines. Victor Somesh Gupta wrote: > > Victor, > > First I would like to address the second case where - based on the > need to be fair - you raise the point that multiple connections per > session is closer to emulating multiple users on a server than a > single connection. That being a valuable point, is probably better > addressed at the upper layers of the host software, where multiple > independent sessions can be created to the same target. The argument > made more often in favor of multiple connections is that the likelyhood > of some connections escaping RED - or other disasters - when you have > multiple connections is good and therefore the overall performance > will be better. > Not sure what you mean by "multiple connections per session." Maybe I was not clear, I was arguing for one TCP connection for one storage connection, which can give multiple TCP connectons for one pair of IP storage gateways. The performance considerations that I presented in my first message regard any possible packet loss, including RED. > However, as Steph pointed out, that skirts the issue of fairness. > > My point about the modifications to congestion control algorithm was > to point out that your work makes an interesting observation(or my > paraphrasing of it) - that it becomes less and less possible to > acheive the maximum possible throughput (in LFNs) as the rate and > distances get larger and larger - and that issue should be addressed > directly. This is true even when a single "data flow" or "application" > might have the entire TCP bandwidth at its disposal. A modification > hopefully will not create super TCPs i.e. would be a reasonable approx > of current equation (flow slow long links or fast short links), but > would provide a better choice (which should propagate to all TCPs) > for 10G and beyond at long distances. > As I was saying in some other message, modifying TCP is very problematic for many reasons, and it is definetly outside the scope of this WG. > There is a cost of synchronizing across multiple connections - both > on the target and the initiator. Someone has to do the work, be it > the host or the adapter. On the initiator side, multiple > connections/adapters (actually their completion routine) will be updating > a single variable MaxCmdRn and the software posting the commands will > have to ensure that MaxCmdRn is not exceeded and has to post the commands > to the appropriate queues (how does it know which). And that is when > there is no stall. Assuming this informations is written/read from > multiple processors (with the pinging of the cache lines), the cost > will escalate rapidly with the number of processors involved. > > There is similar situation on the target. It probably > will prevent different controller boards on the target from sharing > connections in a session (whenever one of the controller decides on a > new increased value of MaxCmdRn, it should be propagated to the other > controller also ??) - maybe this is not so. > > That is why I am very interested in any pointers you or anyone else > could be provide on high-speed implementation of a "single data-flow" > across multiple connections. I am sure this has been tried in the > past so hopefully there are some pointers to results. > > Somesh > I think that what you are saying is somewhat orthogonal to my points. I was arguing for one TCP connection for one storage connection, which is different, simpler and more natural than bundling or anonymizing all storage connections in one big flow and then striping a "single data-flow" across multiple connections. I am not sure about other literature, I'll be looking for it myself. Victor From owner-ips@ece.cmu.edu Wed Jan 17 02:21:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA14289 for ; Wed, 17 Jan 2001 02:21:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0H4nWk00418 for ips-outgoing; Tue, 16 Jan 2001 23:49:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mtiwmhc24.worldnet.att.net (mtiwmhc24.worldnet.att.net [204.127.131.49]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0H4kFs00351 for ; Tue, 16 Jan 2001 23:46:15 -0500 (EST) Received: from SG2351 ([32.102.233.99]) by mtiwmhc24.worldnet.att.net (InterMail vM.4.01.03.10 201-229-121-110) with SMTP id <20010117044608.SYQC1480.mtiwmhc24.worldnet.att.net@SG2351> for ; Wed, 17 Jan 2001 04:46:08 +0000 Reply-To: From: "Somesh Gupta" To: "IPS" Subject: InDataOrder (or not InDataOrder) Date: Tue, 16 Jan 2001 20:43:22 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Looking at the Appendix C of the latest draft, and the InDataOrder key, it seems that both sides can decide that they want data in order (for a single command) - or each side can independently agree to accept data in order. I have the following questions: 1. Does any target return data for a read out of order? Althought the potential benefit is there, in practice is it used? 2. For a write, does this restriction apply to the entire write operation or only to individual r2t's (I assume the r2t) although in practice there seems little reason for the r2t's to be requested out of order? Thanks, Somesh From owner-ips@ece.cmu.edu Wed Jan 17 04:41:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA14978 for ; Wed, 17 Jan 2001 04:41:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0H7G0Y03307 for ips-outgoing; Wed, 17 Jan 2001 02:16:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0H7Fas03299 for ; Wed, 17 Jan 2001 02:15:36 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 3B7C27C3 for ; Tue, 16 Jan 2001 23:15:35 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id XAA15439; Tue, 16 Jan 2001 23:14:36 -0800 (PST) From: Santosh Rao Message-Id: <200101170714.XAA15439@hpcuhe.cup.hp.com> Subject: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports To: ips@ece.cmu.edu (ips) Date: Tue, 16 Jan 2001 23:14:26 -0800 (PST) Cc: santoshr@hpcuhe.cup.hp.com (Santosh Rao) X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I have a question for the FCIP and iFCP folks. Do these 2 protocols still assure in-order delivery of frames to the FC end nodes (N/NL_Ports) when requested through the "Sequential Delivery" bit in FLOGI ? Can this assurance be made even while using multiple TCP connections b/n edge switch/routers ? - Santosh -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Wed Jan 17 13:05:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA23191 for ; Wed, 17 Jan 2001 13:05:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0HF0wu23294 for ips-outgoing; Wed, 17 Jan 2001 10:00:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from radmail.rad.co.il ([192.116.244.108]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0HF0c123284 for ; Wed, 17 Jan 2001 10:00:38 -0500 (EST) Received: from sanrad.com ([172.17.200.73]) by radmail.rad.co.il (Post.Office MTA v3.5.3 release 223 ID# 0-69802U1300L1200S0V35) with ESMTP id il for ; Wed, 17 Jan 2001 16:02:57 +0200 Message-ID: <3A65CFDA.B12F96F@sanrad.com> Date: Wed, 17 Jan 2001 17:01:14 +0000 From: Yaron Klein Organization: SANRAD X-Mailer: Mozilla 4.7 [en] (WinNT; I) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: CRC64 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Since the last draft sets the CRC64 as TBD, a good candidate can be the one defined in: http://www.openusenet.org/diablo/ Which is also applied in many other applications. The polynom and sample code are in the file crctest.c in the latest version: polynom 0x0060034000f050b init 0xfac432b10cd5e44a Regards, Yaron From owner-ips@ece.cmu.edu Wed Jan 17 13:37:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA23848 for ; Wed, 17 Jan 2001 13:37:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0HG5EL25503 for ips-outgoing; Wed, 17 Jan 2001 11:05:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0HG4X125484 for ; Wed, 17 Jan 2001 11:04:33 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id C6R664RH; Wed, 17 Jan 2001 08:04:30 -0800 From: "Y P Cheng" To: "Santosh Rao" , "ips" Cc: "Santosh Rao" Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Wed, 17 Jan 2001 08:03:28 -0800 Message-ID: <000601c0809f$078e5e20$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: <200101170714.XAA15439@hpcuhe.cup.hp.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > I have a question for the FCIP and iFCP folks. Do these 2 protocols still > assure in-order delivery of frames to the FC end nodes (N/NL_Ports) > when requested through the "Sequential Delivery" bit in FLOGI ? > > Can this assurance be made even while using multiple TCP connections b/n > edge switch/routers ? > > - Santosh If I may add that most FC devices out in the market will NOT receive out-of-order delivery because it does not go over the scatter/gather list from beginning on every incoming frame! So, if FCIP and iFCP wish to work with legacy FC devices, they must learn from Brocade switches, i.e. assure in-order delivery. Y.P. Cheng, ConnectCom Solutions From owner-ips@ece.cmu.edu Wed Jan 17 16:31:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA26471 for ; Wed, 17 Jan 2001 16:31:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0HJMB902503 for ips-outgoing; Wed, 17 Jan 2001 14:22:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0HJLJ102476 for ; Wed, 17 Jan 2001 14:21:19 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id C6R664Z1; Wed, 17 Jan 2001 11:21:17 -0800 From: "Y P Cheng" To: Cc: "ips" Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Wed, 17 Jan 2001 11:20:07 -0800 Message-ID: <001701c080ba$80d9be80$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: <3A65EAD5.4DF32CDD@cup.hp.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > It is not only the desire to inter-operate with legacy FC devices > that drives this requirement. > This is a mandatory requirement, since FC-FLA Table 3 mandates > that N/NL_Ports originating FLOGI SHALL request in-order delivery in > the FLOGI. (by setting the "Sequential Delivery" bit in their class > specific parameters for class 2 & 3 to 1.). > - Santosh If in-order-delivery is a MUST, the next question is "How many open sequences?" On a 10 Gb/sec network with long latency, we need many open sequences to fill the pipeline. This is the same problem as TCP window size of iSCSI. To keep all open sequences in order, in the worst case, there will be a huge holding buffer in an iFCP and FCIP device. We have just moved the TCP window problem from an end-point into a bridge or gateway device. Hence, by just changing the protocol, the problem does not go away. Y.P. Cheng, ConnectCom Solutions From owner-ips@ece.cmu.edu Wed Jan 17 16:31:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA26482 for ; Wed, 17 Jan 2001 16:31:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0HIuCq01587 for ips-outgoing; Wed, 17 Jan 2001 13:56:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0HIsb101516 for ; Wed, 17 Jan 2001 13:54:38 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id D6E9A5AB; Wed, 17 Jan 2001 10:54:30 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id KAA22406; Wed, 17 Jan 2001 10:54:20 -0800 (PST) Message-ID: <3A65EAD5.4DF32CDD@cup.hp.com> Date: Wed, 17 Jan 2001 10:56:22 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng Cc: ips Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports References: <000601c0809f$078e5e20$90c809c0@yp_portable.advansys.com> Content-Type: multipart/mixed; boundary="------------5375859CBE75FE13021ED1BC" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------5375859CBE75FE13021ED1BC Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit It is not only the desire to inter-operate with legacy FC devices that drives this requirement. This is a mandatory requirement, since FC-FLA Table 3 mandates that N/NL_Ports originating FLOGI SHALL request in-order delivery in the FLOGI. (by setting the "Sequential Delivery" bit in their class specific parameters for class 2 & 3 to 1.). - Santosh Y P Cheng wrote: > > I have a question for the FCIP and iFCP folks. Do these 2 protocols still > > assure in-order delivery of frames to the FC end nodes (N/NL_Ports) > > when requested through the "Sequential Delivery" bit in FLOGI ? > > > > Can this assurance be made even while using multiple TCP connections b/n > > edge switch/routers ? > > > > - Santosh > > If I may add that most FC devices out in the market will NOT receive > out-of-order delivery because it does not go over the scatter/gather list > from beginning on every incoming frame! So, if FCIP and iFCP wish to work > with legacy FC devices, they must learn from Brocade switches, i.e. assure > in-order delivery. > > Y.P. Cheng, ConnectCom Solutions --------------5375859CBE75FE13021ED1BC Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------5375859CBE75FE13021ED1BC-- From owner-ips@ece.cmu.edu Wed Jan 17 17:19:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA27045 for ; Wed, 17 Jan 2001 17:19:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0HKP5t04801 for ips-outgoing; Wed, 17 Jan 2001 15:25:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0HKOa104787 for ; Wed, 17 Jan 2001 15:24:38 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id C6R6647M; Wed, 17 Jan 2001 12:24:34 -0800 From: "Y P Cheng" To: Cc: "'Ips@Ece. Cmu. Edu'" Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Wed, 17 Jan 2001 12:23:20 -0800 Message-ID: <001901c080c3$55309840$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: <3A65F50A.6E7C1DE5@cup.hp.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > > Hence, by just changing the protocol, the problem does not go away. > > Not sure what you mean here. Changing which protocols ? By reading this reflector, you must know my opinion about the overlapping functionality of iFCP and iSCSI. I consider them as two different protocols solving the same set of problems, i.e. long Internet latency, congestion, lossy connections, duplicated or missing frames, R_A_TOV values, target device naming and discovery, etc., etc. Hence, why two protocols solving the same set of problems? (I must qualify that iFCP and FCIP do not have to solve the message framing issue. Each FCP IU is a separate frame/sequence. But, if iFCP and FCIP use TCP encapsulation, then, we back to square zero.) From owner-ips@ece.cmu.edu Wed Jan 17 17:20:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA27076 for ; Wed, 17 Jan 2001 17:20:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0HJdFC03093 for ips-outgoing; Wed, 17 Jan 2001 14:39:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0HJc4103055 for ; Wed, 17 Jan 2001 14:38:05 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id BA13611CA; Wed, 17 Jan 2001 11:38:01 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id LAA25781; Wed, 17 Jan 2001 11:37:53 -0800 (PST) Message-ID: <3A65F50A.6E7C1DE5@cup.hp.com> Date: Wed, 17 Jan 2001 11:39:55 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng Cc: ips Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports References: <001701c080ba$80d9be80$90c809c0@yp_portable.advansys.com> Content-Type: multipart/mixed; boundary="------------05D56DAD8BEB1816C20568E8" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------05D56DAD8BEB1816C20568E8 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Y P Cheng wrote: > > It is not only the desire to inter-operate with legacy FC devices > > that drives this requirement. > > This is a mandatory requirement, since FC-FLA Table 3 mandates > > that N/NL_Ports originating FLOGI SHALL request in-order delivery in > > the FLOGI. (by setting the "Sequential Delivery" bit in their class > > specific parameters for class 2 & 3 to 1.). > > - Santosh > > If in-order-delivery is a MUST, the next question is "How many open > sequences?" In-order is a requirement that spans across exchanges. (not just within an exchange). > On a 10 Gb/sec network with long latency, we need many open > sequences to fill the pipeline. This is the same problem as TCP window size > of iSCSI. To keep all open sequences in order, in the worst case, there > will be a huge holding buffer in an iFCP and FCIP device. We have just > moved the TCP window problem from an end-point into a bridge or gateway > device. Agreed. The alternative is to use 1 TCP connection per FC (S_ID, D_ID) pair. Which is why I raised the question in the context of multiple TCP connections between the edge switch/routers per (S_ID, D_ID) pair. > Hence, by just changing the protocol, the problem does not go away. Not sure what you mean here. Changing which protocols ? Regards, Santosh --------------05D56DAD8BEB1816C20568E8 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------05D56DAD8BEB1816C20568E8-- From owner-ips@ece.cmu.edu Wed Jan 17 17:55:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA27742 for ; Wed, 17 Jan 2001 17:55:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0HKY5T05505 for ips-outgoing; Wed, 17 Jan 2001 15:34:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0HKXh105495 for ; Wed, 17 Jan 2001 15:33:43 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id VAA110078 for ; Wed, 17 Jan 2001 21:33:29 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id VAA228108 for ; Wed, 17 Jan 2001 21:33:32 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D7.0070EDA2 ; Wed, 17 Jan 2001 21:33:28 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 17 Jan 2001 22:29:11 +0200 Subject: Re: CRC64 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Yaron, We are all familiar with the internet search tools. The polynom you quote was mentioned already by me. It is a 56 bit CRC that we still have to check its theoretical goodness. We know also about the tests run on it. Thanks, Julo Yaron Klein on 17/01/2001 19:01:14 Please respond to Yaron Klein To: ips@ece.cmu.edu cc: Subject: CRC64 Since the last draft sets the CRC64 as TBD, a good candidate can be the one defined in: http://www.openusenet.org/diablo/ Which is also applied in many other applications. The polynom and sample code are in the file crctest.c in the latest version: polynom 0x0060034000f050b init 0xfac432b10cd5e44a Regards, Yaron From owner-ips@ece.cmu.edu Thu Jan 18 11:58:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA28476 for ; Thu, 18 Jan 2001 11:58:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0IEXNa08896 for ips-outgoing; Thu, 18 Jan 2001 09:33:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0IAtR104756 for ; Thu, 18 Jan 2001 05:55:27 -0500 (EST) Received: from divyaroot.India.Sun.COM ([129.158.228.35]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id CAA28391 for ; Thu, 18 Jan 2001 02:55:23 -0800 (PST) Received: from helix (helix [129.158.226.51]) by divyaroot.India.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.0) with SMTP id QAA02425 for ; Thu, 18 Jan 2001 16:25:20 +0530 (IST) Message-Id: <200101181055.QAA02425@divyaroot.India.Sun.COM> Date: Thu, 18 Jan 2001 16:27:05 -0500 (GMT) From: Raghavendra Rao Reply-To: Raghavendra Rao Subject: NOP-Out clarification To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: yoablEauLAQYyqtt7SfoEg== Sender: owner-ips@ece.cmu.edu Precedence: bulk My reading of NOP-out as used for data-in acknowledgements is that it is for the entire task - Not per R2T since I don't see offset/length in the NOP-out header. Is my assumption correct ? It would be nice if the draft can be more explicit. Thanks. -JP From owner-ips@ece.cmu.edu Thu Jan 18 12:02:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA28569 for ; Thu, 18 Jan 2001 12:02:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0IEXP508899 for ips-outgoing; Thu, 18 Jan 2001 09:33:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0IBdt105473 for ; Thu, 18 Jan 2001 06:39:55 -0500 (EST) Received: from divyaroot.India.Sun.COM ([129.158.224.35]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id DAA11917 for ; Thu, 18 Jan 2001 03:39:50 -0800 (PST) Received: from helix (helix [129.158.226.51]) by divyaroot.India.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.0) with SMTP id RAA10580 for ; Thu, 18 Jan 2001 17:09:46 +0530 (IST) Message-Id: <200101181139.RAA10580@divyaroot.India.Sun.COM> Date: Thu, 18 Jan 2001 17:11:32 -0500 (GMT) From: Raghavendra Rao Reply-To: Raghavendra Rao Subject: NOP-Out clarification: Got it To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: QESybexDlmlTfAGCFCVDjA== Sender: owner-ips@ece.cmu.edu Precedence: bulk Sorry about my previous mail. I found the answer in the -03 draft. It's quite clear about DataRN acknowledgements as well as collating NOP-Outs Thanks. -JP ======================== My reading of NOP-out as used for data-in acknowledgements is that it is for the entire task - Not per R2T since I don't see offset/length in the NOP-out header. Is my assumption correct ? It would be nice if the draft can be more explicit. Thanks. -JP From owner-ips@ece.cmu.edu Thu Jan 18 17:30:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA06431 for ; Thu, 18 Jan 2001 17:30:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0IJidV19580 for ips-outgoing; Thu, 18 Jan 2001 14:44:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0IJhj119532 for ; Thu, 18 Jan 2001 14:43:46 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 49F7914DD; Thu, 18 Jan 2001 11:43:43 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id LAA28009; Thu, 18 Jan 2001 11:43:37 -0800 (PST) Message-ID: <3A6747E5.C6990076@cup.hp.com> Date: Thu, 18 Jan 2001 11:45:41 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Raghavendra Rao Cc: ips@ece.cmu.edu Subject: Re: NOP-Out clarification References: <200101181055.QAA02425@divyaroot.India.Sun.COM> Content-Type: multipart/mixed; boundary="------------608104EFC528CFF6547F22A5" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------608104EFC528CFF6547F22A5 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit NOP-OUT was intended to ack a set of DataRNs. It need not be per task if DataNumber > the no. of frames in a task. (though the draft did claim DataRN to be per task). In any case, this topic is pointless, since I heard that the WG voted out DataRN. (?) - Santosh Raghavendra Rao wrote: > My reading of NOP-out as used for data-in acknowledgements is that > it is for the entire task - Not per R2T since I don't see offset/length > in the NOP-out header. Is my assumption correct ? It would be nice > if the draft can be more explicit. > > Thanks. > > -JP --------------608104EFC528CFF6547F22A5 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------608104EFC528CFF6547F22A5-- From owner-ips@ece.cmu.edu Thu Jan 18 19:05:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA07398 for ; Thu, 18 Jan 2001 19:05:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0ILqfj24168 for ips-outgoing; Thu, 18 Jan 2001 16:52:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0ILqR124152 for ; Thu, 18 Jan 2001 16:52:27 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 115C112D4 for ; Thu, 18 Jan 2001 13:52:24 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id NAA06347; Thu, 18 Jan 2001 13:51:33 -0800 (PST) Message-ID: <3A6765E2.F29983D6@cup.hp.com> Date: Thu, 18 Jan 2001 13:53:38 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Reject PDU Concerns. Content-Type: multipart/mixed; boundary="------------A167566E1CA72F4356C7B9A5" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------A167566E1CA72F4356C7B9A5 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Ref : Section 2.19 (Reject PDU), Section 5.5 (Digest Errors) Issue : ===== 1) Reject PDU must return an Initiator Task Tag in the Reject PDU header, which provides the initiator sufficient context to lookup the outbound exchange that was transmitted. 2) An error_byte field should be considered for inclusion in the Reject PDU which contains the byte offset of the errant byte[or word] in the received header/payload that caused the target to send a Reject response to the command. This is a simple solution that provides for quick fault isolation and root cause of the reason for the reject. This also rids the Reject PDU of any detailed elaboration of reason codes meant to allow root cause of the reason for the reject. 3) The Reject PDU should only be used for " Format Error" since the Reject mechanism is a per-task error recovery to deal with header format problems. "Digest Error" handling is too complex dealing with too many situations. Digest error recovery should be simplified to a connection level error recovery which involves Logout or connection close/re-start. The current digest recovery is a ping-pong as follows : - Target sends REJECT - Initiator on seeing REJECT with "Digest Error" sends Logout - Target sends Logout response. (the spec is not clear on who drives the connection close. does the initiator close the connection on receiving logout response, or does the target close connection and send logout reponse on another connection ??). The above procedure is quite convoluted and the followinf simplification should be considered : - On a digest error, target sends Logout, indicating appropriate reason code. - Target follows up the Logout with a TCP connection close. 4) There is no value add in expecting a logout response on a logout which is intended to close a connection and is sent on the same connection. The logout can be followed up with a connection close. Regards, Santosh --------------A167566E1CA72F4356C7B9A5 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------A167566E1CA72F4356C7B9A5-- From owner-ips@ece.cmu.edu Thu Jan 18 19:05:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA07397 for ; Thu, 18 Jan 2001 19:05:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0ILcfi23666 for ips-outgoing; Thu, 18 Jan 2001 16:38:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcars04e.ca.nortel.com (h56s242a129n47.user.nortelnetworks.com [47.129.242.56]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0ILcI123654 for ; Thu, 18 Jan 2001 16:38:19 -0500 (EST) Received: from zbl6c012.corpeast.baynetworks.com by zcars04e.ca.nortel.com; Thu, 18 Jan 2001 16:33:35 -0500 Received: from zbl6c002.corpeast.baynetworks.com ([132.245.205.52]) by zbl6c012.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id ZLK0BPYF; Thu, 18 Jan 2001 16:33:33 -0500 Received: from nortelnetworks.com (dhcp223-140.engeast.baynetworks.com [192.32.223.140]) by zbl6c002.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id CN99D3TG; Thu, 18 Jan 2001 16:33:34 -0500 Message-ID: <3A676146.6C6FD16D@nortelnetworks.com> Date: Thu, 18 Jan 2001 16:33:58 -0500 X-Sybari-Space: 00000000 00000000 00000000 From: "Victor Firoiu" Organization: Nortel Networks X-Mailer: Mozilla 4.76 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: somesh_gupta CC: IPS Reflector Subject: Re: Performance of iSCSI, FCIP and iFCP References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Orig: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Somesh, I was looking more at the comparison between the FCIP case of multiple FC connections (all connections between two FCIP gateways) per TCP connection and the iFCP case of one/few TCP connections per FC connection (thus multiple TCP connections per pair of iFCP gateways). I think that, in general, one TCP connection per storage connection is closer to "one/two TCP connections per user" model, and thus easier to defend as being "fair", compared to multiple TCP connections per storage connection. I agree, this is quite a fuzzy quantification, but is as precise as RFC 2914 and 2309 get in this matter. Regarding modifing TCP congestion control as you suggest, I am again very skeptical. Victor Somesh Gupta wrote: > > Victor, > > I just assumed that you were familiar with the iSCSI specification > of multiple connections per session - which is essentially as you > describe in the second last paragraph - there is one big "flow" > between the initiator and the target (called an iSCSI session) > which is then striped across multiple TCP connections - and I > think people assumed that your statements were in support of > such a proposal. > > Having said that, the math still holds. And also the fact that > fundamental limits to the throughput of a single TCP connection > really do require a change to TCP congestion avoidance (esp when > mistaking link error for congestion) formulae. > > If the link error rate is such that errors occur faster than time > to recover (or of that order), TCP will have to accomodate that > somehow. I don't think the TCP gurus are inflexible in this area > - PAWS, window scaling, and SACK are all example of TCP accomodating > the characteristics of the underlying network. > > Somesh > From owner-ips@ece.cmu.edu Thu Jan 18 19:05:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA07418 for ; Thu, 18 Jan 2001 19:05:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0IM6xf24668 for ips-outgoing; Thu, 18 Jan 2001 17:06:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0IM62124633 for ; Thu, 18 Jan 2001 17:06:03 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id XAA30002 for ; Thu, 18 Jan 2001 23:05:55 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id XAA96268 for ; Thu, 18 Jan 2001 23:05:55 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D8.00795C9F ; Thu, 18 Jan 2001 23:05:35 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 18 Jan 2001 23:32:41 +0200 Subject: Re: iSCSI: header digest error at initiator Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, In the SCSI world there is no scoreboarding at the initiator. The whole operation is master-slave with the target being the master. The status, counters etc. are determined by the target and the initiator has propagate them to the application client. Julo Santosh Rao on 16/01/2001 22:42:47 Please respond to Santosh Rao To: IPS Reflector cc: Subject: Re: iSCSI: header digest error at initiator Matt, My replies inline. Regards, Santosh Matt Wakeley wrote: > Santosh Rao wrote: > > > > If this is the intention of the recommended error recovery, it is the > > result of not allowing score-boarding. By score-boarding an initiator > > would detect an underrun and would just error the affected I/O back. > > Two comments here. First, in your example, the initiator is inventing an > error that really didn't occur in the target. The target completed the I/O > successfully, it was the transport that experienced an error, but you're > treating it like a target error. The LLP (initiator) would return an error to ULP indicating a transport error (service response of "service delivery or target failure") occurred. [due to the data underrun.] > Second, you say that initiators and targets routinely perform scoreboarding. > How is this done today? Buffer(s) are provided to an I/O chip. The I/O chip > writes the data into the buffers. It does not have the memory to determine > that each and every byte has been written to. So how is the initiator/target > supposed to be absolutely sure that every byte was written to? The initiator would use a check along the following lines : (total bytes xfer'ed as indicated by the chip ) = (no. of bytes of data xfer specified by ULP) - (resid reported by the target in SCSI Response PDU). to verify that all the data the target sent is accounted for at the initiator end. From owner-ips@ece.cmu.edu Thu Jan 18 19:05:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA07429 for ; Thu, 18 Jan 2001 19:05:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0ILkgm23960 for ips-outgoing; Thu, 18 Jan 2001 16:46:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0ILjj123925 for ; Thu, 18 Jan 2001 16:45:45 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id B6FB693A for ; Thu, 18 Jan 2001 14:45:41 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 8868617C for ; Thu, 18 Jan 2001 16:45:40 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id NAA05461 for ; Thu, 18 Jan 2001 13:45:35 -0800 (PST) Message-ID: <3A67642E.35140993@agilent.com> Date: Thu, 18 Jan 2001 13:46:22 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports References: <001901c080c3$55309840$90c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Please explain why/how you think iFCP and FCIP do not require the use of TCP? -Matt Wakeley Y P Cheng wrote: > > > > Hence, by just changing the protocol, the problem does not go away. > > > > Not sure what you mean here. Changing which protocols ? > > By reading this reflector, you must know my opinion about the overlapping > functionality of iFCP and iSCSI. I consider them as two different protocols > solving the same set of problems, i.e. long Internet latency, congestion, > lossy connections, duplicated or missing frames, R_A_TOV values, target > device naming and discovery, etc., etc. Hence, why two protocols solving > the same set of problems? (I must qualify that iFCP and FCIP do not have to > solve the message framing issue. Each FCP IU is a separate frame/sequence. > But, if iFCP and FCIP use TCP encapsulation, then, we back to square zero.) From owner-ips@ece.cmu.edu Thu Jan 18 19:09:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA07462 for ; Thu, 18 Jan 2001 19:09:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0IM6vv24664 for ips-outgoing; Thu, 18 Jan 2001 17:06:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0IM61124629 for ; Thu, 18 Jan 2001 17:06:01 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id XAA84584 for ; Thu, 18 Jan 2001 23:05:54 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id XAA116742 for ; Thu, 18 Jan 2001 23:05:54 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D8.00795DEC ; Thu, 18 Jan 2001 23:05:39 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 18 Jan 2001 23:39:27 +0200 Subject: Re: InDataOrder (or not InDataOrder) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Definitely - many high performance controllers, when they have cache misses, return data out of order for operations that involve many blocks. Julo "Somesh Gupta" on 17/01/2001 06:43:22 Please respond to somesh_gupta@silverbacksystems.com To: "IPS" cc: Subject: InDataOrder (or not InDataOrder) Looking at the Appendix C of the latest draft, and the InDataOrder key, it seems that both sides can decide that they want data in order (for a single command) - or each side can independently agree to accept data in order. I have the following questions: 1. Does any target return data for a read out of order? Althought the potential benefit is there, in practice is it used? 2. For a write, does this restriction apply to the entire write operation or only to individual r2t's (I assume the r2t) although in practice there seems little reason for the r2t's to be requested out of order? Thanks, Somesh From owner-ips@ece.cmu.edu Thu Jan 18 19:11:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA07479 for ; Thu, 18 Jan 2001 19:11:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0IM6jg24654 for ips-outgoing; Thu, 18 Jan 2001 17:06:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0IM60124627 for ; Thu, 18 Jan 2001 17:06:00 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id XAA175566 for ; Thu, 18 Jan 2001 23:05:53 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id XAA116736 for ; Thu, 18 Jan 2001 23:05:51 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D8.00795EC7 ; Thu, 18 Jan 2001 23:05:41 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 18 Jan 2001 23:50:31 +0200 Subject: Re: NOP-Out clarification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk JP, The consesus at the intermediate meeting in Orlando was to drop Data Numbering. If you have strong object here is your last chance to object. Also Data Numbering was meant only for incoming data and it is per command ("less than a task" if you take in account linked commands). Regards, Julo Please respond to Raghavendra Rao To: ips@ece.cmu.edu cc: Subject: NOP-Out clarification My reading of NOP-out as used for data-in acknowledgements is that it is for the entire task - Not per R2T since I don't see offset/length in the NOP-out header. Is my assumption correct ? It would be nice if the draft can be more explicit. Thanks. -JP From owner-ips@ece.cmu.edu Thu Jan 18 20:09:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA07812 for ; Thu, 18 Jan 2001 20:09:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0INKhR26908 for ips-outgoing; Thu, 18 Jan 2001 18:20:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0INKI126876 for ; Thu, 18 Jan 2001 18:20:18 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Thu, 18 Jan 2001 15:20:44 -0800 Message-ID: From: Joshua Tseng To: Santosh Rao , ips@ece.cmu.edu Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Thu, 18 Jan 2001 15:19:38 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > > I have a question for the FCIP and iFCP folks. Do these 2 > protocols still > assure in-order delivery of frames to the FC end nodes (N/NL_Ports) > when requested through the "Sequential Delivery" bit in FLOGI ? TCP will deliver the FC frames in the same order that they were passed to TCP by the FC network (FCIP or iFCP) or FC device (iFCP only). As long as the FC network or device passes them in the right order, TCP will do its part to keep the frames in the same order. > > Can this assurance be made even while using multiple TCP > connections b/n > edge switch/routers ? This question is N/A for FCIP, since only a single connection is specified between FCIP gateways. In iFCP, a single TCP connection is specified between each unique pair of N_PORTs. Each gateway may support multiple N_PORTs, but this will not interfere with frame delivery order between each N_PORT pair. So the answer to your question is yes for iFCP. Josh > > - Santosh > > -- > ################################# > Santosh Rao > Software Design Engineer, > HP, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################# > From owner-ips@ece.cmu.edu Thu Jan 18 20:10:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA07828 for ; Thu, 18 Jan 2001 20:10:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0IMxof26193 for ips-outgoing; Thu, 18 Jan 2001 17:59:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0IMxd126182 for ; Thu, 18 Jan 2001 17:59:39 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 8571712A2; Thu, 18 Jan 2001 14:59:38 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id OAA12876; Thu, 18 Jan 2001 14:59:34 -0800 (PST) Message-ID: <3A6775D2.4C800E59@cup.hp.com> Date: Thu, 18 Jan 2001 15:01:39 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI: header digest error at initiator References: Content-Type: multipart/mixed; boundary="------------3BDCD3DFB9FC4868DF33BEAA" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------3BDCD3DFB9FC4868DF33BEAA Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, I think Matt clarified earlier that the operation I was referring to is not score-boarding. I was referring to a check along the following lines to verify if there was I/O underrun : (total bytes xfer'ed as indicated by the chip ) = (no. of bytes of data xfer specified by ULP) - (resid reported by the target in SCSI Response PDU). This is a standard check in an un-reliable scsi transport protocol like FC. One can argue that it is not required with reliable SCSI transports like iSCSI. More importantly, if iSCSI allows overlapping data transfers (which is the case, unlike FC), this type of operation is not possible, as doug pointed out. Thanks All. Regards, Santosh julian_satran@il.ibm.com wrote: > Santosh, > > In the SCSI world there is no scoreboarding at the initiator. > > The whole operation is master-slave with the target being the master. > The status, counters etc. are determined by the target and the initiator > has propagate them to the application client. > > Julo > > Santosh Rao on 16/01/2001 22:42:47 > > Please respond to Santosh Rao > > To: IPS Reflector > cc: > Subject: Re: iSCSI: header digest error at initiator > > Matt, > > My replies inline. > > Regards, > Santosh > > Matt Wakeley wrote: > > > Santosh Rao wrote: > > > > > > If this is the intention of the recommended error recovery, it is the > > > result of not allowing score-boarding. By score-boarding an initiator > > > would detect an underrun and would just error the affected I/O back. > > > > Two comments here. First, in your example, the initiator is inventing an > > error that really didn't occur in the target. The target completed the > I/O > > successfully, it was the transport that experienced an error, but you're > > treating it like a target error. > > The LLP (initiator) would return an error to ULP indicating a transport > error > (service response of "service delivery or target failure") occurred. [due > to the > data underrun.] > > > Second, you say that initiators and targets routinely perform > scoreboarding. > > How is this done today? Buffer(s) are provided to an I/O chip. The I/O > chip > > writes the data into the buffers. It does not have the memory to > determine > > that each and every byte has been written to. So how is the > initiator/target > > supposed to be absolutely sure that every byte was written to? > > The initiator would use a check along the following lines : > > (total bytes xfer'ed as indicated by the chip ) = > (no. of bytes of data xfer specified by ULP) - (resid reported by the > target in > SCSI Response PDU). > > to verify that all the data the target sent is accounted for at the > initiator > end. --------------3BDCD3DFB9FC4868DF33BEAA Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------3BDCD3DFB9FC4868DF33BEAA-- From owner-ips@ece.cmu.edu Thu Jan 18 20:12:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA07843 for ; Thu, 18 Jan 2001 20:12:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0IMwho26169 for ips-outgoing; Thu, 18 Jan 2001 17:58:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antipater.hosting.pacbell.net (antipater.hosting.pacbell.net [216.100.99.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0ILsT124227 for ; Thu, 18 Jan 2001 16:54:35 -0500 (EST) Received: from SG2351 ([206.112.110.142]) by antipater.hosting.pacbell.net id QAA19933; Thu, 18 Jan 2001 16:54:16 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "Victor Firoiu" Cc: "IPS Reflector" Subject: RE: Performance of iSCSI, FCIP and iFCP Date: Thu, 18 Jan 2001 13:51:31 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: <3A676146.6C6FD16D@nortelnetworks.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I am more hopeful on the point of adapting the algorithm as environments change. The TCP gurus are conservative but not totally inflexible. -----Original Message----- From: Victor Firoiu [mailto:vfiroiu@nortelnetworks.com] Sent: Thursday, January 18, 2001 1:34 PM To: somesh_gupta Cc: IPS Reflector Subject: Re: Performance of iSCSI, FCIP and iFCP Somesh, I was looking more at the comparison between the FCIP case of multiple FC connections (all connections between two FCIP gateways) per TCP connection and the iFCP case of one/few TCP connections per FC connection (thus multiple TCP connections per pair of iFCP gateways). I think that, in general, one TCP connection per storage connection is closer to "one/two TCP connections per user" model, and thus easier to defend as being "fair", compared to multiple TCP connections per storage connection. I agree, this is quite a fuzzy quantification, but is as precise as RFC 2914 and 2309 get in this matter. Regarding modifing TCP congestion control as you suggest, I am again very skeptical. Victor Somesh Gupta wrote: > > Victor, > > I just assumed that you were familiar with the iSCSI specification > of multiple connections per session - which is essentially as you > describe in the second last paragraph - there is one big "flow" > between the initiator and the target (called an iSCSI session) > which is then striped across multiple TCP connections - and I > think people assumed that your statements were in support of > such a proposal. > > Having said that, the math still holds. And also the fact that > fundamental limits to the throughput of a single TCP connection > really do require a change to TCP congestion avoidance (esp when > mistaking link error for congestion) formulae. > > If the link error rate is such that errors occur faster than time > to recover (or of that order), TCP will have to accomodate that > somehow. I don't think the TCP gurus are inflexible in this area > - PAWS, window scaling, and SACK are all example of TCP accomodating > the characteristics of the underlying network. > > Somesh > From owner-ips@ece.cmu.edu Thu Jan 18 21:12:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA08159 for ; Thu, 18 Jan 2001 21:12:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0J08je28068 for ips-outgoing; Thu, 18 Jan 2001 19:08:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0J08B128057 for ; Thu, 18 Jan 2001 19:08:11 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id DG4KPTK9; Thu, 18 Jan 2001 16:08:09 -0800 From: "Y P Cheng" To: "Matt Wakeley" , "IPS Reflector" Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Thu, 18 Jan 2001 16:06:53 -0800 Message-ID: <004201c081ab$ba5383c0$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: <3A67642E.35140993@agilent.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > > But, if iFCP and FCIP use TCP encapsulation, then, we back to > > square zero.) > Please explain why/how you think iFCP and FCIP do not require the > use of TCP? > > -Matt Wakeley Matt, Certainly, both drafts have proposed TCP. However, when two iFCP or two FCIP devices talk to another of its own kind, they have the freedom to use any protocol they wish, even SCTP, as long as the protocol provides reliable delivery. This is because the frames are only received by another device of same implementation. There is no interoperability issue with any legacy implementations. Now, does the other protocol obeys the fairness rule of Internet? I assume it would. Y.P. From owner-ips@ece.cmu.edu Thu Jan 18 21:15:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA08215 for ; Thu, 18 Jan 2001 21:15:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0J0ahG28716 for ips-outgoing; Thu, 18 Jan 2001 19:36:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0J0aE128703 for ; Thu, 18 Jan 2001 19:36:14 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id E93E88A2 for ; Thu, 18 Jan 2001 17:36:12 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id CF1B983 for ; Thu, 18 Jan 2001 19:36:11 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id QAA09456 for ; Thu, 18 Jan 2001 16:36:05 -0800 (PST) Message-ID: <3A678C26.A2E34347@agilent.com> Date: Thu, 18 Jan 2001 16:36:54 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports References: <004201c081ab$ba5383c0$90c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Your reply seems to indicate that UDP or some other protocol could be used. Only TCP and SCTP implement the congestion control REQUIRED by the internet. -Matt Y P Cheng wrote: > > > > But, if iFCP and FCIP use TCP encapsulation, then, we back to > > > square zero.) > > Please explain why/how you think iFCP and FCIP do not require the > > use of TCP? > > > > -Matt Wakeley > > Matt, > > Certainly, both drafts have proposed TCP. > > However, when two iFCP or two FCIP devices talk to another of its own kind, > they have the freedom to use any protocol they wish, even SCTP, as long as > the protocol provides reliable delivery. This is because the frames are > only received by another device of same implementation. There is no > interoperability issue with any legacy implementations. Now, does the other > protocol obeys the fairness rule of Internet? I assume it would. > > Y.P. From owner-ips@ece.cmu.edu Thu Jan 18 21:18:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA08280 for ; Thu, 18 Jan 2001 21:18:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0J0jih28946 for ips-outgoing; Thu, 18 Jan 2001 19:45:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0J0jP128929 for ; Thu, 18 Jan 2001 19:45:25 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id E4F6F1838 for ; Thu, 18 Jan 2001 16:45:21 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id QAA22844; Thu, 18 Jan 2001 16:45:02 -0800 (PST) Message-ID: <3A678E89.1F694E2@cup.hp.com> Date: Thu, 18 Jan 2001 16:47:05 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iscsi : Fragmentation & Reassembly issues in iSCSI. Content-Type: multipart/mixed; boundary="------------146167E8DEDC340718D98947" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------146167E8DEDC340718D98947 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, Issue : ===== DataPDULength, PingMaxReplyLength, Fragmentation/Reassembly issues. Problem : ======= 1) What is the relationship b/n DataPDULength and PingMaxReplyLength ? If a simple iSCSI test program decided to use the NOP-OUT as a form of basic testing and began issuing increasing sizes of NOP-OUT, at some point, it will cross the DataPDULength PDU size (provided it was able to negotiate PingMaxReplyLength > DataPDULength, which such a test program would attempt, in the interests of large I/O testing). The DataPDULength introduces fragmentation and reassembly issues at the iSCSI layer for the NOP-OUT, NOP-IN and Login commands. (For the SCSI Data PDUs, reassembly is possible based on the Buffer Offset contained in the data PDU header.) Question : ======== What drives the fundamental requirement for the DataPDULength ? I see no benefit for this other than trying to ensure a small data payload in order to have an effective CRC. Are there any other benefits ? Is there a need to impose an upper bound (MTU) on the size of PDUs other than the SCSI Command PDU (containing immediate data) and SCSI Data PDU ? Solution : ======= Specify explicitly in Appendix C that DataPDULength is only applicable for SCSI Command PDU and SCSI Data PDU. The current definition is open to being mis-interpreted as a form of iSCSI MTU, something that can result in the iSCSI layer having to deal with fragmentation and re-assembly issues for non-SCSI PDUs. Regards, Santosh --------------146167E8DEDC340718D98947 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------146167E8DEDC340718D98947-- From owner-ips@ece.cmu.edu Thu Jan 18 23:11:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA11057 for ; Thu, 18 Jan 2001 23:11:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0J2Gla01042 for ips-outgoing; Thu, 18 Jan 2001 21:16:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0J2Gc101034 for ; Thu, 18 Jan 2001 21:16:38 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id C20A114F4; Thu, 18 Jan 2001 18:16:33 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA28409; Thu, 18 Jan 2001 18:16:28 -0800 (PST) Message-ID: <3A67A3F9.6384A49A@cup.hp.com> Date: Thu, 18 Jan 2001 18:18:34 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Sriram Rupanagunta Cc: ips@ece.cmu.edu Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports References: Content-Type: multipart/mixed; boundary="------------73E42DBE269AB3C7285D672E" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------73E42DBE269AB3C7285D672E Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sriram Rupanagunta wrote: > > > Can this assurance be made even while using multiple TCP > > > connections b/n > > > edge switch/routers ? > > This question is N/A for FCIP, since only a single connection > > is specified between FCIP gateways. In iFCP, a single TCP connection > > Sometime back, FCIP folks mentioned that there is nothing in the > FCIP spec that limits the number of connections to one. My > understanding is FCIP leaves this to implementations, whether > to use one connection or multiple. The above is EXACTLY the reason I have raised this question. I hear conflicting statements being made on this issue. I don't see how FCIP can use multiple TCP connections per (S_ID, D_ID) pair and guarantee in-order delivery without implementing buffering and re-ordering on the FC-IP switch. Both FCIP and iFCP MUST guarantee in-order delivery since all FC end ports (N/NL_Ports) depend on this feature and FC-FLA Table 3 mandates this. Regards, Santosh > > Can someone clarify this once for all on this list please ? > > If there is no limit on the number of connections in FCIP, > Santosh's question is valid. In that case, is it fair to > say that FCIP does not guarentee orderly delivery ? > > - Sriram --------------73E42DBE269AB3C7285D672E Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------73E42DBE269AB3C7285D672E-- From owner-ips@ece.cmu.edu Thu Jan 18 23:13:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA11070 for ; Thu, 18 Jan 2001 23:12:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0J1nki00401 for ips-outgoing; Thu, 18 Jan 2001 20:49:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from rly-ip02.mx.aol.com (rly-ip02.mx.aol.com [152.163.225.160]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0J1n2100384 for ; Thu, 18 Jan 2001 20:49:02 -0500 (EST) Received: from tot-ti.proxy.aol.com (tot-ti.proxy.aol.com [152.163.194.131]) by rly-ip02.mx.aol.com (8.8.8/8.8.8/AOL-5.0.0) with ESMTP id UAA06598 for ; Thu, 18 Jan 2001 20:48:56 -0500 (EST) Received: from vaio (ACA32FDC.ipt.aol.com [172.163.47.220]) by tot-ti.proxy.aol.com (8.10.0/8.10.0) with SMTP id f0J1mqj14759 for ; Thu, 18 Jan 2001 20:48:53 -0500 (EST) From: "Sriram Rupanagunta" To: Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Thu, 18 Jan 2001 19:46:00 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 X-Apparently-From: Yeseswini@aol.com Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > > Can this assurance be made even while using multiple TCP > > connections b/n > > edge switch/routers ? > > This question is N/A for FCIP, since only a single connection > is specified between FCIP gateways. In iFCP, a single TCP connection Sometime back, FCIP folks mentioned that there is nothing in the FCIP spec that limits the number of connections to one. My understanding is FCIP leaves this to implementations, whether to use one connection or multiple. Can someone clarify this once for all on this list please ? If there is no limit on the number of connections in FCIP, Santosh's question is valid. In that case, is it fair to say that FCIP does not guarentee orderly delivery ? - Sriram From owner-ips@ece.cmu.edu Fri Jan 19 00:14:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA11916 for ; Fri, 19 Jan 2001 00:14:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0J2nlK01731 for ips-outgoing; Thu, 18 Jan 2001 21:49:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0J2nT101726 for ; Thu, 18 Jan 2001 21:49:29 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id DG4KPTPT; Thu, 18 Jan 2001 18:49:27 -0800 From: "Y P Cheng" To: "Matt Wakeley" Cc: "'Ips@Ece. Cmu. Edu'" Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Thu, 18 Jan 2001 18:48:01 -0800 Message-ID: <005101c081c2$3d381ce0$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: <3A678C26.A2E34347@agilent.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > Your reply seems to indicate that UDP or some other protocol > could be used. Only TCP and SCTP implement the congestion > control REQUIRED by the internet. > > -Matt I said "they have the freedom to use any protocol they wish, even SCTP, as long as the protocol provides reliable delivery." I also said "does the other protocol obeys the fairness rule of Internet? I assume it would." I don't believe UDP fits these two statements. However, talking to a device of its own kind does give it the freedom in choosing a protocol. I did not say that it has a blank check to do everything. Y.P. > Y P Cheng wrote: > > > > > > But, if iFCP and FCIP use TCP encapsulation, then, we back to > > > > square zero.) > > > Please explain why/how you think iFCP and FCIP do not require the > > > use of TCP? > > > > > > -Matt Wakeley > > > > Matt, > > > > Certainly, both drafts have proposed TCP. > > > > However, when two iFCP or two FCIP devices talk to another of > its own kind, > > they have the freedom to use any protocol they wish, even SCTP, > as long as > > the protocol provides reliable delivery. This is because the frames are > > only received by another device of same implementation. There is no > > interoperability issue with any legacy implementations. Now, > does the other > > protocol obeys the fairness rule of Internet? I assume it would. > > > > Y.P. > From owner-ips@ece.cmu.edu Fri Jan 19 05:40:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA26938 for ; Fri, 19 Jan 2001 05:40:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0J8FrS08346 for ips-outgoing; Fri, 19 Jan 2001 03:15:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0J8FY108341 for ; Fri, 19 Jan 2001 03:15:35 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id DAA58904; Fri, 19 Jan 2001 03:08:43 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id BAA69044; Fri, 19 Jan 2001 01:15:32 -0700 X-Priority: 1 (High) Importance: Normal Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports To: "Y P Cheng" Cc: "Matt Wakeley" , "IPS Reflector" X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Fri, 19 Jan 2001 00:11:41 -0800 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 01/19/2001 01:15:32 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk YP, Not sure where you are going here, the purpose of a standard is to permit interconnection with other manufactures equipment. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403 Internet address: hufferd@us.ibm.com "Y P Cheng" @ece.cmu.edu on 01/18/2001 04:06:53 PM Sent by: owner-ips@ece.cmu.edu To: "Matt Wakeley" , "IPS Reflector" cc: Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports > > But, if iFCP and FCIP use TCP encapsulation, then, we back to > > square zero.) > Please explain why/how you think iFCP and FCIP do not require the > use of TCP? > > -Matt Wakeley Matt, Certainly, both drafts have proposed TCP. However, when two iFCP or two FCIP devices talk to another of its own kind, they have the freedom to use any protocol they wish, even SCTP, as long as the protocol provides reliable delivery. This is because the frames are only received by another device of same implementation. There is no interoperability issue with any legacy implementations. Now, does the other protocol obeys the fairness rule of Internet? I assume it would. Y.P. From owner-ips@ece.cmu.edu Fri Jan 19 12:03:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA02445 for ; Fri, 19 Jan 2001 12:03:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0JET0427565 for ips-outgoing; Fri, 19 Jan 2001 09:29:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0J6A5105835 for ; Fri, 19 Jan 2001 01:10:05 -0500 (EST) Received: from divyaroot.India.Sun.COM ([129.158.224.35]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id WAA01555 for ; Thu, 18 Jan 2001 22:10:03 -0800 (PST) Received: from helix (helix [129.158.226.51]) by divyaroot.India.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1) with SMTP id LAA03488 for ; Fri, 19 Jan 2001 11:40:01 +0530 (IST) Message-Id: <200101190610.LAA03488@divyaroot.India.Sun.COM> Date: Fri, 19 Jan 2001 11:41:47 -0500 (GMT) From: Raghavendra Rao Reply-To: Raghavendra Rao Subject: iSCSI: DataRN To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: 9P66XPHlRqyhmsxCaQjIjw== Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, > The consesus at the intermediate meeting in Orlando was to drop Data > Numbering. > If you have strong object here is your last chance to object. > I support the removal of DataRNs. I assume that this will also result in the elimination of the concept of micro-level restart of data transfer after a connection failover with in a session ? Or is that still present ? Thanks. -JP From owner-ips@ece.cmu.edu Fri Jan 19 12:09:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA02580 for ; Fri, 19 Jan 2001 12:09:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0JEUE627611 for ips-outgoing; Fri, 19 Jan 2001 09:30:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.jlonline.com ([202.102.24.27]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f0J9Sx109606 for ; Fri, 19 Jan 2001 04:28:59 -0500 (EST) Received: from mail.jlonline.com([202.102.24.27]) by mail.jlonline.com(JetMail 2.5.3.0) with SMTP id jm83a68379f; Fri, 19 Jan 2001 09:31:04 -0000 Received: from loki.ietf.org([132.151.1.177]) by mail.jlonline.com(JetMail 2.5.3.0) with SMTP id jm2f3a638b9d; Mon, 15 Jan 2001 15:30:12 -0000 Received: (from adm@localhost) by loki.ietf.org (8.9.1b+Sun/8.9.1) id IAA15898 for ietf-123-outbound.03@ietf.org; Mon, 15 Jan 2001 08:15:01 -0500 (EST) Received: from ietf.org (odin.ietf.org [10.27.2.28]) by loki.ietf.org (8.9.1b+Sun/8.9.1) with ESMTP id GAA15296 for ; Mon, 15 Jan 2001 06:44:36 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA01004; Mon, 15 Jan 2001 06:44:36 -0500 (EST) Message-Id: <200101151144.GAA01004@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-iscsi-boot-01.txt Date: Mon, 15 Jan 2001 06:44:36 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : A Standard for BootStrapping Clients using the iSCSI Protocol Author(s) : P. Sarkar, D. Missimer, C. Sapuntzakis Filename : draft-ietf-ips-iscsi-boot-01.txt Pages : 8 Date : 12-Jan-01 The Small Computer Systems Interface (SCSI) is a popular family of protocols for communicating with I/O devices, especially storage devices. iSCSI is a proposed transport protocol for SCSI that operates on top of TCP[12]. This memo describes a standard mechanism to enable clients to bootstrap themselves using the iSCSI protocol. The goal of this standard is to enable clients to obtain the information to open an iSCSI session with the iSCSI bootstrpping server, assuming this information is not available. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-boot-01.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-boot-01.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-boot-01.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20010112140343.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-iscsi-boot-01.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-iscsi-boot-01.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20010112140343.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Fri Jan 19 13:14:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA03642 for ; Fri, 19 Jan 2001 13:13:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0JFv2m00493 for ips-outgoing; Fri, 19 Jan 2001 10:57:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0JFu0100454 for ; Fri, 19 Jan 2001 10:56:01 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA198256 for ; Fri, 19 Jan 2001 16:55:51 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA146938 for ; Fri, 19 Jan 2001 16:54:36 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D9.0057608D ; Fri, 19 Jan 2001 16:54:23 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 19 Jan 2001 16:18:22 +0200 Subject: Re: NOP-Out clarification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk As far as I know there will be an "action items list" that David Black is supposed to send out and, as usual, the notes (a summary of who said what). Regards, Julo Santosh Rao on 19/01/2001 01:04:05 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Re: NOP-Out clarification Julian, I don't know if this is a question for yourself or David. (but I will ask anyway.). Is somebody going to be sending out a write-up on the discussions conducted and decisions reached at the ips WG meet at Orlando, for the benefit of folks like myself who could'nt make it. Regards, Santosh julian_satran@il.ibm.com wrote: > JP, > > The consesus at the intermediate meeting in Orlando was to drop Data > Numbering. > If you have strong object here is your last chance to object. > > Also Data Numbering was meant only for incoming data and it is per command > ("less than a task" if you take in account linked commands). > > Regards, > Julo > > Please respond to Raghavendra Rao > > To: ips@ece.cmu.edu > cc: > Subject: NOP-Out clarification > > My reading of NOP-out as used for data-in acknowledgements is that > it is for the entire task - Not per R2T since I don't see offset/length > in the NOP-out header. Is my assumption correct ? It would be nice > if the draft can be more explicit. > > Thanks. > > -JP - santoshr.vcf From owner-ips@ece.cmu.edu Fri Jan 19 13:14:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA03655 for ; Fri, 19 Jan 2001 13:14:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0JFt6o00402 for ips-outgoing; Fri, 19 Jan 2001 10:55:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0JFsl100380 for ; Fri, 19 Jan 2001 10:54:47 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAA57150 for ; Fri, 19 Jan 2001 16:54:35 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA146934 for ; Fri, 19 Jan 2001 16:54:35 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D9.0057632C ; Fri, 19 Jan 2001 16:54:30 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 19 Jan 2001 17:13:45 +0200 Subject: Re: iscsi : Fragmentation & Reassembly issues in iSCSI. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk answers in text - Julo Santosh Rao on 19/01/2001 02:47:05 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iscsi : Fragmentation & Reassembly issues in iSCSI. Julian, Issue : ===== DataPDULength, PingMaxReplyLength, Fragmentation/Reassembly issues. Problem : ======= 1) What is the relationship b/n DataPDULength and PingMaxReplyLength ? If a simple iSCSI test program decided to use the NOP-OUT as a form of basic testing and began issuing increasing sizes of NOP-OUT, at some point, it will cross the DataPDULength PDU size (provided it was able to negotiate PingMaxReplyLength > DataPDULength, which such a test program would attempt, in the interests of large I/O testing). I will introduce in 04 a statement saying that that PingMaxRelayLength will be limited by the lowest of the two The DataPDULength introduces fragmentation and reassembly issues at the iSCSI layer for the NOP-OUT, NOP-IN and Login commands. (For the SCSI Data PDUs, reassembly is possible based on the Buffer Offset contained in the data PDU header.) Question : ======== What drives the fundamental requirement for the DataPDULength ? I see no benefit for this other than trying to ensure a small data payload in order to have an effective CRC. Are there any other benefits ? data multiplexing on a connexion is the other reason. Unsolicited data are the only case you are not allowed to multiplex in order to avoid deadlocks. A good resync mechanism, whenever will become available can also operate effectively only on limited sizes unless you use a full-fledged RDMA scheme. Everywhere those reasons are no relevant you can set the maximum sizes to you memory available. And the maximum ULP PDU is not the TCP MTU. Is there a need to impose an upper bound (MTU) on the size of PDUs other than the SCSI Command PDU (containing immediate data) and SCSI Data PDU ? Solution : ======= Specify explicitly in Appendix C that DataPDULength is only applicable for SCSI Command PDU and SCSI Data PDU. The current definition is open to being mis-interpreted as a form of iSCSI MTU, something that can result in the iSCSI layer having to deal with fragmentation and re-assembly issues for non-SCSI PDUs. What would be those? Text commands? Login? Nops? Having a single limit seemms simpler Regards, Santosh - santoshr.vcf From owner-ips@ece.cmu.edu Fri Jan 19 13:14:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA03653 for ; Fri, 19 Jan 2001 13:14:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0JFt9600407 for ips-outgoing; Fri, 19 Jan 2001 10:55:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0JFsU100371 for ; Fri, 19 Jan 2001 10:54:31 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAA67834 for ; Fri, 19 Jan 2001 16:54:24 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA273138 for ; Fri, 19 Jan 2001 16:54:24 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D9.00575E3B ; Fri, 19 Jan 2001 16:54:17 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 19 Jan 2001 15:46:50 +0200 Subject: Re: iSCSI : Reject PDU Concerns. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, my answers in text Julo Santosh Rao on 18/01/2001 23:53:38 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iSCSI : Reject PDU Concerns. Ref : Section 2.19 (Reject PDU), Section 5.5 (Digest Errors) Issue : ===== 1) Reject PDU must return an Initiator Task Tag in the Reject PDU header, which provides the initiator sufficient context to lookup the outbound exchange that was transmitted. reject PDU can have a multitude of causes and those are/will be detailed to certain level (not too much. Whenever the Initiator Task Tag is available and relevant it can be read from the returned information. However I am still looking to what it would take to get it back into the reject PDU itself 2) An error_byte field should be considered for inclusion in the Reject PDU which contains the byte offset of the errant byte[or word] in the received header/payload that caused the target to send a Reject response to the command. This is a simple solution that provides for quick fault isolation and root cause of the reason for the reject. This also rids the Reject PDU of any detailed elaboration of reason codes meant to allow root cause of the reason for the reject. that is no big help since there can be more than one error and it helps mostly in debugging. For implementation errors in the target and/or initiator I am sure the protocol analyzer will be happy to help but running implementation should not be burdened with. For those that are running errors on good implementations we will provide some details when necessary 3) The Reject PDU should only be used for " Format Error" since the Reject mechanism is a per-task error recovery to deal with header format problems. "Digest Error" handling is too complex dealing with too many situations. Digest error recovery should be simplified to a connection level error recovery which involves Logout or connection close/re-start. The current digest recovery is a ping-pong as follows : - Target sends REJECT - Initiator on seeing REJECT with "Digest Error" sends Logout - Target sends Logout response. (the spec is not clear on who drives the connection close. does the initiator close the connection on receiving logout response, or does the target close connection and send logout reponse on another connection ??). The above procedure is quite convoluted and the followinf simplification should be considered : - On a digest error, target sends Logout, indicating appropriate reason code. - Target follows up the Logout with a TCP connection close. No. Command Response flow follows a regular client-server pattern and I am not willing to consider all the consequences of transforming it into a "peer to peer" 4) There is no value add in expecting a logout response on a logout which is intended to close a connection and is sent on the same connection. The logout can be followed up with a connection close. That can be done at extreme anyhow. Logout is a late addition to this protocol mainly to clean-up a failing connection before recovery. Logout response is there for symmetry and convenience. Even if the target follows logout with a connection close - the response can get to the initiator and recovery may proceed faster that in a complete failure in which it would have to wait for a timeout Regards, Santosh From owner-ips@ece.cmu.edu Fri Jan 19 13:15:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA03681 for ; Fri, 19 Jan 2001 13:14:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0JFv4900497 for ips-outgoing; Fri, 19 Jan 2001 10:57:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0JFu0100453 for ; Fri, 19 Jan 2001 10:56:00 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA29194 for ; Fri, 19 Jan 2001 16:55:51 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA146936 for ; Fri, 19 Jan 2001 16:54:35 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569D9.005762E1 ; Fri, 19 Jan 2001 16:54:29 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 19 Jan 2001 16:52:06 +0200 Subject: RE: NOP-Out clarification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Ashish, TCP already guarantees data delivery. Data Numbering was there to enable long running input operation to recover faster (repeating only unacked data after a restart). And many members of this list (not all) felt it was adding to much complexity for what it bought. As for scoreboarding in all SCSI protocols the target is supposed to do its job and the delivery subsystem to deliver. The initiator is not supposed to do scoreboarding (it can do for completeness but it is not required to) and has to report exceptions (including counters for overrun/underrun) as they where conveyed by the target. Julo "Choubal, Ashish" on 19/01/2001 02:07:46 Please respond to "Choubal, Ashish" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: NOP-Out clarification Hi Julian, Scraping Data Numbering will affect how single-initiator-command - multiple data response PDU case is handled. My understanding was that the data numbering mechanism allowed the initiator not to do any score-boarding and put the burden on the target to guarantee that all data requested by initiator was delivered - i.e. target numbers the each of response data PDU and initiator acknowledges it received the PDU via a NOP. When target receives acknowledgements for all the DataRN's it sent, then it sends a status info to complete the command. What am I missing? -Ashish Choubal -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Thursday, January 18, 2001 3:51 PM To: ips@ece.cmu.edu Subject: Re: NOP-Out clarification JP, The consesus at the intermediate meeting in Orlando was to drop Data Numbering. If you have strong object here is your last chance to object. Also Data Numbering was meant only for incoming data and it is per command ("less than a task" if you take in account linked commands). Regards, Julo Please respond to Raghavendra Rao To: ips@ece.cmu.edu cc: Subject: NOP-Out clarification My reading of NOP-out as used for data-in acknowledgements is that it is for the entire task - Not per R2T since I don't see offset/length in the NOP-out header. Is my assumption correct ? It would be nice if the draft can be more explicit. Thanks. -JP From owner-ips@ece.cmu.edu Fri Jan 19 15:24:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA05852 for ; Fri, 19 Jan 2001 15:24:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0JHiCO04477 for ips-outgoing; Fri, 19 Jan 2001 12:44:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ludwig.troikanetworks.com (ludwig.troikanetworks.com [12.31.172.3]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0JHhD104430 for ; Fri, 19 Jan 2001 12:43:13 -0500 (EST) Received: by host03.troikanetworks.com with Internet Mail Service (5.5.2653.19) id ; Fri, 19 Jan 2001 09:43:33 -0800 Message-ID: From: Wayland Jeong To: "'Santosh Rao'" , Sriram Rupanagunta Cc: ips@ece.cmu.edu Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Fri, 19 Jan 2001 09:43:27 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I can't speak for the authors of FCIP, but I will venture to guess, the comment that was made regarding FCIP's ability to open as many connections as desired between FCIP devices, was only meant to imply that an implementation could solve the ordering problem. As you note, an implementation which identified source/destination flows could bind these flows to a single TCP connection and adhere to the sequential delivery requirement of FC fabrics. Similarly, an implementation that allowed multiple TCP connections per src/dest flows could ensure ordering by inserting sequence numbers in the FCIP encapsulation and re-ordering at the destination bridge. In fact, at the interim WG meeting in Orlando, Brocade had a proposal for a new FCIP encapsulation that seemed to include such a sequence number (albeit for a UDP implementation). Now, I'm not saying this would be fun to implement, but the possibility exists. With regards to iFCP, it does ensure ordered delivery. It binds a FC login session to a single TCP connection. My 2 cents anyway. -----Original Message----- From: Santosh Rao [mailto:santoshr@cup.hp.com] Sent: Thursday, January 18, 2001 6:19 PM To: Sriram Rupanagunta Cc: ips@ece.cmu.edu Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Sriram Rupanagunta wrote: > > > Can this assurance be made even while using multiple TCP > > > connections b/n > > > edge switch/routers ? > > This question is N/A for FCIP, since only a single connection > > is specified between FCIP gateways. In iFCP, a single TCP connection > > Sometime back, FCIP folks mentioned that there is nothing in the > FCIP spec that limits the number of connections to one. My > understanding is FCIP leaves this to implementations, whether > to use one connection or multiple. The above is EXACTLY the reason I have raised this question. I hear conflicting statements being made on this issue. I don't see how FCIP can use multiple TCP connections per (S_ID, D_ID) pair and guarantee in-order delivery without implementing buffering and re-ordering on the FC-IP switch. Both FCIP and iFCP MUST guarantee in-order delivery since all FC end ports (N/NL_Ports) depend on this feature and FC-FLA Table 3 mandates this. Regards, Santosh > > Can someone clarify this once for all on this list please ? > > If there is no limit on the number of connections in FCIP, > Santosh's question is valid. In that case, is it fair to > say that FCIP does not guarentee orderly delivery ? > > - Sriram From owner-ips@ece.cmu.edu Fri Jan 19 15:27:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA05903 for ; Fri, 19 Jan 2001 15:27:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0JHr6H04831 for ips-outgoing; Fri, 19 Jan 2001 12:53:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0JHqU104807 for ; Fri, 19 Jan 2001 12:52:30 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id DG4KP4B2; Fri, 19 Jan 2001 09:52:29 -0800 From: "Y P Cheng" To: "John Hufferd" Cc: "IPS Reflector" Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Fri, 19 Jan 2001 09:51:11 -0800 Message-ID: <002401c08240$69231ac0$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > YP, > Not sure where you are going here, the purpose of a standard is to permit > interconnection with other manufactures equipment. > > . > John L. Hufferd John, I guess that I failed miserably in communicating my thinking to this WG. Most people if not all in this WG view iSCSI, iFCP, and FCIP as protocols to be implemented by either hardware or software or the combination of. Because their implementations could be software that takes advantage of existing TCP stacks in different OS's that are proven working, this WG tries real hard to fit the new protocols into the existing implementations. In the world where I live, iSCSI, iFCP, and FCIP will be implemented in a box or an adapter running RTOS or microcode with fresh new implementations. While it is essential to intemperate with the world that runs the existing TCP implementations, nothing prohibits the box and adapter to interoperate with each other running in "fast mode" in correct TCP packets as long as they obey the Internet fairness rule without creating so called "Super TCP". In my adapter, I don't have to live with any old TCP implementations. I asked often how do we streaming data on a 10 Gb/sec network with roundtrip time over 100 milliseconds? I would like to hear discussions providing answers to the above question. The statement "the TCP implementation guarantees in-order delivery and retries lost packets and has the necessary flow control and congestion avoidance" does not answer the question for me. If everyone agrees that this group can put iSCSI, iFCP, and FCIP together by assuming the current TCP implementations having all the solutions, please let me know. Y.P. > To: "Matt Wakeley" , "IPS Reflector" > > cc: > Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports > > > > > > But, if iFCP and FCIP use TCP encapsulation, then, we back to > > > square zero.) > > Please explain why/how you think iFCP and FCIP do not require the > > use of TCP? > > > > -Matt Wakeley > > Matt, > > Certainly, both drafts have proposed TCP. > > However, when two iFCP or two FCIP devices talk to another of its > own kind, > they have the freedom to use any protocol they wish, even SCTP, as long as > the protocol provides reliable delivery. This is because the frames are > only received by another device of same implementation. There is no > interoperability issue with any legacy implementations. Now, does the > other > protocol obeys the fairness rule of Internet? I assume it would. > > Y.P. > > From owner-ips@ece.cmu.edu Fri Jan 19 19:26:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA08430 for ; Fri, 19 Jan 2001 19:26:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0JLwAo14275 for ips-outgoing; Fri, 19 Jan 2001 16:58:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0JLvp114266 for ; Fri, 19 Jan 2001 16:57:52 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 5E04FB6B; Fri, 19 Jan 2001 13:57:46 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id NAA06739; Fri, 19 Jan 2001 13:57:41 -0800 (PST) Message-ID: <3A68B8D5.2C9994FC@cup.hp.com> Date: Fri, 19 Jan 2001 13:59:49 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: Re: iscsi : Fragmentation & Reassembly issues in iSCSI. References: Content-Type: multipart/mixed; boundary="------------92ADF59665BFFF03E71ACEF0" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------92ADF59665BFFF03E71ACEF0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > I will introduce in 04 a statement saying that that PingMaxRelayLength > will be limited by the lowest of the two Julian, The fragmentation issue as explained in the above example is also applicable for the Login and Text Commands. (TotalText > DataPDULength). In the case of these commands, a "F" bit is required in bit 7 of byte 1 of the command & response PDUs to indicate the last command or response PDU due to the fragmentation issues that arise from TotalText & DataPDULength. > Solution : > ======= > Specify explicitly in Appendix C that DataPDULength is only applicable > for SCSI Command PDU and SCSI Data PDU. The current definition is open > to being mis-interpreted as a form of iSCSI MTU, something that can > result in the iSCSI layer having to deal with fragmentation and > re-assembly issues for non-SCSI PDUs. > > What would be those? Text commands? Login? Nops? Having a single limit > seemms simpler On the lines of your above solution, (based on a single limit), removal of PingMaxReplyLength should be considered and implicitly use DataPDULength as the PingMaxReplyLength. This ensures only 1 limit for NOP-OUT/NOP-IN. Regards, Santosh --------------92ADF59665BFFF03E71ACEF0 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------92ADF59665BFFF03E71ACEF0-- From owner-ips@ece.cmu.edu Fri Jan 19 20:41:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA08934 for ; Fri, 19 Jan 2001 20:41:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0JNoCp22069 for ips-outgoing; Fri, 19 Jan 2001 18:50:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0JNnL122045 for ; Fri, 19 Jan 2001 18:49:21 -0500 (EST) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id PAA11054 for ; Fri, 19 Jan 2001 15:49:20 -0800 (PST) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.152.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id PAA20574 for ; Fri, 19 Jan 2001 15:49:19 -0800 (PST) Received: from ebay.sun.com by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id PAA29563; Fri, 19 Jan 2001 15:49:19 -0800 (PST) Message-ID: <3A68BB57.11B1356C@ebay.sun.com> Date: Fri, 19 Jan 2001 14:10:31 -0800 From: David Robinson Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.75 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports References: <002401c08240$69231ac0$90c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Y P Cheng wrote: > In the world where I live, iSCSI, iFCP, and FCIP will be implemented in a > box or an adapter running RTOS or microcode with fresh new implementations. > While it is essential to intemperate with the world that runs the existing > TCP implementations, nothing prohibits the box and adapter to interoperate > with each other running in "fast mode" in correct TCP packets as long as > they obey the Internet fairness rule without creating so called "Super TCP". > In my adapter, I don't have to live with any old TCP implementations. I > asked often how do we streaming data on a 10 Gb/sec network with roundtrip > time over 100 milliseconds? I would like to hear discussions providing > answers to the above question. The statement "the TCP implementation > guarantees in-order delivery and retries lost packets and has the necessary > flow control and congestion avoidance" does not answer the question for me. In general I have not seen people in this WG constraining the design based on TCP implementations, in fact some have been very abstract in their comments and referring to what has been proven in theory (if not limited test implementations) that does not reflect current widely deployed implementations. If I read you correctly, you are asserting that there is a fundemental problem in the design on the TCP *protocol* which prevents it from taking advantage of a 10G/100ms network. If so what exactly do you see as a problem? Since we cannot (ips WG) cannot change TCP how should an IPS protocol work around this problem while still being friendly with other protocols? > If everyone agrees that this group can put iSCSI, iFCP, and FCIP together by > assuming the current TCP implementations having all the solutions, please > let me know. Conversely, if you feel that this group is designing to the TCP implementations instead of the protocol, please let us know. -David From owner-ips@ece.cmu.edu Fri Jan 19 20:44:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA08966 for ; Fri, 19 Jan 2001 20:44:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0JNWBH21587 for ips-outgoing; Fri, 19 Jan 2001 18:32:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0JNW0121581 for ; Fri, 19 Jan 2001 18:32:00 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 19 Jan 2001 18:31:53 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A070410145B@corpmx9.isus.emc.com> From: Black_David@emc.com To: minutes@ietf.org Cc: ips@ece.cmu.edu Subject: IP Storage (ips) San Diego Minutes Date: Fri, 19 Jan 2001 18:31:49 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk IETF IP Storage (ips) Working Group Meeting Minutes San Diego IETF Meeting, December 11-12, 2000 ---------- Monday December 11, 2000 EMC will be sending out an IPR notice regarding a patent related to iSCSI and FCIP to the mailing list. Interim meeting being scheduled for week of January 15, to coincide with T10 in Orlando - Grosvenor resort. -- Framework document - Mark Carlson (Sun) Describes environments for IP Storage. Includes terms, background on various protocols. This is a living document. Currently more of a survey. This document will coordinate with Naming and Discovery. Looking for more co-authors, please contact Mark if you are interested. -- Framing discussion -- Randy Haagens (HP) and Allyn Romanow (Cisco) - Allyn and Randy were asked to compose this presentation by the ADs. Purpose was to try to clarify the problem and present a range of solutions. - Framing is a common challenge with for both iSCSI, FCIP as well as non IPS documents. While framing is not explicitly required, a solution for a more effective iSCSI specification is highly desirable. The focus of the presentation was understanding the requirements of framing (i.e. the problem). Reaching consensus on a solution was not one the goals of the presentation. Allyn started the presentation by pointing out that this topic will also be discussed on Monday night in the TSVWG. - The problem: TCP reassembly can be costly, and in some instances not feasible. Also, there is limited host memory and host bus bandwidth, so one wants to avoid manipulating the data more than once. Best would be one use of the bus and memory - zero copy. Note: This is not the same as TCP zero copy. TCP typically waits for all the data to arrive, and then copies the data to host. - In outbound direction, data can be transferred directly from memory to the protocol controller and out onto the wire. In the inbound direction, when received out of order, data has to be put in a reassembly buffer until all data is received. - One solution: Direct Memory Placement (Payload steering; data steering; RDMA) -- In order to conserve host memory bandwidth, CPU cycles and reduce on-board memory requirements, it is desirable to deliver iSCSI data directly to host buffers, avoiding the overhead of TCP reassembly buffers. The TCP reassembly buffer can be 250MB for a 10Gbps link with 200ms round-trip time. At 1Gbps, reassembly is possible but very costly. But at 10Gbps speeds or above, reassembly is no longer feasible. So, the goal is to get rid of a separate TCP reassembly buffer. Can decode ULP (iSCSI) headers and place payload directly in host memory without intermediate buffers. This would not be a conventional NIC card; instead it would be very iSCSI aware, but it would not necessarily process the iSCSI headers, but just use them to determine where to place the data. As in TCP, the iSCSI stream is presented to the iSCSI protocol processor in-order. - In this solution, must address loss of ULP sync - when a segment containing a ULP header is dropped or delayed, ULP sync is lost. Direct data placement cannot continue; data must be diverted to a reassembly buffer. Goal is to recover ULP sync at the next ULP header. There are both TCP aware and TCP unaware solutions to recovering ULP sync. - TCP unaware approaches: a) SCTP - issues with this include lack of widespread deployment b) Special Characters - requires byte by bytes processing c) Fixed length ULP messages - Inefficient for short ULP messages d) Periodic Marker - Best solution for this class of approaches Sublayer of a framing protocol. Manageable; relatively easy to implement in hardware Marker 4 byte field number of ULP bytes remaining in current PDU. Marker inserted and removed by framing protocol; e.g. iSCSI. After loss of sync, locate next marker; use to locate the next ULP PDU. Markers are transmitted twice in a row; ensures markers cannot be split by stream fragmentation/segmentation. - TCP aware Approaches a) URGent pointer - disallowed b) PSH bit - disallowed - Another TCP aware approach can be considered by the TSV working group. The TSV working group works on small items in the transport area that do not need a full working group as well as TCP/UDP transport issues. - Allyn Romanow presented a technique for demarcating message boundaries using a TCP option. This consists of using one of the reserved bits in the TCP header to extend TCP to support this type of framing. Then can add up to 40 bytes before the TCP payload. Problem is that these reserved bits are a scarce resource; need to evaluate the need for the change. Also any time a change to TCP is proposed, there is tension, e.g. tension between the need to update TCP and stability of TCP. - Procedure for standardizing a TCP option consists of a) The IESG has to approve new work items for the TSV wg. b) Ask the Transport Services (TSV) working group to adopt this as a WG item c) Pros-and cons will be discussed on the TSV wg mailing list. If it supported, hopefully the spec will be wrapped at the next IETF (roughly 3 month time frame). If no support, it's dead. The advantage of the TSV wg is that transport experts will be able to contribute feedback. d) If supported, will be adopted at next IETF meeting. Advantage is that people who are experts in transport will be able to contribute, and that this will not be an iSCSI specific solution. IPS should follow this process and contribute. Make sure that the solution (since not iSCSI specific) meets the needs of this group. This is a very common problem, that is worthy of consideration at the transport layer. Addresses areas beyond IPS. The TCP option is not the only approach. TCP header bits could potentially be used for framing. The flag approach may send many packets that are less than MSS. This is potentially a risky change to TCP. Message Boundary Option Two approaches. Not in drafts yet: - Flag approach -- Costa has written up; will post as draft. The flag approach may send many packets that are less than MSS. This is potentially a risky change to TCP. ULP header is aligned with first byte of TCP payload. - Offset Approach -- 4 bytes. 2 byte offset indicates offset into TCP payload of first ULP header in the segment. Write-up forthcoming. Discussion - Lead by Steve Bellovin Steve Requested the group concentrate on Requirements. The discussion raised the following points: - Another option for alignment - periodic alignment instead of periodic marker. There could be a requirement in iSCSI that an upper-layer header appear every n kbytes in the TCP stream. Padding could be used to make sure this happens. - This is not the first time that this issue has arisen, and there is value in a general solution that is applicable to other protocols, even though this may take longer to deploy. The consensus in the room was that a general approach is preferable to one specific to iSCSI. - Multiple message boundaries in a single TCP segment are not a problem. Once the first boundary is found, the rest are found by examining the iSCSI (or other ULP) headers. - If there is a large gap between message boundaries, the data in the gap will need buffering. Implementations may wish to consider this in setting maximum data size for a PDU. - RDMA is different but related to this topic. Any RDMA protocol will either incorporate or assume framing. It may make sense to spec a generalized RDMA protocol on top of this framing mechanism. - Implementation of this sort of framing would be optional. - A generic data framing protocol may also be a good place to put in a stronger CRC than the 16-bit Internet checksum. Drafts making specific proposals are welcome. Steve Bellovin asked for a hum of the room on whether to solve the "framing problem" in an iSCSI-specific way or whether to pursue a mechanism to add to TCP. The hum in the room was to do it in TCP. -- ISCSI document review - presented by Julian Satran. - Rough consensus has been reached on the session model - Symmetric with optional multiple connections. - Login Session context - good understanding. - Login Security context - more work needed. - Commands, messages, tasks, and tags almost complete. Items open - coding, some layout. - Response numbering scheme is well understood; complete. - The data numbering scheme has received no consensus. It may be removed. Julian's personal opinion is that it's optional and low cost with advantages. - For recovery, command restart and status well understood. No consensus on data recovery. Digest not well understood; needs to be readdressed. - Text commands - negotiation mechanisms done. - Mapping moved to T10 (aliasing). Dropped from iSCSI. - RDMA/Sync, Security/Authentication - all are still open issues. - Authentication - login phase must provide authentication. This was the consensus at the last meeting. Every iSCSI PDU must provide data integrity and authentication. - A mechanism should enable optional end2end data protection/authentication. Would like to use TCP recovery in presence of error. Digests can be activated at a higher level. Need a mechanism that can be activated on demand, ideally at login. - The current digest scheme needs to be changed. Julian suggested using IPSec for data integrity, since all the above mechanisms are provided by IPSec, it is a best fit for what is needed and very cheap if use only what is needed. Can insert own policies, including policies that will verify integrity verses provide security but use same mechanisms. Policies will be addressed in next two weeks. - David: IPSec does negotiation securely. What is currently in the draft is most likely vulnerable to man-in-the-middle attack. - Steve Bellovin indicated that the IPSec WG would be extremely opposed to any insecure non-cryptographic algorithm being defined for IPSec. Silicon must support SHA-1 or MD5 in order to do key negotiation. There are active discussions/proposals on how to do high speed encryption/negotiation. Early in process; drafts not yet standards, but worth looking at this. - Mark Bakke really wants to maintain the separate iSCSI header/iSCSI payload digests. This separation is lost by moving to IPSec. Gained data integrity is only as good as the group is willing to pay. Good integration with encryption. - Can use IPSec in transport mode, which will provide end2end protection. Integrity is required end2end, but security may not be. Security may need to be removed at the firewall/gateway, but need to still be able to verify integrity at the endpoints. Can have multiple layers of IPSec if needed. Comment from audience - not recommended. - David Peterson of Cisco asked whether ACA will be mandated by the draft. The consensus, after the discussion, is that iSCSI must support ACA but that a device need not support ACA (Ralph Weber pointed out that few initiator use ACA today). There was some grumbling because ACA is needed for reliable pipelining of ordered commands in the face of errors. - There was a question on whether asynchronous event notification (AEN) was mandatory to implement in iSCSI. Again, iSCSI transports must support asynchronous events but iSCSI devices need not. Somebody pointed out that SCSI mode pages can be used to regulate whether a device generate AENs. - Ralph Weber (T10 secretary) praised iSCSI for trying to advance the state of the art in SCSI. -- iSCSI requirements --- presented by Marjorie Krueger (HP) T10 work on authorization will not be integrated into iSCSI; to the extent that SCSI provides authorization, that's T10's domain. The fact that IP networks are less secure than typical SCSI environments have been in the past introduces additional issues that need to be addressed here in iSCSI. T10 work will be used and referenced where applicable. It was noted that the point of iSCSI authentication and authorization was to control who was able to get to a target. -- Bootstrapping -- presented by Prasenjit Sarkar (IBM) This document contains guidelines for how iSCSI boot clients connect to iSCSI boot server. Included description of how to use existing techniques. iSCSI boot clients need IP address, iSCSI boot server service delivery port name, default; LUN = 0; iSCSI initiator software. Boot process steps: Client software stage Use PXE or related bootp/tftp protocol to get iSCSI initiator software DHCP stage Use DHCP to configure client IP address Use new DHCP option to configure iSCSI boot server service delivery port name Discovery server stage Use "to be defined" iSCSI delivery service to get iSCSI There was a question on whether the boot client had to have IPsec, in light of the integrity proposal by Julian and security proposals by others. It is not required; bootp is sufficient. The absence of security requirements for boot was pointed out. The current goal of the boot document is to remain neutral on security (neither mandate nor disallow). There was some question on what to do with the iSCSI session once a bootstrap program was done with it. It was noted that it was probably simplest to close it and have the loaded program establish a new iSCSI session, but this is up to implementations. -- MIB presentation - Mark Bakke (Cisco) A group is forming to work on iSCSI MIB. The scope is management of iSCSI as opposed to SCSI. If necessary, a separate SCSI MIB (if one does not already exist) would be addressed separately. The original MIB structure in the current draft is not adequate, and is being redone. These revisions will also bring the MIB up to date with the current iSCSI draft. A question was raised about how FC-style zoning works with the MIB. It's not clear how zoning fits into the iSCSI architecture. The MIB could be implemented on anything running iSCSI including initiator, target, and gateway. The FC HBA API available from SNIA might be of interest to this group. It has a complete list of things management tools want to be able to see out of an initiator. ----- Tuesday, December 12, 2000 -- Naming and Discovery Requirements - Mark Bakke (Cisco) Naming and discovery will specify target discovery but would leave LUN discovery to SCSI mechanisms, such as REPORT LUNs. There was a bit of debate on this; why not go all the way and support LUN discovery in the naming system? The counter-argument is based on layering: "Leave unto SCSI that which is SCSI's". Scaling requirements include both small and large environments. Find targets by querying SNS. Small environments do not require SNS. Hierarchical format, with Naming Authority. World Wide Unique Identifier Address composed of IP addr+TCP port+Target Name, URL like. Plan to apply for well known port for TCP. In such a case, an address w/o TCP specified would default to this well known port. Format includes info on naming authority, including support for 'local' naming authority. Character set to be allowed? Unicode? Recommend UI schemes for naming authority. Need to look at security issues. T10 issues - reservations, reset, LUN naming. Target reset discussion. Noted that T10 is thinking of making target reset optional. Is breaking of a connection in iSCSI equivalent to a target reset? Consensus is no: the end of a session was equivalent to a target reset and would also cause any persistent reservations to be released. Naming scheme will allow multiple port and multiple initiator/target discovery. Will give list of targets + all paths to that target. Draft currently an individual submission - consensus (hum) taken, to be adopted as working group document. No opposition hums. -- iSNS document presented by Josh Tseng, Nishan ISNS describes a scalable information facility for registration, discovery and management of networked facilities. ISNS follows a client/server architecture. If client registers with name server, allows itself to be managed by the name server. Why needed? Simplifies storage management implementations. Allows greater scalability over broadcast/multicast discovery methods. Supports zoning. Next step - incorporate requirements/suggestions from IPS working group. Extend document for FCIP Access control - what is name server role? Targets upload public key to name server. Enforced at the end node/target. Supports both soft and hard zoning. How does it fit into discovery? Naming and discovery team will look at this to see how well it fits. Should this be maintained as a separate document vs incorporated into naming/discovery team? Yes, this is a separate document because it supports more than just iSCSI. In reading the draft, reliance on WWN. This draft would need to be redone to support WWUI of n&d requirements. Direction is one in which naming and discovery team approves of? Yes, close. Is there working group consensus as a base document; working w/ NDT group to produce a revised document, aligned with N&D, which would then be adopted as an official wg document. Rough consensus - next revised version of document will become an official working group document. Not unanimous. -- FCIP - Status and progress of FCIP. - Raj Bhagwat (LightSand) Current status - difference from previous presentation Solution for bridging remote FC SAN islands. From FC point of view, appears to be entirely an FC network. Initially did not have congestion management (previous presentation). Draft overhauled to incorporate TCP as transport in order to address congestion management and recovery mechanisms. In rev -00, PSH flag incorporated. Based on feedback from mailing list, this was eliminated and in -01, a new frame boundary mechanism introduced. Topics under discussion -- QOS, security, MTU/MSS, Framing/synchronization, order of delivery, discovery, error recovery. Alignment with new project in T11 - FC-BB2. FC-BB2 focused on issues outside the scope of the IETF, including link level issues. Target date for completion - June 2001. Much FC/IP work is being done on conference calls. Conference calls are design team calls open to design team members and authors. Public review is on the mailing list. An FCIP device is a gateway between an FC SAN and IP network. Discovery of FCIP gateway (device) and other FCIP gateways is currently via static configuration. Dynamic configuration support is envisioned, perhaps using iSNS. David Black will work with authors on revising the QoS text. -- iFCP - presented by Charles Monia, Nishan What is the difference between iFCP and FCIP? - FCIP is a tunneling model between FC SANs. A conduit for FC frames to flow transparently to FC network over IP backbone. - iFCP network model extends up to the FC storage device itself. Uses a session model. Consolidates FC storage switching and routing functions in the IP fabric. Reduces total cost of ownership, unifies network and storage management domains and exploits IP technology investment. Extend SAN over lan/man/wan distances. Next step -- complete the n_port session model. Encapsulation changes for additional end-to-end error detection. The authors of iFCP would like to see it considered for adoption as a work group item. Adoption of iFCP as a work group item requires modification to the WG charter. David Black requested input on this be set to the WG chairs. Revising of the charter requires consultation of the area directors and working group chairs. After the presentation, Suggestions were made that iFCP and FC/IP should merge since they are so similar. It was pointed out that the two protocols take different approaches. iFCP works by intercepting FC logins (connection requests) and modifying FC frames. In addition, it doesn't run FC routing protocols between FC SANs. Clarification of FCIP and iFCP - the latter is for FCP protocol mapping only, whereas FCIP can transport any FC upper level protocol. FC/IP works at a lower level than iFCP. It doesn't modify FC frames. FC/IP requires running FC routing/switching protocols between FC domains. Some thought that iFCP was a superset of FC/IP. There was a concern that the iFCP gateway would need to run IP routing protocols. It was eventually decided the iFCP gateway was just an IP host and didn't have to run IP routing protocols. Other comments need to be sent to mailing list or chairs directly. -- Adaptation Layer presentation -- Randall Stewart, Cisco Randall Stewart's presentation introduced how the IPS protocols could be architected with an adaptation layer independent of the underlying transport (i.e. at least both SCTP and TCP). To do this, a uniform API boundary between the ULP and transport would need to be defined. This would require many changes to all existing drafts. APIs would need to be a message oriented type of mechanism. Critical path would need to be done so that they would be protocol agnostic. Transport interface would need to provide methods for passing buffers to/from control of transport, e.g. for zero copy. Adaptation layer would need to worry about Framing Zero copy Parallel paths Message retrieval Notifications Must be very careful that this API would not make assumptions about the transport being used. In adaptation model, would need to figure out how to overcome the issues. Randall would be more than glad to help by contributing both advice and/or drafts to bring about this sort of adaptation layer. A concern was expressed that the adaptation layer would add too many layers between iSCSI and TCP and that separate protocol should be done for SCTP. It was suggested that the CAM may be an inspiration for the adaptation layer. Others responded that the CAM is at the wrong layer, above iSCSI. From owner-ips@ece.cmu.edu Fri Jan 19 20:44:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA08989 for ; Fri, 19 Jan 2001 20:44:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0K0UC923104 for ips-outgoing; Fri, 19 Jan 2001 19:30:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0K0Th123088 for ; Fri, 19 Jan 2001 19:29:44 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 19 Jan 2001 19:29:38 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A070410145E@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: iSCSI-related conclusions from Orlando interim Meeting Date: Fri, 19 Jan 2001 19:29:36 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a summary of the conclusions reached in the iSCSI area at the interim meeting in Orlando earlier this week. FCIP-related conclusions will follow under separate cover. These conclusions are stated as preliminary consensus of those in the room in Orlando and are subject to further discussion and revision on this mailing list. (1) Framing - The iSCSI draft will be revised to add a formal iSCSI interface to framing - This will support the current marker approach, SCTP, and SCTP-like chunking for TCP. - The existing description of markers will be moved to an appendix as an example. -The WARP group working on RDMA will write a draft on using SCTP-like chunking for iSCSI without RDMA (2) Error Recovery - Terminology Changes - "Reference Number" and "RN" are used only for SCSI -iSCSI uses "Sequence Number" and "SN" instead - "AER" is used only for SCSI - iSCSI communication of asynchronous events is through a mechanism that is now called "Asynchronous Messages" - iSCSI uses these to implement AER - If a SCSI initiator has disabled AER, iSCSI does not send the corresponding Asynchronous Messages - CmdSN (formerly CmdRN) is mandatory in all cases to simplify things. - This includes single connection sessions - SCSI has defined a new (optional) SCSI Command Ref Number - iSCSI will use byte #2 in iSCSI header (currently Reserved) to transport this - This is not the ideal solution, but matches the level of support in FCP. - We have to check where mode page is to negotiate this - If it's on a transport-specific mode page, iSCSI will use a text key to negotiate this instead - DataSN (formerly DataRN) functionality will be removed. Error recovery is now at the granularity of commands, not within a command. - There will be a significant connection recovery write-up, including details, procedures and examples added to the draft. - Ping/NOP issues - A description of intended usage will be added to the draft as advice to implementers. - A ping is intended to check whether the corresponding protocol is alive - NOP responses are not permitted to request responses to avoid loops - Abort WARNING will be added to the draft. - Immediate Delivery of Aborts and similar task management commands may have unexpected results when multiple TCP connections are in use because Abort, Clear Task Set, etc. may bypass command(s) to be aborted/cleared on other TCP connections. - Ordered Delivery should be used instead when this is a concern. (3) CRC - iSCSI will use separate CRCs on header and data to make header modification easier for systems that need to do that. - Using the same CRC algorithm on header and data is simpler - in particular the idea of using a 16-bit CRC on the header was discarded. - One CRC should cover both the fixed header and optional extensions - Sense of the room on CRC Algorithms - CRC-32 is the obvious first/default choice. - There is some interest in investigating both weaker (Adler-32) and stronger (CRC-64) CRCs (CRC-64 may not be appropriate for header, and is still a research topic). - CRCs are MUST implement - Open issue: whether CRC use is negotiable - Default: Use CRCs - Limiting the amount of data per CRC - Probability of undetected error rises with amount of data covered by one CRC. - There will be at most one data CRC per PDU - This limit SHOULD be enforced limit by fragmenting large data blocks into Multiple Data PDUs - Julian Satran will look for the reference that suggests CRC-32 should not be used on data blocks significantly larger than 8-10k. (4) Security - Current iSCSI security digests will be removed in favor of IPsec and/or TLS - Only reason for digests is data integrity (i.e., CRCs) - Open issue: How does iSCSI negotiate or detect presence of lower level security? - Open issue: What is minimum security required to be used (authentication/integrity) by IETF? David Black - SRP and Kerberos login authentication will remain in the iSCSI draft pending resolution of this issue. (5) Naming - UTF-8 will be used instead of ASCII for text strings in iSCSI login and text commands, naming, discovery, etc. - Binary values will be encoded in UTF-8 rather than adding type/length support. - Localization of iSCSI text is forbidden on the wire for interoperability reasons (e.g., keys/values in login and text commands are the same byte strings, no matter what the locale). Ok, comment away ... Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Jan 19 20:44:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA09000 for ; Fri, 19 Jan 2001 20:44:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0K0GCi22745 for ips-outgoing; Fri, 19 Jan 2001 19:16:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0K0Fa122733 for ; Fri, 19 Jan 2001 19:15:36 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id A71E63BF for ; Fri, 19 Jan 2001 17:15:35 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 0FB94155 for ; Fri, 19 Jan 2001 19:15:34 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id QAA12501 for ; Fri, 19 Jan 2001 16:15:32 -0800 (PST) Message-ID: <3A68D8D8.8DC6462E@agilent.com> Date: Fri, 19 Jan 2001 16:16:24 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports References: <002401c08240$69231ac0$90c809c0@yp_portable.advansys.com> <3A68BB57.11B1356C@ebay.sun.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David Robinson wrote: > If I read you correctly, you are asserting that there is a fundemental > problem > in the design on the TCP *protocol* which prevents it from taking > advantage > of a 10G/100ms network. If so what exactly do you see as a problem? That's easy - The lack of PDU framing in TCP. -Matt From owner-ips@ece.cmu.edu Fri Jan 19 21:26:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA09222 for ; Fri, 19 Jan 2001 21:26:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0K0xCJ23837 for ips-outgoing; Fri, 19 Jan 2001 19:59:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0K0wW123821 for ; Fri, 19 Jan 2001 19:58:33 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 19 Jan 2001 19:58:27 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A070410145F@corpmx9.isus.emc.com> From: Black_David@emc.com To: David.Robinson@EBay.Sun.COM, ips@ece.cmu.edu Subject: RE: iSCSI-related conclusions from Orlando interim Meeting Date: Fri, 19 Jan 2001 19:58:24 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk It's a bit cryptic. The conclusion in the room was to convert binary values to/from text representations for the purpose of negotiation (and use UTF-8 for the text). This was felt to be simpler than defining new formats for binary values. --David > -----Original Message----- > From: David Robinson [SMTP:David.Robinson@EBay.Sun.COM] > Sent: Friday, January 19, 2001 7:47 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI-related conclusions from Orlando interim Meeting > > > (5) Naming > > - UTF-8 will be used instead of ASCII for text strings in iSCSI > > login and > > text commands, naming, discovery, etc. > > - Binary values will be encoded in UTF-8 rather than adding > > type/length support. > > What does this mean? UTF-8 is an encoding of Unicode character sets, not > arbitrary > binary values. It is still just binary data so what > possible advantage is this? > > -David From owner-ips@ece.cmu.edu Fri Jan 19 21:28:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA09237 for ; Fri, 19 Jan 2001 21:28:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0K1BDI24127 for ips-outgoing; Fri, 19 Jan 2001 20:11:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0K1B7124119 for ; Fri, 19 Jan 2001 20:11:08 -0500 (EST) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id RAA09526 for ; Fri, 19 Jan 2001 17:11:07 -0800 (PST) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.142.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id RAA07689 for ; Fri, 19 Jan 2001 17:11:06 -0800 (PST) Received: from ebay.sun.com by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id RAA02615; Fri, 19 Jan 2001 17:11:06 -0800 (PST) Message-ID: <3A68E5C7.A3486D85@ebay.sun.com> Date: Fri, 19 Jan 2001 17:11:35 -0800 From: David Robinson Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.75 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI-related conclusions from Orlando interim Meeting References: <0F31E5C394DAD311B60C00E029101A070410145F@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Black_David@emc.com wrote: > It's a bit cryptic. The conclusion in the room was > to convert binary values to/from text representations > for the purpose of negotiation (and use UTF-8 for the > text). This was felt to be simpler than defining new > formats for binary values. So if I want to send the binary data 0101101011110000 I first turn that into text "5AF0" then encode that into UTF-8 "5AF0" (printable ASCII is a no-op in UTF-8)? I'll buy that. Any statement as to the text encoding? Hex? Decimal? Octal? -David From owner-ips@ece.cmu.edu Fri Jan 19 21:32:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA09293 for ; Fri, 19 Jan 2001 21:31:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0K0lCv23556 for ips-outgoing; Fri, 19 Jan 2001 19:47:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0K0ko123548 for ; Fri, 19 Jan 2001 19:46:51 -0500 (EST) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id QAA02521 for ; Fri, 19 Jan 2001 16:46:50 -0800 (PST) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.154.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id QAA03315 for ; Fri, 19 Jan 2001 16:46:49 -0800 (PST) Received: from ebay.sun.com by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id QAA10782; Fri, 19 Jan 2001 16:46:48 -0800 (PST) Message-ID: <3A68E015.4593E9F5@ebay.sun.com> Date: Fri, 19 Jan 2001 16:47:17 -0800 From: David Robinson Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.75 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI-related conclusions from Orlando interim Meeting References: <0F31E5C394DAD311B60C00E029101A070410145E@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > (5) Naming > - UTF-8 will be used instead of ASCII for text strings in iSCSI > login and > text commands, naming, discovery, etc. > - Binary values will be encoded in UTF-8 rather than adding > type/length support. What does this mean? UTF-8 is an encoding of Unicode character sets, not arbitrary binary values. It is still just binary data so what possible advantage is this? -David From owner-ips@ece.cmu.edu Fri Jan 19 21:35:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA10179 for ; Fri, 19 Jan 2001 21:35:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0K0fCr23420 for ips-outgoing; Fri, 19 Jan 2001 19:41:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0K0eR123400 for ; Fri, 19 Jan 2001 19:40:27 -0500 (EST) Received: from sponge.cisco.com (sponge.cisco.com [171.71.61.25]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id QAA18954; Fri, 19 Jan 2001 16:40:31 -0800 (PST) Received: from DAPW2K (sj-dial-4-26.cisco.com [171.68.181.155]) by sponge.cisco.com (Mirapoint) with SMTP id AFR06709; Fri, 19 Jan 2001 18:40:18 -0600 (CST) From: "David Peterson" To: , Subject: RE: iSCSI-related conclusions from Orlando interim Meeting Date: Fri, 19 Jan 2001 18:38:51 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <0F31E5C394DAD311B60C00E029101A070410145E@corpmx9.isus.emc.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit The SCSI command reference number is negotiated via the logical unit control page (18h). Dave > - SCSI has defined a new (optional) SCSI Command Ref Number > - iSCSI will use byte #2 in iSCSI header (currently Reserved) to > transport this > - This is not the ideal solution, but matches the level of support > in FCP. > - We have to check where mode page is to negotiate this > - If it's on a transport-specific mode page, iSCSI will > use a text key to negotiate this instead > From owner-ips@ece.cmu.edu Fri Jan 19 21:35:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA10181 for ; Fri, 19 Jan 2001 21:35:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0K1LDs24349 for ips-outgoing; Fri, 19 Jan 2001 20:21:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp1.mail.yahoo.com (smtp1.mail.yahoo.com [128.11.69.60]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f0K1Kq124340 for ; Fri, 19 Jan 2001 20:20:52 -0500 (EST) Received: from unknown (HELO somesh) (206.112.104.97) by smtp.mail.vip.suc.yahoo.com with SMTP; 20 Jan 2001 01:20:45 -0000 X-Apparently-From: Reply-To: From: "Somesh Gupta" To: "Santosh Rao" , , Subject: RE: iscsi : Fragmentation & Reassembly issues in iSCSI. Date: Fri, 19 Jan 2001 17:18:15 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: <3A68B8D5.2C9994FC@cup.hp.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit It is better to leave a seperate, possibly lower limit for PingMaxReplyLength -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Santosh Rao Sent: Friday, January 19, 2001 2:00 PM To: julian_satran@il.ibm.com; ips@ece.cmu.edu Subject: Re: iscsi : Fragmentation & Reassembly issues in iSCSI. julian_satran@il.ibm.com wrote: > I will introduce in 04 a statement saying that that PingMaxRelayLength > will be limited by the lowest of the two Julian, The fragmentation issue as explained in the above example is also applicable for the Login and Text Commands. (TotalText > DataPDULength). In the case of these commands, a "F" bit is required in bit 7 of byte 1 of the command & response PDUs to indicate the last command or response PDU due to the fragmentation issues that arise from TotalText & DataPDULength. > Solution : > ======= > Specify explicitly in Appendix C that DataPDULength is only applicable > for SCSI Command PDU and SCSI Data PDU. The current definition is open > to being mis-interpreted as a form of iSCSI MTU, something that can > result in the iSCSI layer having to deal with fragmentation and > re-assembly issues for non-SCSI PDUs. > > What would be those? Text commands? Login? Nops? Having a single limit > seemms simpler On the lines of your above solution, (based on a single limit), removal of PingMaxReplyLength should be considered and implicitly use DataPDULength as the PingMaxReplyLength. This ensures only 1 limit for NOP-OUT/NOP-IN. Regards, Santosh _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From owner-ips@ece.cmu.edu Fri Jan 19 22:54:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA11521 for ; Fri, 19 Jan 2001 22:54:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0K1lEQ24921 for ips-outgoing; Fri, 19 Jan 2001 20:47:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0K1l2124911 for ; Fri, 19 Jan 2001 20:47:02 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id DG4KP445; Fri, 19 Jan 2001 17:46:56 -0800 From: "Y P Cheng" To: "David Robinson" , "IPS Reflector" Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Fri, 19 Jan 2001 17:45:36 -0800 Message-ID: <001701c08282$af6a1be0$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: <3A68BB57.11B1356C@ebay.sun.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > If I read you correctly, you are asserting that there is a fundemental > problem in the design on the TCP *protocol* which prevents it from taking > advantage of a 10G/100ms network. If so what exactly do you see as a problem? I have been very careful in stating the problem is in implementation. The protocol is fine -- as the more and more I learn about the TCP options. I think more new options such as framing, marking, NCK, SACK, and RDMA can be defined and placed in login negotiation to allow the TCP protocol to run well in 10G/100ms network. > Since we cannot (ips WG) cannot change TCP how should an IPS protocol > work around this problem while still being friendly with other protocols? I do believe many people in this group do understand the problems. It new options that facilitate the 10G/100ms network are negotiated, we can still be friendly with older implementation which does not support the new options. In such case, we just run slow. > > If everyone agrees that this group can put iSCSI, iFCP, and FCIP together by > > assuming the current TCP implementations having all the solutions, please > > let me know. > > Conversely, if you feel that this group is designing to the TCP > implementations instead of the protocol, please let us know. > > -David I did sense that some people in this group were worry about compatibility with older implementations and reluctant to discuss or add new TCP options. In general, I do believe lots of people are quite up to speed, pun unintended. My statements made in herein previous postings were saying that two iSCSI or FCIP adapters of same kind -- with same TCP implementations -- should be able to run the new options. Y.P. From owner-ips@ece.cmu.edu Fri Jan 19 22:54:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA11532 for ; Fri, 19 Jan 2001 22:54:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0K20FI25194 for ips-outgoing; Fri, 19 Jan 2001 21:00:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0K1xk125181 for ; Fri, 19 Jan 2001 20:59:46 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id E7B8FE3C; Fri, 19 Jan 2001 17:59:44 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA04738; Fri, 19 Jan 2001 17:59:11 -0800 (PST) Message-ID: <3A68F16F.1F1B589B@cup.hp.com> Date: Fri, 19 Jan 2001 18:01:19 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com, Julian Satran Cc: ips@ece.cmu.edu Subject: Re: iSCSI-related conclusions from Orlando interim Meeting References: <0F31E5C394DAD311B60C00E029101A070410145E@corpmx9.isus.emc.com> Content-Type: multipart/mixed; boundary="------------B5927F735C6BD53B6517F0CF" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------B5927F735C6BD53B6517F0CF Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit I assume the below is referring to targets using NOP-IN as a NOP request in order to test a connection (?) If so, I don't see how this can result in a loop. A NOP-IN is sent with an Initiator Task Tag of 0xFFFFFFFF (assuming the errant usage of 0 as a reserved value were to be fixed in the spec). An initiator on seeing a NOP-IN with an I.T.T. of 0xFFFFFFFF will respond by sending NOP-OUT with I.T.T. of 0xFFFFFFFF. When a target receives a NOP-OUT with an I.T.T. of 0xFFFFFFFF, it is aware that this is a response to a NOP-IN it had originated earlier with a T.T.T that it specifies. How is a loop caused in the above scenario ? However, there is the other issue of the convoluted and non-intuitive logic in the above approach. There exist benefits of simplicity of understanding by the usage of NOP-OUT [or NOP Command] for the request and NOP-IN [or NOP response] for the response. Regards, Santosh Black_David@emc.com wrote: > > - Ping/NOP issues > - NOP responses are not permitted to request responses to avoid > loops > --------------B5927F735C6BD53B6517F0CF Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------B5927F735C6BD53B6517F0CF-- From owner-ips@ece.cmu.edu Fri Jan 19 22:56:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA11544 for ; Fri, 19 Jan 2001 22:56:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0K2JII25609 for ips-outgoing; Fri, 19 Jan 2001 21:19:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0K2JD125604 for ; Fri, 19 Jan 2001 21:19:13 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id CAA1CDEC for ; Fri, 19 Jan 2001 18:19:12 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA05842 for ; Fri, 19 Jan 2001 18:19:08 -0800 (PST) Message-ID: <3A68F61C.EB278310@cup.hp.com> Date: Fri, 19 Jan 2001 18:21:16 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Login Key issues. Content-Type: multipart/mixed; boundary="------------C4F0649652CA77FC9238C26C" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------C4F0649652CA77FC9238C26C Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Issues with iSCSI login keys : ====================== 1) The draft is missing default value specifications for the following login keys in Appendix C : MaxConnections TotalText KeyValueText A uniform policy should be applied to login keys which is either : a) specify default values for all keys OR b) declare a set of login keys as mandatory and respond to a login missing these mandatory key with "Reject Login - Missing Mandatory Parameters" in the Login Response PDU. 2) The default value for MaxOutstandingR2T is set to 256 which is too high. This should be lowered, since this is per task. Regards, Santosh --------------C4F0649652CA77FC9238C26C Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------C4F0649652CA77FC9238C26C-- From owner-ips@ece.cmu.edu Fri Jan 19 22:56:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA11555 for ; Fri, 19 Jan 2001 22:56:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0K1mFW24945 for ips-outgoing; Fri, 19 Jan 2001 20:48:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0K1lR124925 for ; Fri, 19 Jan 2001 20:47:27 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id C7991620; Fri, 19 Jan 2001 17:47:26 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA04084; Fri, 19 Jan 2001 17:47:21 -0800 (PST) Message-ID: <3A68EEA9.157C22C0@cup.hp.com> Date: Fri, 19 Jan 2001 17:49:30 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com, Julian Satran Cc: ips@ece.cmu.edu Subject: Re: iSCSI-related conclusions from Orlando interim Meeting References: <0F31E5C394DAD311B60C00E029101A070410145E@corpmx9.isus.emc.com> Content-Type: multipart/mixed; boundary="------------E3276B26F86DBBA8B83B1834" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------E3276B26F86DBBA8B83B1834 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit All, I have raised this question on the reflector earlier without getting any response. If the target's only means for a logout is to "request a logout" through an AEN (now called Asynch Message) and host SCSI stacks disable AER by default (which is the behaviour today), then, targets DO NOT have a reliable means of logging initiators out, since they cannot send AE messages. Targets MUST be allowed to send logouts or another reliable means be defined in the spec that allow targets to logout initiators. This peer-to-peer model [wherein a target can originate logout] is not a violation of SAM and allowing targets to send logouts is the most expeditious form of error recovery at the target end. Such a peer-to-peer model will also rid the spec of its convoluted way of using NOP-IN [which is a NOP response] as a NOP request when targets wish to check the connection. Targets MUST be allowed to originate Logout and NOP-OUT. Regards, Santosh Black_David@emc.com wrote: > - "AER" is used only for SCSI > - iSCSI communication of asynchronous events is through a > mechanism that is now called "Asynchronous Messages" - iSCSI > uses these to implement AER > - If a SCSI initiator has disabled AER, iSCSI does not send > the corresponding Asynchronous Messages --------------E3276B26F86DBBA8B83B1834 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------E3276B26F86DBBA8B83B1834-- From owner-ips@ece.cmu.edu Sat Jan 20 13:32:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA28808 for ; Sat, 20 Jan 2001 13:32:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0KFp2u18551 for ips-outgoing; Sat, 20 Jan 2001 10:51:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0KFoO618529 for ; Sat, 20 Jan 2001 10:50:25 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA97992 for ; Sat, 20 Jan 2001 16:50:16 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA188602 for ; Sat, 20 Jan 2001 16:50:17 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DA.0056FD39 ; Sat, 20 Jan 2001 16:50:09 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 20 Jan 2001 09:26:21 +0200 Subject: Re: iSCSI: DataRN Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Yes - command restart will have to be just that ... restart. Julo Raghavendra Rao on 19/01/2001 18:41:47 Please respond to Raghavendra Rao To: ips@ece.cmu.edu cc: Subject: iSCSI: DataRN Julian, > The consesus at the intermediate meeting in Orlando was to drop Data > Numbering. > If you have strong object here is your last chance to object. > I support the removal of DataRNs. I assume that this will also result in the elimination of the concept of micro-level restart of data transfer after a connection failover with in a session ? Or is that still present ? Thanks. -JP From owner-ips@ece.cmu.edu Sat Jan 20 21:41:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA01851 for ; Sat, 20 Jan 2001 21:41:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0L008R29174 for ips-outgoing; Sat, 20 Jan 2001 19:00:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail1.atl.bellsouth.net (mail1.atl.bellsouth.net [205.152.0.28]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0KNch628786 for ; Sat, 20 Jan 2001 18:38:44 -0500 (EST) Received: from default (adsl-61-48-44.atl.bellsouth.net [208.61.48.44]) by mail1.atl.bellsouth.net (3.3.5alt/0.75.2) with SMTP id SAA19496 for ; Sat, 20 Jan 2001 18:26:41 -0500 (EST) Message-ID: <001001c08311$d1804120$2c303dd0@default> From: "Sukha Ghosh" To: Subject: Some questions on iSCSI marker Date: Sat, 20 Jan 2001 18:49:36 -0000 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_000D_01C08311.BCA53440" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6600 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------=_NextPart_000_000D_01C08311.BCA53440 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, I have some questions on iSCSI Markers which are not clear to me = from the specification. Can one of you please clarify or answer: 1. The marker interval value like RFMarkInt, SFMarkInt, IFMarkInt etc, = are they value in dwords (for example, a value of 1 means interval of 4 = bytes) or are they in bytes but you have to make sure that they are = multiple of 4? 2. Is the position of the first marker in a TCP connection in a = direction (initial sequence value + IFMarkInt value)? 3. Is the relative offset of the next nearest start of the iSCSI message = header that is in the marker is counting the beginning of the marker or = is it relative to the end of the marker? 4. If there are multiple markers at negotiated marker intervals before = the beginning of next iSCSI message header (in between two iSCSI = messages), is it required to check the content validity of the next = marker with respect to the previous marker or do you take the content of = the latest marker identified as valid value? 5. The iSCSI PDUs being always multiple of 4 bytes, does it imply in any = way that the TCP sequence # needs to be also at a 4-byte boundary = alignment before starting placement of markers? 6. As I understood from the specification -01, the idea of using Urgent = pointer to identfy iSCSI message boundary in TCP stream is also to help = an external protocol analyzer to be able to identify and sync correctly = from a iSCSI message boundary at any point of time when it is asked to = sample. With the marker scheme, it seems that the protocol analyzer = needs to start sampling right at the beginning of the iSCSI log-in phase = to be able to identify the markers and hence the iSCSI message boundary = etc. Is my understanding correct?=20 Thanks =20 ------=_NextPart_000_000D_01C08311.BCA53440 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hi,
 
    I have some = questions on iSCSI=20 Markers which are not clear to me from the specification. Can one of you = please=20 clarify or answer:
 
1. The marker interval value like = RFMarkInt,=20 SFMarkInt, IFMarkInt etc, are they value in dwords (for example, a value = of 1=20 means interval of 4 bytes) or are they in bytes but you have to make = sure that=20 they are multiple of 4?
 
2. Is the position of the first marker = in a=20 TCP connection in a direction (initial sequence value + IFMarkInt=20 value)?
 
3. Is the relative offset of the = next=20 nearest start of the iSCSI message header that is in the marker is = counting=20 the beginning of the marker or is it relative to the end of the=20 marker?
 
4. If there are multiple markers = at negotiated=20 marker intervals before the beginning of next iSCSI message header (in = between=20 two iSCSI messages), is it required to check the content validity = of the=20 next marker with respect to the previous marker or do you take = the content=20 of the latest marker identified as valid value?
 
5. The iSCSI PDUs being always = multiple of 4=20 bytes, does it imply in any way that the TCP sequence # needs to be also = at a 4-byte boundary alignment before starting placement of=20 markers?
 
6. As I understood from the = specification -01, the=20 idea of using Urgent pointer to identfy iSCSI message boundary = in TCP=20 stream is also to help an external protocol analyzer to be = able to=20 identify and sync correctly from a iSCSI message boundary at any = point of=20 time when it is asked to sample. With the marker scheme, it seems that = the=20 protocol analyzer needs to start sampling right at the beginning of the = iSCSI=20 log-in phase to be able to identify the markers and hence the iSCSI = message=20 boundary etc. Is my understanding correct? 
 
Thanks
  
------=_NextPart_000_000D_01C08311.BCA53440-- From owner-ips@ece.cmu.edu Sun Jan 21 05:19:32 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA07552 for ; Sun, 21 Jan 2001 05:19:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0L7oDU06944 for ips-outgoing; Sun, 21 Jan 2001 02:50:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0L7nr606938 for ; Sun, 21 Jan 2001 02:49:53 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA57604 for ; Sun, 21 Jan 2001 08:49:41 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA110832 for ; Sun, 21 Jan 2001 08:49:36 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DB.002AFB55 ; Sun, 21 Jan 2001 08:49:28 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 21 Jan 2001 09:45:07 +0200 Subject: Re: iscsi : Fragmentation & Reassembly issues in iSCSI. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, The intent was to enable lower limits for ping and text. I would be reluctant to use the same limit as in some contexts the PDU limit could be practically very large. PDU will be limited at first by 2 factors - the CRC and the need to limit loss-of-framing temporary memory. If and when the underlying transport will have a good RDMA mechanism and we will have a decent CRC-64 or we are using IPsec the PDU limit could be extremely large. We don't want the same to hold for Ping, Text etc. However we would like to have the PDU limit to hold for all as it is mainly meant for framing and CRC. For all those reasons I suggest limiting the text length also to the lower of the two. Julo Santosh Rao on 19/01/2001 23:59:49 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: Re: iscsi : Fragmentation & Reassembly issues in iSCSI. julian_satran@il.ibm.com wrote: > I will introduce in 04 a statement saying that that PingMaxRelayLength > will be limited by the lowest of the two Julian, The fragmentation issue as explained in the above example is also applicable for the Login and Text Commands. (TotalText > DataPDULength). In the case of these commands, a "F" bit is required in bit 7 of byte 1 of the command & response PDUs to indicate the last command or response PDU due to the fragmentation issues that arise from TotalText & DataPDULength. > Solution : > ======= > Specify explicitly in Appendix C that DataPDULength is only applicable > for SCSI Command PDU and SCSI Data PDU. The current definition is open > to being mis-interpreted as a form of iSCSI MTU, something that can > result in the iSCSI layer having to deal with fragmentation and > re-assembly issues for non-SCSI PDUs. > > What would be those? Text commands? Login? Nops? Having a single limit > seemms simpler On the lines of your above solution, (based on a single limit), removal of PingMaxReplyLength should be considered and implicitly use DataPDULength as the PingMaxReplyLength. This ensures only 1 limit for NOP-OUT/NOP-IN. Regards, Santosh From owner-ips@ece.cmu.edu Sun Jan 21 06:09:52 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA07700 for ; Sun, 21 Jan 2001 06:09:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0L8jDQ07712 for ips-outgoing; Sun, 21 Jan 2001 03:45:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0L8iP607702 for ; Sun, 21 Jan 2001 03:44:25 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id JAA20412 for ; Sun, 21 Jan 2001 09:44:14 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id JAA253728 for ; Sun, 21 Jan 2001 09:44:14 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DB.002FFDB4 ; Sun, 21 Jan 2001 09:44:11 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 21 Jan 2001 10:39:46 +0200 Subject: Re: iSCSI-related conclusions from Orlando interim Meeting Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I think that David (Robinson) has a good point. I think that NDT would want to say text encoding in one of the C-language forms (0x'...' etc.) and those include binaries etc. The alternative is to allow only hex (not always nice nor readable). Julo David Robinson on 20/01/2001 03:11:35 Please respond to David Robinson To: ips@ece.cmu.edu cc: Subject: Re: iSCSI-related conclusions from Orlando interim Meeting Black_David@emc.com wrote: > It's a bit cryptic. The conclusion in the room was > to convert binary values to/from text representations > for the purpose of negotiation (and use UTF-8 for the > text). This was felt to be simpler than defining new > formats for binary values. So if I want to send the binary data 0101101011110000 I first turn that into text "5AF0" then encode that into UTF-8 "5AF0" (printable ASCII is a no-op in UTF-8)? I'll buy that. Any statement as to the text encoding? Hex? Decimal? Octal? -David From owner-ips@ece.cmu.edu Sun Jan 21 10:37:47 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA09458 for ; Sun, 21 Jan 2001 10:37:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0LD3GX21034 for ips-outgoing; Sun, 21 Jan 2001 08:03:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0LD2v621029 for ; Sun, 21 Jan 2001 08:02:57 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id OAA73298 for ; Sun, 21 Jan 2001 14:02:44 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id OAA176636 for ; Sun, 21 Jan 2001 14:01:28 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DB.004789AD ; Sun, 21 Jan 2001 14:01:23 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 21 Jan 2001 14:57:04 +0200 Subject: Re: iSCSI-related conclusions from Orlando interim Meeting Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, The answer is in the current draft. AEN (newly renamed Asynch Messages) can have an iSCSI origin or a SCSI origin. SCSI can control only the SCSI reporting. iSCSI initiators are supposed to accept always iSCSI Asynch Messages. Well behaved targets are not supposed to send SCSI messages when forbiden by SCSI. The iSCSI initiator is not mandated to check conformance to the above rule. Julo Santosh Rao on 20/01/2001 03:49:30 Please respond to Santosh Rao To: Black_David@emc.com, Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI-related conclusions from Orlando interim Meeting All, I have raised this question on the reflector earlier without getting any response. If the target's only means for a logout is to "request a logout" through an AEN (now called Asynch Message) and host SCSI stacks disable AER by default (which is the behaviour today), then, targets DO NOT have a reliable means of logging initiators out, since they cannot send AE messages. Targets MUST be allowed to send logouts or another reliable means be defined in the spec that allow targets to logout initiators. This peer-to-peer model [wherein a target can originate logout] is not a violation of SAM and allowing targets to send logouts is the most expeditious form of error recovery at the target end. Such a peer-to-peer model will also rid the spec of its convoluted way of using NOP-IN [which is a NOP response] as a NOP request when targets wish to check the connection. Targets MUST be allowed to originate Logout and NOP-OUT. Regards, Santosh Black_David@emc.com wrote: > - "AER" is used only for SCSI > - iSCSI communication of asynchronous events is through a > mechanism that is now called "Asynchronous Messages" - iSCSI > uses these to implement AER > - If a SCSI initiator has disabled AER, iSCSI does not send > the corresponding Asynchronous Messages - santoshr.vcf From owner-ips@ece.cmu.edu Sun Jan 21 11:26:13 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA09659 for ; Sun, 21 Jan 2001 11:26:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0LDsGL21701 for ips-outgoing; Sun, 21 Jan 2001 08:54:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0LDs5621693 for ; Sun, 21 Jan 2001 08:54:06 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id OAA199760 for ; Sun, 21 Jan 2001 14:53:59 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id OAA61998 for ; Sun, 21 Jan 2001 14:53:59 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DB.004C583E ; Sun, 21 Jan 2001 14:53:53 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 21 Jan 2001 15:49:33 +0200 Subject: Re: iSCSI : Login Key issues. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, We did attempt to have a default for every key. We will fix the missing ones. Some of the defaults are rather arbitrary (until we gain some hands-on experience) but 256 is not that high for a long fat pipe. Is 255 better ? -:) Julo Santosh Rao on 20/01/2001 04:21:16 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iSCSI : Login Key issues. Issues with iSCSI login keys : ====================== 1) The draft is missing default value specifications for the following login keys in Appendix C : MaxConnections TotalText KeyValueText A uniform policy should be applied to login keys which is either : a) specify default values for all keys OR b) declare a set of login keys as mandatory and respond to a login missing these mandatory key with "Reject Login - Missing Mandatory Parameters" in the Login Response PDU. 2) The default value for MaxOutstandingR2T is set to 256 which is too high. This should be lowered, since this is per task. Regards, Santosh - santoshr.vcf From owner-ips@ece.cmu.edu Sun Jan 21 12:51:41 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA10122 for ; Sun, 21 Jan 2001 12:51:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0LF9Im22927 for ips-outgoing; Sun, 21 Jan 2001 10:09:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0LF8m622919 for ; Sun, 21 Jan 2001 10:08:48 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAA34116 for ; Sun, 21 Jan 2001 16:08:42 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA76138 for ; Sun, 21 Jan 2001 16:08:42 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DB.005326E3 ; Sun, 21 Jan 2001 16:08:14 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 21 Jan 2001 17:03:48 +0200 Subject: Re: Some questions on iSCSI marker Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Answers in text. Julo "Sukha Ghosh" on 20/01/2001 20:49:36 Please respond to "Sukha Ghosh" To: ips@ece.cmu.edu cc: Subject: Some questions on iSCSI marker Hi, I have some questions on iSCSI Markers which are not clear to me from the specification. Can one of you please clarify or answer: 1. The marker interval value like RFMarkInt, SFMarkInt, IFMarkInt etc, are they value in dwords (for example, a value of 1 means interval of 4 bytes) or are they in bytes but you have to make sure that they are multiple of 4? 1 is one word 2. Is the position of the first marker in a TCP connection in a direction (initial sequence value + IFMarkInt value)? correct 3. Is the relative offset of the next nearest start of the iSCSI message header that is in the marker is counting the beginning of the marker or is it relative to the end of the marker? start-to-start I will clarify the text and add an exemple 4. If there are multiple markers at negotiated marker intervals before the beginning of next iSCSI message header (in between two iSCSI messages), is it required to check the content validity of the next marker with respect to the previous marker or do you take the content of the latest marker identified as valid value? that is up to the implemeter; in theory you could ignore them until you need them but you could as well signal a format error is markers are incorrect 5. The iSCSI PDUs being always multiple of 4 bytes, does it imply in any way that the TCP sequence # needs to be also at a 4-byte boundary alignment before starting placement of markers? definitely not! TCP asigns a random sequence number to start with and that is your "virtual-mile-0" 6. As I understood from the specification -01, the idea of using Urgent pointer to identfy iSCSI message boundary in TCP stream is also to help an external protocol analyzer to be able to identify and sync correctly from a iSCSI message boundary at any point of time when it is asked to sample. With the marker scheme, it seems that the protocol analyzer needs to start sampling right at the beginning of the iSCSI log-in phase to be able to identify the markers and hence the iSCSI message boundary etc. Is my understanding correct? Thanks - att1.htm a protocol analyzer will either have to find out by itself what marker interval is being used (trial and error) or ask the operate to key one in From owner-ips@ece.cmu.edu Sun Jan 21 16:41:59 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA11372 for ; Sun, 21 Jan 2001 16:41:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0LJUMl28367 for ips-outgoing; Sun, 21 Jan 2001 14:30:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0LJTZ628338 for ; Sun, 21 Jan 2001 14:29:36 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 96CAD1B84; Sun, 21 Jan 2001 11:29:17 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id LAA20126; Sun, 21 Jan 2001 11:29:12 -0800 (PST) From: Santosh Rao Message-Id: <200101211929.LAA20126@hpcuhe.cup.hp.com> Subject: Re: iscsi : Fragmentation & Reassembly issues in iSCSI. To: julian_satran@il.ibm.com Date: Sun, 21 Jan 2001 11:29:12 -0800 (PST) Cc: ips@ece.cmu.edu In-Reply-To: from "julian_satran@il.ibm.com" at Jan 21, 2001 09:45:07 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > Santosh, > > The intent was to enable lower limits for ping and text. I would be > reluctant to use the same limit as in some contexts the PDU limit > could be practically very large. PDU will be limited at first > by 2 factors - the CRC and the need to limit loss-of-framing > temporary memory.If and when the underlying transport will have a good > RDMA mechanism and we will have a decent CRC-64 or we are using > IPsec the PDU limit could be extremely large. > We don't want the same to hold for Ping, Text etc. > > However we would like to have the PDU limit to hold for all as it is > mainly meant for framing and CRC. > > For all those reasons I suggest limiting the text length also to the > lower of the two. Julian, Agreed. However, it could be the case that the number of login keys being negotiated exceeds TotalText requiring the use of multiple Text Commands and Text Responses. An "F" bit is required in bit 7 of byte 1 of the text and login commands and responses to indicate the last command or response PDU of the login or text operations to deal with this condition. Regards, Santosh > > Julo > > Santosh Rao on 19/01/2001 23:59:49 > > Please respond to Santosh Rao > > The fragmentation issue as explained in the above example is also > applicable for the Login and Text Commands. (TotalText > DataPDULength). > > In the case of these commands, a "F" bit is required in bit 7 of byte > 1 of the command & response PDUs to indicate the last command or > response PDU due to the fragmentation issues that arise from > TotalText & DataPDULength. -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Sun Jan 21 16:42:19 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA11383 for ; Sun, 21 Jan 2001 16:42:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0LJMMU28183 for ips-outgoing; Sun, 21 Jan 2001 14:22:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0LJLt628175 for ; Sun, 21 Jan 2001 14:21:55 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 90A231B11; Sun, 21 Jan 2001 11:21:53 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id LAA19795; Sun, 21 Jan 2001 11:21:46 -0800 (PST) From: Santosh Rao Message-Id: <200101211921.LAA19795@hpcuhe.cup.hp.com> Subject: Re: iSCSI-related conclusions from Orlando interim Meeting To: julian_satran@il.ibm.com Date: Sun, 21 Jan 2001 11:21:46 -0800 (PST) Cc: ips@ece.cmu.edu In-Reply-To: from "julian_satran@il.ibm.com" at Jan 21, 2001 02:57:04 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, ON AER, SAM-2 states that : "Each SCSI Protocol shall describe a mechanism for Asynchronous Event Reporting, including a procedure whereby a SCSI device can selectively enable or disable asynchronous event reports from being sent to it by a specific target." (5.7.4.1) From your response, it sounds like iSCSI intends to describe an AER mechanism, but not define a procedure that allows initiators to selectively enable or disable it. Is this in line with SAM ? On the subject of Async Message "Target requests a logout", which is a ping-pong procedure to achieve a logout, it may be worth considering an alternate Async Message in its place that indicates "Target is logging out for error recovery" and allow the target to follow up the Async Message with a connection close. This will simplify target driven connection level error recovery. Regards, Santosh > > Santosh, > > The answer is in the current draft. AEN (newly renamed Asynch Messages) > can have an iSCSI origin or a SCSI origin. > SCSI can control only the SCSI reporting. iSCSI initiators are supposed to > accept always iSCSI Asynch Messages. Well behaved targets are not supposed > to send SCSI messages when forbiden by SCSI. The iSCSI initiator is not > mandated to check conformance to the above rule. > > Julo > > Santosh Rao on 20/01/2001 03:49:30 > > Please respond to Santosh Rao > > To: Black_David@emc.com, Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: iSCSI-related conclusions from Orlando interim Meeting > > > > > All, > > I have raised this question on the reflector earlier without getting any > response. > > If the target's only means for a logout is to "request a logout" through an > AEN > (now called Asynch Message) and host SCSI stacks disable AER by default > (which > is the behaviour today), then, targets DO NOT have a reliable means of > logging > initiators out, since they cannot send AE messages. > > Targets MUST be allowed to send logouts or another reliable means be > defined in > the spec that allow targets to logout initiators. This peer-to-peer model > [wherein a target can originate logout] is not a violation of SAM and > allowing > targets to send logouts is the most expeditious form of error recovery at > the > target end. > > Such a peer-to-peer model will also rid the spec of its convoluted way of > using > NOP-IN [which is a NOP response] as a NOP request when targets wish to > check > the connection. > > Targets MUST be allowed to originate Logout and NOP-OUT. > > Regards, > Santosh > > Black_David@emc.com wrote: > > > - "AER" is used only for SCSI > > > - iSCSI communication of asynchronous events is through a > > mechanism that is now called "Asynchronous Messages" - iSCSI > > uses these to implement AER > > > - If a SCSI initiator has disabled AER, iSCSI does not send > > the corresponding Asynchronous Messages > > - santoshr.vcf > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Sun Jan 21 21:34:43 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA13658 for ; Sun, 21 Jan 2001 21:34:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0M0YRs05360 for ips-outgoing; Sun, 21 Jan 2001 19:34:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from tiny-teddy.aarnet.edu.au (IDENT:root@tiny-teddy.aarnet.edu.au [203.21.37.30]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0M0C9604921 for ; Sun, 21 Jan 2001 19:12:09 -0500 (EST) Received: from aarnet.edu.au (bush.aarnet.adelaide.edu.au [129.127.80.130]) by tiny-teddy.aarnet.edu.au (8.9.3/8.9.3) with ESMTP id KAA10111; Mon, 22 Jan 2001 10:41:53 +1030 Message-ID: <3A6B7B04.76117B4F@aarnet.edu.au> Date: Mon, 22 Jan 2001 10:42:52 +1030 From: Glen Turner Organization: Australian Academic and Research Network X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.4.0 i686) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com CC: ips@ece.cmu.edu Subject: Re: iSCSI-related conclusions from Orlando interim Meeting References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > > I think that David (Robinson) has a good point. I think that NDT would want > to say text encoding in one of the C-language forms (0x'...' etc.) and those > include binaries etc. Hi Julo, Perhaps not the C method, as this has caused problems in the past (most notoriously with scanning IP addresses). The difference between "10" and "010" isn't apparent to the typical author of a configuraton file (one number is in decimal, the other number is in octal). People that align strings by zero-filling them are burnt. If using UTF-8 you may also want to constrain valid strings to the shortest UTF-8 representation of the string[1]. This makes checking for values much simpler and allows the writing of reliable regular expressions: eg: allow volumes in "*.example.au" > The alternative is to allow only hex (not always nice nor readable). Or decimal. Probably better if values will propogate up into configuration files. One of the advantages of a textual representation of numbers is that you can simply pass down the user's parameters unaltered. This allows additional parameters to be passed to the server without the client needing to be updated. Regards, Glen [1] I can't find the reference for the right word for "shortest", it might be a word like "canonical" instead? From owner-ips@ece.cmu.edu Sun Jan 21 21:34:54 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA13760 for ; Sun, 21 Jan 2001 21:34:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0M0YPp05357 for ips-outgoing; Sun, 21 Jan 2001 19:34:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from tiny-teddy.aarnet.edu.au (IDENT:root@tiny-teddy.aarnet.edu.au [203.21.37.30]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0M0C9604921 for ; Sun, 21 Jan 2001 19:12:09 -0500 (EST) Received: from aarnet.edu.au (bush.aarnet.adelaide.edu.au [129.127.80.130]) by tiny-teddy.aarnet.edu.au (8.9.3/8.9.3) with ESMTP id KAA10111; Mon, 22 Jan 2001 10:41:53 +1030 Message-ID: <3A6B7B04.76117B4F@aarnet.edu.au> Date: Mon, 22 Jan 2001 10:42:52 +1030 From: Glen Turner Organization: Australian Academic and Research Network X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.4.0 i686) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com CC: ips@ece.cmu.edu Subject: Re: iSCSI-related conclusions from Orlando interim Meeting References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > > I think that David (Robinson) has a good point. I think that NDT would want > to say text encoding in one of the C-language forms (0x'...' etc.) and those > include binaries etc. Hi Julo, Perhaps not the C method, as this has caused problems in the past (most notoriously with scanning IP addresses). The difference between "10" and "010" isn't apparent to the typical author of a configuraton file (one number is in decimal, the other number is in octal). People that align strings by zero-filling them are burnt. If using UTF-8 you may also want to constrain valid strings to the shortest UTF-8 representation of the string[1]. This makes checking for values much simpler and allows the writing of reliable regular expressions: eg: allow volumes in "*.example.au" > The alternative is to allow only hex (not always nice nor readable). Or decimal. Probably better if values will propogate up into configuration files. One of the advantages of a textual representation of numbers is that you can simply pass down the user's parameters unaltered. This allows additional parameters to be passed to the server without the client needing to be updated. Regards, Glen [1] I can't find the reference for the right word for "shortest", it might be a word like "canonical" instead? From owner-ips@ece.cmu.edu Sun Jan 21 21:37:56 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA13833 for ; Sun, 21 Jan 2001 21:37:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0M06QW04803 for ips-outgoing; Sun, 21 Jan 2001 19:06:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0M05a604790 for ; Sun, 21 Jan 2001 19:05:36 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 212BCD64; Sun, 21 Jan 2001 16:05:35 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id QAA01948; Sun, 21 Jan 2001 16:05:22 -0800 (PST) From: Santosh Rao Message-Id: <200101220005.QAA01948@hpcuhe.cup.hp.com> Subject: Re: iSCSI-related conclusions from Orlando interim Meeting To: Black_David@emc.com Date: Sun, 21 Jan 2001 16:05:22 -0800 (PST) Cc: ips@ece.cmu.edu In-Reply-To: <0F31E5C394DAD311B60C00E029101A070410145E@corpmx9.isus.emc.com> from "Black_David@emc.com" at Jan 19, 2001 07:29:36 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > - Abort WARNING will be added to the draft. > - Immediate Delivery of Aborts and similar task management commands > may have unexpected results when multiple TCP connections are in use > because Abort, Clear Task Set, etc. may bypass command(s) to be > aborted/cleared on other TCP connections. > - Ordered Delivery should be used instead when this is a concern. Is the above referring to ordered delivery as in CmdSN based ordering or the use of the Ordered Task Tag ? I see the following issues with this : a) CmdSN defines the order of delivery from the iSCSI layer to the SCSI ULP at the target. Abortion of active tasks will involve the invalidation of the SCSI State information [being accessed by either adapter micro-code or the FPGA/ASIC] maintained for these tasks. This process is kicked off in the iSCSI layer and this abort functionality in the iSCSI layer will also need to be subject to the ordering constraints of CmdSN. (in addition to the ordering of delivery to the SCSI ULP). b) A received Task Management command [on, say, connection 'x'] cannot be performed due to previous CmdSNs not yet having arrived. (In the case where another connection 'y' in the session is faulty and CmdSNs sent on connection 'y' have'nt arrived). c) Ordered Task Tags should not be used with error recovery Task Management commands, since this can result in a deadlock of the task mgmt command in case the older tasks are'nt completing in the SCSI ULP due to some ULP problem. ("An Ordered Task cannot enter the enabled state until all older tasks have completed.") d) Given that Immediate command delivery (CmdSN of 0) is un-usable for task management commands, is there any other application for this feature? If not, all references to it can be removed from the draft and CmdSN can start from 0. Regards, Santosh -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Mon Jan 22 00:12:22 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA16286 for ; Mon, 22 Jan 2001 00:12:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0M33Rh08402 for ips-outgoing; Sun, 21 Jan 2001 22:03:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0M32a608388 for ; Sun, 21 Jan 2001 22:02:36 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 6562C34A; Sun, 21 Jan 2001 19:02:35 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id TAA09749; Sun, 21 Jan 2001 19:02:30 -0800 (PST) Message-ID: <3A6BA34D.D0C23457@cup.hp.com> Date: Sun, 21 Jan 2001 19:04:45 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI: DataRN References: Content-Type: multipart/mixed; boundary="------------D22673B0921E9182548A815D" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------D22673B0921E9182548A815D Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, The iSCSI draft should refrain from advocating any retry policies at the iSCSI layer, such as being currently done for connection failures and digest errors. I/O retry policies are decided by the SCSI ULP based on the class of the target. (disk, tape, changer, etc). For a tape class of device, the SCSI ULP may not wish to retry on an error [without a prior rewind operation]. In such situations, iSCSI attempting to retry on connection failures or digest errors can result in problems with sequential access type of media. Regards, Santosh julian_satran@il.ibm.com wrote: > Yes - command restart will have to be just that ... restart. > > Julo > > Raghavendra Rao on 19/01/2001 18:41:47 > > Please respond to Raghavendra Rao > > To: ips@ece.cmu.edu > cc: > Subject: iSCSI: DataRN > > Julian, > > > The consesus at the intermediate meeting in Orlando was to drop Data > > Numbering. > > If you have strong object here is your last chance to object. > > > > I support the removal of DataRNs. I assume that this will also result in > the elimination of the concept of micro-level restart of data transfer > after > a connection failover with in a session ? Or is that still present ? > > Thanks. > > -JP --------------D22673B0921E9182548A815D Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------D22673B0921E9182548A815D-- From owner-ips@ece.cmu.edu Mon Jan 22 00:13:54 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA16308 for ; Mon, 22 Jan 2001 00:13:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0M2rRk08228 for ips-outgoing; Sun, 21 Jan 2001 21:53:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0M2rP608223 for ; Sun, 21 Jan 2001 21:53:25 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 31B6ED53; Sun, 21 Jan 2001 18:53:24 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAB09385; Sun, 21 Jan 2001 18:53:19 -0800 (PST) Message-ID: <3A6BA125.B43EC181@cup.hp.com> Date: Sun, 21 Jan 2001 18:55:33 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI : Login Key issues. References: Content-Type: multipart/mixed; boundary="------------E72878DE8464F216EE76201E" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------E72878DE8464F216EE76201E Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, Based on Fibre Channel's FCP_XFER_RDY experiences and the WRITE I/O breakups that occur in Fibre Channel, this value is too high. The number of FCP_XFER_RDYs per WRITE I/O of the sizes of 256K does not require more than 4 - 5 FCP_XFER_RDYs. We are most likely speaking ballpark figures of total no. of R2Ts for 256K WRITE I/O to be in the range of < 10. Therefore, the number of concurrent R2Ts per task that an initiator can handle should be in that range. It is highly unlikely that targets are expected to fragment typical WRITE I/Os into 256 R2Ts. Regards, Santosh julian_satran@il.ibm.com wrote: > Santosh, > > Some of the defaults are rather arbitrary (until we gain some hands-on > experience) but 256 is not that high for a long fat pipe. Is 255 better ? > -:) > > Julo > --------------E72878DE8464F216EE76201E Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------E72878DE8464F216EE76201E-- From owner-ips@ece.cmu.edu Mon Jan 22 03:31:33 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA00453 for ; Mon, 22 Jan 2001 03:31:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0M6JUx12147 for ips-outgoing; Mon, 22 Jan 2001 01:19:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0M6JN612141 for ; Mon, 22 Jan 2001 01:19:23 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA33738 for ; Mon, 22 Jan 2001 07:19:17 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id HAA81052 for ; Mon, 22 Jan 2001 07:19:17 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DC.0022B87A ; Mon, 22 Jan 2001 07:19:14 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 22 Jan 2001 08:14:54 +0200 Subject: Re: iscsi : Fragmentation & Reassembly issues in iSCSI. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, The F bit is used to indicate the end of a sequence that is "sort-of" atomic (like an R2T or the initial burst). There might be some parameter negotiations that we want to be atomic (like the login phase) and we might end-up using the F bit there. But I will NOT mandate the use of the F bit in any sequence only due to the (rather academic) fragmentation aspect you mention. Initiator and target builders should be able to have enough buffer space for all the keys they use. Julo Santosh Rao on 21/01/2001 21:29:12 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iscsi : Fragmentation & Reassembly issues in iSCSI. > Santosh, > > The intent was to enable lower limits for ping and text. I would be > reluctant to use the same limit as in some contexts the PDU limit > could be practically very large. PDU will be limited at first > by 2 factors - the CRC and the need to limit loss-of-framing > temporary memory.If and when the underlying transport will have a good > RDMA mechanism and we will have a decent CRC-64 or we are using > IPsec the PDU limit could be extremely large. > We don't want the same to hold for Ping, Text etc. > > However we would like to have the PDU limit to hold for all as it is > mainly meant for framing and CRC. > > For all those reasons I suggest limiting the text length also to the > lower of the two. Julian, Agreed. However, it could be the case that the number of login keys being negotiated exceeds TotalText requiring the use of multiple Text Commands and Text Responses. An "F" bit is required in bit 7 of byte 1 of the text and login commands and responses to indicate the last command or response PDU of the login or text operations to deal with this condition. Regards, Santosh > > Julo > > Santosh Rao on 19/01/2001 23:59:49 > > Please respond to Santosh Rao > > The fragmentation issue as explained in the above example is also > applicable for the Login and Text Commands. (TotalText > DataPDULength). > > In the case of these commands, a "F" bit is required in bit 7 of byte > 1 of the command & response PDUs to indicate the last command or > response PDU due to the fragmentation issues that arise from > TotalText & DataPDULength. -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Mon Jan 22 03:31:36 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA00464 for ; Mon, 22 Jan 2001 03:31:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0M6VUu12420 for ips-outgoing; Mon, 22 Jan 2001 01:31:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0M6Us612405 for ; Mon, 22 Jan 2001 01:30:54 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA73922; Mon, 22 Jan 2001 07:30:36 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id HAA176420; Mon, 22 Jan 2001 07:29:04 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DC.00239CF1 ; Mon, 22 Jan 2001 07:28:59 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu cc: "Barry Reinhold" Message-ID: Date: Mon, 22 Jan 2001 08:24:38 +0200 Subject: Re: Coverage of Data Digest when using Header Digests Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Barry, Considering that one of the reasons to have a separate header and data digest was to enable data to carried through proxies, virtualizers etc. the current thinking is that the data digest will cover only the data and the header (including extensions) will be covered by the header digest. Julo "Barry Reinhold" on 21/01/2001 21:50:04 Please respond to "Barry Reinhold" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Coverage of Data Digest when using Header Digests Julian, Is the Data Digest intended to cover whatever follows the 48 byte iSCSI header? In particular in a command frame which has a CDB > 16 bytes, uses bidi, has immediate data, and is using both header and data digests - what would the data digest cover? - barry reinhold From owner-ips@ece.cmu.edu Mon Jan 22 03:33:03 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA00491 for ; Mon, 22 Jan 2001 03:33:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0M6AUC12002 for ips-outgoing; Mon, 22 Jan 2001 01:10:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0M69Y611971 for ; Mon, 22 Jan 2001 01:09:35 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA25606 for ; Mon, 22 Jan 2001 07:09:27 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id HAA24068 for ; Mon, 22 Jan 2001 07:08:11 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DC.0021B40A ; Mon, 22 Jan 2001 07:08:07 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 22 Jan 2001 08:03:43 +0200 Subject: Re: iSCSI-related conclusions from Orlando interim Meeting Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, We have discussed the things at some length with Ralph Weber. As I explained in my previous note we use the Asynch Message both to convey to the initiator iSCSI messages and SCSI messages. the target is supposed to behave and stop sending events that the iSCSI initiator should pop up to the SCSI layer. An iSCSI initiator may fileter (check for compliance) SCSI messages but it is NOT REQUIRED to do so (in order to do this it will have to have knowledge of the SCSI state). iSCSI has no provision to stop async messages. All those are (except the names) already in the current draft (and some before). I am reluctant to consider any procedure that originates a command at the target. Note however that a target in trouble may drop a TCP connection and force an initiator to recover. Julo Santosh Rao on 21/01/2001 21:21:46 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI-related conclusions from Orlando interim Meeting Julian, ON AER, SAM-2 states that : "Each SCSI Protocol shall describe a mechanism for Asynchronous Event Reporting, including a procedure whereby a SCSI device can selectively enable or disable asynchronous event reports from being sent to it by a specific target." (5.7.4.1) From your response, it sounds like iSCSI intends to describe an AER mechanism, but not define a procedure that allows initiators to selectively enable or disable it. Is this in line with SAM ? On the subject of Async Message "Target requests a logout", which is a ping-pong procedure to achieve a logout, it may be worth considering an alternate Async Message in its place that indicates "Target is logging out for error recovery" and allow the target to follow up the Async Message with a connection close. This will simplify target driven connection level error recovery. Regards, Santosh > > Santosh, > > The answer is in the current draft. AEN (newly renamed Asynch Messages) > can have an iSCSI origin or a SCSI origin. > SCSI can control only the SCSI reporting. iSCSI initiators are supposed to > accept always iSCSI Asynch Messages. Well behaved targets are not supposed > to send SCSI messages when forbiden by SCSI. The iSCSI initiator is not > mandated to check conformance to the above rule. > > Julo > > Santosh Rao on 20/01/2001 03:49:30 > > Please respond to Santosh Rao > > To: Black_David@emc.com, Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: iSCSI-related conclusions from Orlando interim Meeting > > > > > All, > > I have raised this question on the reflector earlier without getting any > response. > > If the target's only means for a logout is to "request a logout" through an > AEN > (now called Asynch Message) and host SCSI stacks disable AER by default > (which > is the behaviour today), then, targets DO NOT have a reliable means of > logging > initiators out, since they cannot send AE messages. > > Targets MUST be allowed to send logouts or another reliable means be > defined in > the spec that allow targets to logout initiators. This peer-to-peer model > [wherein a target can originate logout] is not a violation of SAM and > allowing > targets to send logouts is the most expeditious form of error recovery at > the > target end. > > Such a peer-to-peer model will also rid the spec of its convoluted way of > using > NOP-IN [which is a NOP response] as a NOP request when targets wish to > check > the connection. > > Targets MUST be allowed to originate Logout and NOP-OUT. > > Regards, > Santosh > > Black_David@emc.com wrote: > > > - "AER" is used only for SCSI > > > - iSCSI communication of asynchronous events is through a > > mechanism that is now called "Asynchronous Messages" - iSCSI > > uses these to implement AER > > > - If a SCSI initiator has disabled AER, iSCSI does not send > > the corresponding Asynchronous Messages > > - santoshr.vcf > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Mon Jan 22 06:00:55 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA01467 for ; Mon, 22 Jan 2001 06:00:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0M8YVf14594 for ips-outgoing; Mon, 22 Jan 2001 03:34:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0M8Xx614583 for ; Mon, 22 Jan 2001 03:33:59 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id JAA123022 for ; Mon, 22 Jan 2001 09:33:52 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id JAA14540 for ; Mon, 22 Jan 2001 09:32:36 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DC.002EEA65 ; Mon, 22 Jan 2001 09:32:26 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 22 Jan 2001 10:28:02 +0200 Subject: Re: iSCSI-related conclusions from Orlando interim Meeting Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, I am not sure that I understand your points. Point c - is irrelevant as task management function are just that task management functions and not commands (i.e., have no ordering, queueing etc. associated with its SCSI execution) we choose to get a task management function a task-id only to simplify request response handling in the iSCSI layer - and not because there is a SCSI task context associated with it the side effects that David (Black) was referring I think are the following: if at the target you get to abort a task that has not arrived yet and since you are bound to use service response function complete you should (at the target) dutifully note the referenced-id and abort it when it arrives - as you are forbidden (by SAM) to send any more responses for this task; this may lead to interesting cleanup problem if the referenced id was never sent - and you will have to handle this (practically you can remove the mark when CmdSN arrives or after an interval if you don't have a CmdSN and either inform or not the initiator) if at the target you get a clear task set you may end not clearing tasks you where supposed too or clearing tasks you where not supposed too (as there is no ordering implied) The solution suggested - order the task management at iSCSI level is straightforward task management functions request can be delivered to SCSI and executed IMMEDIATELY and the CmdSN they carry should be viewed only as a "boundary" for their applicability Immediate delivery was meant to be used in cases in which you are confident that ordered delivery does not matter. This might be a drastic action (like target rest) or things that we did not consider yet. Julo Santosh Rao on 22/01/2001 02:05:22 Please respond to Santosh Rao To: Black_David@emc.com cc: ips@ece.cmu.edu Subject: Re: iSCSI-related conclusions from Orlando interim Meeting > - Abort WARNING will be added to the draft. > - Immediate Delivery of Aborts and similar task management commands > may have unexpected results when multiple TCP connections are in use > because Abort, Clear Task Set, etc. may bypass command(s) to be > aborted/cleared on other TCP connections. > - Ordered Delivery should be used instead when this is a concern. Is the above referring to ordered delivery as in CmdSN based ordering or the use of the Ordered Task Tag ? I see the following issues with this : a) CmdSN defines the order of delivery from the iSCSI layer to the SCSI ULP at the target. Abortion of active tasks will involve the invalidation of the SCSI State information [being accessed by either adapter micro-code or the FPGA/ASIC] maintained for these tasks. This process is kicked off in the iSCSI layer and this abort functionality in the iSCSI layer will also need to be subject to the ordering constraints of CmdSN. (in addition to the ordering of delivery to the SCSI ULP). b) A received Task Management command [on, say, connection 'x'] cannot be performed due to previous CmdSNs not yet having arrived. (In the case where another connection 'y' in the session is faulty and CmdSNs sent on connection 'y' have'nt arrived). c) Ordered Task Tags should not be used with error recovery Task Management commands, since this can result in a deadlock of the task mgmt command in case the older tasks are'nt completing in the SCSI ULP due to some ULP problem. ("An Ordered Task cannot enter the enabled state until all older tasks have completed.") d) Given that Immediate command delivery (CmdSN of 0) is un-usable for task management commands, is there any other application for this feature? If not, all references to it can be removed from the draft and CmdSN can start from 0. Regards, Santosh -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Mon Jan 22 12:34:02 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA09684 for ; Mon, 22 Jan 2001 12:33:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0MFK4U01646 for ips-outgoing; Mon, 22 Jan 2001 10:20:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0MFJl601637 for ; Mon, 22 Jan 2001 10:19:47 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id KAA20366; Mon, 22 Jan 2001 10:19:37 -0500 (EST) Message-ID: <3A6C500E.CD92908@cisco.com> Date: Mon, 22 Jan 2001 09:21:50 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: julian_satran@il.ibm.com CC: ips@ece.cmu.edu, Barry Reinhold Subject: Re: Coverage of Data Digest when using Header Digests References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Barry- In particular, the data digest covers only the SCSI Data part of an iSCSI message; the header digest covers everything else. This means that in an 8k write, the data digest will cover only the 8k, and the header digest will cover everything else. Hope this helps, Mark julian_satran@il.ibm.com wrote: > > Barry, > > Considering that one of the reasons to have a separate header and data > digest was to enable data > to carried through proxies, virtualizers etc. the current thinking is that > the data digest will cover only the data and the header (including > extensions) will be covered by the header digest. > > Julo > > "Barry Reinhold" on 21/01/2001 21:50:04 > > Please respond to "Barry Reinhold" > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: Coverage of Data Digest when using Header Digests > > Julian, > Is the Data Digest intended to cover whatever follows the 48 byte > iSCSI > header? In particular in a command frame which has a CDB > 16 bytes, uses > bidi, has immediate data, and is using both header and data digests - what > would the data digest cover? > - barry > reinhold -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Mon Jan 22 12:35:40 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA09719 for ; Mon, 22 Jan 2001 12:35:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0MEEs729592 for ips-outgoing; Mon, 22 Jan 2001 09:14:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0MEEn629587 for ; Mon, 22 Jan 2001 09:14:50 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id JAA14746; Mon, 22 Jan 2001 09:14:42 -0500 (EST) Message-ID: <3A6C40D7.A269EEE9@cisco.com> Date: Mon, 22 Jan 2001 08:16:55 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: David Robinson CC: ips@ece.cmu.edu Subject: Re: iSCSI-related conclusions from Orlando interim Meeting References: <0F31E5C394DAD311B60C00E029101A070410145F@corpmx9.isus.emc.com> <3A68E5C7.A3486D85@ebay.sun.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit That's right. The N&D team will send a new draft soon with the encodings. Some of the options for encoding binary data are hex, decimal, octal, and uuencode. We will probably suggest hex; it uses more space than unicode, but for most binary names, is the simplest and most readable. Please stay tuned. David Robinson wrote: > > Black_David@emc.com wrote: > > It's a bit cryptic. The conclusion in the room was > > to convert binary values to/from text representations > > for the purpose of negotiation (and use UTF-8 for the > > text). This was felt to be simpler than defining new > > formats for binary values. > > So if I want to send the binary data 0101101011110000 > I first turn that into text "5AF0" then encode that into > UTF-8 "5AF0" (printable ASCII is a no-op in UTF-8)? > > I'll buy that. > > Any statement as to the text encoding? Hex? Decimal? Octal? > > -David -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Mon Jan 22 12:35:52 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA09736 for ; Mon, 22 Jan 2001 12:35:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0MG30D03276 for ips-outgoing; Mon, 22 Jan 2001 11:03:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0MG2W603258 for ; Mon, 22 Jan 2001 11:02:32 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 22 Jan 2001 11:01:53 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101468@corpmx9.isus.emc.com> From: Black_David@emc.com To: santoshr@cup.hp.com, Black_David@emc.com, julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI-related conclusions from Orlando interim Meeting Date: Mon, 22 Jan 2001 11:01:51 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > I assume the below is referring to targets using NOP-IN as a NOP request in > order to test a connection (?) > > If so, I don't see how this can result in a loop. Both NOP-IN and NOP-OUT contain a flag requesting a response (i.e., send the other sort of NOP back). If every NOP has that flag set, the two sides will trade NOPs forever. Forbidding that flag to be set when a NOP is sent in response to a NOP is the most direct way of making sure this doesn't happen. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Jan 22 12:36:08 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA09748 for ; Mon, 22 Jan 2001 12:36:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0MG4v203347 for ips-outgoing; Mon, 22 Jan 2001 11:04:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0MG4N603325 for ; Mon, 22 Jan 2001 11:04:23 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id RAA09234 for ; Mon, 22 Jan 2001 17:04:16 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA52952 for ; Mon, 22 Jan 2001 17:04:16 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DC.005845DD ; Mon, 22 Jan 2001 17:04:10 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 22 Jan 2001 17:59:55 +0200 Subject: RE: Coverage of Data Digest when using Header Digests Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk yes, Julo "Barry Reinhold" on 22/01/2001 17:49:29 Please respond to "Barry Reinhold" To: mbakke@cisco.com, Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: RE: Coverage of Data Digest when using Header Digests Ok, So if the bidi-read length and ADDCDB fields are not in the data digest then I assume we have to figure them into the header digest even though they are located past the header digets. Is that the expected behavior? -----Original Message----- From: mbakke@cisco.com [mailto:mbakke@cisco.com] Sent: Monday, January 22, 2001 10:22 AM To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu; Barry Reinhold Subject: Re: Coverage of Data Digest when using Header Digests Barry- In particular, the data digest covers only the SCSI Data part of an iSCSI message; the header digest covers everything else. This means that in an 8k write, the data digest will cover only the 8k, and the header digest will cover everything else. Hope this helps, Mark julian_satran@il.ibm.com wrote: > > Barry, > > Considering that one of the reasons to have a separate header and data > digest was to enable data > to carried through proxies, virtualizers etc. the current thinking is that > the data digest will cover only the data and the header (including > extensions) will be covered by the header digest. > > Julo > > "Barry Reinhold" on 21/01/2001 21:50:04 > > Please respond to "Barry Reinhold" > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: Coverage of Data Digest when using Header Digests > > Julian, > Is the Data Digest intended to cover whatever follows the 48 byte > iSCSI > header? In particular in a command frame which has a CDB > 16 bytes, uses > bidi, has immediate data, and is using both header and data digests - what > would the data digest cover? > - barry > reinhold -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Mon Jan 22 12:36:20 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA09760 for ; Mon, 22 Jan 2001 12:36:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0MFp7w02790 for ips-outgoing; Mon, 22 Jan 2001 10:51:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from chmls05.mediaone.net (chmls05.mediaone.net [24.147.1.143]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0MFo9602736 for ; Mon, 22 Jan 2001 10:50:09 -0500 (EST) Received: from breinhold (h00e09871f366.ne.mediaone.net [24.128.216.253]) by chmls05.mediaone.net (8.11.1/8.11.1) with SMTP id f0MFnYS29510; Mon, 22 Jan 2001 10:49:34 -0500 (EST) From: "Barry Reinhold" To: , Cc: Subject: RE: Coverage of Data Digest when using Header Digests Date: Mon, 22 Jan 2001 10:49:29 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Importance: Normal In-Reply-To: <3A6C500E.CD92908@cisco.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Ok, So if the bidi-read length and ADDCDB fields are not in the data digest then I assume we have to figure them into the header digest even though they are located past the header digets. Is that the expected behavior? -----Original Message----- From: mbakke@cisco.com [mailto:mbakke@cisco.com] Sent: Monday, January 22, 2001 10:22 AM To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu; Barry Reinhold Subject: Re: Coverage of Data Digest when using Header Digests Barry- In particular, the data digest covers only the SCSI Data part of an iSCSI message; the header digest covers everything else. This means that in an 8k write, the data digest will cover only the 8k, and the header digest will cover everything else. Hope this helps, Mark julian_satran@il.ibm.com wrote: > > Barry, > > Considering that one of the reasons to have a separate header and data > digest was to enable data > to carried through proxies, virtualizers etc. the current thinking is that > the data digest will cover only the data and the header (including > extensions) will be covered by the header digest. > > Julo > > "Barry Reinhold" on 21/01/2001 21:50:04 > > Please respond to "Barry Reinhold" > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: Coverage of Data Digest when using Header Digests > > Julian, > Is the Data Digest intended to cover whatever follows the 48 byte > iSCSI > header? In particular in a command frame which has a CDB > 16 bytes, uses > bidi, has immediate data, and is using both header and data digests - what > would the data digest cover? > - barry > reinhold -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Mon Jan 22 12:49:29 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA10141 for ; Mon, 22 Jan 2001 12:49:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0MFx1Q03096 for ips-outgoing; Mon, 22 Jan 2001 10:59:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0MFwQ603072 for ; Mon, 22 Jan 2001 10:58:26 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 22 Jan 2001 10:58:21 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101467@corpmx9.isus.emc.com> From: Black_David@emc.com To: santoshr@cup.hp.com, Black_David@emc.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI-related conclusions from Orlando interim Meeting Date: Mon, 22 Jan 2001 10:58:17 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > > - Abort WARNING will be added to the draft. > > - Immediate Delivery of Aborts and similar task management commands > > may have unexpected results when multiple TCP connections are in use > > because Abort, Clear Task Set, etc. may bypass command(s) to be > > aborted/cleared on other TCP connections. > > - Ordered Delivery should be used instead when this is a concern. > > Is the above referring to ordered delivery as in CmdSN based ordering or > the use of the Ordered Task Tag ? It's CmdSN, not Task Tag. Santosh's issues appear to be with the Task Tag, which is not what is being pursued. Using the Task Tag in this fashion would cause all sorts of problems. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Jan 22 12:53:09 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA10271 for ; Mon, 22 Jan 2001 12:53:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0MFtvx02990 for ips-outgoing; Mon, 22 Jan 2001 10:55:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0MFtN602961 for ; Mon, 22 Jan 2001 10:55:24 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 22 Jan 2001 10:55:18 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101466@corpmx9.isus.emc.com> From: Black_David@emc.com To: santoshr@cup.hp.com, julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI: DataRN Date: Mon, 22 Jan 2001 10:55:16 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > The iSCSI draft should refrain from advocating any retry policies at the > iSCSI layer, such as being currently done for connection failures and > digest errors. If the above had said "at the SCSI layer", I'd agree. The issue being addressed here is hiding failure of a connection in an iSCSI session from SCSI (i.e., transparently recover at the iSCSI layer from failure of an iSCSI connection). The following discussion of retry is correct for SCSI, but doesn't apply to iSCSI because iSCSI will deliver the retried command to the SCSI layer at the device at most once, and hence the described problem caused by the command being executed at the device twice can't happen. > I/O retry policies are decided by the SCSI ULP based on the class of the > target. (disk, tape, changer, etc). > For a tape class of device, the SCSI ULP may not wish to retry on an error > [without a prior rewind operation]. In such situations, iSCSI attempting to > retry on connection failures or digest errors can result in problems with > sequential access type of media. OTOH, I'm sympathetic to the argument that it's up to an iSCSI implementation to decide how aggressively to recover a failed connection - notice that SCSI works quite happily over Fibre Channel where any individual command can be dropped without losing the session, and there's no retry (although FCP-2 has had to do some things for tape). The mechanisms needed for transparent recovery should be documented, and then we can figure out if they are MUST, SHOULD, or MAY implement. Getting back to the original point - the reason for dropping DataRN is that the gain from the optimization it provides for this sort of recovery situation doesn't seem to justify the added complexity. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Jan 22 14:19:05 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA12405 for ; Mon, 22 Jan 2001 14:19:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0MGCwF03667 for ips-outgoing; Mon, 22 Jan 2001 11:12:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0MGCG603642 for ; Mon, 22 Jan 2001 11:12:16 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id RAA119198 for ; Mon, 22 Jan 2001 17:12:08 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA114194 for ; Mon, 22 Jan 2001 17:12:08 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DC.0058FE00 ; Mon, 22 Jan 2001 17:12:01 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 22 Jan 2001 18:07:47 +0200 Subject: RE: iSCSI-related conclusions from Orlando interim Meeting Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Besides there is no way (no SAM defined way) to order task management functions. Task management is handled by the task manager while tasks (commands) are handled by the device server. Julo Black_David@emc.com on 22/01/2001 17:58:17 Please respond to Black_David@emc.com To: santoshr@cup.hp.com, Black_David@emc.com cc: ips@ece.cmu.edu Subject: RE: iSCSI-related conclusions from Orlando interim Meeting > > - Abort WARNING will be added to the draft. > > - Immediate Delivery of Aborts and similar task management commands > > may have unexpected results when multiple TCP connections are in use > > because Abort, Clear Task Set, etc. may bypass command(s) to be > > aborted/cleared on other TCP connections. > > - Ordered Delivery should be used instead when this is a concern. > > Is the above referring to ordered delivery as in CmdSN based ordering or > the use of the Ordered Task Tag ? It's CmdSN, not Task Tag. Santosh's issues appear to be with the Task Tag, which is not what is being pursued. Using the Task Tag in this fashion would cause all sorts of problems. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Jan 22 15:14:47 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA13452 for ; Mon, 22 Jan 2001 15:14:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0MHs1J08033 for ips-outgoing; Mon, 22 Jan 2001 12:54:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-1.cisco.com (sj-msg-core-1.cisco.com [171.71.163.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0MHqu607997 for ; Mon, 22 Jan 2001 12:52:56 -0500 (EST) Received: from sponge.cisco.com (sponge.cisco.com [171.71.61.25]) by sj-msg-core-1.cisco.com (8.9.3/8.9.1) with ESMTP id JAA13179; Mon, 22 Jan 2001 09:53:01 -0800 (PST) Received: from DAPW2K (dhcp-161-44-68-159.cisco.com [161.44.68.159]) by sponge.cisco.com (Mirapoint) with SMTP id AFT03813; Mon, 22 Jan 2001 11:52:40 -0600 (CST) From: "David Peterson" To: "Wayland Jeong" , "'Santosh Rao'" , "Sriram Rupanagunta" Cc: Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Mon, 22 Jan 2001 11:51:13 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit To make sure there are no misconceptions, this new proposal is targeted as a TCP/IP based solution. The field you have referred to is a reserved field that may be used for other protocols (e.g. UDP). Dave > Orlando, Brocade had a proposal for a new FCIP encapsulation that seemed to > include such a sequence number (albeit for a UDP implementation). Now, I'm > not saying this would be fun to implement, but the possibility exists. > From owner-ips@ece.cmu.edu Mon Jan 22 15:14:51 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA13456 for ; Mon, 22 Jan 2001 15:14:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0MHKwW06560 for ips-outgoing; Mon, 22 Jan 2001 12:20:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0MHKV606540 for ; Mon, 22 Jan 2001 12:20:31 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 4B8561004; Mon, 22 Jan 2001 09:20:30 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id JAA18684; Mon, 22 Jan 2001 09:20:25 -0800 (PST) From: Santosh Rao Message-Id: <200101221720.JAA18684@hpcuhe.cup.hp.com> Subject: Re: iSCSI: DataRN To: Black_David@emc.com Date: Mon, 22 Jan 2001 09:20:24 -0800 (PST) Cc: santoshr@cup.hp.com, julian_satran@il.ibm.com, ips@ece.cmu.edu In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704101466@corpmx9.isus.emc.com> from "Black_David@emc.com" at Jan 22, 2001 10:55:16 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > > > The iSCSI draft should refrain from advocating any retry policies at the > > iSCSI layer, such as being currently done for connection failures and > > digest errors. > > If the above had said "at the SCSI layer", I'd agree. The issue being > addressed here is hiding failure of a connection in an iSCSI session from > SCSI (i.e., transparently recover at the iSCSI layer from failure of an > iSCSI > connection). The following discussion of retry is correct for SCSI, but > doesn't > apply to iSCSI because iSCSI will deliver the retried command to the SCSI > layer at the device at most once, and hence the described problem caused > by the command being executed at the device twice can't happen. David, The above is correct if the digest error or connection failure occurred on delivery of the command. If a digest error were to be detected by an initiator on the response PDU (by which time the target has already completed the operation and the TCP layer at the initiator has already sent the ACK), then, the command is complete from the device perspective and should not be retried. Similarly, if the command had completed at the target and the Response PDU in transit was affected by a connection failure, retries should not be performed by the initiator. Rather than distingush these corner cases in its retry policies, iSCSI should refrain from advocating retries at its layer. Connection failures can be considered as a gross error and recovery can be performed at the SCSI ULP. Regards, Santosh > > > I/O retry policies are decided by the SCSI ULP based on the class of the > > target. (disk, tape, changer, etc). > > For a tape class of device, the SCSI ULP may not wish to retry on an error > > [without a prior rewind operation]. In such situations, iSCSI attempting > to > > retry on connection failures or digest errors can result in problems with > > sequential access type of media. > > OTOH, I'm sympathetic to the argument that it's up to an iSCSI > implementation > to decide how aggressively to recover a failed connection - notice that SCSI > works quite happily over Fibre Channel where any individual command can be > dropped without losing the session, and there's no retry (although FCP-2 has > had to do some things for tape). The mechanisms needed for transparent > recovery should be documented, and then we can figure out if they are MUST, > SHOULD, or MAY implement. Getting back to the original point - the reason > for > dropping DataRN is that the gain from the optimization it provides for this > sort > of recovery situation doesn't seem to justify the added complexity. > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Mon Jan 22 15:16:50 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA13488 for ; Mon, 22 Jan 2001 15:16:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0MHWwM06984 for ips-outgoing; Mon, 22 Jan 2001 12:32:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0MHWn606971 for ; Mon, 22 Jan 2001 12:32:49 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id DC9D86E2; Mon, 22 Jan 2001 09:32:48 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id JAA19499; Mon, 22 Jan 2001 09:32:44 -0800 (PST) From: Santosh Rao Message-Id: <200101221732.JAA19499@hpcuhe.cup.hp.com> Subject: Re: iSCSI-related conclusions from Orlando interim Meeting To: julian_satran@il.ibm.com Date: Mon, 22 Jan 2001 09:32:43 -0800 (PST) Cc: ips@ece.cmu.edu In-Reply-To: from "julian_satran@il.ibm.com" at Jan 22, 2001 08:03:43 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > I am reluctant to consider any procedure that originates a command at the > target. Julian, iSCSI targets are already allowed to originate Async Messages. This is not a new procedure being proposed. > > Note however that a target in trouble may drop a TCP connection and force > an initiator to recover. This does not communicate to the initiator the reason for the connection close. It is better to use an Async Message for this purpose than abruptly terminate the connection, for the same reason that one would rather use a Lgout than close a connection. [in the case where a Logout is sent on the same connection that is being cleaned up.] Regards, Santosh > > Santosh Rao on 21/01/2001 21:21:46 > > Please respond to Santosh Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: iSCSI-related conclusions from Orlando interim Meeting > > > On the subject of Async Message "Target requests a logout", which is a > ping-pong procedure to achieve a logout, it may be worth considering an > alternate Async Message in its place that indicates "Target is logging out > for error recovery" and allow the target to follow up the Async Message > with a connection close. > > This will simplify target driven connection level error recovery. > > Regards, > Santosh > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Mon Jan 22 16:58:08 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA15403 for ; Mon, 22 Jan 2001 16:58:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0MJvJR12926 for ips-outgoing; Mon, 22 Jan 2001 14:57:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mallory.overx.com (mallory.overx.com [63.82.155.179] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0MJuY612891 for ; Mon, 22 Jan 2001 14:56:35 -0500 (EST) Received: from everest.overx.com (everest.overx.com [63.93.29.10]) by mallory.overx.com (Postfix) with ESMTP id D0C56E37F; Mon, 22 Jan 2001 14:13:54 -0600 (CST) Received: from cs.uchicago.edu (marquette.overx.com [63.93.29.16]) by everest.overx.com (Postfix) with ESMTP id 22B37567F; Mon, 22 Jan 2001 13:56:14 -0600 (CST) To: ips@ece.cmu.edu Cc: somesh_gupta@silverbacksystems.com Subject: iSCSI: Re: InDataOrder (or not InDataOrder) In-Reply-To: Message from "Somesh Gupta" of "Tue, 16 Jan 2001 20:43:22 PST." References: Date: Mon, 22 Jan 2001 13:55:53 -0600 From: Stephen Bailey Message-Id: <20010122195614.22B37567F@everest.overx.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, > 1. Does any target return data for a read out of order? Althought the > potential benefit is there, in practice is it used? Historically, there HAVE been targets that return read data out of order. This history predates the widespread adoption of SCSI. I don't know any modern SCSI target that returns read data out of order. I expect this capability is primarily there for people from big, old companies, (with 3 letter names, one of which doesn't exist anymore) who remember those cool products where data (cylinders) read out-of-order substantially reduced random I/O latency, increasing TP throughput. I believe that's the rough story behind FCP's support of the feature, and we may as well keep bowing to the shrine. Certainly, the reasons to support it haven't gone away, but implementors still find so many other, easier avenues of improvement that it's unclear whether it ever will be implemented again. I've never seen the feature enabled. It's cute that I started this draft, and then Julian said: > Definitely - many high performance controllers, when they have cache > misses, return data out of order for operations that involve many > blocks. That's irony for you. Julian (or anybody else), can you please list some SCSI (of FC) controller/targets being sold today that actually do return read data out of order? That aside, there's really no reason to toss the feature if it might become useful in the future, particularly given that it has been around for time immemorial. Steph From owner-ips@ece.cmu.edu Mon Jan 22 17:03:48 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA15483 for ; Mon, 22 Jan 2001 17:03:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0MJ11N10805 for ips-outgoing; Mon, 22 Jan 2001 14:01:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail5.atl.bellsouth.net (mail5.atl.bellsouth.net [205.152.0.93]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0MJ0A610739 for ; Mon, 22 Jan 2001 14:00:10 -0500 (EST) Received: from sukhaghosh (host-216-78-37-119.ath.bellsouth.net [216.78.37.119]) by mail5.atl.bellsouth.net (3.3.5alt/0.75.2) with SMTP id OAA16203; Mon, 22 Jan 2001 14:03:08 -0500 (EST) Reply-To: From: "Sukha Ghosh" To: , Cc: "Barry Reinhold" Subject: RE: Coverage of Data Digest when using Header Digests Date: Mon, 22 Jan 2001 14:05:25 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, In this case of optional extended header (Header length > 48 bytes, AddCDB having non-zero value), since the digest coverage is through the Header digest, does it also mean that the Length field will cover the length after the header extension or the length is all the bytes except the 48 byte header and the padding bytes? Thanks Sukha -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Monday, January 22, 2001 1:25 AM To: ips@ece.cmu.edu Cc: Barry Reinhold Subject: Re: Coverage of Data Digest when using Header Digests Barry, Considering that one of the reasons to have a separate header and data digest was to enable data to carried through proxies, virtualizers etc. the current thinking is that the data digest will cover only the data and the header (including extensions) will be covered by the header digest. Julo "Barry Reinhold" on 21/01/2001 21:50:04 Please respond to "Barry Reinhold" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Coverage of Data Digest when using Header Digests Julian, Is the Data Digest intended to cover whatever follows the 48 byte iSCSI header? In particular in a command frame which has a CDB > 16 bytes, uses bidi, has immediate data, and is using both header and data digests - what would the data digest cover? - barry reinhold From owner-ips@ece.cmu.edu Mon Jan 22 20:03:54 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA18459 for ; Mon, 22 Jan 2001 20:03:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0MM9Pd18194 for ips-outgoing; Mon, 22 Jan 2001 17:09:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0MM8a618165 for ; Mon, 22 Jan 2001 17:08:36 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id RAA50828 for ; Mon, 22 Jan 2001 17:01:44 -0500 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id PAA35940 for ; Mon, 22 Jan 2001 15:08:30 -0700 Importance: Normal Subject: Re: iSCSI-related conclusions from Orlando interim Meeting To: ips@ece.cmu.edu From: "Jim Hafner" Date: Mon, 22 Jan 2001 14:08:29 -0800 Message-ID: X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.6 |December 14, 2000) at 01/22/2001 02:08:30 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Mark and David, Though we haven't discussed this within NDT, this issue is a bit broader (applicable to all key:value pairs in login and text messages). Some have suggested full C syntax, others have said that hex is sufficient. I'd like to support the two most important cases: 1) representation of 'integer' type values with represent a quantity (like length, size, count, etc.) 2) binary strings (as might occur in WWUIs, or other binary entities that don't necessarily represent a 'quantity' For the first, decimal seems the most natural. For the latter, hex seems best suited. For WWUI's, we've already weakly proposed a format that implies hex. Perhaps in other cases, the prefix '0x' or the prefix 'hex=' would be a good way to indicate non-decimal format. Jim Hafner Mark Bakke @ece.cmu.edu on 01-22-2001 06:16:55 AM Sent by: owner-ips@ece.cmu.edu To: David Robinson cc: ips@ece.cmu.edu Subject: Re: iSCSI-related conclusions from Orlando interim Meeting That's right. The N&D team will send a new draft soon with the encodings. Some of the options for encoding binary data are hex, decimal, octal, and uuencode. We will probably suggest hex; it uses more space than unicode, but for most binary names, is the simplest and most readable. Please stay tuned. David Robinson wrote: > > Black_David@emc.com wrote: > > It's a bit cryptic. The conclusion in the room was > > to convert binary values to/from text representations > > for the purpose of negotiation (and use UTF-8 for the > > text). This was felt to be simpler than defining new > > formats for binary values. > > So if I want to send the binary data 0101101011110000 > I first turn that into text "5AF0" then encode that into > UTF-8 "5AF0" (printable ASCII is a no-op in UTF-8)? > > I'll buy that. > > Any statement as to the text encoding? Hex? Decimal? Octal? > > -David -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Tue Jan 23 02:27:33 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA06546 for ; Tue, 23 Jan 2001 02:27:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0N4nFo29455 for ips-outgoing; Mon, 22 Jan 2001 23:49:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([208.50.99.84]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0N4mm629448 for ; Mon, 22 Jan 2001 23:48:48 -0500 (EST) Received: from cx418298b (cx418298-b.orng1.occa.home.com [24.1.179.117]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id UAA24638; Mon, 22 Jan 2001 20:48:36 -0800 (PST) Message-ID: <01bd01c084f9$1738dea0$75b30118@orng1.occa.home.com> From: "Murali Rajagopal" To: "David Robinson" , "IPS Reflector" References: <002401c08240$69231ac0$90c809c0@yp_portable.advansys.com> <3A68BB57.11B1356C@ebay.sun.com> Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Mon, 22 Jan 2001 20:58:08 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit A point of clarification on FCIP Ordering and multiple TCP connections: The FCIP draft does not preclude multiple TCP connections. Somesh is right in pointing that we could result in out-of-order if we have more than one TCP connections between same two FCIP Gateways. However, an FCIP gateway (say FCIP-A) can carry on simultaneous TCP connections say with FCIP-B and FCIP-C gateways without the danger of the out-of-order issue. Out-of-order is therotically possible even with FC Switch fabrics running a dynamic routing protocol such as FSPF, although in practice FC switches vendors seldom run into this condition. In summary, multiple TCP connections is not precluded but a solution is specified in the current FCIP draft. The authors of the FCIP draft will take an action to clarify this in the next version. Regards, Murali Rajagopal LightSand Communication ----- Original Message ----- From: "David Robinson" To: "IPS Reflector" Sent: Friday, January 19, 2001 2:10 PM Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports > Y P Cheng wrote: > > In the world where I live, iSCSI, iFCP, and FCIP will be implemented in a > > box or an adapter running RTOS or microcode with fresh new implementations. > > While it is essential to intemperate with the world that runs the existing > > TCP implementations, nothing prohibits the box and adapter to interoperate > > with each other running in "fast mode" in correct TCP packets as long as > > they obey the Internet fairness rule without creating so called "Super TCP". > > In my adapter, I don't have to live with any old TCP implementations. I > > asked often how do we streaming data on a 10 Gb/sec network with roundtrip > > time over 100 milliseconds? I would like to hear discussions providing > > answers to the above question. The statement "the TCP implementation > > guarantees in-order delivery and retries lost packets and has the necessary > > flow control and congestion avoidance" does not answer the question for me. > > In general I have not seen people in this WG constraining the design > based > on TCP implementations, in fact some have been very abstract in their > comments > and referring to what has been proven in theory (if not limited test > implementations) > that does not reflect current widely deployed implementations. > > If I read you correctly, you are asserting that there is a fundemental > problem > in the design on the TCP *protocol* which prevents it from taking > advantage > of a 10G/100ms network. If so what exactly do you see as a problem? > Since we > cannot (ips WG) cannot change TCP how should an IPS protocol work around > this > problem while still being friendly with other protocols? > > > If everyone agrees that this group can put iSCSI, iFCP, and FCIP together by > > assuming the current TCP implementations having all the solutions, please > > let me know. > > Conversely, if you feel that this group is designing to the TCP > implementations > instead of the protocol, please let us know. > > -David > > > From owner-ips@ece.cmu.edu Tue Jan 23 02:27:34 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA06547 for ; Tue, 23 Jan 2001 02:27:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0N58BW29905 for ips-outgoing; Tue, 23 Jan 2001 00:08:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0N57c629893 for ; Tue, 23 Jan 2001 00:07:38 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Mon, 22 Jan 2001 21:08:52 -0800 Message-ID: From: Charles Monia To: Matt Wakeley Cc: ips@ece.cmu.edu Subject: RE: iFCP as an IP Storage Work Item, Reset Date: Mon, 22 Jan 2001 21:06:56 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Matt: FWIW: The iFCP presentation clearly shows iFCP gateways at the terminus of the IP network. This can be confirmed from the PDF, which has been submitted for inclusion in the yet-to-be published proceedings. Charles > -----Original Message----- > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > Sent: Tuesday, January 16, 2001 7:53 AM > To: ips@ece.cmu.edu > Subject: Re: iFCP as an IP Storage Work Item, Reset > > > John Hufferd wrote: > > > The iFCP draft says that it is intended to be a Gateway to Gateway > > protocol. > > Just for clarification, that's certainly not the picture that > was presented in > San Diego. > > -Matt > From owner-ips@ece.cmu.edu Tue Jan 23 03:24:27 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id DAA07125 for ; Tue, 23 Jan 2001 03:24:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0N5lCc00921 for ips-outgoing; Tue, 23 Jan 2001 00:47:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0N5kb600906 for ; Tue, 23 Jan 2001 00:46:37 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 8664989; Mon, 22 Jan 2001 21:46:36 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id VAA08730; Mon, 22 Jan 2001 21:46:31 -0800 (PST) From: Santosh Rao Message-Id: <200101230546.VAA08730@hpcuhe.cup.hp.com> Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports To: muralir@lightsand.com (Murali Rajagopal) Date: Mon, 22 Jan 2001 21:46:31 -0800 (PST) Cc: David.Robinson@EBay.Sun.COM (David Robinson), ips@ece.cmu.edu (IPS Reflector) In-Reply-To: <01bd01c084f9$1738dea0$75b30118@orng1.occa.home.com> from "Murali Rajagopal" at Jan 22, 2001 08:58:08 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > > Out-of-order is therotically possible even with FC Switch fabrics running a > dynamic routing protocol such as FSPF, although in practice FC switches > vendors seldom run into this condition. A FC switch that guarantees in-order delivery by ACCing an FLOGI which is requesting sequential delivery and then causes out-of-order traffic is violating the FC-FS and FC-FLA standards. Out-of-order issues are avoided in FC by requesting in-order delivery in the class sp. params of FLOGI. - Santosh -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Tue Jan 23 09:37:53 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id JAA13183 for ; Tue, 23 Jan 2001 09:37:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NBhJr17113 for ips-outgoing; Tue, 23 Jan 2001 06:43:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NBgQ617095 for ; Tue, 23 Jan 2001 06:42:26 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id MAA38210 for ; Tue, 23 Jan 2001 12:42:19 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id MAA155578 for ; Tue, 23 Jan 2001 12:42:20 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DD.004048A8 ; Tue, 23 Jan 2001 12:42:09 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 23 Jan 2001 08:01:15 +0200 Subject: RE: Coverage of Data Digest when using Header Digests Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Barry, We are well aware of this catch-22. You'll have to assume the additional length field is OK and then check the CRC. Julo "Barry Reinhold" on 22/01/2001 18:40:06 Please respond to "Barry Reinhold" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: Coverage of Data Digest when using Header Digests Well, If that's the case then you have to use the information in the unchecked header to figure out the how to check the header. -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Monday, January 22, 2001 11:00 AM To: ips@ece.cmu.edu Subject: RE: Coverage of Data Digest when using Header Digests yes, Julo "Barry Reinhold" on 22/01/2001 17:49:29 Please respond to "Barry Reinhold" To: mbakke@cisco.com, Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: RE: Coverage of Data Digest when using Header Digests Ok, So if the bidi-read length and ADDCDB fields are not in the data digest then I assume we have to figure them into the header digest even though they are located past the header digets. Is that the expected behavior? -----Original Message----- From: mbakke@cisco.com [mailto:mbakke@cisco.com] Sent: Monday, January 22, 2001 10:22 AM To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu; Barry Reinhold Subject: Re: Coverage of Data Digest when using Header Digests Barry- In particular, the data digest covers only the SCSI Data part of an iSCSI message; the header digest covers everything else. This means that in an 8k write, the data digest will cover only the 8k, and the header digest will cover everything else. Hope this helps, Mark julian_satran@il.ibm.com wrote: > > Barry, > > Considering that one of the reasons to have a separate header and data > digest was to enable data > to carried through proxies, virtualizers etc. the current thinking is that > the data digest will cover only the data and the header (including > extensions) will be covered by the header digest. > > Julo > > "Barry Reinhold" on 21/01/2001 21:50:04 > > Please respond to "Barry Reinhold" > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: Coverage of Data Digest when using Header Digests > > Julian, > Is the Data Digest intended to cover whatever follows the 48 byte > iSCSI > header? In particular in a command frame which has a CDB > 16 bytes, uses > bidi, has immediate data, and is using both header and data digests - what > would the data digest cover? > - barry > reinhold -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Tue Jan 23 12:04:36 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA17859 for ; Tue, 23 Jan 2001 12:04:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NF5Pn22403 for ips-outgoing; Tue, 23 Jan 2001 10:05:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NF5J622395 for ; Tue, 23 Jan 2001 10:05:19 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA13154 for ; Tue, 23 Jan 2001 16:05:05 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA98738 for ; Tue, 23 Jan 2001 16:05:05 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DD.0052DAA0 ; Tue, 23 Jan 2001 16:04:59 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 23 Jan 2001 17:00:35 +0200 Subject: Re: iSCSI-related conclusions from Orlando interim Meeting Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I'll think about it. Julo Santosh Rao on 22/01/2001 19:32:43 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI-related conclusions from Orlando interim Meeting > I am reluctant to consider any procedure that originates a command at the > target. Julian, iSCSI targets are already allowed to originate Async Messages. This is not a new procedure being proposed. > > Note however that a target in trouble may drop a TCP connection and force > an initiator to recover. This does not communicate to the initiator the reason for the connection close. It is better to use an Async Message for this purpose than abruptly terminate the connection, for the same reason that one would rather use a Lgout than close a connection. [in the case where a Logout is sent on the same connection that is being cleaned up.] Regards, Santosh > > Santosh Rao on 21/01/2001 21:21:46 > > Please respond to Santosh Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: iSCSI-related conclusions from Orlando interim Meeting > > > On the subject of Async Message "Target requests a logout", which is a > ping-pong procedure to achieve a logout, it may be worth considering an > alternate Async Message in its place that indicates "Target is logging out > for error recovery" and allow the target to follow up the Async Message > with a connection close. > > This will simplify target driven connection level error recovery. > > Regards, > Santosh > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Tue Jan 23 12:13:11 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA18101 for ; Tue, 23 Jan 2001 12:13:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NENOY21025 for ips-outgoing; Tue, 23 Jan 2001 09:23:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NEMM620998 for ; Tue, 23 Jan 2001 09:22:22 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id PAA46950 for ; Tue, 23 Jan 2001 15:22:14 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id PAA154150 for ; Tue, 23 Jan 2001 15:22:14 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DD.004EED24 ; Tue, 23 Jan 2001 15:22:05 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 23 Jan 2001 16:17:40 +0200 Subject: Re: iSCSI-related conclusions from Orlando interim Meeting Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I support this. Julo "Jim Hafner" on 23/01/2001 00:08:29 Please respond to "Jim Hafner" To: ips@ece.cmu.edu cc: Subject: Re: iSCSI-related conclusions from Orlando interim Meeting Mark and David, Though we haven't discussed this within NDT, this issue is a bit broader (applicable to all key:value pairs in login and text messages). Some have suggested full C syntax, others have said that hex is sufficient. I'd like to support the two most important cases: 1) representation of 'integer' type values with represent a quantity (like length, size, count, etc.) 2) binary strings (as might occur in WWUIs, or other binary entities that don't necessarily represent a 'quantity' For the first, decimal seems the most natural. For the latter, hex seems best suited. For WWUI's, we've already weakly proposed a format that implies hex. Perhaps in other cases, the prefix '0x' or the prefix 'hex=' would be a good way to indicate non-decimal format. Jim Hafner Mark Bakke @ece.cmu.edu on 01-22-2001 06:16:55 AM Sent by: owner-ips@ece.cmu.edu To: David Robinson cc: ips@ece.cmu.edu Subject: Re: iSCSI-related conclusions from Orlando interim Meeting That's right. The N&D team will send a new draft soon with the encodings. Some of the options for encoding binary data are hex, decimal, octal, and uuencode. We will probably suggest hex; it uses more space than unicode, but for most binary names, is the simplest and most readable. Please stay tuned. David Robinson wrote: > > Black_David@emc.com wrote: > > It's a bit cryptic. The conclusion in the room was > > to convert binary values to/from text representations > > for the purpose of negotiation (and use UTF-8 for the > > text). This was felt to be simpler than defining new > > formats for binary values. > > So if I want to send the binary data 0101101011110000 > I first turn that into text "5AF0" then encode that into > UTF-8 "5AF0" (printable ASCII is a no-op in UTF-8)? > > I'll buy that. > > Any statement as to the text encoding? Hex? Decimal? Octal? > > -David -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Tue Jan 23 12:15:18 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA18210 for ; Tue, 23 Jan 2001 12:15:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NF3R322337 for ips-outgoing; Tue, 23 Jan 2001 10:03:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NF37622329 for ; Tue, 23 Jan 2001 10:03:07 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAA93002 for ; Tue, 23 Jan 2001 16:03:00 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA182264 for ; Tue, 23 Jan 2001 16:03:00 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DD.0052A813 ; Tue, 23 Jan 2001 16:02:49 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 23 Jan 2001 16:58:22 +0200 Subject: Re: iSCSI: DataRN Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk some comments in text - Julo Santosh Rao on 22/01/2001 19:20:24 Please respond to Santosh Rao To: Black_David@emc.com cc: santoshr@cup.hp.com, Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu Subject: Re: iSCSI: DataRN > > > The iSCSI draft should refrain from advocating any retry policies at the > > iSCSI layer, such as being currently done for connection failures and > > digest errors. > > If the above had said "at the SCSI layer", I'd agree. The issue being > addressed here is hiding failure of a connection in an iSCSI session from > SCSI (i.e., transparently recover at the iSCSI layer from failure of an > iSCSI > connection). The following discussion of retry is correct for SCSI, but > doesn't > apply to iSCSI because iSCSI will deliver the retried command to the SCSI > layer at the device at most once, and hence the described problem caused > by the command being executed at the device twice can't happen. David, The above is correct if the digest error or connection failure occurred on delivery of the command. If a digest error were to be detected by an initiator on the response PDU (by which time the target has already completed the operation and the TCP layer at the initiator has already sent the ACK), then, the command is complete from the device perspective and should not be retried. how would that happen ? Similarly, if the command had completed at the target and the Response PDU in transit was affected by a connection failure, retries should not be performed by the initiator. that is basically what StatSN is there to help to. The only thing the target has to do is resend the status is the ExpStatSN is not advancing. If there is no new command coming to the target - the later may want to enquire through a NOP before getting rid of the status Rather than distinguish these corner cases in its retry policies, iSCSI should refrain from advocating retries at its layer. Connection failures can be considered as a gross error and recovery can be performed at the SCSI ULP. That argument was heard far in the past and a consensus was reached that recovery at the iSCSI level for many of the transport errors would be valuable. If you read carefully the draft you will also see that implementations have a large degree of freedom in implementing recovery. The only open question, IMHO, is if we want to define classes (or profiles) of recovery - and I would say that given that the recovery is so basic classes are not worth the effort On basic issues it will also help the discussion if you get some "history context" from one of your colleagues or read some of the archived mail - or both Regards, Santosh > > > I/O retry policies are decided by the SCSI ULP based on the class of the > > target. (disk, tape, changer, etc). > > For a tape class of device, the SCSI ULP may not wish to retry on an error > > [without a prior rewind operation]. In such situations, iSCSI attempting > to > > retry on connection failures or digest errors can result in problems with > > sequential access type of media. > > OTOH, I'm sympathetic to the argument that it's up to an iSCSI > implementation > to decide how aggressively to recover a failed connection - notice that SCSI > works quite happily over Fibre Channel where any individual command can be > dropped without losing the session, and there's no retry (although FCP-2 has > had to do some things for tape). The mechanisms needed for transparent > recovery should be documented, and then we can figure out if they are MUST, > SHOULD, or MAY implement. Getting back to the original point - the reason > for > dropping DataRN is that the gain from the optimization it provides for this > sort > of recovery situation doesn't seem to justify the added complexity. > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Tue Jan 23 12:16:16 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA18230 for ; Tue, 23 Jan 2001 12:16:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NEkPj21704 for ips-outgoing; Tue, 23 Jan 2001 09:46:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NEk1621688 for ; Tue, 23 Jan 2001 09:46:01 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id JAA24524; Tue, 23 Jan 2001 09:45:52 -0500 (EST) Message-ID: <3A6D99A3.FBE3B81@cisco.com> Date: Tue, 23 Jan 2001 08:48:03 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: julian_satran@il.ibm.com CC: ips@ece.cmu.edu Subject: Re: iSCSI-related conclusions from Orlando interim Meeting References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I agree as well. Anything that's a number in a text message header should be decimal. Anything that is a binary string (WWUI, authentication info, etc) should be hex. -- Mark julian_satran@il.ibm.com wrote: > > I support this. > > Julo > > "Jim Hafner" on 23/01/2001 00:08:29 > > Please respond to "Jim Hafner" > > To: ips@ece.cmu.edu > cc: > Subject: Re: iSCSI-related conclusions from Orlando interim Meeting > > Mark and David, > > Though we haven't discussed this within NDT, this issue is a bit broader > (applicable to all key:value pairs in login and text messages). > > Some have suggested full C syntax, others have said that hex is sufficient. > > I'd like to support the two most important cases: > 1) representation of 'integer' type values with represent a quantity (like > length, size, count, etc.) > 2) binary strings (as might occur in WWUIs, or other binary entities that > don't necessarily represent a 'quantity' > > For the first, decimal seems the most natural. For the latter, hex seems > best suited. For WWUI's, we've already weakly proposed a format that > implies hex. Perhaps in other cases, the prefix '0x' or the prefix 'hex=' > would be a good way to indicate non-decimal format. > > Jim Hafner > > Mark Bakke @ece.cmu.edu on 01-22-2001 06:16:55 AM > > Sent by: owner-ips@ece.cmu.edu > > To: David Robinson > cc: ips@ece.cmu.edu > Subject: Re: iSCSI-related conclusions from Orlando interim Meeting > > That's right. The N&D team will send a new draft soon with the > encodings. Some of the options for encoding binary data are hex, > decimal, octal, and uuencode. We will probably suggest hex; it > uses more space than unicode, but for most binary names, is the > simplest and most readable. Please stay tuned. > > David Robinson wrote: > > > > Black_David@emc.com wrote: > > > It's a bit cryptic. The conclusion in the room was > > > to convert binary values to/from text representations > > > for the purpose of negotiation (and use UTF-8 for the > > > text). This was felt to be simpler than defining new > > > formats for binary values. > > > > So if I want to send the binary data 0101101011110000 > > I first turn that into text "5AF0" then encode that into > > UTF-8 "5AF0" (printable ASCII is a no-op in UTF-8)? > > > > I'll buy that. > > > > Any statement as to the text encoding? Hex? Decimal? Octal? > > > > -David > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Tue Jan 23 14:18:33 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA21969 for ; Tue, 23 Jan 2001 14:18:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NGLXU25011 for ips-outgoing; Tue, 23 Jan 2001 11:21:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NGKX624977 for ; Tue, 23 Jan 2001 11:20:33 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA47434 for ; Tue, 23 Jan 2001 17:20:26 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA82260 for ; Tue, 23 Jan 2001 17:20:26 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DD.0059C138 ; Tue, 23 Jan 2001 17:20:21 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 23 Jan 2001 18:15:58 +0200 Subject: RE: Coverage of Data Digest when using Header Digests Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Sukha, The length includes the ADCB and whatever fields are there after the 48 bytes. It is not "clean" ... nor final :)- Julo "Sukha Ghosh" on 22/01/2001 21:05:25 Please respond to sukha_ghosh@ivivity.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: "Barry Reinhold" Subject: RE: Coverage of Data Digest when using Header Digests Julian, In this case of optional extended header (Header length > 48 bytes, AddCDB having non-zero value), since the digest coverage is through the Header digest, does it also mean that the Length field will cover the length after the header extension or the length is all the bytes except the 48 byte header and the padding bytes? Thanks Sukha -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Monday, January 22, 2001 1:25 AM To: ips@ece.cmu.edu Cc: Barry Reinhold Subject: Re: Coverage of Data Digest when using Header Digests Barry, Considering that one of the reasons to have a separate header and data digest was to enable data to carried through proxies, virtualizers etc. the current thinking is that the data digest will cover only the data and the header (including extensions) will be covered by the header digest. Julo "Barry Reinhold" on 21/01/2001 21:50:04 Please respond to "Barry Reinhold" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Coverage of Data Digest when using Header Digests Julian, Is the Data Digest intended to cover whatever follows the 48 byte iSCSI header? In particular in a command frame which has a CDB > 16 bytes, uses bidi, has immediate data, and is using both header and data digests - what would the data digest cover? - barry reinhold From owner-ips@ece.cmu.edu Tue Jan 23 14:25:00 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA22133 for ; Tue, 23 Jan 2001 14:24:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NGRS825220 for ips-outgoing; Tue, 23 Jan 2001 11:27:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antipater.hosting.pacbell.net (antipater.hosting.pacbell.net [216.100.99.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NGRB625201 for ; Tue, 23 Jan 2001 11:27:12 -0500 (EST) Received: from VENKAT1 (64-160-62-200.rhapsodynetworks.com [64.160.62.200] (may be forged)) by antipater.hosting.pacbell.net id LAA03809; Tue, 23 Jan 2001 11:26:57 -0500 (EST) [ConcentricHost SMTP Relay 1.7] From: "Venkat Rangan" To: "Murali Rajagopal" , "David Robinson" , "IPS Reflector" Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Tue, 23 Jan 2001 08:29:22 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: <01bd01c084f9$1738dea0$75b30118@orng1.occa.home.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > Out-of-order is therotically possible even with FC Switch fabrics running a > dynamic routing protocol such as FSPF, although in practice FC switches > vendors seldom run into this condition. FC switches do not run into this condition by taking special steps. One approach is to institute a new route (in response to a link going down) after waiting for at least R_A_TOV time. This way, there are no frames within that time window whose route may change. It does mean less responsiveness to link failures, but it preserves the in-order semantics negotiated at FLOGI. Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Murali Rajagopal Sent: Monday, January 22, 2001 8:58 PM To: David Robinson; IPS Reflector Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports A point of clarification on FCIP Ordering and multiple TCP connections: The FCIP draft does not preclude multiple TCP connections. Somesh is right in pointing that we could result in out-of-order if we have more than one TCP connections between same two FCIP Gateways. However, an FCIP gateway (say FCIP-A) can carry on simultaneous TCP connections say with FCIP-B and FCIP-C gateways without the danger of the out-of-order issue. Out-of-order is therotically possible even with FC Switch fabrics running a dynamic routing protocol such as FSPF, although in practice FC switches vendors seldom run into this condition. In summary, multiple TCP connections is not precluded but a solution is specified in the current FCIP draft. The authors of the FCIP draft will take an action to clarify this in the next version. Regards, Murali Rajagopal LightSand Communication ----- Original Message ----- From: "David Robinson" To: "IPS Reflector" Sent: Friday, January 19, 2001 2:10 PM Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports > Y P Cheng wrote: > > In the world where I live, iSCSI, iFCP, and FCIP will be implemented in a > > box or an adapter running RTOS or microcode with fresh new implementations. > > While it is essential to intemperate with the world that runs the existing > > TCP implementations, nothing prohibits the box and adapter to interoperate > > with each other running in "fast mode" in correct TCP packets as long as > > they obey the Internet fairness rule without creating so called "Super TCP". > > In my adapter, I don't have to live with any old TCP implementations. I > > asked often how do we streaming data on a 10 Gb/sec network with roundtrip > > time over 100 milliseconds? I would like to hear discussions providing > > answers to the above question. The statement "the TCP implementation > > guarantees in-order delivery and retries lost packets and has the necessary > > flow control and congestion avoidance" does not answer the question for me. > > In general I have not seen people in this WG constraining the design > based > on TCP implementations, in fact some have been very abstract in their > comments > and referring to what has been proven in theory (if not limited test > implementations) > that does not reflect current widely deployed implementations. > > If I read you correctly, you are asserting that there is a fundemental > problem > in the design on the TCP *protocol* which prevents it from taking > advantage > of a 10G/100ms network. If so what exactly do you see as a problem? > Since we > cannot (ips WG) cannot change TCP how should an IPS protocol work around > this > problem while still being friendly with other protocols? > > > If everyone agrees that this group can put iSCSI, iFCP, and FCIP together by > > assuming the current TCP implementations having all the solutions, please > > let me know. > > Conversely, if you feel that this group is designing to the TCP > implementations > instead of the protocol, please let us know. > > -David > > > From owner-ips@ece.cmu.edu Tue Jan 23 15:36:06 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA23963 for ; Tue, 23 Jan 2001 15:36:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NHLSr27362 for ips-outgoing; Tue, 23 Jan 2001 12:21:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NHLH627351 for ; Tue, 23 Jan 2001 12:21:17 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id DG4KPXCT; Tue, 23 Jan 2001 09:21:15 -0800 From: "Y P Cheng" To: "Murali Rajagopal" , "David Robinson" , "IPS Reflector" Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Tue, 23 Jan 2001 09:19:48 -0800 Message-ID: <001a01c08560$b053a400$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: <01bd01c084f9$1738dea0$75b30118@orng1.occa.home.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > A point of clarification on FCIP Ordering and multiple TCP connections: > > The FCIP draft does not preclude multiple TCP connections. > Somesh is right in pointing that we could result in out-of-order if we > have more than one TCP connections between same two FCIP Gateways. Even with just one TCP connection, when frames travel through the "IP cloud", don't we risk out-of-order and duplicated frames at the other end? For multiple connections, it is a sure thing. > However, an FCIP gateway (say FCIP-A) can carry on simultaneous TCP > connections say with FCIP-B and FCIP-C gateways without the > danger of the out-of-order issue. FCIP-A can take responsibility for in-order delivery of outgoing frames. But, there is no guarantee for in-order and non-duplicated reception on either FCIP-B or FCIP-C. > Out-of-order is therotically possible even with FC Switch fabrics > running a dynamic routing protocol such as FSPF, although in practice > FC switches vendors seldom run into this condition. I am not an expert in implementing switches. But, in talking to both Finisar and Brocade, I was told that the switches do see this problem and ensure in-order delivery by buffering the frames. > In summary, multiple TCP connections is not precluded but a solution is > specified in the current FCIP draft. The authors of the FCIP > draft will take an action to clarify this in the next version. > > Murali Rajagopal > LightSand Communication I always thought that FCIP can count on an E-port for frame buffering and ordered delivery. If not, then, FCIP even using TCP must address the same problem of moving large amount of data on a 10Gb/100ms network like that of iFCP and iSCSI. One just can not assume that the TCP without help of RDMA header and frame marker will run fast on a 10Gb/100ms network. I thought FCIP by leverage the effort in the fibre channel switches gets away with the out-of-order reception problem. But, I am not a switch expert. Just my 2 cents' worth. Y.P. Cheng, ConnectCom Solutions. From owner-ips@ece.cmu.edu Tue Jan 23 16:44:33 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA25159 for ; Tue, 23 Jan 2001 16:44:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NIteb00893 for ips-outgoing; Tue, 23 Jan 2001 13:55:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NItE600855 for ; Tue, 23 Jan 2001 13:55:14 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id B9F508F3 for ; Tue, 23 Jan 2001 11:55:13 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 16224288 for ; Tue, 23 Jan 2001 13:55:00 -0500 (EST) Received: from agilent.com (cos1nai131036.cos.agilent.com [141.184.131.36]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id KAA26185 for ; Tue, 23 Jan 2001 10:54:57 -0800 (PST) Message-ID: <3A6DD37E.B3C1D29F@agilent.com> Date: Tue, 23 Jan 2001 10:54:54 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Reject PDU Concerns. References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > 2) An error_byte field should be considered for inclusion in the Reject > PDU which contains the byte offset of the errant byte[or word] in the > received header/payload that caused the target to send a Reject response > to the command. > > This is a simple solution that provides for quick fault isolation and > root cause of the reason for the reject. This also rids the Reject PDU > of any detailed elaboration of reason codes meant to allow root cause of > the reason for the reject. > > that is no big help since there can be more than one error and it > helps mostly in debugging. For implementation errors in the target and/or > initiator I am sure the protocol analyzer will be happy to help but running > implementation should not be burdened with. For those that are running > errors on good implementations we will provide some details when necessary > Julian, With only in-band framing [the markers that are currently defined in the spec], there is no way a protocol analyzer can be used in the traditional way. In order for the protocol analyzer to work, it would have to be continuously running since the tcp connection(s) was established. Besides, early on there will not be any protocol analyzers. -Matt From owner-ips@ece.cmu.edu Tue Jan 23 16:44:59 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA25174 for ; Tue, 23 Jan 2001 16:44:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NJ4SA01238 for ips-outgoing; Tue, 23 Jan 2001 14:04:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NJ4I601233 for ; Tue, 23 Jan 2001 14:04:19 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 07BD6AE3 for ; Tue, 23 Jan 2001 12:04:15 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id DB589138 for ; Tue, 23 Jan 2001 14:04:13 -0500 (EST) Received: from agilent.com (cos1nai131036.cos.agilent.com [141.184.131.36]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id LAA27225 for ; Tue, 23 Jan 2001 11:04:11 -0800 (PST) Message-ID: <3A6DD5A8.6753D2A2@agilent.com> Date: Tue, 23 Jan 2001 11:04:08 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports References: <001701c08282$af6a1be0$90c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit YP, You are correct that most people in *this* group understand the problems. However, *this* group cannot invent new TCP options. That is controlled by the end2end group, and *they* do not understand the problems (or if they do, they are unwilling to help solve them). -Matt Y P Cheng wrote: > > > If I read you correctly, you are asserting that there is a fundemental > > problem in the design on the TCP *protocol* which prevents it from taking > > advantage of a 10G/100ms network. If so what exactly do you see as a > problem? > > I have been very careful in stating the problem is in implementation. The > protocol is fine -- as the more and more I learn about the TCP options. I > think more new options such as framing, marking, NCK, SACK, and RDMA can be > defined and placed in login negotiation to allow the TCP protocol to run > well in 10G/100ms network. > > > Since we cannot (ips WG) cannot change TCP how should an IPS protocol > > work around this problem while still being friendly with other protocols? > > I do believe many people in this group do understand the problems. It new > options that facilitate the 10G/100ms network are negotiated, we can still > be friendly with older implementation which does not support the new > options. In such case, we just run slow. > > > > If everyone agrees that this group can put iSCSI, iFCP, and FCIP > together by > > > assuming the current TCP implementations having all the solutions, > please > > > let me know. > > > > Conversely, if you feel that this group is designing to the TCP > > implementations instead of the protocol, please let us know. > > > > -David > > I did sense that some people in this group were worry about compatibility > with older implementations and reluctant to discuss or add new TCP options. > In general, I do believe lots of people are quite up to speed, pun > unintended. My statements made in herein previous postings were saying that > two iSCSI or FCIP adapters of same kind -- with same TCP implementations -- > should be able to run the new options. > > Y.P. From owner-ips@ece.cmu.edu Tue Jan 23 16:46:26 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA25194 for ; Tue, 23 Jan 2001 16:46:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NItgH00898 for ips-outgoing; Tue, 23 Jan 2001 13:55:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NIsK600813 for ; Tue, 23 Jan 2001 13:54:20 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id CD4271165; Tue, 23 Jan 2001 10:54:19 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id KAA23295; Tue, 23 Jan 2001 10:54:15 -0800 (PST) Message-ID: <3A6DD3E2.F63CC265@cup.hp.com> Date: Tue, 23 Jan 2001 10:56:34 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI: DataRN References: Content-Type: multipart/mixed; boundary="------------95651E28C166BED42D091BB4" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------95651E28C166BED42D091BB4 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > > The above is correct if the digest error or connection failure occurred on > delivery of the command. If a digest error were to be detected by an > initiator on the response PDU (by which time the target has already > completed the operation and the TCP layer at the initiator has already > sent the ACK), then, the command is complete from the device perspective > and should not be retried. > > how would that happen ? Julian, If the initiator detected a digest error on the Response PDU[and the target has completed all or part of the data phase], such an operation cannot be retried without a prior rewind command, in the case of sequential media. The iSCSI layer cannot retry commands destined to sequential media when there is a possibility that some or all of the data phase is complete. Regards, Santosh --------------95651E28C166BED42D091BB4 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------95651E28C166BED42D091BB4-- From owner-ips@ece.cmu.edu Tue Jan 23 16:56:22 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA25276 for ; Tue, 23 Jan 2001 16:56:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NJDVK01570 for ips-outgoing; Tue, 23 Jan 2001 14:13:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NJDL601558 for ; Tue, 23 Jan 2001 14:13:21 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0NKNu049896; Tue, 23 Jan 2001 12:23:58 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Murali Rajagopal" , "David Robinson" , "IPS Reflector" Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Tue, 23 Jan 2001 11:12:02 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <01bd01c084f9$1738dea0$75b30118@orng1.occa.home.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Murali, To allow detection of stale frames due from a disrupted IP network and to maintaining ordering on multiple connections, a local time-stamp that resolves to the packet rate would solve both these issues. To ensure ordered delivery, especially with multiple connections, both situations must be accommodated. The R_A_TOV fabric time-outs are meaningless unless there is a means to police stale frames finding their way across IP space. A locally generated time-stamp would allow a means to both ensure ordering over multiple connections as well as a means of rejecting stale frames held up by some IP disruption otherwise frame ordering is not assured. Doug > A point of clarification on FCIP Ordering and multiple TCP connections: > > The FCIP draft does not preclude multiple TCP connections. > Somesh is right in pointing that we could result in out-of-order if we > have more than one TCP connections between same two FCIP Gateways. > > However, an FCIP gateway (say FCIP-A) can carry on simultaneous TCP > connections > say with FCIP-B and FCIP-C gateways without the danger of the out-of-order > issue. > > Out-of-order is therotically possible even with FC Switch fabrics > running a > dynamic routing protocol such as FSPF, although in practice FC switches > vendors seldom run into this condition. > > In summary, multiple TCP connections is not precluded but a solution is > specified in the current FCIP draft. The authors of the FCIP > draft will take > an action to clarify this in the next version. > > Regards, > > Murali Rajagopal > LightSand Communication > > ----- Original Message ----- > From: "David Robinson" > To: "IPS Reflector" > Sent: Friday, January 19, 2001 2:10 PM > Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports > > > > Y P Cheng wrote: > > > In the world where I live, iSCSI, iFCP, and FCIP will be > implemented in > a > > > box or an adapter running RTOS or microcode with fresh new > implementations. > > > While it is essential to intemperate with the world that runs the > existing > > > TCP implementations, nothing prohibits the box and adapter to > interoperate > > > with each other running in "fast mode" in correct TCP packets > as long as > > > they obey the Internet fairness rule without creating so called "Super > TCP". > > > In my adapter, I don't have to live with any old TCP > implementations. I > > > asked often how do we streaming data on a 10 Gb/sec network with > roundtrip > > > time over 100 milliseconds? I would like to hear discussions > providing > > > answers to the above question. The statement "the TCP implementation > > > guarantees in-order delivery and retries lost packets and has the > necessary > > > flow control and congestion avoidance" does not answer the > question for > me. > > > > In general I have not seen people in this WG constraining the design > > based > > on TCP implementations, in fact some have been very abstract in their > > comments > > and referring to what has been proven in theory (if not limited test > > implementations) > > that does not reflect current widely deployed implementations. > > > > If I read you correctly, you are asserting that there is a fundemental > > problem > > in the design on the TCP *protocol* which prevents it from taking > > advantage > > of a 10G/100ms network. If so what exactly do you see as a problem? > > Since we > > cannot (ips WG) cannot change TCP how should an IPS protocol work around > > this > > problem while still being friendly with other protocols? > > > > > If everyone agrees that this group can put iSCSI, iFCP, and FCIP > together by > > > assuming the current TCP implementations having all the solutions, > please > > > let me know. > > > > Conversely, if you feel that this group is designing to the TCP > > implementations > > instead of the protocol, please let us know. > > > > -David > > > > > > > > From owner-ips@ece.cmu.edu Tue Jan 23 16:57:32 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA25291 for ; Tue, 23 Jan 2001 16:57:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NJXUN02299 for ips-outgoing; Tue, 23 Jan 2001 14:33:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NJWq602273 for ; Tue, 23 Jan 2001 14:32:53 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id AEC931C52; Tue, 23 Jan 2001 11:32:51 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id LAA27922; Tue, 23 Jan 2001 11:32:46 -0800 (PST) Message-ID: <3A6DDCE9.777D86FD@cup.hp.com> Date: Tue, 23 Jan 2001 11:35:06 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Matt Wakeley , Julian Satran Cc: IPS Reflector Subject: Re: iSCSI : Reject PDU Concerns. References: <3A6DD37E.B3C1D29F@agilent.com> Content-Type: multipart/mixed; boundary="------------FCFEC5D4F1A6A17AFD8ACD7B" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------FCFEC5D4F1A6A17AFD8ACD7B Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Matt Wakeley wrote: > julian_satran@il.ibm.com wrote: > > > 2) An error_byte field should be considered for inclusion in the Reject > > PDU which contains the byte offset of the errant byte[or word] in the > > received header/payload that caused the target to send a Reject response > > to the command. > > > > This is a simple solution that provides for quick fault isolation and > > root cause of the reason for the reject. This also rids the Reject PDU > > of any detailed elaboration of reason codes meant to allow root cause of > > the reason for the reject. > > > > that is no big help since there can be more than one error and it > > helps mostly in debugging. Julian, 1) The iSCSI layer, on parsing a PDU, will stop on detecting the first error. So, there should really be only one error to report and that is the first one discovered. 2) Helping in debugging is a big issue since one of the major problems iSCSI implementations will need to deal with is inter-operability. This is not a one-time problem as new implmentations of devices and initiators keep coming along. The iSCSI draft must attempt to provide some aids for quick fault isolation. 3) The cost of providing this error_byte based fault indication feature is really minimal, compared to the benefits it provides. Regards, Santosh > For implementation errors in the target and/or > > initiator I am sure the protocol analyzer will be happy to help but running > > implementation should not be burdened with. For those that are running > > errors on good implementations we will provide some details when necessary > > > > Julian, > > With only in-band framing [the markers that are currently defined in the > spec], there is no way a protocol analyzer can be used in the traditional > way. In order for the protocol analyzer to work, it would have to be > continuously running since the tcp connection(s) was established. Besides, > early on there will not be any protocol analyzers. > > -Matt --------------FCFEC5D4F1A6A17AFD8ACD7B Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------FCFEC5D4F1A6A17AFD8ACD7B-- From owner-ips@ece.cmu.edu Tue Jan 23 19:08:47 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA26745 for ; Tue, 23 Jan 2001 19:08:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NLKff06427 for ips-outgoing; Tue, 23 Jan 2001 16:20:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NLKK606393 for ; Tue, 23 Jan 2001 16:20:20 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 369048A5 for ; Tue, 23 Jan 2001 14:20:19 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 9379D250 for ; Tue, 23 Jan 2001 16:20:17 -0500 (EST) Received: from agilent.com (cos1nai131036.cos.agilent.com [141.184.131.36]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id NAA10094 for ; Tue, 23 Jan 2001 13:20:13 -0800 (PST) Message-ID: <3A6DF588.B88657D@agilent.com> Date: Tue, 23 Jan 2001 13:20:08 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Ordered delivery capability notification References: <3A5D2C4A.EB8CC024@hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit A better solution is to change "SHOULD deliver the commands to the target SCSI layer in the order specified by CmdSN" to "MUST". -Matt Pierre Labat wrote: > > Julian, > > Where: > ====== > > 1.2.2.1 Command numbering > > "iSCSI supports ordered command delivery within a session" > > "The target iSCSI layer SHOULD deliver the commands to the target > SCSI layer in the order specified by CmdRN". > > Problem: > ======= > An initiator is unable to know if the iSCSI target layer support ordered > > command delivery. Hence it cannot trust iSCSI to have ordered delivery. > > Hence all the iSCSI ordering mechanism becomes useless because > it can't be trusted. > The problem is not that iSCSI target layers can't support ordered > delivery, > the problem is that the initiator/application doesn't know anything > about it. > > Solution: > ====== > > a) Add a Login/Text key > Ordered: > > b) Put the explanation in for the key: > "If the target answered "yes" and is unable to maintain the order for > whatever reason, it must notify the initiator and drop the session" > > If the target answered "yes", when it receives a "retried" command > (retry bit), > it must be able to retry the command in such a way that the effect > on the initiator and target device when the command completes > is the same as if the command would have not needed to be retried. > This doesn't mean always restart from where left, a READ > can have all the data re-transmitted because it is transparent > a completion time. From owner-ips@ece.cmu.edu Tue Jan 23 19:09:00 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA26766 for ; Tue, 23 Jan 2001 19:08:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NLHkq06282 for ips-outgoing; Tue, 23 Jan 2001 16:17:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NLGW606240 for ; Tue, 23 Jan 2001 16:16:32 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 83938808 for ; Tue, 23 Jan 2001 14:16:31 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 946772B1 for ; Tue, 23 Jan 2001 16:16:29 -0500 (EST) Received: from agilent.com (cos1nai131036.cos.agilent.com [141.184.131.36]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id NAA09471 for ; Tue, 23 Jan 2001 13:16:21 -0800 (PST) Message-ID: <3A6DF4A1.5B1D3445@agilent.com> Date: Tue, 23 Jan 2001 13:16:17 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI: remove CDB from iSCSI header References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I don't think I like this. Header followed by digest followed by more header. I think this is a good opportunity to think about removing the CDB from the header. This will reduce the amount of dead space used by headers that do not contain a CDB. This will be beneficial when sending multiple smaller PDUs (in order to keep the CRC coverage high) by reducing the iSCSI header overhead. Change the iSCSI command so that there is an iSCSI header (verified by the header digest) followed by CDB "payload" (verified by the data digest). No options for immediate data. Keep it simple, like it is in Fibre Channel. -Matt Wakeley Agilent Technologies julian_satran@il.ibm.com wrote: > > yes, > > Julo > > "Barry Reinhold" on 22/01/2001 17:49:29 > > Please respond to "Barry Reinhold" > > To: mbakke@cisco.com, Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: RE: Coverage of Data Digest when using Header Digests > > Ok, > So if the bidi-read length and ADDCDB fields are not in the data > digest > then I assume we have to figure them into the header digest even though > they > are located past the header digets. Is that the expected behavior? > > -----Original Message----- > From: mbakke@cisco.com [mailto:mbakke@cisco.com] > Sent: Monday, January 22, 2001 10:22 AM > To: julian_satran@il.ibm.com > Cc: ips@ece.cmu.edu; Barry Reinhold > Subject: Re: Coverage of Data Digest when using Header Digests > > Barry- > > In particular, the data digest covers only the SCSI Data part of an > iSCSI message; the header digest covers everything else. This means > that in an 8k write, the data digest will cover only the 8k, and the > header digest will cover everything else. > > Hope this helps, > > Mark > > julian_satran@il.ibm.com wrote: > > > > Barry, > > > > Considering that one of the reasons to have a separate header and data > > digest was to enable data > > to carried through proxies, virtualizers etc. the current thinking is > that > > the data digest will cover only the data and the header (including > > extensions) will be covered by the header digest. > > > > Julo > > > > "Barry Reinhold" on 21/01/2001 21:50:04 > > > > Please respond to "Barry Reinhold" > > > > To: Julian Satran/Haifa/IBM@IBMIL > > cc: > > Subject: Coverage of Data Digest when using Header Digests > > > > Julian, > > Is the Data Digest intended to cover whatever follows the 48 byte > > iSCSI > > header? In particular in a command frame which has a CDB > 16 bytes, uses > > bidi, has immediate data, and is using both header and data digests - > what > > would the data digest cover? > > - barry > > reinhold > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 From owner-ips@ece.cmu.edu Tue Jan 23 19:10:17 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA26787 for ; Tue, 23 Jan 2001 19:10:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NLHdl06278 for ips-outgoing; Tue, 23 Jan 2001 16:17:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NLGf606247 for ; Tue, 23 Jan 2001 16:16:42 -0500 (EST) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id DG4KPX3T; Tue, 23 Jan 2001 13:16:36 -0800 From: "Y P Cheng" To: "Venkat Rangan" , "IPS Reflector" Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Tue, 23 Jan 2001 13:15:08 -0800 Message-ID: <001201c08581$902086a0$90c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > FC switches do not run into this condition by taking special steps. One > approach is to institute a new route (in response to a link going down) > after waiting for at least R_A_TOV time. This way, there are no frames > within that time window whose route may change. It does mean less > responsiveness to link failures, but it preserves the in-order semantics > negotiated at FLOGI. > > Venkat Rangan If I read you correctly, even switches do not buffer the frames to ensure in-order delivery. Instead it relies on R_A_TOV timeout. So, it always passes a sequence of in-order incoming frames quickly to an ISL. If the ISL fails, it waits until R_A_TOV timeout that forces all unfinished sequences being thrown away, then, allocates a new link. Does the switch perform any checking for out-of-order incoming frames? Or, is it simply garbage-in-garbage-out? Now, if all iFCP, iSCSI, and FCIP must rely on TCP implementation for ordered delivery, I am the poor guy trying real hard to put TCP in the microcode of the I/O interface chips in every host, target, and switch port. If I solve the performance problem of TCP on a 10Gb/100Ms network without requiring 50 megabytes of reassembly buffer for every single port, I can corner this market? This is becomes more interesting every minute. Thank God, that should a standard that allows me to write the microcode for this chip only once. Other switch experts care to comment? Y.P. Cheng, ConnectCom Solutions. From owner-ips@ece.cmu.edu Tue Jan 23 19:10:29 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA26798 for ; Tue, 23 Jan 2001 19:10:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NLZWw06959 for ips-outgoing; Tue, 23 Jan 2001 16:35:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NLZ9606924 for ; Tue, 23 Jan 2001 16:35:13 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 44938B10 for ; Tue, 23 Jan 2001 14:35:09 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 52EE91C3 for ; Tue, 23 Jan 2001 16:35:07 -0500 (EST) Received: from agilent.com (cos1nai131036.cos.agilent.com [141.184.131.36]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id NAA10684 for ; Tue, 23 Jan 2001 13:34:59 -0800 (PST) Message-ID: <3A6DF8FF.FD1C4577@agilent.com> Date: Tue, 23 Jan 2001 13:34:55 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iscsi : Fragmentation & Reassembly issues in iSCSI. References: <3A678E89.1F694E2@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Ok, There is a DataPDULength which specifies max size of a *DATA* PDU, and a PingMaxReplyLength field which indicates how much buffer space an implementation is willing to dedicate to Ping messages. Text strings that will not fit in a single Text command such that it's <= DataPDULength can be sent as multiple text commands. That leaves Login messages. Since these maximum length fields are exchanged during login, there really isn't a maximum (unless there is a default value) to specify the maximum length of the login before the login. So, the login should just be sent as a single message, regardless of it's length. Forget all this F bit stuff. -Matt From owner-ips@ece.cmu.edu Tue Jan 23 19:12:30 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA26850 for ; Tue, 23 Jan 2001 19:12:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NLpX707495 for ips-outgoing; Tue, 23 Jan 2001 16:51:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NLoY607461 for ; Tue, 23 Jan 2001 16:50:34 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0NN1H050018; Tue, 23 Jan 2001 15:01:18 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Matt Wakeley" , "IPS Reflector" Subject: RE: iSCSI : Reject PDU Concerns. Date: Tue, 23 Jan 2001 13:49:22 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <3A6DD37E.B3C1D29F@agilent.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt, Unless encoding headers to hide typical PDU signatures, an analyzer will have little difficulty in discovering PDU boundaries. Should a scheme force PDU alignment every 8192 bytes as example, discovery would be a certainty within 8k bytes. To prevent blockage, the size of the PDU is limited and so the search is not impossible as you imply nor even difficult. Either markers or alignment seems a good solution for minimizing impact of segment loss to that of the interval rather than 2RTT x rate. For either a periodic marker or alignment scheme to work for sending, no changes to TCP is required. As for the diagnostics, I would suggest there is no reason to encode data into decimal text for the same reason and so a hexadecimal encoding should be the default for all numeric encoded settings. This information is not for human consumption and such decimal conversion requires additional processing for dubious benefit. If there is an analyzer dissecting content, such human aid can be added much as text is appended to a code dump. Doug > julian_satran@il.ibm.com wrote: > > > 2) An error_byte field should be considered for inclusion in the Reject > > PDU which contains the byte offset of the errant byte[or word] in the > > received header/payload that caused the target to send a Reject response > > to the command. > > > > This is a simple solution that provides for quick fault isolation and > > root cause of the reason for the reject. This also rids the Reject PDU > > of any detailed elaboration of reason codes meant to allow root cause of > > the reason for the reject. > > > > that is no big help since there can be more than one error and it > > helps mostly in debugging. For implementation errors in the > target and/or > > initiator I am sure the protocol analyzer will be happy to help > but running > > implementation should not be burdened with. For those that are running > > errors on good implementations we will provide some details > when necessary > > > > Julian, > > With only in-band framing [the markers that are currently defined in the > spec], there is no way a protocol analyzer can be used in the traditional > way. In order for the protocol analyzer to work, it would have to be > continuously running since the tcp connection(s) was > established. Besides, > early on there will not be any protocol analyzers. > > -Matt > From owner-ips@ece.cmu.edu Tue Jan 23 20:18:31 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA27561 for ; Tue, 23 Jan 2001 20:18:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NMmaD09631 for ips-outgoing; Tue, 23 Jan 2001 17:48:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NMmV609625 for ; Tue, 23 Jan 2001 17:48:31 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id RAA18909; Tue, 23 Jan 2001 17:48:25 -0500 (EST) Message-ID: <3A6E0ABA.D33FD3A2@cisco.com> Date: Tue, 23 Jan 2001 16:50:34 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Matt Wakeley CC: ips@ece.cmu.edu Subject: Re: iSCSI: remove CDB from iSCSI header References: <3A6DF4A1.5B1D3445@agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I agree with Matt. Basically, then, the iSCSI header digest should cover everything in the first 48 bytes. All of the messages, with the exception of the SCSI Command Request, could easily be done this way, and I think that we should modify that to fit Matt's idea of treating the CDB (and sense data) the same as SCSI data for digest purposes. Here's what they would all look like: 01 - SCSI Command Request The iSCSI header digest currently covers the 48-byte iSCSI header, which includes the CDB. The data digest covers everything else, which includes extended CDB, bidi-read transfer length, and immediate data. Since the command request is the only operation that really breaks the notion of Matt's model, I would suggest that we start the CDB after the 48-byte iSCSI header, and that it not be covered by the header digest. We can move the bidi-read length inside the iSCSI header, and end up with: +---------------------------------------------------- 0| Normal iSCSI header stuff for command request +---------------------------------------------------- 32| Bidi-read length +-------------------------------------------------- 36/ Reserved +/ +--------------------------------------------------- | (optional header digest goes here) +------------------------------------------------- 48| CDB +-------------------------------------------------- | Additional CDB +--------------------------------------------------- | Immediate Data +---------------------------------------------------- | (optional data digest goes here) +----------------------------------------------------- The header digest would cover the first 48 bytes, and would exactly match the rest of the iSCSI operations. The data digest would include the CDB (now contiguous with the additional CDB, as an added bonus) and the immediate data. Anyone concerned with having a separate data integrity check for just the data when an additional CDB is being used should refrain from using the immediate data feature. While we are on the topic, is anyone really planning to make use of bidi-read and immediate data?  Here is my understanding of what the digests cover with the remainder of the iSCSI operations: 81 - SCSI Response 91 - Asynchronous Event iSCSI header digest covers 48-byte iSCSI header iSCSI data digest covers the sense data, if present 02 - Task management command 82 - Task management response 06 - Logout command 86 - Logout response 90 - R2T iSCSI header digest covers 48-byte header no data, so no data digest present 05 - SCSI Data write 85 - SCSI Data read 00 - NOP-Out 80 - NOP-In iSCSI header digest covers 48-byte header iSCSI data digest covers the data 04 - Text command 84 - Text response 03 - Login command 83 - Login response iSCSI header digest covers 48-byte iSCSI header iSCSI data digest covers the text parameters ??? Should the text be padded to a 4-byte boundary before adding the data CRC? I think that it already says so somewhere in the spec. -- Mark Matt Wakeley wrote: > > I don't think I like this. Header followed by digest followed by more header. > > I think this is a good opportunity to think about removing the CDB from the > header. This will reduce the amount of dead space used by headers that do not > contain a CDB. This will be beneficial when sending multiple smaller PDUs (in > order to keep the CRC coverage high) by reducing the iSCSI header overhead. > > Change the iSCSI command so that there is an iSCSI header (verified by the > header digest) followed by CDB "payload" (verified by the data digest). No > options for immediate data. Keep it simple, like it is in Fibre Channel. > > -Matt Wakeley > Agilent Technologies > > julian_satran@il.ibm.com wrote: > > > > yes, > > > > Julo > > > > "Barry Reinhold" on 22/01/2001 17:49:29 > > > > Please respond to "Barry Reinhold" > > > > To: mbakke@cisco.com, Julian Satran/Haifa/IBM@IBMIL > > cc: ips@ece.cmu.edu > > Subject: RE: Coverage of Data Digest when using Header Digests > > > > Ok, > > So if the bidi-read length and ADDCDB fields are not in the data > > digest > > then I assume we have to figure them into the header digest even though > > they > > are located past the header digets. Is that the expected behavior? > > > > -----Original Message----- > > From: mbakke@cisco.com [mailto:mbakke@cisco.com] > > Sent: Monday, January 22, 2001 10:22 AM > > To: julian_satran@il.ibm.com > > Cc: ips@ece.cmu.edu; Barry Reinhold > > Subject: Re: Coverage of Data Digest when using Header Digests > > > > Barry- > > > > In particular, the data digest covers only the SCSI Data part of an > > iSCSI message; the header digest covers everything else. This means > > that in an 8k write, the data digest will cover only the 8k, and the > > header digest will cover everything else. > > > > Hope this helps, > > > > Mark > > > > julian_satran@il.ibm.com wrote: > > > > > > Barry, > > > > > > Considering that one of the reasons to have a separate header and data > > > digest was to enable data > > > to carried through proxies, virtualizers etc. the current thinking is > > that > > > the data digest will cover only the data and the header (including > > > extensions) will be covered by the header digest. > > > > > > Julo > > > > > > "Barry Reinhold" on 21/01/2001 21:50:04 > > > > > > Please respond to "Barry Reinhold" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL > > > cc: > > > Subject: Coverage of Data Digest when using Header Digests > > > > > > Julian, > > > Is the Data Digest intended to cover whatever follows the 48 byte > > > iSCSI > > > header? In particular in a command frame which has a CDB > 16 bytes, uses > > > bidi, has immediate data, and is using both header and data digests - > > what > > > would the data digest cover? > > > - barry > > > reinhold > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Tue Jan 23 20:20:36 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA27581 for ; Tue, 23 Jan 2001 20:20:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NMRdW08837 for ips-outgoing; Tue, 23 Jan 2001 17:27:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0N5UR600562 for ; Tue, 23 Jan 2001 00:30:27 -0500 (EST) Received: from divyaroot.India.Sun.COM ([129.158.225.35]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id VAA06939 for ; Mon, 22 Jan 2001 21:30:24 -0800 (PST) Received: from helix (helix [129.158.226.51]) by divyaroot.India.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1) with SMTP id LAA21295 for ; Tue, 23 Jan 2001 11:00:23 +0530 (IST) Message-Id: <200101230530.LAA21295@divyaroot.India.Sun.COM> Date: Tue, 23 Jan 2001 11:02:05 -0500 (GMT) From: Raghavendra Rao Reply-To: Raghavendra Rao Subject: iSCSI: INQUIRY page 0x83 identifier To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: AoXg0dORyv6jZ/TIoL1UPw== Sender: owner-ips@ece.cmu.edu Precedence: bulk This may seem like a T10 issue at a high level, but I'm concerned that SPC-2 (SCSI primary command set) defines in the INQUIRY page 0x83 identifier type field, a FC identifier type (value = 3). What identifer type should an iSCSI target use to report a LU identifier when queried for page 0x83 ? Luckily there seem to be a generic IEEE extended identifier type (value = 2). Is the naming and discovery document going to provide a clear defintion about LU names ? I understand that there should be NO interconnect specific identifier at the SCSI layer - But SPC-2 is already polluted with a FC identifier type. -JP From owner-ips@ece.cmu.edu Tue Jan 23 20:20:56 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA27592 for ; Tue, 23 Jan 2001 20:20:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NMTYN08917 for ips-outgoing; Tue, 23 Jan 2001 17:29:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NMSf608872 for ; Tue, 23 Jan 2001 17:28:42 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 4E95AFE8 for ; Tue, 23 Jan 2001 15:28:38 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 7E40C31B for ; Tue, 23 Jan 2001 17:28:37 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id OAA13372 for ; Tue, 23 Jan 2001 14:28:36 -0800 (PST) Message-ID: <3A6E05D9.59C82573@agilent.com> Date: Tue, 23 Jan 2001 14:29:45 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Reject PDU Concerns. References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I disagree with you Doug, Douglas Otis wrote: > > Matt, > > Unless encoding headers to hide typical PDU signatures, an analyzer will > have little difficulty in discovering PDU boundaries. I think that at 10Gig, it will be very difficult for an analyzer to find an iSCSI header in a byte stream and keep up with the incoming traffic while it's doing it. Furthermore, I could come up with a data pattern that would make it really difficult for your searching analyzer to synchronize on the real PDUs vs the fake ones in the data. > Should a scheme force > PDU alignment every 8192 bytes as example, discovery would be a certainty > within 8k bytes. How is that? Remember, TCP's initial sequence number is random, so how will you know where the 8K boundary is? > To prevent blockage, the size of the PDU is limited and so > the search is not impossible as you imply nor even difficult. Either > markers or alignment seems a good solution for minimizing impact of segment > loss to that of the interval rather than 2RTT x rate. For either a periodic > marker or alignment scheme to work for sending, no changes to TCP is > required. Tweeking TCP a little by adding even an option bit in the header signalling a start of a new PDU would be a lot better. TCP and any other protocol should evolve to track the evolution of the rest of computer technology. -Matt From owner-ips@ece.cmu.edu Tue Jan 23 20:22:19 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA27607 for ; Tue, 23 Jan 2001 20:22:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NMRgi08842 for ips-outgoing; Tue, 23 Jan 2001 17:27:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NMQY608785 for ; Tue, 23 Jan 2001 17:26:34 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id D53E01272; Tue, 23 Jan 2001 14:26:31 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id OAA14111; Tue, 23 Jan 2001 14:26:12 -0800 (PST) Message-ID: <3A6E058F.F5409E7F@cup.hp.com> Date: Tue, 23 Jan 2001 14:28:32 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Matt Wakeley , Julian Satran Cc: IPS Reflector Subject: Re: iscsi : Fragmentation & Reassembly issues in iSCSI. References: <3A678E89.1F694E2@cup.hp.com> <3A6DF8FF.FD1C4577@agilent.com> Content-Type: multipart/mixed; boundary="------------40B00A2204304E336E0DEB5C" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------40B00A2204304E336E0DEB5C Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Matt Wakeley wrote: > Ok, > > There is a DataPDULength which specifies max size of a *DATA* PDU, The DataPDULength is nebulous in its definition. It is'nt clear from its definition if it is the size of only a [scsi] *DATA* PDU, or it specifies the max size of the data portion of all command and data PDUs. Also, what is the exact definition of "data payload" of command PDUs ? Does this include or exclude the additional header spanning beyond 48 bytes ? > and a > PingMaxReplyLength field which indicates how much buffer space an > implementation is willing to dedicate to Ping messages. > > Text strings that will not fit in a single Text command such that it's <= > DataPDULength can be sent as multiple text commands. If the login key negotiation can be performed as multiple text commands, how does the target know the initiator has sent its last text command of login keys to be negotiated ? The login phase is negotiated as follows : ============================== Initiator Target ============================== 1) Login -----------> <---------- 1) Login response of "accept login" with F=0 2) Text ------------> <----------- 2) Text Response : : <------------ 3) Login Response of "accept login" with F=1 If the Text phase can consist of multiple text commands, how does a target know when to send in the login accept response with F=1 ? The target needs an indication that the initiator has sent in the last text command of the login phase and the "F" bit in the login command can provide that. > Forget all this F bit stuff. If the login phase can comprise multiple text commands, the target needs a way to know the initiator has sent its final text command and the "F" bit provides this. OTOH, one could take the stance that the total number of login keys cannot exceed DataPDULength (and TotalText shall be >= DataPDULength), in which case the login phase cannot span beyond 1 text command. Regards, Santosh --------------40B00A2204304E336E0DEB5C Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------40B00A2204304E336E0DEB5C-- From owner-ips@ece.cmu.edu Tue Jan 23 21:42:07 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA29316 for ; Tue, 23 Jan 2001 21:42:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0NNKbW10742 for ips-outgoing; Tue, 23 Jan 2001 18:20:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0NNK7610700 for ; Tue, 23 Jan 2001 18:20:07 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 23 Jan 2001 18:20:01 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A070410147F@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Tue, 23 Jan 2001 18:19:59 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > > The FCIP draft does not preclude multiple TCP connections. > > Somesh is right in pointing that we could result in out-of-order if we > > have more than one TCP connections between same two FCIP Gateways. > > Even with just one TCP connection, when frames travel through the "IP > cloud", don't we risk out-of-order and duplicated frames at the other end? > For multiple connections, it is a sure thing. Not a problem - TCP sits above IP and takes care of both of these. FWIW, TCP tends not to be good at dealing with out-of-order packets (e.g., it asks for retransmits) resulting in a significant incentive to the IP cloud to not deliver out of order very often. > > However, an FCIP gateway (say FCIP-A) can carry on simultaneous TCP > > connections say with FCIP-B and FCIP-C gateways without the > > danger of the out-of-order issue. > > FCIP-A can take responsibility for in-order delivery of outgoing frames. > But, there is no guarantee for in-order and non-duplicated reception on > either FCIP-B or FCIP-C. TCP on FCIP-B and FCIP-C will ensure in-order and non-duplicated reception on the FCIP to FCIP links. The FC fabric is responsible for in-order delivery across the FC fabric based on this link property. Beyond this, FCIP may have framing issues (identification and placement of inbound data in the presence of drops) that are similar to iSCSI depending on how the FCIP nodes are implemented. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Tue Jan 23 23:29:11 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA01467 for ; Tue, 23 Jan 2001 23:29:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0O1dcB14340 for ips-outgoing; Tue, 23 Jan 2001 20:39:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ludwig.troikanetworks.com (ludwig.troikanetworks.com [12.31.172.3]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0O1dU614330 for ; Tue, 23 Jan 2001 20:39:30 -0500 (EST) Received: by host03.troikanetworks.com with Internet Mail Service (5.5.2653.19) id ; Tue, 23 Jan 2001 17:40:01 -0800 Message-ID: From: Wayland Jeong To: "'Black_David@emc.com'" , ips@ece.cmu.edu Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Tue, 23 Jan 2001 17:39:54 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > Beyond this, FCIP may have framing issues (identification and placement > of inbound data in the presence of drops) that are similar to iSCSI > depending > on how the FCIP nodes are implemented. > The issue wrt. to framing in an FCIP device is a bit puzzling to me. Framing for an iSCSI HBA for implementing zero-copy DMA semantics I can understand. The notification of delivery and the delivery of data is de-coupled, so order is preserved in this case. But, aggressively forwarding through drops in an FCIP bridge is not only dangerous, it violates TCP layering. Forwarding around dropped PDU's will introduce all sorts of ordering issues. It seems like an FCIP device would want to take the conservative approach and allow TCP to recover and deliver the PDU's in the order that they were sent. Hence, I don't see a need to specify a framing method for an FCIP-type device. > --David > -Wayland From owner-ips@ece.cmu.edu Tue Jan 23 23:31:21 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA01776 for ; Tue, 23 Jan 2001 23:31:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0O1sbs14722 for ips-outgoing; Tue, 23 Jan 2001 20:54:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0O1s4614712 for ; Tue, 23 Jan 2001 20:54:04 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 23 Jan 2001 20:53:58 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101482@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Tue, 23 Jan 2001 20:53:56 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > > Beyond this, FCIP may have framing issues (identification and placement > > of inbound data in the presence of drops) that are similar to iSCSI depending > > on how the FCIP nodes are implemented. > > > The issue wrt. to framing in an FCIP device is a bit puzzling to me. Framing > for an iSCSI HBA for implementing zero-copy DMA semantics I can understand. > The notification of delivery and the delivery of data is de-coupled, so order is > preserved in this case. > But, aggressively forwarding through drops in an FCIP bridge is not only > dangerous, it violates TCP layering. Forwarding around dropped PDU's > will introduce all sorts of ordering issues. It seems like an FCIP device > would want to take the conservative approach and allow TCP to recover > and deliver the PDU's in the order that they were sent. Hence, I don't see > a need to specify a framing method for an FCIP-type device. Depending on the implementation approach to memory management, an FCIP device may have issues similar to an iSCSI HBA, and this is still the case even though an FCIP device MUST NOT perform the sort of aggressive forwarding described in the above paragraph. YMMV depending on the physical configuration and software management of memory in the specific implementation. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Tue Jan 23 23:32:35 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA01900 for ; Tue, 23 Jan 2001 23:32:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0O12kR13415 for ips-outgoing; Tue, 23 Jan 2001 20:02:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0O124613397 for ; Tue, 23 Jan 2001 20:02:04 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0O2CE050168; Tue, 23 Jan 2001 18:12:14 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Mark Bakke" , "Matt Wakeley" Cc: Subject: RE: iSCSI: remove CDB from iSCSI header Date: Tue, 23 Jan 2001 17:00:19 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <3A6E0ABA.D33FD3A2@cisco.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Mark, By removing the CDB from the header, perhaps FCIP, iFCP and iSCSI could converge on a common header structure after all for a common encapsulation. The SCSI portion could be defined to contain the same information as FCP. Doug > I agree with Matt. Basically, then, the iSCSI header digest should cover > everything in the first 48 bytes. All of the messages, with the exception > of the SCSI Command Request, could easily be done this way, and I think > that we should modify that to fit Matt's idea of treating the CDB (and > sense data) the same as SCSI data for digest purposes. > > Here's what they would all look like: > > 01 - SCSI Command Request > > The iSCSI header digest currently covers the 48-byte iSCSI header, > which includes the CDB. > > The data digest covers everything else, which includes extended CDB, > bidi-read transfer length, and immediate data. > > Since the command request is the only operation that really breaks > the notion of Matt's model, I would suggest that we start the CDB > after the 48-byte iSCSI header, and that it not be covered by the > header digest. We can move the bidi-read length inside the iSCSI > header, and end up with: > > +---------------------------------------------------- > 0| Normal iSCSI header stuff for command request > +---------------------------------------------------- > 32| Bidi-read length > +-------------------------------------------------- > 36/ Reserved > +/ > +--------------------------------------------------- > | (optional header digest goes here) > +------------------------------------------------- > 48| CDB > +-------------------------------------------------- > | Additional CDB > +--------------------------------------------------- > | Immediate Data > +---------------------------------------------------- > | (optional data digest goes here) > +----------------------------------------------------- > > The header digest would cover the first 48 bytes, and would exactly > match the rest of the iSCSI operations. The data digest would include > the CDB (now contiguous with the additional CDB, as an added bonus) > and the immediate data. Anyone concerned with having a separate > data integrity check for just the data when an additional CDB is > being used should refrain from using the immediate data feature. > > While we are on the topic, is anyone really planning to make use > of bidi-read > and immediate data? > >  > Here is my understanding of what the digests cover with the remainder > of the iSCSI operations: > > 81 - SCSI Response > 91 - Asynchronous Event > > iSCSI header digest covers 48-byte iSCSI header > iSCSI data digest covers the sense data, if present > > 02 - Task management command > 82 - Task management response > 06 - Logout command > 86 - Logout response > 90 - R2T > > iSCSI header digest covers 48-byte header > no data, so no data digest present > > 05 - SCSI Data write > 85 - SCSI Data read > 00 - NOP-Out > 80 - NOP-In > > iSCSI header digest covers 48-byte header > iSCSI data digest covers the data > > 04 - Text command > 84 - Text response > 03 - Login command > 83 - Login response > > iSCSI header digest covers 48-byte iSCSI header > iSCSI data digest covers the text parameters > > ??? Should the text be padded to a 4-byte boundary before adding > the data CRC? > I think that it already says so somewhere in the spec. > > -- > Mark > > Matt Wakeley wrote: > > > > I don't think I like this. Header followed by digest followed > by more header. > > > > I think this is a good opportunity to think about removing the > CDB from the > > header. This will reduce the amount of dead space used by > headers that do not > > contain a CDB. This will be beneficial when sending multiple > smaller PDUs (in > > order to keep the CRC coverage high) by reducing the iSCSI > header overhead. > > > > Change the iSCSI command so that there is an iSCSI header > (verified by the > > header digest) followed by CDB "payload" (verified by the data > digest). No > > options for immediate data. Keep it simple, like it is in > Fibre Channel. > > > > -Matt Wakeley > > Agilent Technologies > > > > julian_satran@il.ibm.com wrote: > > > > > > yes, > > > > > > Julo > > > > > > "Barry Reinhold" on 22/01/2001 17:49:29 > > > > > > Please respond to "Barry Reinhold" > > > > > > To: mbakke@cisco.com, Julian Satran/Haifa/IBM@IBMIL > > > cc: ips@ece.cmu.edu > > > Subject: RE: Coverage of Data Digest when using Header Digests > > > > > > Ok, > > > So if the bidi-read length and ADDCDB fields are not in the data > > > digest > > > then I assume we have to figure them into the header digest > even though > > > they > > > are located past the header digets. Is that the expected behavior? > > > > > > -----Original Message----- > > > From: mbakke@cisco.com [mailto:mbakke@cisco.com] > > > Sent: Monday, January 22, 2001 10:22 AM > > > To: julian_satran@il.ibm.com > > > Cc: ips@ece.cmu.edu; Barry Reinhold > > > Subject: Re: Coverage of Data Digest when using Header Digests > > > > > > Barry- > > > > > > In particular, the data digest covers only the SCSI Data part of an > > > iSCSI message; the header digest covers everything else. This means > > > that in an 8k write, the data digest will cover only the 8k, and the > > > header digest will cover everything else. > > > > > > Hope this helps, > > > > > > Mark > > > > > > julian_satran@il.ibm.com wrote: > > > > > > > > Barry, > > > > > > > > Considering that one of the reasons to have a separate > header and data > > > > digest was to enable data > > > > to carried through proxies, virtualizers etc. the current > thinking is > > > that > > > > the data digest will cover only the data and the header (including > > > > extensions) will be covered by the header digest. > > > > > > > > Julo > > > > > > > > "Barry Reinhold" on 21/01/2001 21:50:04 > > > > > > > > Please respond to "Barry Reinhold" > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL > > > > cc: > > > > Subject: Coverage of Data Digest when using Header Digests > > > > > > > > Julian, > > > > Is the Data Digest intended to cover whatever follows > the 48 byte > > > > iSCSI > > > > header? In particular in a command frame which has a CDB > > 16 bytes, uses > > > > bidi, has immediate data, and is using both header and data > digests - > > > what > > > > would the data digest cover? > > > > - barry > > > > reinhold > > > > > > -- > > > Mark A. Bakke > > > Cisco Systems > > > mbakke@cisco.com > > > 763.398.1054 > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 > From owner-ips@ece.cmu.edu Tue Jan 23 23:35:08 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA01957 for ; Tue, 23 Jan 2001 23:35:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0O0xbB13329 for ips-outgoing; Tue, 23 Jan 2001 19:59:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0O0wl613310 for ; Tue, 23 Jan 2001 19:58:47 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0O29W050161; Tue, 23 Jan 2001 18:09:32 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Matt Wakeley" , "IPS Reflector" Subject: RE: iSCSI : Reject PDU Concerns. Date: Tue, 23 Jan 2001 16:57:38 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <3A6E05D9.59C82573@agilent.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt, As both schemes are periodic, once an interval has been deciphered, a boundary is discovered and can be applied to later data already in progress. Once a sample has been scanned for typical PDU signatures, a trial intercept should confirm success with little effort even at OC-192 rates. By making the intervals a modulo that is a power of 2, you would not need to be concerned about potential wraps between efforts. The evolution is called SCTP for transforming TCP into a frame aligned object based protocol. Such changes to TCP as you suggest will hardly end with a flag here or there. Doug > I disagree with you Doug, > > Douglas Otis wrote: > > > > Matt, > > > > Unless encoding headers to hide typical PDU signatures, an analyzer will > > have little difficulty in discovering PDU boundaries. > > I think that at 10Gig, it will be very difficult for an analyzer > to find an > iSCSI header in a byte stream and keep up with the incoming > traffic while it's > doing it. Furthermore, I could come up with a data pattern that > would make it > really difficult for your searching analyzer to synchronize on > the real PDUs > vs the fake ones in the data. > > > Should a scheme force > > PDU alignment every 8192 bytes as example, discovery would be a > certainty > > within 8k bytes. > > How is that? Remember, TCP's initial sequence number is random, > so how will > you know where the 8K boundary is? > > > To prevent blockage, the size of the PDU is limited and so > > the search is not impossible as you imply nor even difficult. Either > > markers or alignment seems a good solution for minimizing > impact of segment > > loss to that of the interval rather than 2RTT x rate. For > either a periodic > > marker or alignment scheme to work for sending, no changes to TCP is > > required. > > Tweeking TCP a little by adding even an option bit in the header > signalling a > start of a new PDU would be a lot better. TCP and any other > protocol should > evolve to track the evolution of the rest of computer technology. > > -Matt > From owner-ips@ece.cmu.edu Wed Jan 24 00:45:43 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA02916 for ; Wed, 24 Jan 2001 00:45:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0O2pdf16180 for ips-outgoing; Tue, 23 Jan 2001 21:51:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0O2oc616160 for ; Tue, 23 Jan 2001 21:50:38 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 40F90137F; Tue, 23 Jan 2001 18:50:37 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA07034; Tue, 23 Jan 2001 18:50:32 -0800 (PST) Message-ID: <3A6E4384.582861D3@cup.hp.com> Date: Tue, 23 Jan 2001 18:52:52 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: Re: iSCSI: DataRN References: <3A6DD3E2.F63CC265@cup.hp.com> Content-Type: multipart/mixed; boundary="------------4B0A17E36B99F6C2CEEE9013" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------4B0A17E36B99F6C2CEEE9013 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Santosh Rao wrote: > julian_satran@il.ibm.com wrote: > > > > > The above is correct if the digest error or connection failure occurred on > > delivery of the command. If a digest error were to be detected by an > > initiator on the response PDU (by which time the target has already > > completed the operation and the TCP layer at the initiator has already > > sent the ACK), then, the command is complete from the device perspective > > and should not be retried. > > > > how would that happen ? > > Julian, > > If the initiator detected a digest error on the Response PDU[and the target > has completed all or part of the data phase], such an operation cannot be > retried without a prior rewind command, in the case of sequential media. > > The iSCSI layer cannot retry commands destined to sequential media when there > is a possibility that some or all of the data phase is complete. > > Regards, > Santosh Before anybody points out StatSN and the fact that a target need not re-do the operation but just send the Status, based on the un-acknowledged StatSN, do take note of the fact that status recovery is optional. The initiator does NOT know if the target does support status recovery and is not guaranteed that the target will not re-do the operation. In the absence of such a guarantee, the initiator cannot retry the command. - Santosh --------------4B0A17E36B99F6C2CEEE9013 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------4B0A17E36B99F6C2CEEE9013-- From owner-ips@ece.cmu.edu Wed Jan 24 01:42:59 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA07393 for ; Wed, 24 Jan 2001 01:42:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0O4MgK18441 for ips-outgoing; Tue, 23 Jan 2001 23:22:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0O4M3618426 for ; Tue, 23 Jan 2001 23:22:04 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0O5We050343; Tue, 23 Jan 2001 21:32:42 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: , Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Tue, 23 Jan 2001 20:20:46 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704101482@corpmx9.isus.emc.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, The FC frame is placed on FC network within a short time of its arrival or there is no advantage in allowing out of sequence processing. With that in mind, multiple connections can be synchronized with a slight delay by looking at a single time-stamp field and stale frames dropped if their arrival is deemed too late by means of a TOV. In each case, a time-stamp offers a simple solution to this problem of potential out of sequence placement. With a small window looking across multiple connections, frames can be placed in order. If a frame arrives with a relative time-stamp in excess of this time window, bye-bye frame. At least the fabric assumptions remain with respect delinquent frames. The size of the "small" window may increase in time to allow for 2RTT+ but it would be a sizeable increase in system latency and require about 1 Mbyte of buffer for a MAN. One wonders if such buffering is desired for FC or if one could live with the results of occasional out of sequence frames where a R_A_TOV limit is used to dismiss frames. The aspect of ensuring a restart is enabled by a means to police old strays. Without a time windowing scheme, R_A_TOV is not valid and the size of the buffer required is unknown. What do you think? Doug > > > Beyond this, FCIP may have framing issues (identification and > placement > > > of inbound data in the presence of drops) that are similar to iSCSI > depending > > > on how the FCIP nodes are implemented. > > > > > The issue wrt. to framing in an FCIP device is a bit puzzling to me. > Framing > > for an iSCSI HBA for implementing zero-copy DMA semantics I can > understand. > > The notification of delivery and the delivery of data is de-coupled, so > order is > > preserved in this case. > > > But, aggressively forwarding through drops in an FCIP bridge is not only > > dangerous, it violates TCP layering. Forwarding around dropped PDU's > > will introduce all sorts of ordering issues. It seems like an > FCIP device > > would want to take the conservative approach and allow TCP to recover > > and deliver the PDU's in the order that they were sent. Hence, > I don't see > > a need to specify a framing method for an FCIP-type device. > > Depending on the implementation approach to memory management, an > FCIP device may have issues similar to an iSCSI HBA, and this is still > the case even though an FCIP device MUST NOT perform the sort of > aggressive forwarding described in the above paragraph. YMMV depending > on the physical configuration and software management of memory in > the specific implementation. > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Wed Jan 24 04:38:15 2001 Received: from ece.cmu.edu ([128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA17257 for ; Wed, 24 Jan 2001 04:38:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0O7Ija22466 for ips-outgoing; Wed, 24 Jan 2001 02:18:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0O7Ia622459 for ; Wed, 24 Jan 2001 02:18:36 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 75C23BF7 for ; Tue, 23 Jan 2001 23:18:35 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id XAA20406; Tue, 23 Jan 2001 23:18:30 -0800 (PST) From: Santosh Rao Message-Id: <200101240718.XAA20406@hpcuhe.cup.hp.com> Subject: iSCSI : Login Phase Problems To: ips@ece.cmu.edu (ips) Date: Tue, 23 Jan 2001 23:18:30 -0800 (PST) Cc: santoshr@hpcuhe.cup.hp.com (Santosh Rao) X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian & All, Problem : ========= The login phase is intended to be driven by the initiator and it is the one to decide if only a Login PDU is sufficient or it needs a Login + Text PDU. The initiator needs a mechanism to indicate to the target that a Text command will [or will not] follow the login command. However, by placing the "F" bit in the login response and NOT the login command, the iSCSI login phase opens up a number of questions : Reference ======== Section 4, 4.1 of rev 03 iSCSI draft. The login phase is negotiated as follows : =========================================== Initiator Target =========================================== 1) Login -----------> <---------- 1) Login response of "accept login" with F=0 2) Text ------------> <----------- 2) Text Response : : <------------ 3) Login Response of "accept login" with F=1 Issues : ======== 1) Without a communication from the initiator on whether it intends to follow the login command with a text command, the target may send back a login accept with the "final" (F) bit set, when the initiator was still interested in negotiating more keys. 2) The target may send back an interim login accept ("F" bit is not set), expecting the initiator to continue negotiation with a text command. The initiator may have no further keys to negotiate and so, does nothing. When is the login phase then deemed to be complete ? (i.e. what causes the tgt to do a final login accept (F bit set)). 3) If multiple Text commands are to be allowed in the login phase, how does a target know that the initiator is done with all its key negotiations and has sent all its text commands ? If multiple Text commands are NOT allowed in the login phase, then, the draft MUST state this explicitly. It is the initiator that drives the Login/Text negotiations and hence, it is the initiator that must decide whether a login is sufficient or a login will be followed by a text phase. Therefore, the "F" bit currently in the login response should actually be in login command. The initiator is the master of the login phase and it should drive the login + text phases. Solution : ========== Case I (only login command used ) --------------------------------- Initiator Target ====================================== Login(F bit set) ------> <----- Login Accept (F bit set) Case (II) (login + text commands used in login phase) ----------------------------------------------------- Initiator Target ====================================== Login(F bit NOT set) ------> <----- Login Accept (F bit NOT set) Text (F bit NOT set) ------> <----- Text Response : : Text (F bit set) -----> <---- Text Response <---- Login Response (F bit set) Comments ? Regards, Santosh -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Wed Jan 24 10:38:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA23584 for ; Wed, 24 Jan 2001 10:38:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0ODUq808529 for ips-outgoing; Wed, 24 Jan 2001 08:30:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0ODUW608521 for ; Wed, 24 Jan 2001 08:30:32 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id OAA18598 for ; Wed, 24 Jan 2001 14:30:24 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id OAA153244 for ; Wed, 24 Jan 2001 14:30:24 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DE.004A2E70 ; Wed, 24 Jan 2001 14:30:15 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 24 Jan 2001 15:25:42 +0200 Subject: Re: iSCSI : Reject PDU Concerns. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk That is not completely accurate. You could enter the marker intervals if you know them or, assuming that you have CRCs the analyzer can look for 48byte sequence (or 56 byte sequences) with a correct CRC and after finding several block that make sense consider itself synched. Consider also that after any longer silence the ULP will be aligned with TCP. The analyzer could work as it can loose data! Julo Matt Wakeley on 23/01/2001 20:54:54 Please respond to Matt Wakeley To: IPS Reflector cc: Subject: Re: iSCSI : Reject PDU Concerns. julian_satran@il.ibm.com wrote: > 2) An error_byte field should be considered for inclusion in the Reject > PDU which contains the byte offset of the errant byte[or word] in the > received header/payload that caused the target to send a Reject response > to the command. > > This is a simple solution that provides for quick fault isolation and > root cause of the reason for the reject. This also rids the Reject PDU > of any detailed elaboration of reason codes meant to allow root cause of > the reason for the reject. > > that is no big help since there can be more than one error and it > helps mostly in debugging. For implementation errors in the target and/or > initiator I am sure the protocol analyzer will be happy to help but running > implementation should not be burdened with. For those that are running > errors on good implementations we will provide some details when necessary > Julian, With only in-band framing [the markers that are currently defined in the spec], there is no way a protocol analyzer can be used in the traditional way. In order for the protocol analyzer to work, it would have to be continuously running since the tcp connection(s) was established. Besides, early on there will not be any protocol analyzers. -Matt From owner-ips@ece.cmu.edu Wed Jan 24 10:39:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA23612 for ; Wed, 24 Jan 2001 10:39:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0ODJs808268 for ips-outgoing; Wed, 24 Jan 2001 08:19:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0ODJV608259 for ; Wed, 24 Jan 2001 08:19:31 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id OAA90338 for ; Wed, 24 Jan 2001 14:19:18 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id OAA101232 for ; Wed, 24 Jan 2001 14:19:18 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DE.00492AC1 ; Wed, 24 Jan 2001 14:19:10 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 24 Jan 2001 15:14:43 +0200 Subject: Re: iSCSI: DataRN Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, The ACK for a response is when status is ACKed (by the status numbering). Before this ack a target has several options for restart: to keep all the data (recall we have dropped DataRN) and "restart sending". not to keep data and reject the command restart with a service response (that we have to specify) of restart reject (your tape target may want to do just this) Alternatively we may want in the restart to indicate the phase the command was in (Data Transfer or Status) and to let the target react accordingly but I am not sure it is worth the effort. Regards, Julo Santosh Rao on 23/01/2001 20:56:34 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI: DataRN julian_satran@il.ibm.com wrote: > > The above is correct if the digest error or connection failure occurred on > delivery of the command. If a digest error were to be detected by an > initiator on the response PDU (by which time the target has already > completed the operation and the TCP layer at the initiator has already > sent the ACK), then, the command is complete from the device perspective > and should not be retried. > > how would that happen ? Julian, If the initiator detected a digest error on the Response PDU[and the target has completed all or part of the data phase], such an operation cannot be retried without a prior rewind command, in the case of sequential media. The iSCSI layer cannot retry commands destined to sequential media when there is a possibility that some or all of the data phase is complete. Regards, Santosh - santoshr.vcf From owner-ips@ece.cmu.edu Wed Jan 24 10:39:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA23629 for ; Wed, 24 Jan 2001 10:39:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0ODY1I08596 for ips-outgoing; Wed, 24 Jan 2001 08:34:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0ODWq608573 for ; Wed, 24 Jan 2001 08:32:53 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id OAA16836 for ; Wed, 24 Jan 2001 14:32:49 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id OAA07532 for ; Wed, 24 Jan 2001 14:32:49 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DE.004A688C ; Wed, 24 Jan 2001 14:32:44 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 24 Jan 2001 15:28:16 +0200 Subject: Re: iSCSI : Reject PDU Concerns. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk will consider - Julo Santosh Rao on 23/01/2001 21:35:06 Please respond to Santosh Rao To: Matt Wakeley , Julian Satran/Haifa/IBM@IBMIL cc: IPS Reflector Subject: Re: iSCSI : Reject PDU Concerns. Matt Wakeley wrote: > julian_satran@il.ibm.com wrote: > > > 2) An error_byte field should be considered for inclusion in the Reject > > PDU which contains the byte offset of the errant byte[or word] in the > > received header/payload that caused the target to send a Reject response > > to the command. > > > > This is a simple solution that provides for quick fault isolation and > > root cause of the reason for the reject. This also rids the Reject PDU > > of any detailed elaboration of reason codes meant to allow root cause of > > the reason for the reject. > > > > that is no big help since there can be more than one error and it > > helps mostly in debugging. Julian, 1) The iSCSI layer, on parsing a PDU, will stop on detecting the first error. So, there should really be only one error to report and that is the first one discovered. 2) Helping in debugging is a big issue since one of the major problems iSCSI implementations will need to deal with is inter-operability. This is not a one-time problem as new implmentations of devices and initiators keep coming along. The iSCSI draft must attempt to provide some aids for quick fault isolation. 3) The cost of providing this error_byte based fault indication feature is really minimal, compared to the benefits it provides. Regards, Santosh > For implementation errors in the target and/or > > initiator I am sure the protocol analyzer will be happy to help but running > > implementation should not be burdened with. For those that are running > > errors on good implementations we will provide some details when necessary > > > > Julian, > > With only in-band framing [the markers that are currently defined in the > spec], there is no way a protocol analyzer can be used in the traditional > way. In order for the protocol analyzer to work, it would have to be > continuously running since the tcp connection(s) was established. Besides, > early on there will not be any protocol analyzers. > > -Matt - santoshr.vcf From owner-ips@ece.cmu.edu Wed Jan 24 10:41:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA23709 for ; Wed, 24 Jan 2001 10:41:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0ODqti09075 for ips-outgoing; Wed, 24 Jan 2001 08:52:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0ODpr609049 for ; Wed, 24 Jan 2001 08:51:53 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id OAA41164 for ; Wed, 24 Jan 2001 14:51:44 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id OAA94410 for ; Wed, 24 Jan 2001 14:51:44 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DE.004C23CE ; Wed, 24 Jan 2001 14:51:39 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 24 Jan 2001 15:47:07 +0200 Subject: Re: iSCSI: remove CDB from iSCSI header Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, Considering that most of the traffic is SCSI commands and responses the actual space waste is minor. What you just suggest (based on misreading the accompanying text - I think) will force several reads for most of the traffic and we found this harmful to performance. Considering that most of the traffic has 48 byte headers the outlay we have now is optimal. The digest comes anyhow after the total header (including the additional fields). Julo Matt Wakeley on 23/01/2001 23:16:17 Please respond to Matt Wakeley To: ips@ece.cmu.edu cc: Subject: iSCSI: remove CDB from iSCSI header I don't think I like this. Header followed by digest followed by more header. I think this is a good opportunity to think about removing the CDB from the header. This will reduce the amount of dead space used by headers that do not contain a CDB. This will be beneficial when sending multiple smaller PDUs (in order to keep the CRC coverage high) by reducing the iSCSI header overhead. Change the iSCSI command so that there is an iSCSI header (verified by the header digest) followed by CDB "payload" (verified by the data digest). No options for immediate data. Keep it simple, like it is in Fibre Channel. -Matt Wakeley Agilent Technologies julian_satran@il.ibm.com wrote: > > yes, > > Julo > > "Barry Reinhold" on 22/01/2001 17:49:29 > > Please respond to "Barry Reinhold" > > To: mbakke@cisco.com, Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: RE: Coverage of Data Digest when using Header Digests > > Ok, > So if the bidi-read length and ADDCDB fields are not in the data > digest > then I assume we have to figure them into the header digest even though > they > are located past the header digets. Is that the expected behavior? > > -----Original Message----- > From: mbakke@cisco.com [mailto:mbakke@cisco.com] > Sent: Monday, January 22, 2001 10:22 AM > To: julian_satran@il.ibm.com > Cc: ips@ece.cmu.edu; Barry Reinhold > Subject: Re: Coverage of Data Digest when using Header Digests > > Barry- > > In particular, the data digest covers only the SCSI Data part of an > iSCSI message; the header digest covers everything else. This means > that in an 8k write, the data digest will cover only the 8k, and the > header digest will cover everything else. > > Hope this helps, > > Mark > > julian_satran@il.ibm.com wrote: > > > > Barry, > > > > Considering that one of the reasons to have a separate header and data > > digest was to enable data > > to carried through proxies, virtualizers etc. the current thinking is > that > > the data digest will cover only the data and the header (including > > extensions) will be covered by the header digest. > > > > Julo > > > > "Barry Reinhold" on 21/01/2001 21:50:04 > > > > Please respond to "Barry Reinhold" > > > > To: Julian Satran/Haifa/IBM@IBMIL > > cc: > > Subject: Coverage of Data Digest when using Header Digests > > > > Julian, > > Is the Data Digest intended to cover whatever follows the 48 byte > > iSCSI > > header? In particular in a command frame which has a CDB > 16 bytes, uses > > bidi, has immediate data, and is using both header and data digests - > what > > would the data digest cover? > > - barry > > reinhold > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 From owner-ips@ece.cmu.edu Wed Jan 24 12:32:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA26041 for ; Wed, 24 Jan 2001 12:32:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0OFSt112113 for ips-outgoing; Wed, 24 Jan 2001 10:28:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0OFSi612106 for ; Wed, 24 Jan 2001 10:28:45 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAA92732; Wed, 24 Jan 2001 16:28:33 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA161580; Wed, 24 Jan 2001 16:28:28 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DE.0054FEF2 ; Wed, 24 Jan 2001 16:28:22 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Charles Monia cc: ips@ece.cmu.edu Message-ID: Date: Wed, 24 Jan 2001 17:16:04 +0200 Subject: Re: ISCSI: Target-initiated logout Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, I think that the difference between the iSCSI Asynch Message and the SCSI AER is now clear and the current mechanism is OK. There is only one dissenting voice (Santosh Rao) that would like to see the iSCSI Asynch Message having the semantics "I am going to drop the connection - no need for you to log me out" and he has a point about this being simpler. However I am still concerned about the effect of messages in flight - mainly about the status-acking part of them - on the closing connection. If I'll feel confident that I have them all covered I will add this change. Regards, Julo Charles Monia on 24/01/2001 04:49:20 Please respond to Charles Monia To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: ISCSI: Target-initiated logout Hi Julo: How did the WG resolve the issue of target-initiated logout? Charles Charles Monia Senior Technology Consultant Nishan Systems email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ece.cmu.edu Wed Jan 24 12:35:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA26163 for ; Wed, 24 Jan 2001 12:35:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0OExtW11149 for ips-outgoing; Wed, 24 Jan 2001 09:59:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0OExm611143 for ; Wed, 24 Jan 2001 09:59:49 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 2B3FB6D5 for ; Wed, 24 Jan 2001 07:59:48 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 788F334A for ; Wed, 24 Jan 2001 09:59:47 -0500 (EST) Received: from agilent.com (cos1nai129089.cos.agilent.com [141.184.129.89]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id GAA26955 for ; Wed, 24 Jan 2001 06:59:45 -0800 (PST) Message-ID: <3A6EEDDD.601AD740@agilent.com> Date: Wed, 24 Jan 2001 06:59:41 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: remove CDB from iSCSI header References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > Matt, > > Considering that most of the traffic is SCSI commands and responses the > actual space waste is minor. Assuming short reads/writes... If you are performing the large tape I/Os that you like to refer to (your reason for having DataRN) then it's a larger overhead. > What you just suggest (based on misreading the accompanying text - I think) I did not mis-read anything - perhaps you did: Barry said: B: So if the bidi-read length and ADDCDB fields are not in the data digest B: then I assume we have to figure them into the header digest even though B: they are located past the header digets. Is that the expected behavior? Barry asked/stated if the bidi-read length and ADDCDB fields are past the header digest to which you replied: J: yes, J: J: Julo > will force several reads for most of the traffic and we found this harmful > to performance. > Considering that most of the traffic has 48 byte headers the outlay we have > now is optimal. > The digest comes anyhow after the total header (including the additional > fields). As I stated earlier, that was not what your "yes" reply to Barry indicated. > > Julo > > Matt Wakeley on 23/01/2001 23:16:17 > > Please respond to Matt Wakeley > > To: ips@ece.cmu.edu > cc: > Subject: iSCSI: remove CDB from iSCSI header > > I don't think I like this. Header followed by digest followed by more > header. > > I think this is a good opportunity to think about removing the CDB from the > header. This will reduce the amount of dead space used by headers that do > not > contain a CDB. This will be beneficial when sending multiple smaller PDUs > (in > order to keep the CRC coverage high) by reducing the iSCSI header overhead. > > Change the iSCSI command so that there is an iSCSI header (verified by the > header digest) followed by CDB "payload" (verified by the data digest). No > options for immediate data. Keep it simple, like it is in Fibre Channel. > > -Matt Wakeley > Agilent Technologies > > julian_satran@il.ibm.com wrote: > > > > yes, > > > > Julo > > > > "Barry Reinhold" on 22/01/2001 17:49:29 > > > > Please respond to "Barry Reinhold" > > > > To: mbakke@cisco.com, Julian Satran/Haifa/IBM@IBMIL > > cc: ips@ece.cmu.edu > > Subject: RE: Coverage of Data Digest when using Header Digests > > > > Ok, > > So if the bidi-read length and ADDCDB fields are not in the data > > digest > > then I assume we have to figure them into the header digest even though > > they > > are located past the header digets. Is that the expected behavior? > > > > -----Original Message----- > > From: mbakke@cisco.com [mailto:mbakke@cisco.com] > > Sent: Monday, January 22, 2001 10:22 AM > > To: julian_satran@il.ibm.com > > Cc: ips@ece.cmu.edu; Barry Reinhold > > Subject: Re: Coverage of Data Digest when using Header Digests > > > > Barry- > > > > In particular, the data digest covers only the SCSI Data part of an > > iSCSI message; the header digest covers everything else. This means > > that in an 8k write, the data digest will cover only the 8k, and the > > header digest will cover everything else. > > > > Hope this helps, > > > > Mark > > > > julian_satran@il.ibm.com wrote: > > > > > > Barry, > > > > > > Considering that one of the reasons to have a separate header and data > > > digest was to enable data > > > to carried through proxies, virtualizers etc. the current thinking is > > that > > > the data digest will cover only the data and the header (including > > > extensions) will be covered by the header digest. > > > > > > Julo > > > > > > "Barry Reinhold" on 21/01/2001 21:50:04 > > > > > > Please respond to "Barry Reinhold" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL > > > cc: > > > Subject: Coverage of Data Digest when using Header Digests > > > > > > Julian, > > > Is the Data Digest intended to cover whatever follows the 48 byte > > > iSCSI > > > header? In particular in a command frame which has a CDB > 16 bytes, > uses > > > bidi, has immediate data, and is using both header and data digests - > > what > > > would the data digest cover? > > > - barry > > > reinhold > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 From owner-ips@ece.cmu.edu Wed Jan 24 12:38:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA26234 for ; Wed, 24 Jan 2001 12:38:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0OF30P11252 for ips-outgoing; Wed, 24 Jan 2001 10:03:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0OF1j611220 for ; Wed, 24 Jan 2001 10:01:45 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAA26622 for ; Wed, 24 Jan 2001 16:01:37 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA66840 for ; Wed, 24 Jan 2001 16:01:35 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DE.005286B6 ; Wed, 24 Jan 2001 16:01:24 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 24 Jan 2001 16:56:48 +0200 Subject: Re: iSCSI: remove CDB from iSCSI header Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk It is nice and neat... and we went over it, discussed at length in the past and rejected because commands are the most frequent outgoing message and you want them handled in one operation at target (initiator is easy). And 16 byte CDBs cover most of the commands. We expect also Data Base operations with data lengths of 4-8kbytes to be done using immediate data and reducing the number of operations to implement them is critically important. On the other hand the CDB extensions and other field could go with the data (as they are not modified by any proxy) and we could keep header digest strictly for the first 48 bytes. I am not so much concerned about having to compute a CRC on the variable header as much as by the thought that if you keep a digest after the 48 bytes for commands with longer CDBs but no immediate data you will have a digest after 48 bytes and another one after the extension. Julo Mark Bakke on 24/01/2001 00:50:34 Please respond to Mark Bakke To: Matt Wakeley cc: ips@ece.cmu.edu Subject: Re: iSCSI: remove CDB from iSCSI header I agree with Matt. Basically, then, the iSCSI header digest should cover everything in the first 48 bytes. All of the messages, with the exception of the SCSI Command Request, could easily be done this way, and I think that we should modify that to fit Matt's idea of treating the CDB (and sense data) the same as SCSI data for digest purposes. Here's what they would all look like: 01 - SCSI Command Request The iSCSI header digest currently covers the 48-byte iSCSI header, which includes the CDB. The data digest covers everything else, which includes extended CDB, bidi-read transfer length, and immediate data. Since the command request is the only operation that really breaks the notion of Matt's model, I would suggest that we start the CDB after the 48-byte iSCSI header, and that it not be covered by the header digest. We can move the bidi-read length inside the iSCSI header, and end up with: +---------------------------------------------------- 0| Normal iSCSI header stuff for command request +---------------------------------------------------- 32| Bidi-read length +-------------------------------------------------- 36/ Reserved +/ +--------------------------------------------------- | (optional header digest goes here) +------------------------------------------------- 48| CDB +-------------------------------------------------- | Additional CDB +--------------------------------------------------- | Immediate Data +---------------------------------------------------- | (optional data digest goes here) +----------------------------------------------------- The header digest would cover the first 48 bytes, and would exactly match the rest of the iSCSI operations. The data digest would include the CDB (now contiguous with the additional CDB, as an added bonus) and the immediate data. Anyone concerned with having a separate data integrity check for just the data when an additional CDB is being used should refrain from using the immediate data feature. While we are on the topic, is anyone really planning to make use of bidi-read and immediate data?  Here is my understanding of what the digests cover with the remainder of the iSCSI operations: 81 - SCSI Response 91 - Asynchronous Event iSCSI header digest covers 48-byte iSCSI header iSCSI data digest covers the sense data, if present 02 - Task management command 82 - Task management response 06 - Logout command 86 - Logout response 90 - R2T iSCSI header digest covers 48-byte header no data, so no data digest present 05 - SCSI Data write 85 - SCSI Data read 00 - NOP-Out 80 - NOP-In iSCSI header digest covers 48-byte header iSCSI data digest covers the data 04 - Text command 84 - Text response 03 - Login command 83 - Login response iSCSI header digest covers 48-byte iSCSI header iSCSI data digest covers the text parameters ??? Should the text be padded to a 4-byte boundary before adding the data CRC? I think that it already says so somewhere in the spec. -- Mark Matt Wakeley wrote: > > I don't think I like this. Header followed by digest followed by more header. > > I think this is a good opportunity to think about removing the CDB from the > header. This will reduce the amount of dead space used by headers that do not > contain a CDB. This will be beneficial when sending multiple smaller PDUs (in > order to keep the CRC coverage high) by reducing the iSCSI header overhead. > > Change the iSCSI command so that there is an iSCSI header (verified by the > header digest) followed by CDB "payload" (verified by the data digest). No > options for immediate data. Keep it simple, like it is in Fibre Channel. > > -Matt Wakeley > Agilent Technologies > > julian_satran@il.ibm.com wrote: > > > > yes, > > > > Julo > > > > "Barry Reinhold" on 22/01/2001 17:49:29 > > > > Please respond to "Barry Reinhold" > > > > To: mbakke@cisco.com, Julian Satran/Haifa/IBM@IBMIL > > cc: ips@ece.cmu.edu > > Subject: RE: Coverage of Data Digest when using Header Digests > > > > Ok, > > So if the bidi-read length and ADDCDB fields are not in the data > > digest > > then I assume we have to figure them into the header digest even though > > they > > are located past the header digets. Is that the expected behavior? > > > > -----Original Message----- > > From: mbakke@cisco.com [mailto:mbakke@cisco.com] > > Sent: Monday, January 22, 2001 10:22 AM > > To: julian_satran@il.ibm.com > > Cc: ips@ece.cmu.edu; Barry Reinhold > > Subject: Re: Coverage of Data Digest when using Header Digests > > > > Barry- > > > > In particular, the data digest covers only the SCSI Data part of an > > iSCSI message; the header digest covers everything else. This means > > that in an 8k write, the data digest will cover only the 8k, and the > > header digest will cover everything else. > > > > Hope this helps, > > > > Mark > > > > julian_satran@il.ibm.com wrote: > > > > > > Barry, > > > > > > Considering that one of the reasons to have a separate header and data > > > digest was to enable data > > > to carried through proxies, virtualizers etc. the current thinking is > > that > > > the data digest will cover only the data and the header (including > > > extensions) will be covered by the header digest. > > > > > > Julo > > > > > > "Barry Reinhold" on 21/01/2001 21:50:04 > > > > > > Please respond to "Barry Reinhold" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL > > > cc: > > > Subject: Coverage of Data Digest when using Header Digests > > > > > > Julian, > > > Is the Data Digest intended to cover whatever follows the 48 byte > > > iSCSI > > > header? In particular in a command frame which has a CDB > 16 bytes, uses > > > bidi, has immediate data, and is using both header and data digests - > > what > > > would the data digest cover? > > > - barry > > > reinhold > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Wed Jan 24 13:36:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA27394 for ; Wed, 24 Jan 2001 13:36:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0OFpwj12966 for ips-outgoing; Wed, 24 Jan 2001 10:51:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0OFp9612935 for ; Wed, 24 Jan 2001 10:51:14 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA79100 for ; Wed, 24 Jan 2001 16:50:56 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA268450 for ; Wed, 24 Jan 2001 16:50:57 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DE.00570E60 ; Wed, 24 Jan 2001 16:50:53 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 24 Jan 2001 17:46:22 +0200 Subject: Re: iSCSI : Login Phase Problems Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, That is already in in 04 as I was refining the security negotiation. Regards, Julo Santosh Rao on 24/01/2001 09:18:30 Please respond to Santosh Rao To: ips@ece.cmu.edu (ips) cc: santoshr@hpcuhe.cup.hp.com (Santosh Rao) Subject: iSCSI : Login Phase Problems Julian & All, Problem : ========= The login phase is intended to be driven by the initiator and it is the one to decide if only a Login PDU is sufficient or it needs a Login + Text PDU. The initiator needs a mechanism to indicate to the target that a Text command will [or will not] follow the login command. However, by placing the "F" bit in the login response and NOT the login command, the iSCSI login phase opens up a number of questions : Reference ======== Section 4, 4.1 of rev 03 iSCSI draft. The login phase is negotiated as follows : =========================================== Initiator Target =========================================== 1) Login -----------> <---------- 1) Login response of "accept login" with F=0 2) Text ------------> <----------- 2) Text Response : : <------------ 3) Login Response of "accept login" with F=1 Issues : ======== 1) Without a communication from the initiator on whether it intends to follow the login command with a text command, the target may send back a login accept with the "final" (F) bit set, when the initiator was still interested in negotiating more keys. 2) The target may send back an interim login accept ("F" bit is not set), expecting the initiator to continue negotiation with a text command. The initiator may have no further keys to negotiate and so, does nothing. When is the login phase then deemed to be complete ? (i.e. what causes the tgt to do a final login accept (F bit set)). 3) If multiple Text commands are to be allowed in the login phase, how does a target know that the initiator is done with all its key negotiations and has sent all its text commands ? If multiple Text commands are NOT allowed in the login phase, then, the draft MUST state this explicitly. It is the initiator that drives the Login/Text negotiations and hence, it is the initiator that must decide whether a login is sufficient or a login will be followed by a text phase. Therefore, the "F" bit currently in the login response should actually be in login command. The initiator is the master of the login phase and it should drive the login + text phases. Solution : ========== Case I (only login command used ) --------------------------------- Initiator Target ====================================== Login(F bit set) ------> <----- Login Accept (F bit set) Case (II) (login + text commands used in login phase) ----------------------------------------------------- Initiator Target ====================================== Login(F bit NOT set) ------> <----- Login Accept (F bit NOT set) Text (F bit NOT set) ------> <----- Text Response : : Text (F bit set) -----> <---- Text Response <---- Login Response (F bit set) Comments ? Regards, Santosh -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Wed Jan 24 13:39:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA27510 for ; Wed, 24 Jan 2001 13:39:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0OGH0A13908 for ips-outgoing; Wed, 24 Jan 2001 11:17:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antigonus.hosting.pacbell.net (antigonus.hosting.pacbell.net [216.100.98.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0OGG5613879 for ; Wed, 24 Jan 2001 11:16:05 -0500 (EST) Received: from VENKAT1 (64-160-62-200.rhapsodynetworks.com [64.160.62.200] (may be forged)) by antigonus.hosting.pacbell.net id LAA02178; Wed, 24 Jan 2001 11:15:59 -0500 (EST) [ConcentricHost SMTP Relay 1.7] From: "Venkat Rangan" To: "Mark Bakke" , "Matt Wakeley" Cc: Subject: RE: iSCSI: remove CDB from iSCSI header Date: Wed, 24 Jan 2001 08:18:29 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: <3A6E0ABA.D33FD3A2@cisco.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit We agree with the proposed changes as well. On the "optional" open issue, we would like to have two bits in the flags portion of the header which indicates whether digests are present or not. The digests should be "mandatory" for implementation and "optional" for deployment with "ON" as the default. Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Mark Bakke Sent: Tuesday, January 23, 2001 2:51 PM To: Matt Wakeley Cc: ips@ece.cmu.edu Subject: Re: iSCSI: remove CDB from iSCSI header I agree with Matt. Basically, then, the iSCSI header digest should cover everything in the first 48 bytes. All of the messages, with the exception of the SCSI Command Request, could easily be done this way, and I think that we should modify that to fit Matt's idea of treating the CDB (and sense data) the same as SCSI data for digest purposes. Here's what they would all look like: 01 - SCSI Command Request The iSCSI header digest currently covers the 48-byte iSCSI header, which includes the CDB. The data digest covers everything else, which includes extended CDB, bidi-read transfer length, and immediate data. Since the command request is the only operation that really breaks the notion of Matt's model, I would suggest that we start the CDB after the 48-byte iSCSI header, and that it not be covered by the header digest. We can move the bidi-read length inside the iSCSI header, and end up with: +---------------------------------------------------- 0| Normal iSCSI header stuff for command request +---------------------------------------------------- 32| Bidi-read length +-------------------------------------------------- 36/ Reserved +/ +--------------------------------------------------- | (optional header digest goes here) +------------------------------------------------- 48| CDB +-------------------------------------------------- | Additional CDB +--------------------------------------------------- | Immediate Data +---------------------------------------------------- | (optional data digest goes here) +----------------------------------------------------- The header digest would cover the first 48 bytes, and would exactly match the rest of the iSCSI operations. The data digest would include the CDB (now contiguous with the additional CDB, as an added bonus) and the immediate data. Anyone concerned with having a separate data integrity check for just the data when an additional CDB is being used should refrain from using the immediate data feature. While we are on the topic, is anyone really planning to make use of bidi-read and immediate data?  Here is my understanding of what the digests cover with the remainder of the iSCSI operations: 81 - SCSI Response 91 - Asynchronous Event iSCSI header digest covers 48-byte iSCSI header iSCSI data digest covers the sense data, if present 02 - Task management command 82 - Task management response 06 - Logout command 86 - Logout response 90 - R2T iSCSI header digest covers 48-byte header no data, so no data digest present 05 - SCSI Data write 85 - SCSI Data read 00 - NOP-Out 80 - NOP-In iSCSI header digest covers 48-byte header iSCSI data digest covers the data 04 - Text command 84 - Text response 03 - Login command 83 - Login response iSCSI header digest covers 48-byte iSCSI header iSCSI data digest covers the text parameters ??? Should the text be padded to a 4-byte boundary before adding the data CRC? I think that it already says so somewhere in the spec. -- Mark Matt Wakeley wrote: > > I don't think I like this. Header followed by digest followed by more header. > > I think this is a good opportunity to think about removing the CDB from the > header. This will reduce the amount of dead space used by headers that do not > contain a CDB. This will be beneficial when sending multiple smaller PDUs (in > order to keep the CRC coverage high) by reducing the iSCSI header overhead. > > Change the iSCSI command so that there is an iSCSI header (verified by the > header digest) followed by CDB "payload" (verified by the data digest). No > options for immediate data. Keep it simple, like it is in Fibre Channel. > > -Matt Wakeley > Agilent Technologies > > julian_satran@il.ibm.com wrote: > > > > yes, > > > > Julo > > > > "Barry Reinhold" on 22/01/2001 17:49:29 > > > > Please respond to "Barry Reinhold" > > > > To: mbakke@cisco.com, Julian Satran/Haifa/IBM@IBMIL > > cc: ips@ece.cmu.edu > > Subject: RE: Coverage of Data Digest when using Header Digests > > > > Ok, > > So if the bidi-read length and ADDCDB fields are not in the data > > digest > > then I assume we have to figure them into the header digest even though > > they > > are located past the header digets. Is that the expected behavior? > > > > -----Original Message----- > > From: mbakke@cisco.com [mailto:mbakke@cisco.com] > > Sent: Monday, January 22, 2001 10:22 AM > > To: julian_satran@il.ibm.com > > Cc: ips@ece.cmu.edu; Barry Reinhold > > Subject: Re: Coverage of Data Digest when using Header Digests > > > > Barry- > > > > In particular, the data digest covers only the SCSI Data part of an > > iSCSI message; the header digest covers everything else. This means > > that in an 8k write, the data digest will cover only the 8k, and the > > header digest will cover everything else. > > > > Hope this helps, > > > > Mark > > > > julian_satran@il.ibm.com wrote: > > > > > > Barry, > > > > > > Considering that one of the reasons to have a separate header and data > > > digest was to enable data > > > to carried through proxies, virtualizers etc. the current thinking is > > that > > > the data digest will cover only the data and the header (including > > > extensions) will be covered by the header digest. > > > > > > Julo > > > > > > "Barry Reinhold" on 21/01/2001 21:50:04 > > > > > > Please respond to "Barry Reinhold" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL > > > cc: > > > Subject: Coverage of Data Digest when using Header Digests > > > > > > Julian, > > > Is the Data Digest intended to cover whatever follows the 48 byte > > > iSCSI > > > header? In particular in a command frame which has a CDB > 16 bytes, uses > > > bidi, has immediate data, and is using both header and data digests - > > what > > > would the data digest cover? > > > - barry > > > reinhold > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Wed Jan 24 15:50:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA00417 for ; Wed, 24 Jan 2001 15:50:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0OI3LD17932 for ips-outgoing; Wed, 24 Jan 2001 13:03:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0OI2T617896 for ; Wed, 24 Jan 2001 13:02:30 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id 96B74372 for ; Wed, 24 Jan 2001 10:02:28 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id KAA21332; Wed, 24 Jan 2001 10:00:56 -0800 (PST) Message-ID: <3A6F186E.839908D8@hp.com> Date: Wed, 24 Jan 2001 10:01:18 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI: CRCs Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David (Black), At the Orlando interim meeting, you said that the iSCSI header could be changed on the way to the target. Because the "bidi" information could be changed. (It is the justification for 2 CRCs). Could you elaborate on that? What are the situations where the iSCSI header could be modified? What is the value added to do that? Regards, Pierre From owner-ips@ece.cmu.edu Wed Jan 24 19:50:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA02651 for ; Wed, 24 Jan 2001 19:50:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0OMH9a27736 for ips-outgoing; Wed, 24 Jan 2001 17:17:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0OMGR627712 for ; Wed, 24 Jan 2001 17:16:27 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 8F8D2109B for ; Wed, 24 Jan 2001 14:16:10 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id OAA15262 for ; Wed, 24 Jan 2001 14:15:48 -0800 (PST) Message-ID: <3A6F54A3.4BC65504@cup.hp.com> Date: Wed, 24 Jan 2001 14:18:11 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : draft contradicts itself in sections 1.2.5 & 5.5 Content-Type: multipart/mixed; boundary="------------090006E48789708929F6C228" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------090006E48789708929F6C228 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, In Section 1.2.5, the draft states that : "A target SHOULD NOT silently discard data and request re-transmission through R2T." In Section 5.5 the draft encourages the above to be performed by stating : "When a target receives an iSCSI Data PDU with a data payload digest error, it MUST discard it and request retransmission with a R2T." What is the intent of the former statement in Section 1.2.5, when digest error recovery requires just the opposite ? Regards, Santosh --------------090006E48789708929F6C228 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------090006E48789708929F6C228-- From owner-ips@ece.cmu.edu Wed Jan 24 19:50:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA02662 for ; Wed, 24 Jan 2001 19:50:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0OMDGU27605 for ips-outgoing; Wed, 24 Jan 2001 17:13:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0OMCA627564 for ; Wed, 24 Jan 2001 17:12:10 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id E4FE5BDA; Wed, 24 Jan 2001 14:12:04 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id OAA14771; Wed, 24 Jan 2001 14:11:30 -0800 (PST) Message-ID: <3A6F53A1.ED12D892@cup.hp.com> Date: Wed, 24 Jan 2001 14:13:53 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI: DataRN References: Content-Type: multipart/mixed; boundary="------------727EA3C29133B6B5B4693EFF" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------727EA3C29133B6B5B4693EFF Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > Santosh, > > The ACK for a response is when status is ACKed (by the status numbering). > > Before this ack a target has several options for restart: > > to keep all the data (recall we have dropped DataRN) and "restart > sending". This may work for a READ if the iSCSI layer has buffered the data until it gets a StatSN ACK. Note that even for the READ, status recovery is optional and a tape may not implement status / data recovery and just retry the command. In such a case, a retry results in the target re-doing the operation. For a WRITE however, I don't see how this is going to work. Are you suggesting that the iSCSI layer NOT deliver data to the SCSI layer until the Response is acknowleged ??? This is ruled out, since the Response itself comes from the SCSI layer upon completion of the SCSI command. If, on the WRITE, the data was delivered to the SCSI layer which then wrote it to media and advanced the tape, what good does the above do ?? The tape has advanced following the WRITE and the initiator can corrupt the tape by retrying the WRITE. > not to keep data and reject the command restart with a service response > (that we have to specify) of restart reject (your tape target may want > to do just this) You seem to suggest moving the responsibility of ensuring data integrity from the host to the target !! This is not suitable for the following reasons : a) Often, tape is the last class of storage products to migrate to new storage transport technologies. IOW, the majority of tape backup is going to attain iSCSI connectivity through a iSCSI-FC bridge or iSCS-pSCSI bridge. In such a case, the service response management is done in the bridge and this level of intelligence is now being moved into the bridge. The responsiblity of ensuring data integrity on a SCSI target lies with the initiator issuing the I/O. iSCSI MUST not retry a command to sequential media when there is the possiblity that some or all of the data phase has completed. Regards, Santosh --------------727EA3C29133B6B5B4693EFF Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------727EA3C29133B6B5B4693EFF-- From owner-ips@ece.cmu.edu Wed Jan 24 19:51:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA02673 for ; Wed, 24 Jan 2001 19:51:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0OMJBw27805 for ips-outgoing; Wed, 24 Jan 2001 17:19:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0OMIS627782 for ; Wed, 24 Jan 2001 17:18:28 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Wed, 24 Jan 2001 17:18:22 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101486@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Wed, 24 Jan 2001 17:18:17 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > The FC frame is placed on FC network within a short time of its arrival or > there is no advantage in allowing out of sequence processing. The "there is no advantage" statement depends on the architecture of the implementation. As usual, Doug continues to confuse out of order placement and processing - some FCIP implementations may be able to take advantage of out of order placement; out-of-order processing and forwarding would violate TCP's in order delivery semantics and cause other problems as described earlier in this thread. Some sort of timestamp approach from FCIP to FCIP (i.e., across TCP transmission and delivery) seems like a reasonable way to deal with issues that could cause an issue with FC fabric timeouts. It's also the case that the FC fabric timeouts need to be set with some regard to the presence and characteristics of FCIP links. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Wed Jan 24 19:51:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA02684 for ; Wed, 24 Jan 2001 19:51:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0OMOCs28034 for ips-outgoing; Wed, 24 Jan 2001 17:24:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0OMNa627994 for ; Wed, 24 Jan 2001 17:23:36 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id D4A8AEE0; Wed, 24 Jan 2001 14:23:31 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id OAA15718; Wed, 24 Jan 2001 14:22:45 -0800 (PST) Message-ID: <3A6F5644.4B5E7FC1@cup.hp.com> Date: Wed, 24 Jan 2001 14:25:08 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI : Login Phase Problems References: Content-Type: multipart/mixed; boundary="------------8F6417A98C9B1E7D8BD8C8DF" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------8F6417A98C9B1E7D8BD8C8DF Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, Thanks for the clarification. Can you comment on which of the below 2 alternatives is the approach the draft intends to take : 3) If multiple Text commands are to be allowed in the login phase, how does a target know that the initiator is done with all its key negotiations and has sent all its text commands ? If multiple Text commands are NOT allowed in the login phase, then, the draft MUST state this explicitly. Regards, Santosh julian_satran@il.ibm.com wrote: > Santosh, > > That is already in in 04 as I was refining the security negotiation. > > Regards, > Julo > > Santosh Rao on 24/01/2001 09:18:30 > > Please respond to Santosh Rao > > To: ips@ece.cmu.edu (ips) > cc: santoshr@hpcuhe.cup.hp.com (Santosh Rao) > Subject: iSCSI : Login Phase Problems > > Julian & All, > > Problem : > ========= > The login phase is intended to be driven by the initiator and it is the > one to decide if only a Login PDU is sufficient or it needs a Login + Text > PDU. The initiator needs a mechanism to indicate to the target that a Text > command will [or will not] follow the login command. > > However, by placing the "F" bit in the login response and NOT the login > command, the iSCSI login phase opens up a number of questions : --------------8F6417A98C9B1E7D8BD8C8DF Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------8F6417A98C9B1E7D8BD8C8DF-- From owner-ips@ece.cmu.edu Wed Jan 24 22:08:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA05163 for ; Wed, 24 Jan 2001 22:08:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0P0mC902437 for ips-outgoing; Wed, 24 Jan 2001 19:48:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0P0lf602420 for ; Wed, 24 Jan 2001 19:47:41 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Wed, 24 Jan 2001 19:47:35 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A070410148A@corpmx9.isus.emc.com> From: Black_David@emc.com To: pierre_labat@hp.com, ips@ece.cmu.edu Subject: RE: iSCSI: CRCs Date: Wed, 24 Jan 2001 19:47:33 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk As I recall the discussion, it was about facilitating iSCSI to iSCSI "middle boxes" - gateways, proxies, etc. of various forms that may want to muck with the header for some reason. Such a box is a target on one iSCSI connection and initiator on another, and only having to regenerate the header CRC makes such a box easier to implement, and improves the protection obtained from the data CRC. --David > -----Original Message----- > From: Pierre Labat [SMTP:pierre_labat@hp.com] > Sent: Wednesday, January 24, 2001 1:01 PM > To: ips@ece.cmu.edu > Subject: iSCSI: CRCs > > David (Black), > > > At the Orlando interim meeting, you said that the iSCSI header > could be changed on the way to the target. Because the "bidi" > information could be changed. (It is the justification > for 2 CRCs). > > Could you elaborate on that? > > What are the situations where the iSCSI header could be modified? > What is the value added to do that? > > > Regards, > > Pierre From owner-ips@ece.cmu.edu Wed Jan 24 22:12:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA05236 for ; Wed, 24 Jan 2001 22:12:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0P00EK01044 for ips-outgoing; Wed, 24 Jan 2001 19:00:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0OBw8606536 for ; Wed, 24 Jan 2001 06:58:08 -0500 (EST) Received: from divyaroot.India.Sun.COM ([129.158.226.35]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id DAA25278 for ; Wed, 24 Jan 2001 03:58:06 -0800 (PST) Received: from helix (helix [129.158.226.51]) by divyaroot.India.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1) with SMTP id RAA18353 for ; Wed, 24 Jan 2001 17:28:04 +0530 (IST) Message-Id: <200101241158.RAA18353@divyaroot.India.Sun.COM> Date: Wed, 24 Jan 2001 17:29:45 -0500 (GMT) From: Raghavendra Rao Reply-To: Raghavendra Rao Subject: Re: iSCSI : Login Phase Problems To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: HCZ9ENfzAVuiqETgi7I/eg== Sender: owner-ips@ece.cmu.edu Precedence: bulk > The login phase is intended to be driven by the initiator and it is the > one to decide if only a Login PDU is sufficient or it needs a Login + Text > PDU. The initiator needs a mechanism to indicate to the target that a Text > command will [or will not] follow the login command. > What is being achieved by preparing the target to expect a text command after login ? It is the responsibility of the initiator to negotiate before getting into serious stuff - If it does serious stuff without negotiations, or before negotiations the defaults will take effect. I don't see a reason to change this. -JP From owner-ips@ece.cmu.edu Wed Jan 24 23:12:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA06754 for ; Wed, 24 Jan 2001 23:12:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0P25G504487 for ips-outgoing; Wed, 24 Jan 2001 21:05:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0P24i604473 for ; Wed, 24 Jan 2001 21:04:44 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 8A482C1D for ; Wed, 24 Jan 2001 18:04:43 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA07002 for ; Wed, 24 Jan 2001 18:04:35 -0800 (PST) Message-ID: <3A6F8A42.789B1B1F@cup.hp.com> Date: Wed, 24 Jan 2001 18:06:59 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Content-Type: multipart/mixed; boundary="------------845BFF605983A6EBD9670081" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------845BFF605983A6EBD9670081 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian & All, There's a number of problems with the current iSCSI handling of digest errors (Section 5.5) : "When an initiator receives an iSCSI data PDU with a digest error or any other PDU with a header or data digest error, it must discard and re-start the task -provided it can recognize the Initiator Task Tag". 1) The initiator task tag cannot be trusted when a header digest error is seen. What does the phrase "provided it can recognize the initiator task tag" mean ? How can an initiator reliably claim that the initiator task tag is trustworthy ? 2) The use of a "discard and re-start" policy can cause some problems. The target is un-aware that the initiator has discarded the task and will then receive a 2nd command with the same Initiator Task Tag and CmdSN for a task already in progress, which can cause transmission of duplicate sets of data on the same I.T.T. Any policy of "discard and re-start" must be modified to "discard, abort and then, re-start, and the draft MUST specify connection allegiance of the command and its Abort Task". 3) The policy of "discard and restart" is also subject some race conditions in the CmdSN sliding window. At the time the digest error was detected at the initiator, the ExpCmdSN may not yet have acknowledged that command. This causes the initiator to restart the command with the same Initiator Task Tag, CmdSN and "retry" bit set. However, by the time the command gets to the target, the target may have updated its ExpCmdSN window having sent a later PDU which updated the ExpCmdSN. This results in the target discarding the received CmdSN since it is outside the expected window of (ExpCmdSN, MaxCmdSN). The "discard & restart" policy cannot be reliably used unless it is on a connection failure, in which case, the failed connection is first logged out and an updated ExpCmdSN is received on the logout response. Following the logout response, the "retry" commands CmdSN can be based on the ExpCmdSN. In the case of digest errors, the CmdSN window can be moving forward and an attempt to "discard and restart" will result in a re-send of a CmdSN that is outside the CmdSN window, causing the target to discard the retried command. Regards, Santosh --------------845BFF605983A6EBD9670081 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------845BFF605983A6EBD9670081-- From owner-ips@ece.cmu.edu Wed Jan 24 23:12:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA06765 for ; Wed, 24 Jan 2001 23:12:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0P1rFa04183 for ips-outgoing; Wed, 24 Jan 2001 20:53:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0P1qv604172 for ; Wed, 24 Jan 2001 20:52:57 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 3C55310CB; Wed, 24 Jan 2001 17:52:56 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA06364; Wed, 24 Jan 2001 17:52:51 -0800 (PST) Message-ID: <3A6F8782.E1B7AADA@cup.hp.com> Date: Wed, 24 Jan 2001 17:55:14 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Raghavendra Rao Cc: ips@ece.cmu.edu Subject: Re: iSCSI : Login Phase Problems References: <200101241158.RAA18353@divyaroot.India.Sun.COM> Content-Type: multipart/mixed; boundary="------------088866297F0DE7E52680FBC0" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------088866297F0DE7E52680FBC0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Raghavendra Rao wrote: > > The login phase is intended to be driven by the initiator and it is the > > one to decide if only a Login PDU is sufficient or it needs a Login + Text > > PDU. The initiator needs a mechanism to indicate to the target that a Text > > command will [or will not] follow the login command. > > > > What is being achieved by preparing the target to expect a text command after > login ? > > It is the responsibility of the initiator to negotiate before getting into > serious stuff - If it does serious stuff without negotiations, or before > negotiations the defaults will take effect. Your approach above is to continue login negotiation after going into full feature phase (as indicated by login response with "F" bit). The draft's intent seems to be to complete all negotiations in the login phase (prior to login response with "F" bit set). The current specification of login phase provides an "F" bit in the login response. This is of no use and provides no value-add. ( 1) when/how does the target set or not set the "F" bit ? ) In any case, the simplest approach would be to just use a single login command for all negotiations. Why add all this complexity of login, followed by partial response, followed by text, text response and then final login response ?? - Santosh --------------088866297F0DE7E52680FBC0 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------088866297F0DE7E52680FBC0-- From owner-ips@ece.cmu.edu Thu Jan 25 00:48:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA07906 for ; Thu, 25 Jan 2001 00:48:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0P3YMj06712 for ips-outgoing; Wed, 24 Jan 2001 22:34:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0P3XN606690 for ; Wed, 24 Jan 2001 22:33:23 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id WAA64152 for ; Wed, 24 Jan 2001 22:26:29 -0500 Received: from d03nmx42.almaden.ibm.com (d03nmx42.almaden.ibm.com [9.1.24.146]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id UAA117478 for ; Wed, 24 Jan 2001 20:33:18 -0700 Importance: Normal Subject: Re: iSCSI: INQUIRY page 0x83 identifier To: ips@ece.cmu.edu From: "Jim Hafner" Date: Wed, 24 Jan 2001 19:33:17 -0800 Message-ID: X-MIMETrack: Serialize by Router on D03NMX42/03/M/IBM(Build V60_01102001|January 10, 2001) at 01/24/2001 07:33:18 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk JP, The identifier in this page of Inquiry is NOT at the 'target device' or 'iSCSI device' layer. It is an identifier for the logical unit to which the inquiry command was sent. The FC format was meant primarily for FC disk drives that have only one logical unit (though it can be used in some ways by FC controllers, etc.). The Device (as in a container of one or more logical units) identifier is yet to be fully defined in SCSI, which is one reason that iSCSI NDT chose to define such a name for the iSCSI device. As for an iSCSI type or format for logical unit identifiers: 1) SCSI already allows for an ascii formatted identifier (typically this includes vendor, model, serial number). Since NDT has defined WWUIs for DEVICES and these have a defined UTF-8 format, there is nothing to *prevent* a vendor from using these strings as the starting point for LU page 83 identifiers of the logical units within that device. 2) But..., iSCSI and NDT should not attempt to make a rule under which these are used. FC's intervention in this was more as a strong suggestion, and is certainly not a requirement. Did I confuse things even more? Jim Hafner Raghavendra Rao @ece.cmu.edu on 01/23/2001 08:02:05 AM Please respond to Raghavendra Rao Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: iSCSI: INQUIRY page 0x83 identifier This may seem like a T10 issue at a high level, but I'm concerned that SPC-2 (SCSI primary command set) defines in the INQUIRY page 0x83 identifier type field, a FC identifier type (value = 3). What identifer type should an iSCSI target use to report a LU identifier when queried for page 0x83 ? Luckily there seem to be a generic IEEE extended identifier type (value = 2). Is the naming and discovery document going to provide a clear defintion about LU names ? I understand that there should be NO interconnect specific identifier at the SCSI layer - But SPC-2 is already polluted with a FC identifier type. -JP From owner-ips@ece.cmu.edu Thu Jan 25 04:23:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA22368 for ; Thu, 25 Jan 2001 04:23:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0P6pNa11060 for ips-outgoing; Thu, 25 Jan 2001 01:51:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0P6p8611051 for ; Thu, 25 Jan 2001 01:51:08 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0P81i051918; Thu, 25 Jan 2001 00:01:44 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: , Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Wed, 24 Jan 2001 22:49:51 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704101486@corpmx9.isus.emc.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, In the case of FCIP, a scheme for framing is not to allow buffering in user space as perhaps could be said for iSCSI. The only option allowed would be to place the FC frame onto the FC media. In this case, there is no confusion and you are right such scheme would be a violation of TCP. In reality, reading the content of a TCP segment to move the data portion of the segment into user space is also not in keeping with TCP in the purist sense either. If you are right about FCIP not allowed to "place" FC frames prior to the reception of a missing TCP segment, then there is absolutely no reason to have any type of framing for FCIP. Everything will be required to be buffered anyway for as long as it takes. Doug > > The FC frame is placed on FC network within a short time of its > arrival or > > there is no advantage in allowing out of sequence processing. > > The "there is no advantage" statement depends on > the architecture of the implementation. As usual, > Doug continues to confuse out of order placement > and processing - some FCIP implementations may be > able to take advantage of out of order placement; > out-of-order processing and forwarding would violate > TCP's in order delivery semantics and cause other > problems as described earlier in this thread. > > Some sort of timestamp approach from FCIP to FCIP > (i.e., across TCP transmission and delivery) seems > like a reasonable way to deal with issues that could > cause an issue with FC fabric timeouts. It's also > the case that the FC fabric timeouts need to be set > with some regard to the presence and characteristics > of FCIP links. > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Thu Jan 25 07:34:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA24332 for ; Thu, 25 Jan 2001 07:34:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PA6Uv23688 for ips-outgoing; Thu, 25 Jan 2001 05:06:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from exchange-1.xiotech.com (ftp.xiotech.com [209.46.118.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PA6K623113 for ; Thu, 25 Jan 2001 05:06:21 -0500 (EST) Message-ID: From: "Peglar, Robert" To: ips@ece.cmu.edu Subject: RE: iSCSI: remove CDB from iSCSI header Date: Thu, 25 Jan 2001 04:07:09 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree with Mark and Matt. As for Mark's question about padding, 2.2.3 seems to indicate so, but padding is a good idea and the text portion of the 04/84 and 03/83 should indicate so. Aligned CRCs are a good thing. Rob Rob Peglar XIOtech Corporation (314) 308-6983 > -----Original Message----- > From: Mark Bakke [mailto:mbakke@cisco.com] > Sent: Tuesday, January 23, 2001 4:51 PM > To: Matt Wakeley > Cc: ips@ece.cmu.edu > Subject: Re: iSCSI: remove CDB from iSCSI header > > > > I agree with Matt. Basically, then, the iSCSI header digest > should cover > everything in the first 48 bytes. All of the messages, with > the exception > of the SCSI Command Request, could easily be done this way, > and I think > that we should modify that to fit Matt's idea of treating the CDB (and > sense data) the same as SCSI data for digest purposes. > > Here's what they would all look like: > > 01 - SCSI Command Request > > The iSCSI header digest currently covers the 48-byte iSCSI header, > which includes the CDB. > > The data digest covers everything else, which includes extended CDB, > bidi-read transfer length, and immediate data. > > Since the command request is the only operation that really breaks > the notion of Matt's model, I would suggest that we start the CDB > after the 48-byte iSCSI header, and that it not be covered by the > header digest. We can move the bidi-read length inside the iSCSI > header, and end up with: > > +---------------------------------------------------- > 0| Normal iSCSI header stuff for command request > +---------------------------------------------------- > 32| Bidi-read length > +-------------------------------------------------- > 36/ Reserved > +/ > +--------------------------------------------------- > | (optional header digest goes here) > +------------------------------------------------- > 48| CDB > +-------------------------------------------------- > | Additional CDB > +--------------------------------------------------- > | Immediate Data > +---------------------------------------------------- > | (optional data digest goes here) > +----------------------------------------------------- > > The header digest would cover the first 48 bytes, and would exactly > match the rest of the iSCSI operations. The data digest would include > the CDB (now contiguous with the additional CDB, as an added bonus) > and the immediate data. Anyone concerned with having a separate > data integrity check for just the data when an additional CDB is > being used should refrain from using the immediate data feature. > > While we are on the topic, is anyone really planning to make > use of bidi-read > and immediate data? > >  > Here is my understanding of what the digests cover with the remainder > of the iSCSI operations: > > 81 - SCSI Response > 91 - Asynchronous Event > > iSCSI header digest covers 48-byte iSCSI header > iSCSI data digest covers the sense data, if present > > 02 - Task management command > 82 - Task management response > 06 - Logout command > 86 - Logout response > 90 - R2T > > iSCSI header digest covers 48-byte header > no data, so no data digest present > > 05 - SCSI Data write > 85 - SCSI Data read > 00 - NOP-Out > 80 - NOP-In > > iSCSI header digest covers 48-byte header > iSCSI data digest covers the data > > 04 - Text command > 84 - Text response > 03 - Login command > 83 - Login response > > iSCSI header digest covers 48-byte iSCSI header > iSCSI data digest covers the text parameters > > ??? Should the text be padded to a 4-byte boundary before > adding the data CRC? > I think that it already says so somewhere in the spec. > > -- > Mark > > Matt Wakeley wrote: > > > > I don't think I like this. Header followed by digest > followed by more header. > > > > I think this is a good opportunity to think about removing > the CDB from the > > header. This will reduce the amount of dead space used by > headers that do not > > contain a CDB. This will be beneficial when sending > multiple smaller PDUs (in > > order to keep the CRC coverage high) by reducing the iSCSI > header overhead. > > > > Change the iSCSI command so that there is an iSCSI header > (verified by the > > header digest) followed by CDB "payload" (verified by the > data digest). No > > options for immediate data. Keep it simple, like it is in > Fibre Channel. > > > > -Matt Wakeley > > Agilent Technologies > > > > julian_satran@il.ibm.com wrote: > > > > > > yes, > > > > > > Julo > > > > > > "Barry Reinhold" on 22/01/2001 17:49:29 > > > > > > Please respond to "Barry Reinhold" > > > > > > To: mbakke@cisco.com, Julian Satran/Haifa/IBM@IBMIL > > > cc: ips@ece.cmu.edu > > > Subject: RE: Coverage of Data Digest when using Header Digests > > > > > > Ok, > > > So if the bidi-read length and ADDCDB fields are not > in the data > > > digest > > > then I assume we have to figure them into the header > digest even though > > > they > > > are located past the header digets. Is that the expected behavior? > > > > > > -----Original Message----- > > > From: mbakke@cisco.com [mailto:mbakke@cisco.com] > > > Sent: Monday, January 22, 2001 10:22 AM > > > To: julian_satran@il.ibm.com > > > Cc: ips@ece.cmu.edu; Barry Reinhold > > > Subject: Re: Coverage of Data Digest when using Header Digests > > > > > > Barry- > > > > > > In particular, the data digest covers only the SCSI Data > part of an > > > iSCSI message; the header digest covers everything else. > This means > > > that in an 8k write, the data digest will cover only the > 8k, and the > > > header digest will cover everything else. > > > > > > Hope this helps, > > > > > > Mark > > > > > > julian_satran@il.ibm.com wrote: > > > > > > > > Barry, > > > > > > > > Considering that one of the reasons to have a separate > header and data > > > > digest was to enable data > > > > to carried through proxies, virtualizers etc. the > current thinking is > > > that > > > > the data digest will cover only the data and the header > (including > > > > extensions) will be covered by the header digest. > > > > > > > > Julo > > > > > > > > "Barry Reinhold" on 21/01/2001 21:50:04 > > > > > > > > Please respond to "Barry Reinhold" > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL > > > > cc: > > > > Subject: Coverage of Data Digest when using Header Digests > > > > > > > > Julian, > > > > Is the Data Digest intended to cover whatever > follows the 48 byte > > > > iSCSI > > > > header? In particular in a command frame which has a > CDB > 16 bytes, uses > > > > bidi, has immediate data, and is using both header and > data digests - > > > what > > > > would the data digest cover? > > > > > - barry > > > > reinhold > > > > > > -- > > > Mark A. Bakke > > > Cisco Systems > > > mbakke@cisco.com > > > 763.398.1054 > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 > From owner-ips@ece.cmu.edu Thu Jan 25 12:13:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA04396 for ; Thu, 25 Jan 2001 12:13:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PEgca01354 for ips-outgoing; Thu, 25 Jan 2001 09:42:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from glatton.cnchost.com (glatton.cnchost.com [207.155.248.47]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PEgP601347 for ; Thu, 25 Jan 2001 09:42:25 -0500 (EST) Received: from BSNCBINFORD (service.recreationresource.com [208.189.17.82] (may be forged)) by glatton.cnchost.com id JAA23934; Thu, 25 Jan 2001 09:42:22 -0500 (EST) [ConcentricHost SMTP Relay 1.10] From: "Charles Binford" To: Subject: RE: iSCSI: INQUIRY page 0x83 identifier Date: Thu, 25 Jan 2001 08:45:31 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Jim, You didn't confuse me more - you made a very good point. The identifier is for the LU and is *independent* of the transport. In fact, it is vital that the same identifier be given over any path, any transport. Consider a storage device that has multiple host interfaces, some may be FC, some may be iSCSI, some may be parallel SCSI. If a host does an INQUIRY page x83 to the same LU through any of the above mentioned interfaces, it better get the same identifier back. Using an "FC type identifier" does not imply the interface is FC. It just specifies the format of the data. For the binary identifiers, even the format of the identifier is a 'don't care' to the host as it should treat them as opaque fields that can be matched, but not parsed. By following the format rules, however, the LU can ensure the identifier it gives is world-wide unique. Charles Binford Blue Spruce Networks office/cell: (316) 210-6404 e-fax: (509) 756-4425 -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Jim Hafner Sent: Wednesday, January 24, 2001 9:33 PM To: ips@ece.cmu.edu Subject: Re: iSCSI: INQUIRY page 0x83 identifier JP, The identifier in this page of Inquiry is NOT at the 'target device' or 'iSCSI device' layer. It is an identifier for the logical unit to which the inquiry command was sent. The FC format was meant primarily for FC disk drives that have only one logical unit (though it can be used in some ways by FC controllers, etc.). [deleted] Did I confuse things even more? Jim Hafner From owner-ips@ece.cmu.edu Thu Jan 25 12:13:26 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA04400 for ; Thu, 25 Jan 2001 12:13:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PFLcb02674 for ips-outgoing; Thu, 25 Jan 2001 10:21:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PFKp602655 for ; Thu, 25 Jan 2001 10:20:51 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 25 Jan 2001 10:21:09 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A070410148E@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Thu, 25 Jan 2001 10:20:22 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk >In the case of FCIP, a scheme for framing is not to allow buffering in > user space as perhaps could be said for iSCSI. Again, that statement is implementation-dependent/specific, and there may be advantages to placement even when there isn't any notion of "user space" depending on the hardware and software architecture of the implementation. It's definitely possible to build FCIP nodes that don't require framing support, BUT that doesn't mean that all nodes will be built in that fashion (the same statement applies to iSCSI targets, FWIW). --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Thu Jan 25 12:14:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA04428 for ; Thu, 25 Jan 2001 12:14:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PFhu703412 for ips-outgoing; Thu, 25 Jan 2001 10:43:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PFgi603362 for ; Thu, 25 Jan 2001 10:42:44 -0500 (EST) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id 5DFD38E5; Thu, 25 Jan 2001 07:42:43 -0800 (PST) Received: from mk731913.cup.hp.com (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA13330; Thu, 25 Jan 2001 07:43:58 -0800 (PST) Message-Id: <5.0.0.25.2.20010125072913.025721b0@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.0 Date: Thu, 25 Jan 2001 07:37:12 -0800 To: Black_David@emc.com From: Michael Krause Subject: RE: iSCSI: CRCs Cc: ips@ece.cmu.edu In-Reply-To: <0F31E5C394DAD311B60C00E029101A070410148A@corpmx9.isus.emc. com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 07:47 PM 1/24/2001 -0500, Black_David@emc.com wrote: >As I recall the discussion, it was about facilitating >iSCSI to iSCSI "middle boxes" - gateways, proxies, >etc. of various forms that may want to muck with >the header for some reason. Such a box is a target >on one iSCSI connection and initiator on another, >and only having to regenerate the header CRC >makes such a box easier to implement, and improves >the protection obtained from the data CRC. This is extremely dangerous since the header could be corrupted in the intermediate element and one acts upon the header for determining where the payload is going to be placed. To support such a solution would require one to use a similar scheme to what is provided in InfiniBand: - Clear delineation of what fields are variant and what are not. - A variant CRC that covers the variant fields and the rest of the PDU. - An invariant CRC that does not cover the variant fields but does cover the invariant fields and the rest of the PDU. This scheme insures there is no time that critical header / data fields are not protected end-to-end allowing one to have a strong data integrity solution. BTW, InfiniBand uses a 16-bit variant CRC and a 32-bit invariant CRC. InfiniBand also does this per packet which goes to another issue with the current draft which is the problem of a PDU spanning multiple TCP segments - more on that in a separate e-mail. Mike From owner-ips@ece.cmu.edu Thu Jan 25 12:19:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA04546 for ; Thu, 25 Jan 2001 12:19:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PF5jF02169 for ips-outgoing; Thu, 25 Jan 2001 10:05:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PF53602114 for ; Thu, 25 Jan 2001 10:05:04 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAA27240 for ; Thu, 25 Jan 2001 16:04:35 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA21630 for ; Thu, 25 Jan 2001 16:04:35 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DF.0052CE4C ; Thu, 25 Jan 2001 16:04:27 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 25 Jan 2001 16:59:58 +0200 Subject: Re: iSCSI: DataRN Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, The current draft has status numbering as SHOULD and that is (as it was pointed out to me) almost mandatory. Considering that the consensus at Orlando and on the list was to make CmdRN (now CmdSN) mandatory even for one connection I don't see any reason not to make StatSN support also mandatory . On Writes our assumptions was that the target can at any point in time either reject the restart (if as you outline it can't do it) or if it can reacquire the data it misses trough R2T and resend the status (on writes the target is assumed to know what it needs). In case it rejects the restart you can report a "transport failure" and you are as good as a parallel packetized SCSI. Julo Santosh Rao on 25/01/2001 00:13:53 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI: DataRN julian_satran@il.ibm.com wrote: > Santosh, > > The ACK for a response is when status is ACKed (by the status numbering). > > Before this ack a target has several options for restart: > > to keep all the data (recall we have dropped DataRN) and "restart > sending". This may work for a READ if the iSCSI layer has buffered the data until it gets a StatSN ACK. Note that even for the READ, status recovery is optional and a tape may not implement status / data recovery and just retry the command. In such a case, a retry results in the target re-doing the operation. For a WRITE however, I don't see how this is going to work. Are you suggesting that the iSCSI layer NOT deliver data to the SCSI layer until the Response is acknowleged ??? This is ruled out, since the Response itself comes from the SCSI layer upon completion of the SCSI command. If, on the WRITE, the data was delivered to the SCSI layer which then wrote it to media and advanced the tape, what good does the above do ?? The tape has advanced following the WRITE and the initiator can corrupt the tape by retrying the WRITE. > not to keep data and reject the command restart with a service response > (that we have to specify) of restart reject (your tape target may want > to do just this) You seem to suggest moving the responsibility of ensuring data integrity from the host to the target !! This is not suitable for the following reasons : a) Often, tape is the last class of storage products to migrate to new storage transport technologies. IOW, the majority of tape backup is going to attain iSCSI connectivity through a iSCSI-FC bridge or iSCS-pSCSI bridge. In such a case, the service response management is done in the bridge and this level of intelligence is now being moved into the bridge. The responsiblity of ensuring data integrity on a SCSI target lies with the initiator issuing the I/O. iSCSI MUST not retry a command to sequential media when there is the possiblity that some or all of the data phase has completed. Regards, Santosh - santoshr.vcf From owner-ips@ece.cmu.edu Thu Jan 25 12:49:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA04979 for ; Thu, 25 Jan 2001 12:49:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PFhrt03408 for ips-outgoing; Thu, 25 Jan 2001 10:43:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PFh7603375 for ; Thu, 25 Jan 2001 10:43:07 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA57930 for ; Thu, 25 Jan 2001 16:42:58 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA48946 for ; Thu, 25 Jan 2001 16:42:59 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DF.00565163 ; Thu, 25 Jan 2001 16:42:49 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 25 Jan 2001 17:38:22 +0200 Subject: iSCSI PDU outline Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear all, Please hold your fire on digest placement, CDB inclusion/exclusion etc. The reasons we choose a specific outlay are sound and we have to hold on to many of the even if are going to change. As I said several times in the past some changes in the SCSI formats, like the advent of the bi-directional operations required as to have more and more exceptions and whole outline became ugly but I did not have a sound replacement and we have many more open issues. I have now a pretty good idea of how this new outline will look and I will publish it here - hopefully TODAY. In fact it is a minor change - although it has a rather large editorial impact as I had to make clear how it works now and how it can be extended in the future. It still keeps most of the packets with one fixed length packet (effective) but it is regular and can be extended. And as a matter of debate civility as long as a vote is not called for any of us would appreciate new technical arguments for or against something rather than "I too". And again - I am sorry it took me so long to get out a new format - and cause you such an agravation, but formats where low on my list of priorities and I had to consider many alternatives (many more than where discussed on the list). Julo From owner-ips@ece.cmu.edu Thu Jan 25 12:51:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA05021 for ; Thu, 25 Jan 2001 12:51:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PFae203174 for ips-outgoing; Thu, 25 Jan 2001 10:36:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PFa7603146 for ; Thu, 25 Jan 2001 10:36:07 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 25 Jan 2001 10:36:13 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A070410148F@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: iSCSI: retries and SCSI Date: Thu, 25 Jan 2001 10:34:19 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk If one thinks about retries wrt layering, the iSCSI notion of retry is at the iSCSI layer and hence is aimed primarily at causing the target to retransmit stuff (data, responses) that didn't make it to the initiator previously for some reason (e.g., connection or CRC failure). That's an architectural purist viewpoint that views an iSCSI retry as being about iSCSI, not SCSI, and hence about having the target retransmit iSCSI stuff that it's retained, as opposed to re- executing SCSI operations. A practical engineering viewpoint then notices that if the SCSI operation is idempotent (doing it over has no ill effects - generally the case for disk operations), then a target that has not retained the stuff required to respond to a retry could re-execute the operation as opposed to failing the retry. The motivation for doing this is that an iSCSI recovery on the initiator side should in general be cheaper (time/resources) than a SCSI recovery. When the operation is not idempotent (doing it over has ill effects - tape reads and writes are examples), then this optimization is not applicable and if the target hasn't retained the stuff required to respond to the retry, it has to fail/reject the retry. This is among the topics that should be explained in the more detailed description of error recovery that was recognized as needed in the draft as part of the Orlando meeting. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Thu Jan 25 14:46:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA07261 for ; Thu, 25 Jan 2001 14:45:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PI4kL08108 for ips-outgoing; Thu, 25 Jan 2001 13:04:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PI48608078 for ; Thu, 25 Jan 2001 13:04:08 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 20FEE1593; Thu, 25 Jan 2001 10:04:07 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id KAA25728; Thu, 25 Jan 2001 10:02:49 -0800 (PST) Message-ID: <3A706ADA.A10B7D9@cup.hp.com> Date: Thu, 25 Jan 2001 10:05:14 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI : draft contradicts itself in sections 1.2.5 & 5.5 References: Content-Type: multipart/mixed; boundary="------------D2DC6CA799C38E26467CFC25" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------D2DC6CA799C38E26467CFC25 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > SHOULD NOT means NOT unless you have a good reason not too. And with a > digest error you have a good reason to - don't you? Not if you consider all the problems raised regarding this "discard and retry" policy on digest errors. See : http://ips.pdl.cs.cmu.edu/mail/msg03110.html - Santosh > > Santosh Rao on 25/01/2001 00:18:11 > > Please respond to Santosh Rao > > To: IPS Reflector > cc: > Subject: iSCSI : draft contradicts itself in sections 1.2.5 & 5.5 > > Julian, > > In Section 1.2.5, the draft states that : > "A target SHOULD NOT silently discard data and request re-transmission > through R2T." > > In Section 5.5 the draft encourages the above to be performed by stating > : > "When a target receives an iSCSI Data PDU with a data payload digest > error, it MUST discard it and request retransmission with a R2T." > > What is the intent of the former statement in Section 1.2.5, when digest > error recovery requires just the opposite ? > > Regards, > Santosh > > - santoshr.vcf --------------D2DC6CA799C38E26467CFC25 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------D2DC6CA799C38E26467CFC25-- From owner-ips@ece.cmu.edu Thu Jan 25 14:47:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA07290 for ; Thu, 25 Jan 2001 14:47:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PH4hg06161 for ips-outgoing; Thu, 25 Jan 2001 12:04:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PH3c606121 for ; Thu, 25 Jan 2001 12:03:39 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA49048 for ; Thu, 25 Jan 2001 18:03:11 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA204160 for ; Thu, 25 Jan 2001 18:03:09 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DF.005DA9EC ; Thu, 25 Jan 2001 18:03:03 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 25 Jan 2001 18:58:37 +0200 Subject: new iSCSI PDU outline Mime-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=dTddHe2TkczLRtdjwcs4m2zeG39EF81uuPaLRjQhE1XVge0qIgBFlkJ4" Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=dTddHe2TkczLRtdjwcs4m2zeG39EF81uuPaLRjQhE1XVge0qIgBFlkJ4 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Dear all, Here is the new outline. Julo (See attached file: iSCSI-PDU-outline.txt) --0__=dTddHe2TkczLRtdjwcs4m2zeG39EF81uuPaLRjQhE1XVge0qIgBFlkJ4 Content-type: application/octet-stream; name="iSCSI-PDU-outline.txt" Content-Disposition: attachment; filename="iSCSI-PDU-outline.txt" Content-Description: Text - character set unknown Content-Transfer-Encoding: base64 Mi4gaVNDU0kgUERVIEZvcm1hdHMNCg0KQWxsIG11bHRpLWJ5dGUgaW50ZWdlcnMgc3BlY2lmaWVk IGluIGZvcm1hdHMgZGVmaW5lZCBpbiB0aGlzIGRvY3VtZW50IGFyZSB0byBiZSByZXByZXNlbnRl ZCBpbiBuZXR3b3JrIGJ5dGUgb3JkZXIgKGkuZS4sIGJpZyBlbmRpYW4pLiAgQW55IGJpdHMgbm90 IGRlZmluZWQgTVVTVCBiZSBzZXQgdG8gemVyby4gQW55IHJlc2VydmVkIGZpZWxkcyBhbmQgdmFs dWVzIE1VU1QgYmUgMCB1bmxlc3Mgc3BlY2lmaWVkIG90aGVyd2lzZS4NCg0KMi4xIGlTQ1NJIFBE VSBsZW5ndGggYW5kIHBhZGRpbmcNCg0KaVNDU0kgUERVcyBhcmUgcGFkZGVkIHRvIGFuIGludGVn ZXIgbnVtYmVyIG9mIDQgYnl0ZSB3b3Jkcy4NCg0KMi4yIFBEVSBUZW1wbGF0ZSwgSGVhZGVyIGFu ZCBPcGNvZGVzDQoNCkFsbCBpU0NTSSBQRFVzIGJlZ2luIHdpdGggb25lIG9yIG1vcmUgaGVhZGVy IHNlZ21lbnRzIGZvbGxvd2VkIGJ5IDAgb3IgMSBkYXRhIHNlZ21lbnRzLiAgQWZ0ZXIgdGhlIGVu dGlyZSBoZWFkZXIgc2VnbWVudCBncm91cCB0aGVyZSBNQVkgYmUgYSBoZWFkZXItZGlnZXN0LiBU aGUgZGF0YSBzZWdtZW50IE1BWSBhbHNvIGJlIGZvbGxvd2VkIGJ5IGEgZGF0YS1kaWdlc3QuICAN Cg0KVGhlIGZpcnN0IHNlZ21lbnQgLSBhbmQgaW4gbWFueSBjYXNlcyB0aGUgb25seSBzZWdtZW50 IC0gKEJhc2ljIEhlYWRlciBTZWdtZW50IG9yIEJIUykgaXMgYSBmaXhlZC1sZW5ndGggNDQtYnl0 ZSBoZWFkZXIgc2VnbWVudC4NCkl0IG1heSBiZSBmb2xsb3dlZCBieSBBZGRpdGlvbmFsIEhlYWRl ciBTZWdtZW50cyAoQUhTKS4gRWFjaCBzZWdtZW50IGlzIHByZWNlZGVkIGJ5IGEgNCBieXRlIE5l eHQtUXVhbGlmaWVyLiAgVGh1cyB3aGVuIHdlIGhhdmUgb25seSBhIEJIUyAod2l0aCBubyBkYXRh IG9yIGRpZ2VzdHMpIHRoZSBuZXQgc2l6ZSBvZiB0aGUgaVNDU0kgUERVIGlzIDQ4IGJ5dGVzLg0K IA0KDQpUaGUgb3ZlcmFsbCBzdHJ1Y3R1cmUgb2YgYSBQRFUgaXM6DQoNCg0KQnl0ZSAvICAgIDAg ICAgICAgfCAgICAgICAxICAgICAgIHwgICAgICAgMiAgICAgICB8ICAgICAgIDMgICAgICAgfA0K ICAgLyAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAgICAg ICAgICAgICAgfA0KICB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMg MiAxIDB8NyA2IDUgNCAzIDIgMSAwfCAgICAgICAgICAgDQogICstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogMHwgV04gICAg ICAgICAgICB8V04gc3BlY2lmaWMgZmllbGRzICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8 DQogICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rDQogNC8gQkhTICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAvDQogKy8gICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQogICstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQo0OHwgV04g ICAgICAgICAgICB8V04gc3BlY2lmaWMgZmllbGRzICAgICAgICAgICAgICAgICAgICAgICAgICAg ICB8DQogICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rDQo1Mi8gQUhTICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAvDQogKy8gICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQogICstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQotLS0t ICANCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLSsNCiBtLyBIZWFkZXItRGlnZXN0IChvcHRpb25hbCkgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiArLyAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAgKy0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiBuLyBE YXRhIFNlZ21lbnQob3B0aW9uYWwpICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIC8NCiArLyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIC8NCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiBtLyBEYXRhLURpZ2VzdCAob3B0aW9uYWwp ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiArLyAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAg Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSsNCg0KQWxsIFBEVSBzZWdtZW50cyBhbmQgZGlnZXN0cyBhcmUgcGFkZGVkIHRvIGFu IGludGVnZXIgbnVtYmVyIG9mIDQgYnl0ZSB3b3Jkcy4NCg0KMi4yLjEgV2hhdCdzIE5leHQgKFdO KSANCg0KVGhpcyBpcyBhbiBlbmNvZGVkIGZpZWxkIGluZGljYXRpbmcgd2hhdCBpcyB0aGUgbmV4 dCBzZWdtZW50IGFzIGZvbGxvd3M6DQoNCmJpdCA3ICAtIDAgIE5leHQgaXMgYW5vdGhlciBoZWFk ZXIgc2VnbWVudC4NCmJpdCA2LTQgIE5leHQgaGVhZGVyIHR5cGUgY29kZQ0KICAgMCAgRXh0ZW5k ZWQgQ0RCDQogICAxICBCaS1kaXJlY3Rpb25hbCByZWFkLWRhdGEgdHJhbnNmZXIgaGVhZGVyDQog ICAyICBMb25nIERhdGEgSGVhZGVyDQogMy03ICBSZXNlcnZlZA0KYml0IDcgIC0gMSAgTmV4dCBp cyBhIGRhdGEgc2VnbWVudCBvciBubyBhZGRpdGlvbmFsIHNlZ21lbnQgKGVtcHR5IGRhdGEgc2Vn bWVudCkNCmJpdCA2LTQgIFJlc2VydmVkIA0KYml0IDMtMiAgMCAgTm8gZGlnZXN0IGZvbGxvd3Mg VEhJUyBzZWdtZW50DQogICAgICAgICAxICBBIENSQy0zMlEgZGlnZXN0IGZvbGxvd3MgVEhJUyBz ZWdtZW50DQogICAgICAgMiwzICBSZXNlcnZlZA0KYml0IDEtMCAgMCAgTm8gZGlnZXN0IGZvbGxv d3MgTmV4dCBzZWdtZW50DQogICAgICAgICAxICBBIENSQy0zMlEgZGlnZXN0IGZvbGxvd3MgTmV4 dCBzZWdtZW50DQogICAgICAgICAyICBBIENSQy02NCBkaWdlc3QgZm9sbG93cyBOZXh0IHNlZ21l bnQgICAgICAgICANCiAgICAgICAgIDMgIFJlc2VydmVkDQoNCk4uQi4gQW4gZW1wdHkgZGF0YSBz ZWdtZW50IE1VU1QgTk9UIGJlIGZvbGxvd2VkIGJ5IGEgQ1JDIGRpZ2VzdA0KDQoyLjIuMiBXTiBz cGVjaWZpYyBmaWVsZHMNCg0KVGhlc2UgYXJlIGZpZWxkcyB0aGF0IHdpbGwgY2FycnkgaW5mb3Jt YXRpb24gc3BlY2lmaWMgdG8gdGhlIG5leHQgc2VnbWVudCB0eXBlLg0KDQoyLjIuMi4xIFdOIHNw ZWNpZmljIGZpZWxkcyBmb3IgYSBuZXh0IEV4dGVuZGVkIENEQiBoZWFkZXIgc2VnbWVudA0KDQpC eXRlIC8gICAgMCAgICAgICB8ICAgICAgIDEgICAgICAgfCAgICAgICAyICAgICAgIHwgICAgICAg MyAgICAgICB8DQogICAvICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCAgICAgICAgICAg ICAgIHwgICAgICAgICAgICAgICB8DQogIHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAw fDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8ICAgICAgICAgICANCiAgKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsN CiAwfCBXTiAgICAgICAgICAgIHwgUmVzZXJ2ZWQgICAgICAgICAgICAgICAgICAgICAgfCBBZGRD REIgICAgICAgIHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCg0KV2hlcmUgQWRkQ0RCIGlzIHRoZSBhZGRpdGlvbmFs IENEQiBsZW5ndGggaW4gdW5pdHMgb2YgNCBieXRlIHdvcmRzIGJleW9uZCB0aGUgZmlyc3QgZXh0 ZW5zaW9uIHdvcmQgKGkuZS4sIEFkZENEQiAwIG1lYW5zIGEgMjAgYnl0ZSBDREIsIDEgYSAyNCBi eXRlIGV0Yy4pLg0KDQoyLjIuMi4yIFdOIHNwZWNpZmljIGZpZWxkcyBmb3IgbmV4dCBCaS1kaXJl Y3Rpb25hbCBkYXRhIGhlYWRlciBzZWdtZW50IGFuZCBMb25nIERhdGEgVHJhbnNmZXIgSGVhZGVy DQoNCkJ5dGUgLyAgICAwICAgICAgIHwgICAgICAgMSAgICAgICB8ICAgICAgIDIgICAgICAgfCAg ICAgICAzICAgICAgIHwNCiAgIC8gICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAgICAg ICAgICAgICAgfCAgICAgICAgICAgICAgIHwNCiAgfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMg MiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHwgICAgICAgICAgIA0KICArLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKw0KIDB8IFdOICAgICAgICAgICAgfCBSZXNlcnZlZCAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgfA0KICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KDQoNCjIuMi4yLjMgV04gc3BlY2lmaWMgZmll bGRzIGZvciBuZXh0IERhdGEgaGVhZGVyIHNlZ21lbnQNCg0KQnl0ZSAvICAgIDAgICAgICAgfCAg ICAgICAxICAgICAgIHwgICAgICAgMiAgICAgICB8ICAgICAgIDMgICAgICAgfA0KICAgLyAgICAg ICAgICAgICAgfCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAg fA0KICB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8NyA2 IDUgNCAzIDIgMSAwfCAgICAgICAgICAgDQogICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogMHwgV04gICAgICAgICAgICB8 IERhdGEgTGVuZ3RoIG9yIFJlc2VydmVkICAgICAgICAgICAgICAgICAgICAgICB8DQogICstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rDQoNCldoZW5ldmVyIHRoaXMgaXMgdGhlIE5leHQtUXVhbGlmaWVyIG9mIGEgTG9uZyBEYXRh IEhlYWRlciBvciBhIExvbmcgRGF0YSBIZWFkZXIgYXBwZWFyZWQgZWFybGllciBpbiB0aGUgc2Vx dWVuY2UgdGhlIGRhdGEgbGVuZ3RoIGlzIHRha2VuIGZyb20gd2l0aGluIHRoaXMgaGVhZGVyIChh IDMyIGJpdCBmaWVsZCkuIFdpdGhvdXQgYSBMb25nIERhdGEgSGVhZGVyIHRoZSBtYXhpbXVtIGxl bmd0aCBvZiBhIGRhdGEgc2VnbWVudCBpcyAxNk1ieXRlcy4NCg0KMi4yLjMgSGVhZGVyIERpZ2Vz dCBhbmQgRGF0YSBEaWdlc3QNCg0KT3B0aW9uYWwgaGVhZGVyIGFuZCBkYXRhIGRpZ2VzdHMgcHJv dGVjdCB0aGUgaW50ZWdyaXR5IGFuZCBhdXRoZW50aWNpdHkgb2YgaGVhZGVyIGFuZCBkYXRhLCBy ZXNwZWN0aXZlbHkuIFRoZSBkaWdlc3RzLCBpZiBwcmVzZW50LCBhcHBlYXIgYXMgdHJhaWxlcnMg bG9jYXRlZCwgcmVzcGVjdGl2ZWx5LCBhZnRlciB0aGUgaGVhZGVyIGFuZCBQRFUtc3BlY2lmaWMg ZGF0YS4gDQoNClRoZSBkaWdlc3QgdHlwZXMgYXJlIG5lZ290aWF0ZWQgZHVyaW5nIHRoZSBsb2dp biBwaGFzZS4NCg0KVGhlIHNlcGFyYXRpb24gb2YgdGhlIGhlYWRlciBhbmQgZGF0YSBkaWdlc3Rz IGlzIHVzZWZ1bCBpbiBpU0NTSSByb3V0aW5nIGFwcGxpY2F0aW9ucywgd2hlcmUgb25seSB0aGUg aGVhZGVyIGNoYW5nZXMgd2hlbiBhIG1lc3NhZ2UgaXMgZm9yd2FyZGVkLiBJbiB0aGlzIGNhc2Us IG9ubHkgdGhlIGhlYWRlciBkaWdlc3Qgc2hvdWxkIGJlIHJlLWNhbGN1bGF0ZWQuDQogDQoNCg0K Mi4yLjQgQmFzaWMgSGVhZGVyIFNlZ21lbnQgKEJIUykNCg0KVGhlIEJhc2ljIEhlYWRlciBTZWdt ZW50IGlzIDQ0IGJ5dGVzIGxvbmcuIA0KVGhlIGZpZWxkIG9mIE9wY29kZSBhcHBlYXJzIGluIGFs bCBpU0NTSSBQRFVzLiBJbiBhZGRpdGlvbiwgdGhlIEluaXRpYXRvciBUYXNrIFRhZywgTG9naWNh bCBVbml0IE51bWJlciwgYW5kIEZsYWdzIGZpZWxkcywgd2hlbiB1c2VkLCBhbHdheXMgYXBwZWFy IGluIHRoZSBzYW1lIGxvY2F0aW9uIGluIHRoZSBoZWFkZXIuDQoNCg0KDQpCeXRlIC8gICAgMCAg ICAgICB8ICAgICAgIDEgICAgICAgfCAgICAgICAyICAgICAgIHwgICAgICAgMyAgICAgICB8DQog ICAvICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwgICAgICAg ICAgICAgICB8DQogIHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAy IDEgMHw3IDYgNSA0IDMgMiAxIDB8ICAgICAgICAgICANCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAwfCBPcGNvZGUg ICAgICAgIHxYfCBPcGNvZGUtc3BlY2lmaWMgZmllbGRzICAgICAgICAgICAgICAgICAgICAgIHwN CiAgfCAgICAgICAgICAgICAgIHxQfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiA0fCBMVU4gb3IgT3Bjb2RlLXNwZWNpZmljIGZpZWxk cyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgKyAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICsNCiA4fCAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLSsNCjEyfCBJbml0aWF0b3IgVGFzayBUYWcgb3IgT3Bjb2RlLXNwZWNpZmlj IGZpZWxkcyAgICAgICAgICAgICAgICAgIHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCjE2LyBPcGNvZGUtc3BlY2lm aWMgZmllbGRzICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiArLyAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIC8NCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSsNCjQ0ICAgICAgICAgICAgICAgIA0KDQoyLjIuNC4xIE9wY29kZQ0K DQpUaGUgT3Bjb2RlIGluZGljYXRlcyB3aGF0IHR5cGUgb2YgaVNDU0kgUERVIHRoZSBoZWFkZXIg ZW5jYXBzdWxhdGVzLiBUaGUgT3Bjb2RlIGlzIGZ1cnRoZXIgZW5jb2RlZCBhcyBmb2xsb3dzOg0K DQpiNyAgIFJlc3BvbnNlDQpiNi0wIE9wZXJhdGlvbg0KDQpUaGUgb3Bjb2RlcyBhcmUgZGl2aWRl ZCBpbnRvIHR3byBjYXRlZ29yaWVzOiBpbml0aWF0b3Igb3Bjb2RlcyBhbmQgdGFyZ2V0IG9wY29k ZXMuIEluaXRpYXRvciBvcGNvZGVzIGFyZSBpbiBQRFVzIHNlbnQgYnkgdGhlIGluaXRpYXRvcnMs IGFuZCB0YXJnZXQgb3Bjb2RlcyBhcmUgaW4gUERVcyBzZW50IGJ5IHRoZSB0YXJnZXQuIFRoZSBp bml0aWF0b3IgTVVTVCBOT1Qgc2VuZCB0YXJnZXQgb3Bjb2RlcyBhbmQgdGhlIHRhcmdldCBNVVNU IE5PVCBzZW5kIGluaXRpYXRvciBvcGNvZGVzLiAgVGFyZ2V0IG9wY29kZXMgYXJlIGFsc28gY2Fs bGVkIHJlc3BvbnNlcyBhbmQgYXJlIGRpc3Rpbmd1aXNoZWQgYnkgaGF2aW5nIHRoZSBSZXNwb25z ZSBiaXQgKGJpdCA2KSBzZXQgdG8gMS4NCg0KVmFsaWQgaW5pdGlhdG9yIG9wY29kZXMgZGVmaW5l ZCBpbiB0aGlzIHNwZWNpZmljYXRpb24gYXJlOg0KDQoNCjB4MDAgTk9QLU91dCAoZnJvbSBpbml0 aWF0b3IgdG8gdGFyZ2V0KQ0KMHgwMSBTQ1NJIENvbW1hbmQgKGVuY2Fwc3VsYXRlcyBhIFNDU0kg Q29tbWFuZCBEZXNjcmlwdG9yIEJsb2NrKQ0KMHgwMiBTQ1NJIFRhc2sgTWFuYWdlbWVudCBDb21t YW5kDQoweDAzIExvZ2luIENvbW1hbmQNCjB4MDQgVGV4dCBDb21tYW5kDQoweDA1IFNDU0kgRGF0 YSAoZm9yIFdSSVRFIG9wZXJhdGlvbikNCjB4MDYgTG9nb3V0IENvbW1hbmQNCg0KDQpWYWxpZCB0 YXJnZXQgb3Bjb2RlcyBhcmU6DQoNCg0KMHg4MCBOT1AtSW4gKGZyb20gdGFyZ2V0IHRvIGluaXRp YXRvcikNCjB4ODEgU0NTSSBSZXNwb25zZSAoY29udGFpbnMgU0NTSSBzdGF0dXMgYW5kIHBvc3Np Ymx5IHNlbnNlIGluZm9ybWF0aW9uIG9yIG90aGVyIHJlc3BvbnNlIGluZm9ybWF0aW9uKQ0KMHg4 MiBTQ1NJIFRhc2sgTWFuYWdlbWVudCBSZXNwb25zZQ0KMHg4MyBMb2dpbiBSZXNwb25zZQ0KMHg4 NCBUZXh0IFJlc3BvbnNlDQoweDg1IFNDU0kgRGF0YSAoZm9yIFJFQUQgb3BlcmF0aW9uKQ0KMHg4 NiBMb2dvdXQgUmVzcG9uc2UNCjB4OTAgUmVhZHkgVG8gVHJhbnNmZXIgKFIyVCAtIHNlbnQgYnkg dGFyZ2V0IHRvIGluaXRpYXRvciB3aGVuIGl0IGlzIHJlYWR5IHRvIHJlY2VpdmUgZGF0YSBmcm9t IGluaXRpYXRvcikNCjB4OTEgQXN5bmNocm9ub3VzIE1lc3NhZ2UgKHNlbnQgYnkgdGFyZ2V0IHRv IGluaXRpYXRvciB0byBpbmRpY2F0ZSBjZXJ0YWluIHNwZWNpYWwgY29uZGl0aW9ucykNCjB4ZWYg UmVqZWN0DQoNCkluaXRpYXRvciBvcGNvZGVzIDB4NzAtMHg3ZiBhbmQgdGFyZ2V0IG9wY29kZXMg MHhmMC0weGZmIGFyZSB2ZW5kb3Igc3BlY2lmaWMgY29kZXMuDQoNCg0KDQoyLjIuNC4yIE9wY29k ZS1zcGVjaWZpYyBmaWVsZHMNCg0KVGhlc2UgZmllbGRzIGhhdmUgZGlmZmVyZW50IG1lYW5pbmdz IGZvciBkaWZmZXJlbnQgbWVzc2FnZXMuDQpCaXQgNyBvZiB0aGUgc2Vjb25kIGJ5dGUgaXMgdXNl ZCBhcyBhIHJldHJ5IGluZGljYXRvciBmb3IgY29tbWFuZHMgKFggYml0KSBvciBQb2xsIGJpdCAo UCBiaXQpIGFuZCBtdXN0IGJlIDAgaW4gYWxsIG90aGVyIGlTQ1NJIFBEVXMNCg0KMi4yLjQuMyBM VU4NCg0KU29tZSBvcGNvZGVzIG9wZXJhdGUgb24gYSBzcGVjaWZpYyBMb2dpY2FsIFVuaXQuIFRo ZSBMb2dpY2FsIFVuaXQgTnVtYmVyIChMVU4pIGZpZWxkIGlkZW50aWZpZXMgd2hpY2ggTG9naWNh bCBVbml0LiAgSWYgdGhlIG9wY29kZSBkb2VzIG5vdCByZWxhdGUgdG8gYSBMb2dpY2FsIFVuaXQs IHRoaXMgZmllbGQgZWl0aGVyIGlzIGlnbm9yZWQgb3IgbWF5IGJlIHVzZWQgZm9yIHNvbWUgb3Ro ZXIgcHVycG9zZS4gIFRoZSBMVU4gZmllbGQgaXMgNjQtYml0cyBpbiBhY2NvcmRhbmNlIHdpdGgg W1NBTTJdLiBUaGUgZXhhY3QgZm9ybWF0IG9mIHRoaXMgZmllbGQgY2FuIGJlIGZvdW5kIGluIHRo ZSBbU0FNMl0gZG9jdW1lbnQuDQoNCjIuMi40LjQgSW5pdGlhdG9yIFRhc2sgVGFnIA0KDQpUaGUg aW5pdGlhdG9yIGFzc2lnbnMgYSBUYXNrIFRhZyB0byBlYWNoIFNDU0kgdGFzayB0aGF0IGl0IGlz c3Vlcy4gIFRoaXMgdGFnIGlzIGEgc2Vzc2lvbi13aWRlIHVuaXF1ZSBpZGVudGlmaWVyIHRoYXQg Y2FuIGJlIHVzZWQgdG8gdW5pcXVlbHkgaWRlbnRpZnkgdGhlIFRhc2suDQoNCg0KMi4yLjUgRXh0 ZW5kZWQgQ0RCIEFkZGl0aW9uYWwgSGVhZGVyIFNlZ21lbnQNCg0KQnl0ZSAvICAgIDAgICAgICAg fCAgICAgICAxICAgICAgIHwgICAgICAgMiAgICAgICB8ICAgICAgIDMgICAgICAgfA0KICAgLyAg ICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAg ICAgfA0KICB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8 NyA2IDUgNCAzIDIgMSAwfA0KICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KIDAvIEV4dGVuZGVkIENEQiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0KICsvICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0KICAr LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKw0KDQoNCjIuMi42IEJpLWRpcmVjdGlvbmFsIFJlYWQgQWRkaXRpb25hbCBIZWFkZXIg U2VnbWVudA0KDQpCeXRlIC8gICAgMCAgICAgICB8ICAgICAgIDEgICAgICAgfCAgICAgICAyICAg ICAgIHwgICAgICAgMyAgICAgICB8DQogICAvICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAg fCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8DQogIHw3IDYgNSA0IDMgMiAxIDB8NyA2 IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8DQogICstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r DQogMHwgQmktZGlyZWN0aW9uYWwgUmVhZCBFeHBlY3RlZCBEYXRhIExlbmd0aCAgICAgICAgICAg ICAgICAgICAgICB8DQogICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQoNCjIuMi43IExvbmcgRGF0YSBBZGRpdGlvbmFsIEhl YWRlciBTZWdtZW50DQoNCkJ5dGUgLyAgICAwICAgICAgIHwgICAgICAgMSAgICAgICB8ICAgICAg IDIgICAgICAgfCAgICAgICAzICAgICAgIHwNCiAgIC8gICAgICAgICAgICAgIHwgICAgICAgICAg ICAgICB8ICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwNCiAgfDcgNiA1IDQgMyAyIDEg MHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHwNCiAgKy0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSsNCiAwfCBEYXRhIExlbmd0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCg0KIA0KDQogDQoNCjIuMyBTQ1NJIENvbW1h bmQgDQoNCkJ5dGUgLyAgICAwICAgICAgIHwgICAgICAgMSAgICAgICB8ICAgICAgIDIgICAgICAg fCAgICAgICAzICAgICAgIHwNCiAgIC8gICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAg ICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwNCiAgfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0 IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHwNCiAgKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAw fCAweDAxICAgICAgICAgIHxYfFJ8V3wwIDB8QVRUUiB8IFJlc2VydmVkICAgICAgfCBDbWRSTiBv ciBSc3ZkIHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSsNCiA0fCBMZW5ndGggICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiA4fCBMb2dpY2Fs IFVuaXQgTnVtYmVyIChMVU4pICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwN CiAgKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICsNCjEyfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCjE2fCBJbml0aWF0b3IgVGFzayBU YWcgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSsNCjIwfCBFeHBlY3RlZCBEYXRhIFRyYW5zZmVyIExlbmd0aCAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCjI0fCBDbWRTTiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgKy0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCjI4fCBF eHBTdGF0U04gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSsNCjMyLyBTQ1NJIENvbW1hbmQgRGVzY3JpcHRvciBCbG9jayAoQ0RC KSAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiArLyAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAgKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCjQ0 DQoNCg0KMi4zLjEgRmxhZ3MgJiBUYXNrIEF0dHJpYnV0ZXMNCg0KVGhlIGZsYWdzIGZpZWxkIGZv ciBhIFNDU0kgQ29tbWFuZCBpczoNCg0KDQpiNyAgIFJldHJ5IChYKSANCmI2ICAgKFIpIHNldCB0 byAxIHdoZW4gaW5wdXQgZGF0YSBpcyBleHBlY3RlZCAgDQpiNSAgIChXKSBzZXQgdG8gMSB3aGVu IG91dHB1dCBkYXRhIGlzIGV4cGVjdGVkDQpiMy00IFJlc2VydmVkIChNVVNUIGJlIDApDQpiMC0y IHVzZWQgdG8gaW5kaWNhdGUgVGFzayBBdHRyaWJ1dGVzIA0KDQpUaGUgVGFzayBBdHRyaWJ1dGVz IChBVFRSKSBjYW4gaGF2ZSBvbmUgb2YgdGhlIGZvbGxvd2luZyBpbnRlZ2VyIHZhbHVlcyAoc2Vl IFtTQU0yXSBmb3IgZGV0YWlscyk6DQoNCjAgICAgVW50YWdnZWQNCjEgICAgU2ltcGxlDQoyICAg IE9yZGVyZWQNCjMgICAgSGVhZCBvZiBRdWV1ZQ0KNCAgICBBQ0ENCg0KMi4zLjIgQ21kUk4gDQoN ClNDU0kgY29tbWFuZCByZWZlcmVuY2UgbnVtYmVyIC0gaWYgcHJlc2VudCBpbiB0aGUgU0NTSSBl eGVjdXRlIGFyZ3VtZW50cw0KDQoyLjMuMyBDbWRTTiAtIENvbW1hbmQgU2VxdWVuY2UgTnVtYmVy DQoNCkVuYWJsZXMgb3JkZXJlZCBkZWxpdmVyeSBhY3Jvc3MgbXVsdGlwbGUgY29ubmVjdGlvbnMg aW4gYSBzaW5nbGUgc2Vzc2lvbi4NCg0KMi4zLjQgRXhwU3RhdFNOIC0gRXhwZWN0ZWQgU3RhdHVz IFNlcXVlbmNlIE51bWJlciANCg0KQ29tbWFuZCByZXNwb25zZXMgdXAgdG8gRXhwU3RhdFNOLTEg KG1vZCAyKiozMikgaGF2ZSBiZWVuIHJlY2VpdmVkIChhY2tub3dsZWRnZXMgc3RhdHVzKSBvbiB0 aGUgY29ubmVjdGlvbi4gIA0KDQoyLjMuNSBFeHBlY3RlZCBEYXRhIFRyYW5zZmVyIExlbmd0aA0K DQpGb3IgdW5pZGlyZWN0aW9uYWwgb3BlcmF0aW9ucywgdGhlIEV4cGVjdGVkIERhdGEgVHJhbnNm ZXIgTGVuZ3RoIGZpZWxkIHN0YXRlcyB0aGUgbnVtYmVyIG9mIGJ5dGVzIG9mIGRhdGEgaW52b2x2 ZWQgaW4gdGhpcyBTQ1NJIG9wZXJhdGlvbi4gIEZvciBhIFdSSVRFIG9wZXJhdGlvbiwgdGhlIGlu aXRpYXRvciB1c2VzIHRoaXMgZmllbGQgdG8gc3BlY2lmeSB0aGUgbnVtYmVyIG9mIGJ5dGVzIG9m IGRhdGEgaXQgZXhwZWN0cyB0byB0cmFuc2ZlciBmb3IgdGhpcyBvcGVyYXRpb24uICBGb3IgYSBS RUFEIG9wZXJhdGlvbiwgdGhlIGluaXRpYXRvciB1c2VzIHRoaXMgZmllbGQgdG8gc3BlY2lmeSB0 aGUgbnVtYmVyIG9mIGJ5dGVzIG9mIGRhdGEgaXQgZXhwZWN0cyB0aGUgdGFyZ2V0IHRvIHRyYW5z ZmVyIHRvIHRoZSBpbml0aWF0b3IuICBJdCBjb3JyZXNwb25kcyB0byB0aGUgU0FNLTIgYnl0ZSBj b3VudC4NCg0KSWYgdGhlIEV4cGVjdGVkIERhdGEgdHJhbnNmZXIgTGVuZ3RoIGZvciBhIFdSSVRF IGFuZCB0aGUgbGVuZ3RoIG9mIGltbWVkaWF0ZSBkYXRhIHBhcnQgdGhhdCBmb2xsb3dzIHRoZSBj b21tYW5kIChpZiBhbnkpIGFyZSB0aGUgc2FtZSB0aGVuIG5vIG1vcmUgZGF0YSBQRFVzIGFyZSBl eHBlY3RlZCB0byBmb2xsb3cuDQoNCklmIG5vIGRhdGEgd2lsbCBiZSB0cmFuc2ZlcnJlZCBpbiBT Q1NJIERhdGEgcGFja2V0cyBmb3IgdGhpcyBTQ1NJIG9wZXJhdGlvbiwgdGhpcyBmaWVsZCBzaG91 bGQgYmUgc2V0IHRvIHplcm8uDQoNCkZvciBiaS1kaXJlY3Rpb25hbCBvcGVyYXRpb25zLCB0aGlz IGZpZWxkIHN0YXRlcyB0aGUgbnVtYmVyIG9mIGRhdGEgYnl0ZXMgaW52b2x2ZWQgaW4gdGhlIG91 dGJvdW5kIHRyYW5zZmVyLiBGb3IgYmktZGlyZWN0aW9uYWwgb3BlcmF0aW9ucywgYW4gYWRkaXRp b25hbCBoZWFkZXIgc2VnbWVudCBNVVNUIGJlIHByZXNlbnQgaW4gdGhlIGhlYWRlciBzZXF1ZW5j ZSBpbmRpY2F0aW5nIHRoZSBFeHBlY3RlZCBCaS1kaXJlY3Rpb25hbCBSZWFkIERhdGEgTGVuZ3Ro LiAgSWYgdGhpcyBhZGRpdGlvbmFsIGhlYWRlciBzZWdtZW50IGlzIGFic2VudCB0aGUgRXhwZWN0 ZWQgQmktZGlyZWN0aW9uYWwgUmVhZCBEYXRhIExlbmd0aCBNVVNUIGJlIGFzc3VtZWQgMC4NCg0K VXBvbiBjb21wbGV0aW9uIG9mIGEgZGF0YSB0cmFuc2ZlciwgdGhlIHRhcmdldCB3aWxsIGluZm9y bSB0aGUgaW5pdGlhdG9yIG9mIGhvdyBtYW55IGJ5dGVzIHdlcmUgYWN0dWFsbHkgcHJvY2Vzc2Vk IChzZW50IG9yIHJlY2VpdmVkKSBieSB0aGUgdGFyZ2V0LiAgVGhpcyB3aWxsIGJlIGRvbmUgdGhy b3VnaCByZXNpZHVhbCBjb3VudHMuDQoNCjIuMy42IENEQiAtIFNDU0kgQ29tbWFuZCBEZXNjcmlw dG9yIEJsb2NrDQoNClRoZXJlIGFyZSAxNiBieXRlcyBpbiB0aGUgQ0RCIGZpZWxkIHRvIGFjY29t bW9kYXRlIHRoZSBjb21tb25seSB1c2VkIENEQi4gIFdoZW5ldmVyIGxhcmdlciBDREJzIGFyZSB1 c2VkLCB0aGUgQ0RCIHNwaWxsb3ZlciBNQVkgZXh0ZW5kIGJleW9uZCB0aGUgNDgtYnl0ZSBoZWFk ZXIuDQoNCjIuMy43IENvbW1hbmQtRGF0YSANCg0KU29tZSBTQ1NJIGNvbW1hbmRzIHJlcXVpcmUg YWRkaXRpb25hbCBwYXJhbWV0ZXIgZGF0YSB0byBhY2NvbXBhbnkgdGhlIFNDU0kgY29tbWFuZC4g VGhpcyBkYXRhIG1heSBiZSBwbGFjZWQgYmV5b25kIHRoZSBib3VuZGFyeSBvZiB0aGUgaVNDU0kg aGVhZGVyIChhIGRhdGEgc2VnbWVudCkuICBBbHRlcm5hdGl2ZWx5LCB1c2VyIGRhdGEgKGFzIGZy b20gYSBXUklURSBvcGVyYXRpb24pIGNhbiBiZSBwbGFjZWQgaW4gdGhlIHNhbWUgUERVIChib3Ro IGNhc2VzIHJlZmVycmVkIHRvIGFzIGltbWVkaWF0ZSBkYXRhKS4NCg0K --0__=dTddHe2TkczLRtdjwcs4m2zeG39EF81uuPaLRjQhE1XVge0qIgBFlkJ4-- From owner-ips@ece.cmu.edu Thu Jan 25 14:47:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA07304 for ; Thu, 25 Jan 2001 14:47:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PHjko07449 for ips-outgoing; Thu, 25 Jan 2001 12:45:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PHj4607410 for ; Thu, 25 Jan 2001 12:45:07 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA51754 for ; Thu, 25 Jan 2001 18:44:46 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA83460 for ; Thu, 25 Jan 2001 18:44:46 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DF.006178F2 ; Thu, 25 Jan 2001 18:44:39 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 25 Jan 2001 19:40:12 +0200 Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk answers in text - Julo Santosh Rao on 25/01/2001 04:06:59 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Julian & All, There's a number of problems with the current iSCSI handling of digest errors (Section 5.5) : "When an initiator receives an iSCSI data PDU with a digest error or any other PDU with a header or data digest error, it must discard and re-start the task -provided it can recognize the Initiator Task Tag". 1) The initiator task tag cannot be trusted when a header digest error is seen. What does the phrase "provided it can recognize the initiator task tag" mean ? How can an initiator reliably claim that the initiator task tag is trustworthy ? an initiator may choose to provide some redundancy in the tag itself 2) The use of a "discard and re-start" policy can cause some problems. The target is un-aware that the initiator has discarded the task and will then receive a 2nd command with the same Initiator Task Tag and CmdSN for a task already in progress, which can cause transmission of duplicate sets of data on the same I.T.T. Any policy of "discard and re-start" must be modified to "discard, abort and then, re-start, and the draft MUST specify connection allegiance of the command and its Abort Task". a restart is recognized as such through the restart bit. Why would you require abort? please explain 3) The policy of "discard and restart" is also subject some race conditions in the CmdSN sliding window. At the time the digest error was detected at the initiator, the ExpCmdSN may not yet have acknowledged that command. This causes the initiator to restart the command with the same Initiator Task Tag, CmdSN and "retry" bit set. However, by the time the command gets to the target, the target may have updated its ExpCmdSN window having sent a later PDU which updated the ExpCmdSN. This results in the target discarding the received CmdSN since at command restart you never really rely on the CmdSN. you will want to check the Initiator Task Tag and accept it in the above case. it is outside the expected window of (ExpCmdSN, MaxCmdSN). The "discard & restart" policy cannot be reliably used unless it is on a connection failure, in which case, the failed connection is first logged out and an updated ExpCmdSN is received on the logout response. Following the logout response, the "retry" commands CmdSN can be based on the ExpCmdSN. see above In the case of digest errors, the CmdSN window can be moving forward and an attempt to "discard and restart" will result in a re-send of a CmdSN that is outside the CmdSN window, causing the target to discard the retried command. the target has enough information to discard only true replicas Regards, Santosh - santoshr.vcf From owner-ips@ece.cmu.edu Thu Jan 25 14:47:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA07324 for ; Thu, 25 Jan 2001 14:47:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PHMgb06788 for ips-outgoing; Thu, 25 Jan 2001 12:22:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PHLp606759 for ; Thu, 25 Jan 2001 12:21:52 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA61660 for ; Thu, 25 Jan 2001 18:21:32 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA186176 for ; Thu, 25 Jan 2001 18:21:32 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DF.005F57BC ; Thu, 25 Jan 2001 18:21:23 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 25 Jan 2001 19:16:56 +0200 Subject: Re: iSCSI : draft contradicts itself in sections 1.2.5 & 5.5 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk SHOULD NOT means NOT unless you have a good reason not too. And with a digest error you have a good reason to - don't you? Julo Santosh Rao on 25/01/2001 00:18:11 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iSCSI : draft contradicts itself in sections 1.2.5 & 5.5 Julian, In Section 1.2.5, the draft states that : "A target SHOULD NOT silently discard data and request re-transmission through R2T." In Section 5.5 the draft encourages the above to be performed by stating : "When a target receives an iSCSI Data PDU with a data payload digest error, it MUST discard it and request retransmission with a R2T." What is the intent of the former statement in Section 1.2.5, when digest error recovery requires just the opposite ? Regards, Santosh - santoshr.vcf From owner-ips@ece.cmu.edu Thu Jan 25 16:10:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA08850 for ; Thu, 25 Jan 2001 16:10:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PJCki10589 for ips-outgoing; Thu, 25 Jan 2001 14:12:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PJCe610582 for ; Thu, 25 Jan 2001 14:12:40 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0PKNB052881; Thu, 25 Jan 2001 12:23:11 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: , Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Thu, 25 Jan 2001 11:11:17 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <0F31E5C394DAD311B60C00E029101A070410148E@corpmx9.isus.emc.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, For what it's worth, I am not making such fine distinctions. I was attempting to discuss a reason for including a time-stamp should such placement be onto FC media. (Looking to the day when there is a means to handle out of sequence frames.) Doug > > >In the case of FCIP, a scheme for framing is not to allow buffering in > > user space as perhaps could be said for iSCSI. > > Again, that statement is implementation-dependent/specific, and there > may be advantages to placement even when there isn't any notion > of "user space" depending on the hardware and software architecture > of the implementation. It's definitely possible to build FCIP nodes > that don't require framing support, BUT that doesn't mean that > all nodes will be built in that fashion (the same statement applies > to iSCSI targets, FWIW). > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Thu Jan 25 16:13:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA08912 for ; Thu, 25 Jan 2001 16:13:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PJZqI11618 for ips-outgoing; Thu, 25 Jan 2001 14:35:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PJZi611608 for ; Thu, 25 Jan 2001 14:35:44 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 8112C15C0 for ; Thu, 25 Jan 2001 11:35:43 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id LAA04367 for ; Thu, 25 Jan 2001 11:35:39 -0800 (PST) Message-ID: <3A70809C.3607E19F@cup.hp.com> Date: Thu, 25 Jan 2001 11:38:04 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Bridging missing CmdSNs and Abort I/O Error recovery Content-Type: multipart/mixed; boundary="------------74CDB8071EFBB39EA9ED7183" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------74CDB8071EFBB39EA9ED7183 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian & All, The draft is currently lacking a section that addresses abort I/O error recovery. Specifically, how is CmdSN bridging issues to be handled in the case where an initiator chooses not to retry an I/O [that failed on a connection failure that affects the delivery of the command to the target or a digest error at the target] because its ULP timer may have expired. In such cases, the initiator can send an Abort Task to inform the target that the I.T.T is being aborted and its corresponding CmdSN can be bridged, instead of having the target stall infinitely in its attempt to enforce ordering and await the missing CmdSN [which is'nt going to arrive, because the initiator did not retry the command]. Regards, Santosh --------------74CDB8071EFBB39EA9ED7183 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------74CDB8071EFBB39EA9ED7183-- From owner-ips@ece.cmu.edu Thu Jan 25 16:16:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA09001 for ; Thu, 25 Jan 2001 16:16:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PJFuD10725 for ips-outgoing; Thu, 25 Jan 2001 14:15:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PJFE610685 for ; Thu, 25 Jan 2001 14:15:14 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 2D33050A; Thu, 25 Jan 2001 11:15:12 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id LAA02853; Thu, 25 Jan 2001 11:15:07 -0800 (PST) Message-ID: <3A707BCC.7A322FD0@cup.hp.com> Date: Thu, 25 Jan 2001 11:17:33 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues References: Content-Type: multipart/mixed; boundary="------------9BC36FEAF46E75CC053F378A" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------9BC36FEAF46E75CC053F378A Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, My responses enclosed inline. Regards, Santosh julian_satran@il.ibm.com wrote: > 1) The initiator task tag cannot be trusted when a header digest error > is seen. What does the phrase "provided it can recognize the initiator > task tag" mean ? > How can an initiator reliably claim that the initiator task tag is > trustworthy ? > > an initiator may choose to provide some redundancy in the tag itself > All of these excessive "mays" only serve to complicate the standard and end up as un-used features. They are better off removed. > > 2) The use of a "discard and re-start" policy can cause some problems. > The target is un-aware that the initiator has discarded the task and > will then receive a 2nd command with the same Initiator Task Tag and > CmdSN for a task already in progress, which can cause transmission of > duplicate sets of data on the same I.T.T. > > Any policy of "discard and re-start" must be modified to "discard, abort > > and then, re-start, and the draft MUST specify connection allegiance of > the command and its Abort Task". > > a restart is recognized as such through the restart bit. Why would you > require abort? Where does the draft EXPLICITLY state that the "retry" bit implies an implicit abort of the command followed by re-start ? In the absence of such explicit clarifications, the draft is opening up opportunities for numerous creative mis-interpretations. Despite this, there is still a window b/n the command being re-started at the initiator and the target receiving the command with "retry" bit set, when stale frames continue to arrive at the initiator for that I.T.T. Therefore, in order to avoid stale frames from the previous command continuing to arrive at the initiator in this window, there needs to be an abort followed by a re-start. All of this complexity is only in the case of digest error handling [and not in the case of connection failures]. The draft can be rid of all this complexity if digest errors were to be recovered at a connection level, instead of this "discard & restart" policy which adds all these complexities. > > please explain > > 3) The policy of "discard and restart" is also subject some race > conditions in the CmdSN sliding window. At the time the digest error was > > detected at the initiator, the ExpCmdSN may not yet have acknowledged > that > command. This causes the initiator to restart the command with the same > Initiator Task Tag, CmdSN and "retry" bit set. > > However, by the time the command gets to the target, the target may have > > updated its ExpCmdSN window having sent a later PDU which updated the > ExpCmdSN. This results in the target discarding the received CmdSN since > > at command restart you never really rely on the CmdSN. you will want > to check > the Initiator Task Tag and accept it in the above case. The draft states the following on this subject : Section 5.1 ========= - the initiator will reissue all outstanding commands with their original Initiator Task Tag and their original CmdRN if they are not acknowledged yet or a CmdRN of 0 (not-numbered) if they were acknowledged; the retry (X) flag in the command PDU will be set Section 1.2.2.1 =========== - The target MUST silently ignore any command outside this range or duplicates within the range not flagged with the retry bit (the X bit in the opcode). This, to me, means : - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. - ignore all commands within the range that are not flagged with the retry bit. Can you please clarify the intent of the text ? --------------9BC36FEAF46E75CC053F378A Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------9BC36FEAF46E75CC053F378A-- From owner-ips@ece.cmu.edu Thu Jan 25 16:16:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA09061 for ; Thu, 25 Jan 2001 16:16:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PIkjq09587 for ips-outgoing; Thu, 25 Jan 2001 13:46:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PIjj609547 for ; Thu, 25 Jan 2001 13:45:45 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id TAA07618 for ; Thu, 25 Jan 2001 19:45:29 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id TAA284244 for ; Thu, 25 Jan 2001 19:45:29 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569DF.006707E2 ; Thu, 25 Jan 2001 19:45:22 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 25 Jan 2001 20:40:53 +0200 Subject: Re: iSCSI : draft contradicts itself in sections 1.2.5 & 5.5 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I would also kindly invite you to read the answer. Julo Santosh Rao on 25/01/2001 20:05:14 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI : draft contradicts itself in sections 1.2.5 & 5.5 julian_satran@il.ibm.com wrote: > SHOULD NOT means NOT unless you have a good reason not too. And with a > digest error you have a good reason to - don't you? Not if you consider all the problems raised regarding this "discard and retry" policy on digest errors. See : http://ips.pdl.cs.cmu.edu/mail/msg03110.html - Santosh > > Santosh Rao on 25/01/2001 00:18:11 > > Please respond to Santosh Rao > > To: IPS Reflector > cc: > Subject: iSCSI : draft contradicts itself in sections 1.2.5 & 5.5 > > Julian, > > In Section 1.2.5, the draft states that : > "A target SHOULD NOT silently discard data and request re-transmission > through R2T." > > In Section 5.5 the draft encourages the above to be performed by stating > : > "When a target receives an iSCSI Data PDU with a data payload digest > error, it MUST discard it and request retransmission with a R2T." > > What is the intent of the former statement in Section 1.2.5, when digest > error recovery requires just the opposite ? > > Regards, > Santosh > > - santoshr.vcf - santoshr.vcf From owner-ips@ece.cmu.edu Thu Jan 25 16:22:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA09149 for ; Thu, 25 Jan 2001 16:22:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PJptr12322 for ips-outgoing; Thu, 25 Jan 2001 14:51:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PJpA612298 for ; Thu, 25 Jan 2001 14:51:11 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 2972C4C8; Thu, 25 Jan 2001 11:51:10 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id LAA05808; Thu, 25 Jan 2001 11:51:05 -0800 (PST) Message-ID: <3A70843A.4CC7582C@cup.hp.com> Date: Thu, 25 Jan 2001 11:53:30 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI: retries and SCSI References: <0F31E5C394DAD311B60C00E029101A070410148F@corpmx9.isus.emc.com> Content-Type: multipart/mixed; boundary="------------D2D4AED685E345EFBDD73280" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------D2D4AED685E345EFBDD73280 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Black_David@emc.com wrote: > When the operation is not idempotent > (doing it over has ill effects - tape reads and writes > are examples), then this optimization is not applicable > and if the target hasn't retained the stuff required > to respond to the retry, it has to fail/reject the retry. David, Thanks for the clarification on this. This approach of rejecting the retry, rather than not retrying for sequential devices, poses a couple of problems : a) The responsibility of dealing with the reject will lie with a iSCSI-pSCSI or iSCSI-FC bridge in most of the tape connectivity cases. b) The target iSCSI layer now needs to share SCSI state information with its ULP on the device's class and abilities. (something that would normally be exchanged b/n the peer SCSI ULPs on either side.) I believe the draft already intends to address the issue of retries on a connection failure being made optional [if the initiator detects ULP timeout and wishes not to retry]. I would like to suggest that the same policy be applied to digest errors as well. (as initiators may not wish to retry on a digest error, when it sees a ULP timeout.). This allows initiators the flexibility to not retry the I/O to non-idempotent target media. Thanks & Regards, Santosh --------------D2D4AED685E345EFBDD73280 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------D2D4AED685E345EFBDD73280-- From owner-ips@ece.cmu.edu Thu Jan 25 16:32:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA09306 for ; Thu, 25 Jan 2001 16:32:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PJmsh12194 for ips-outgoing; Thu, 25 Jan 2001 14:48:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PJmZ612169 for ; Thu, 25 Jan 2001 14:48:35 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 6F68C7A7 for ; Thu, 25 Jan 2001 12:48:34 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 238F6365 for ; Thu, 25 Jan 2001 14:48:33 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id LAA07047 for ; Thu, 25 Jan 2001 11:48:31 -0800 (PST) Message-ID: <3A708363.4E3406EB@agilent.com> Date: Thu, 25 Jan 2001 11:49:55 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Bridging missing CmdSNs and Abort I/O Error recovery References: <3A70809C.3607E19F@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit The target iSCSI is the only entity that has knowledge of CmdSN. When it receives a commands, it delivers them to the SCSI layer in CmdSN order (with the CmdSN stripped off). iSCSI does not handle the abort commands. If an I/O times out (because of a connection failure) and is aborted by the upper level driver such that it is dequeued from the iSCSI driver (and therefore cannot be retransmitted on a new connection), then the iSCSI driver can simply send a NOP command in it's place. -Matt Santosh Rao wrote: > > Julian & All, > > The draft is currently lacking a section that addresses abort I/O error > recovery. Specifically, how is CmdSN bridging issues to be handled in > the case where an initiator chooses not to retry an I/O [that failed on > a connection failure that affects the delivery of the command to the > target or a digest error at the target] because its ULP timer may have > expired. > > In such cases, the initiator can send an Abort Task to inform the target > that the I.T.T is being aborted and its corresponding CmdSN can be > bridged, instead of having the target stall infinitely in its attempt to > enforce ordering and await the missing CmdSN [which is'nt going to > arrive, because the initiator did not retry the command]. > > Regards, > Santosh From owner-ips@ece.cmu.edu Thu Jan 25 16:49:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA09585 for ; Thu, 25 Jan 2001 16:49:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PK6mS13030 for ips-outgoing; Thu, 25 Jan 2001 15:06:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PK5s612994 for ; Thu, 25 Jan 2001 15:05:54 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 0E6821640; Thu, 25 Jan 2001 12:05:52 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id MAA07152; Thu, 25 Jan 2001 12:05:47 -0800 (PST) Message-ID: <3A7087AC.424C6E96@cup.hp.com> Date: Thu, 25 Jan 2001 12:08:12 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Matt Wakeley Cc: IPS Reflector Subject: Re: iSCSI : Bridging missing CmdSNs and Abort I/O Error recovery References: <3A70809C.3607E19F@cup.hp.com> <3A708363.4E3406EB@agilent.com> Content-Type: multipart/mixed; boundary="------------5CDC3EBF1CADDC3449D7AAC1" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------5CDC3EBF1CADDC3449D7AAC1 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Matt Wakeley wrote: > The target iSCSI is the only entity that has knowledge of CmdSN. When it > receives a commands, it delivers them to the SCSI layer in CmdSN order (with > the CmdSN stripped off). iSCSI does not handle the abort commands. The Abort Task should also involve some functionality in the iSCSI layer, which is to invalidate and clean up the iSCSI resources for that task (?) - Santosh --------------5CDC3EBF1CADDC3449D7AAC1 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------5CDC3EBF1CADDC3449D7AAC1-- From owner-ips@ece.cmu.edu Thu Jan 25 16:51:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA09657 for ; Thu, 25 Jan 2001 16:51:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PKEo913398 for ips-outgoing; Thu, 25 Jan 2001 15:14:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PKEI613362 for ; Thu, 25 Jan 2001 15:14:18 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id CE683B9A; Thu, 25 Jan 2001 12:14:13 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id MAA07730; Thu, 25 Jan 2001 12:14:08 -0800 (PST) Message-ID: <3A7089A1.D72ECF2@cup.hp.com> Date: Thu, 25 Jan 2001 12:16:34 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI: DataRN References: Content-Type: multipart/mixed; boundary="------------9D1D2FB55BD9528649F1D5F9" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------9D1D2FB55BD9528649F1D5F9 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > Santosh, > > The current draft has status numbering as SHOULD and that is (as it was > pointed out to me) almost mandatory. > Considering that the consensus at Orlando and on the list was to make CmdRN > (now CmdSN) mandatory even for one connection I don't see any reason not to > make StatSN support also mandatory . Julian, Using the above reasoning, one could also suggest that, considering the consensus to remove DataSN and all its complexity and the choice to re-start a command from scratch, the StatSN also could be considered for removal. (?) The StatSN is comparable with DataSN, more than CmdSN (which is required to enforce ordering.). Both DataSN and StatSN were defined to allow for partial recovery and the WG consensus in Orlando was that the complexity of doing a partial recovery was not worth the benefit it may provide. Regards, Santosh --------------9D1D2FB55BD9528649F1D5F9 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------9D1D2FB55BD9528649F1D5F9-- From owner-ips@ece.cmu.edu Thu Jan 25 16:53:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA09699 for ; Thu, 25 Jan 2001 16:53:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PKbxt14241 for ips-outgoing; Thu, 25 Jan 2001 15:37:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PKaw614195 for ; Thu, 25 Jan 2001 15:36:58 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 07ADCB47 for ; Thu, 25 Jan 2001 13:36:57 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 8D9DE45B for ; Thu, 25 Jan 2001 15:36:54 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id MAA08912 for ; Thu, 25 Jan 2001 12:36:52 -0800 (PST) Message-ID: <3A708EB8.4DD51C12@agilent.com> Date: Thu, 25 Jan 2001 12:38:16 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: DataRN References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, The current draft says: 1.2.2.2 Response/Status numbering Responses in transit from the target to the initiator are numbered. ... Initiators and Targets MUST support the response-numbering scheme regardless of the support for command recovery. The words "are numbered" and "MUST support" don't sound like "should" to me... -Matt julian_satran@il.ibm.com wrote: > > Santosh, > > The current draft has status numbering as SHOULD and that is (as it was > pointed out to me) almost mandatory. > Considering that the consensus at Orlando and on the list was to make CmdRN > (now CmdSN) mandatory even for one connection I don't see any reason not to > make StatSN support also mandatory . > > On Writes our assumptions was that the target can at any point in time > either reject the restart (if as you outline it can't do it) or if it can > reacquire the data it misses trough R2T and resend the status (on writes > the target is assumed to know what it needs). In case it rejects the > restart you can report a "transport failure" and you are as good as a > parallel packetized SCSI. > > Julo > > Santosh Rao on 25/01/2001 00:13:53 > > Please respond to Santosh Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: iSCSI: DataRN > > julian_satran@il.ibm.com wrote: > > > Santosh, > > > > The ACK for a response is when status is ACKed (by the status numbering). > > > > Before this ack a target has several options for restart: > > > > to keep all the data (recall we have dropped DataRN) and "restart > > sending". > > This may work for a READ if the iSCSI layer has buffered the data until it > gets a StatSN ACK. Note that even for the READ, status recovery is optional > and a tape may not implement status / data recovery and just retry the > command. In such a case, a retry results in the target re-doing the > operation. > > For a WRITE however, I don't see how this is going to work. Are you > suggesting > that the iSCSI layer NOT deliver data to the SCSI layer until the Response > is > acknowleged ??? This is ruled out, since the Response itself comes from the > SCSI layer upon completion of the SCSI command. > > If, on the WRITE, the data was delivered to the SCSI layer which then wrote > it > to media and advanced the tape, what good does the above do ?? The tape has > advanced following the WRITE and the initiator can corrupt the tape by > retrying the WRITE. > > > not to keep data and reject the command restart with a service > response > > (that we have to specify) of restart reject (your tape target may want > > to do just this) > > You seem to suggest moving the responsibility of ensuring data integrity > from > the host to the target !! This is not suitable for the following reasons : > a) Often, tape is the last class of storage products to migrate to new > storage > transport technologies. IOW, the majority of tape backup is going to attain > iSCSI connectivity through a iSCSI-FC bridge or iSCS-pSCSI bridge. In such > a > case, the service response management is done in the bridge and this level > of > intelligence is now being moved into the bridge. > > The responsiblity of ensuring data integrity on a SCSI target lies with the > initiator issuing the I/O. > > iSCSI MUST not retry a command to sequential media when there is the > possiblity that some or all of the data phase has completed. > > Regards, > Santosh > > - santoshr.vcf From owner-ips@ece.cmu.edu Thu Jan 25 16:57:26 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA09730 for ; Thu, 25 Jan 2001 16:57:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PKfo714396 for ips-outgoing; Thu, 25 Jan 2001 15:41:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PKew614359 for ; Thu, 25 Jan 2001 15:41:02 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 20B72B67 for ; Thu, 25 Jan 2001 13:40:58 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id B9498473 for ; Thu, 25 Jan 2001 15:40:56 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id MAA08955 for ; Thu, 25 Jan 2001 12:40:55 -0800 (PST) Message-ID: <3A708FAB.819018F3@agilent.com> Date: Thu, 25 Jan 2001 12:42:19 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues References: <3A707BCC.7A322FD0@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > The draft states the following on this subject : > > Section 5.1 > ========= > - the initiator will reissue all outstanding commands with their > original Initiator Task Tag and their original CmdRN if they > are not acknowledged yet or a CmdRN of 0 (not-numbered) if they > were acknowledged; the retry (X) flag in the command PDU will > be set > > Section 1.2.2.1 > =========== > - The target MUST silently ignore any command > outside this range or duplicates within the range not flagged with > the retry bit (the X bit in the opcode). > > This, to me, means : > - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. > - ignore all commands within the range that are not flagged with the retry > bit. I think you missed the word "duplicates"... > > Can you please clarify the intent of the text ? -Matt From owner-ips@ece.cmu.edu Thu Jan 25 17:01:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA09775 for ; Thu, 25 Jan 2001 17:01:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PKW2v14011 for ips-outgoing; Thu, 25 Jan 2001 15:32:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PKV3613976 for ; Thu, 25 Jan 2001 15:31:03 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 6574EB28 for ; Thu, 25 Jan 2001 13:31:02 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 929EC3FF for ; Thu, 25 Jan 2001 15:31:00 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id MAA08777 for ; Thu, 25 Jan 2001 12:30:58 -0800 (PST) Message-ID: <3A708D56.FF028D7A@agilent.com> Date: Thu, 25 Jan 2001 12:32:22 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Bridging missing CmdSNs and Abort I/O Error recovery References: <3A70809C.3607E19F@cup.hp.com> <3A708363.4E3406EB@agilent.com> <3A7087AC.424C6E96@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit No, that would require iSCSI to interpret iSCSI commands. There is nothing at the target iSCSI to clean up (except the "hole" in the command stream). The target iSCSI receives iSCSI commands and sends them to the SCSI layer, what is there to clean up? -Matt Santosh Rao wrote: > > Matt Wakeley wrote: > > > The target iSCSI is the only entity that has knowledge of CmdSN. When it > > receives a commands, it delivers them to the SCSI layer in CmdSN order (with > > the CmdSN stripped off). iSCSI does not handle the abort commands. > > The Abort Task should also involve some functionality in the iSCSI layer, which > is to invalidate and clean up the iSCSI resources for that task (?) > > - Santosh From owner-ips@ece.cmu.edu Thu Jan 25 18:45:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA11617 for ; Thu, 25 Jan 2001 18:45:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PM08b17293 for ips-outgoing; Thu, 25 Jan 2001 17:00:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zmamail04.zma.compaq.com (zmamail04.zma.compaq.com [161.114.64.104]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PLxk617275 for ; Thu, 25 Jan 2001 16:59:46 -0500 (EST) Received: by zmamail04.zma.compaq.com (Postfix, from userid 12345) id 6D7224849; Thu, 25 Jan 2001 16:59:41 -0500 (EST) Received: from exchou-gh02.cca.cpqcorp.net (exchou-gh02.cca.cpqcorp.net [16.110.248.202]) by zmamail04.zma.compaq.com (Postfix) with ESMTP id 239CA481A for ; Thu, 25 Jan 2001 16:59:41 -0500 (EST) Received: by exchou-gh02.cca.cpqcorp.net with Internet Mail Service (5.5.2652.78) id ; Thu, 25 Jan 2001 15:59:40 -0600 Message-ID: <78AF3C342AEAEF4BA33B35A8A15668C659C0B2@cceexc17.americas.cpqcorp.net> From: "Elliott, Robert" To: ips@ece.cmu.edu Subject: RE: iSCSI: INQUIRY page 0x83 identifier Date: Thu, 25 Jan 2001 16:02:51 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2652.78) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree that SPC should not reference FC-PH, FC-PH3, and FC-FS for VPD "Device Identifier" type 3h, especially given that iSCSI and InfiniBand SRP devices will want to use that format. We could describe the format in SPC-3 itself, making sure to match the existing FC formats. We could leave out some of the choices, perhaps only documenting the 64-bit IEEE Registered and 128-bit IEEE Registered Extended formats. For reference, the FC-FS formats are: * IEEE (48 bits, 802.1A universal LAN MAC address) * IEEE Extended (48 bits + 12 more bits) * Locally assigned (60 bits) * IP (32-bit IPv4 + 32 bits of padding) * IEEE registered (24-bit company ID + 36-bit vendor-specified identifier) * IEEE registered extended (24-bit company ID + 36-bit vendor-specified identifier + 64 bit vendor-specified identifier extension) * CCITT individual address (60 bits) * CCITT group address (60 bits) (the 4 bit NAA field identifying the naming authority increases these to 64 bits each, except for registered extended which is 128 bits) > -----Original Message----- > From: Charles Binford [mailto:Charles.Binford@BlueSpruceNet.com] > Sent: Thursday, January 25, 2001 8:46 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI: INQUIRY page 0x83 identifier > > Jim, You didn't confuse me more - you made a very good point. The > identifier is for the LU and is *independent* of the > transport. In fact, it > is vital that the same identifier be given over any path, any > transport. > Consider a storage device that has multiple host interfaces, > some may be FC, > some may be iSCSI, some may be parallel SCSI. If a host does > an INQUIRY > page x83 to the same LU through any of the above mentioned > interfaces, it > better get the same identifier back. > > Using an "FC type identifier" does not imply the interface is > FC. It just > specifies the format of the data. For the binary > identifiers, even the > format of the identifier is a 'don't care' to the host as it > should treat > them as opaque fields that can be matched, but not parsed. > By following the > format rules, however, the LU can ensure the identifier it gives is > world-wide unique. > > > Charles Binford > Blue Spruce Networks > office/cell: (316) 210-6404 > e-fax: (509) 756-4425 > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Jim Hafner > Sent: Wednesday, January 24, 2001 9:33 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: INQUIRY page 0x83 identifier > > JP, > > The identifier in this page of Inquiry is NOT at the 'target > device' or > 'iSCSI device' layer. It is an identifier for the logical > unit to which > the inquiry command was sent. The FC format was meant > primarily for FC > disk drives that have only one logical unit (though it can be > used in some > ways by FC controllers, etc.). > [deleted] > > Did I confuse things even more? > > Jim Hafner > From owner-ips@ece.cmu.edu Thu Jan 25 18:46:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA11645 for ; Thu, 25 Jan 2001 18:46:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PLuqT17155 for ips-outgoing; Thu, 25 Jan 2001 16:56:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PLuZ617144 for ; Thu, 25 Jan 2001 16:56:35 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 89F1A6A9 for ; Thu, 25 Jan 2001 14:56:34 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 25B0D178 for ; Thu, 25 Jan 2001 16:56:33 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id NAA11580 for ; Thu, 25 Jan 2001 13:56:31 -0800 (PST) Message-ID: <3A70A163.61048309@agilent.com> Date: Thu, 25 Jan 2001 13:57:55 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Bridging missing CmdSNs and Abort I/O Error recovery References: <3A70809C.3607E19F@cup.hp.com> <3A708363.4E3406EB@agilent.com> <3A7087AC.424C6E96@cup.hp.com> <3A708D56.FF028D7A@agilent.com> <3A709A3D.A7C51C3B@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > > Matt, > > Surely, an active task at a target involves some resources allocated by > the iSCSI layer firmware on the target ! > > Are you saying that target firmware at the SCSI Transport protocol > layer (FC, pSCSI, iSCSI) do NOT allocate ANY resources > (like in per-task data structures ) to execute a task, and thereby, > have no clean up to perform ?? Such as what? iSCSI just hands the CDB to the SCSI layer. After the SCSI layer decodes the CDB it then communicates to the iSCSI layer what resources (if any) need to be allocated to handle the command. -Matt From owner-ips@ece.cmu.edu Thu Jan 25 18:47:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA11687 for ; Thu, 25 Jan 2001 18:47:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PLPxH16056 for ips-outgoing; Thu, 25 Jan 2001 16:25:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PLPI616016 for ; Thu, 25 Jan 2001 16:25:18 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id C69F11887; Thu, 25 Jan 2001 13:25:08 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id NAA12852; Thu, 25 Jan 2001 13:24:59 -0800 (PST) Message-ID: <3A709A3D.A7C51C3B@cup.hp.com> Date: Thu, 25 Jan 2001 13:27:25 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Matt Wakeley Cc: IPS Reflector Subject: Re: iSCSI : Bridging missing CmdSNs and Abort I/O Error recovery References: <3A70809C.3607E19F@cup.hp.com> <3A708363.4E3406EB@agilent.com> <3A7087AC.424C6E96@cup.hp.com> <3A708D56.FF028D7A@agilent.com> Content-Type: multipart/mixed; boundary="------------CD1BA7C450EEBC4853ED9A5B" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------CD1BA7C450EEBC4853ED9A5B Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Matt, Surely, an active task at a target involves some resources allocated by the iSCSI layer firmware on the target ! Are you saying that target firmware at the SCSI Transport protocol layer (FC, pSCSI, iSCSI) do NOT allocate ANY resources (like in per-task data structures ) to execute a task, and thereby, have no clean up to perform ?? - Santosh Matt Wakeley wrote: > No, that would require iSCSI to interpret iSCSI commands. There is nothing at > the target iSCSI to clean up (except the "hole" in the command stream). > > The target iSCSI receives iSCSI commands and sends them to the SCSI layer, > what is there to clean up? > > -Matt > > Santosh Rao wrote: > > > > Matt Wakeley wrote: > > > > > The target iSCSI is the only entity that has knowledge of CmdSN. When it > > > receives a commands, it delivers them to the SCSI layer in CmdSN order (with > > > the CmdSN stripped off). iSCSI does not handle the abort commands. > > > > The Abort Task should also involve some functionality in the iSCSI layer, which > > is to invalidate and clean up the iSCSI resources for that task (?) > > > > - Santosh --------------CD1BA7C450EEBC4853ED9A5B Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------CD1BA7C450EEBC4853ED9A5B-- From owner-ips@ece.cmu.edu Thu Jan 25 18:47:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA11700 for ; Thu, 25 Jan 2001 18:47:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PLquS16987 for ips-outgoing; Thu, 25 Jan 2001 16:52:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PLqb616967 for ; Thu, 25 Jan 2001 16:52:37 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id B3452AEA for ; Thu, 25 Jan 2001 14:52:29 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 8849D59B for ; Thu, 25 Jan 2001 16:52:03 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id NAA11301 for ; Thu, 25 Jan 2001 13:52:01 -0800 (PST) Message-ID: <3A70A055.7123FE1E@agilent.com> Date: Thu, 25 Jan 2001 13:53:25 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues References: <3A707BCC.7A322FD0@cup.hp.com> <3A708FAB.819018F3@agilent.com> <3A7098F3.4451FB97@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > > Matt Wakeley wrote: > > > > Section 1.2.2.1 > > > =========== > > > - The target MUST silently ignore any command > > > outside this range or duplicates within the range not flagged with > > > the retry bit (the X bit in the opcode). > > > > > > This, to me, means : > > > - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. > > > - ignore all commands within the range that are not flagged with the retry > > > bit. > > I think you missed the word "duplicates"... > > Alright (so the cut & paste did'nt work) ;-} > > The issue under discussion is a retried command [and thereby, a duplicate > CmdSN] that is OUTSIDE the (ExpCmdSn, MaxCmdSN) range. That falls under the > first criteria. And the point is? What am I missing? -Matt > > - Santosh From owner-ips@ece.cmu.edu Thu Jan 25 18:47:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA11713 for ; Thu, 25 Jan 2001 18:47:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PLtqt17113 for ips-outgoing; Thu, 25 Jan 2001 16:55:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from fujitsuII.fujitsu.com (fujitsuII.fujitsu.com [133.164.253.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PLtJ617088 for ; Thu, 25 Jan 2001 16:55:20 -0500 (EST) Received: from fujitsuII.fujitsu.com (localhost [127.0.0.1]) by fujitsuII.fujitsu.com (8.9.3/8.9.3) with ESMTP id NAA16023 for ; Thu, 25 Jan 2001 13:55:12 -0800 (PST) Received: from central.fmi.fujitsu.com (central.fmi.fujitsu.com [162.35.28.137]) by fujitsuII.fujitsu.com (8.9.3/8.9.3) with ESMTP id NAA16009 for ; Thu, 25 Jan 2001 13:55:11 -0800 (PST) Received: from france.fmi.fujitsu.com (localhost [127.0.0.1]) by central.fmi.fujitsu.com (8.8.6/8.8.6) with ESMTP id NAA00860 for ; Thu, 25 Jan 2001 13:55:11 -0800 (PST) Received: from fmi.fujitsu.com (localhost [127.0.0.1]) by france.fmi.fujitsu.com (8.8.8+Sun/8.8.8) with ESMTP id NAA15362 for ; Thu, 25 Jan 2001 13:55:11 -0800 (PST) Message-ID: <3A70A0BE.C65BE03@fmi.fujitsu.com> Date: Thu, 25 Jan 2001 13:55:10 -0800 From: Laxmiprasad Nandipati X-Mailer: Mozilla 4.7 [en] (X11; U; SunOS 5.6 sun4u) X-Accept-Language: en MIME-Version: 1.0 CC: IPS Reflector Subject: Basic question. References: <3A70809C.3607E19F@cup.hp.com> <3A708363.4E3406EB@agilent.com> <3A7087AC.424C6E96@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hello all, I'm very new to iSCSI. Pls let me know where, i get the latest release (draft) version of the iSCSI spec. Thanx Prasad. From owner-ips@ece.cmu.edu Thu Jan 25 18:49:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA11749 for ; Thu, 25 Jan 2001 18:49:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PLLpk15883 for ips-outgoing; Thu, 25 Jan 2001 16:21:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PLLN615869 for ; Thu, 25 Jan 2001 16:21:23 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 97A43236; Thu, 25 Jan 2001 13:19:40 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id NAA12416; Thu, 25 Jan 2001 13:19:30 -0800 (PST) Message-ID: <3A7098F3.4451FB97@cup.hp.com> Date: Thu, 25 Jan 2001 13:21:56 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Matt Wakeley Cc: IPS Reflector Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues References: <3A707BCC.7A322FD0@cup.hp.com> <3A708FAB.819018F3@agilent.com> Content-Type: multipart/mixed; boundary="------------B31EC5418332A20B3C204F7B" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------B31EC5418332A20B3C204F7B Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Matt Wakeley wrote: > > Section 1.2.2.1 > > =========== > > - The target MUST silently ignore any command > > outside this range or duplicates within the range not flagged with > > the retry bit (the X bit in the opcode). > > > > This, to me, means : > > - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. > > - ignore all commands within the range that are not flagged with the retry > > bit. > I think you missed the word "duplicates"... Alright (so the cut & paste did'nt work) ;-} The issue under discussion is a retried command [and thereby, a duplicate CmdSN] that is OUTSIDE the (ExpCmdSn, MaxCmdSN) range. That falls under the first criteria. - Santosh --------------B31EC5418332A20B3C204F7B Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------B31EC5418332A20B3C204F7B-- From owner-ips@ece.cmu.edu Thu Jan 25 19:49:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA12666 for ; Thu, 25 Jan 2001 19:49:36 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PMgvR18807 for ips-outgoing; Thu, 25 Jan 2001 17:42:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PMgT618789 for ; Thu, 25 Jan 2001 17:42:29 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0PNqu053035; Thu, 25 Jan 2001 15:52:56 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: , Subject: RE: new iSCSI PDU outline Date: Thu, 25 Jan 2001 14:41:02 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Would you consider splitting headers. As was done with MTF, there was just a simple 32 bit simple XOR checksum for header fields. Here is an example with a CDB appended. The WN could expand the appended type into SCSI_CMND, SCSI_RSP, SCSI_R2T, iSCSI_Specific, etc. |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| WN | Reserved | AddCDB | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8| ->CmdSN or <-StatRN | +---------------+---------------+---------------+---------------+ 12| ->ExpStatSN or <-ExpCmdSN | +---------------+---------------+---------------+---------------+ 16| ->MaxStatRN? <-MaxCmdSN | +---------------+---------------+---------------+---------------+ 20| XOR Checksum | +---------------+---------------+---------------+---------------+ +---------------+---------------+---------------+---------------+ 24| SCSI_CMND ... | +- | 32| | +---------------+---------------+---------------+---------------+ 36| CDB ... | +---------------+---------------+---------------+---------------+ x | (Data ...) | +---------------+---------------+---------------+---------------+ y | CRC | +---------------+---------------+---------------+---------------+ Doug From owner-ips@ece.cmu.edu Thu Jan 25 19:50:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA12690 for ; Thu, 25 Jan 2001 19:50:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PMwwK19365 for ips-outgoing; Thu, 25 Jan 2001 17:58:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PMwT619351 for ; Thu, 25 Jan 2001 17:58:29 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id RAA70574; Thu, 25 Jan 2001 17:51:37 -0500 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id PAA59020; Thu, 25 Jan 2001 15:58:27 -0700 Importance: Normal Subject: Re: iSCSI: retries and SCSI To: Black_David@emc.com Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.5 September 22, 2000 Message-ID: From: "Prasenjit Sarkar" Date: Thu, 25 Jan 2001 14:58:22 -0800 X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.6 |December 14, 2000) at 01/25/2001 02:58:27 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk The motivation for doing this is that an iSCSI recovery on the initiator side should in general be cheaper (time/resources) than a SCSI recovery. ------------------------------------------------------------------------------------------------------------------------------------------------- From owner-ips@ece.cmu.edu Thu Jan 25 19:52:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA12726 for ; Thu, 25 Jan 2001 19:52:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PN49L19553 for ips-outgoing; Thu, 25 Jan 2001 18:04:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PN3o619535 for ; Thu, 25 Jan 2001 18:03:50 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 53FF99B5; Thu, 25 Jan 2001 15:03:49 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id PAA24093; Thu, 25 Jan 2001 15:03:44 -0800 (PST) Message-ID: <3A70B161.1990AAF2@cup.hp.com> Date: Thu, 25 Jan 2001 15:06:09 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Matt Wakeley Cc: IPS Reflector Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues References: <3A707BCC.7A322FD0@cup.hp.com> <3A708FAB.819018F3@agilent.com> <3A7098F3.4451FB97@cup.hp.com> <3A70A055.7123FE1E@agilent.com> Content-Type: multipart/mixed; boundary="------------0BB8A53F9DEE9BB4625AABB6" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------0BB8A53F9DEE9BB4625AABB6 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Matt, (You may want to take another look at the thread.). Here's the details again : > 3) The policy of "discard and restart" is also subject some race > conditions in the CmdSN sliding window. At the time the digest error was > > detected at the initiator, the ExpCmdSN may not yet have acknowledged > that > command. This causes the initiator to restart the command with the same > Initiator Task Tag, CmdSN and "retry" bit set. > > However, by the time the command gets to the target, the target may have > > updated its ExpCmdSN window having sent a later PDU which updated the > ExpCmdSN. This results in the target discarding the received CmdSN since > > at command restart you never really rely on the CmdSN. you will want > to check > the Initiator Task Tag and accept it in the above case. The draft states the following on this subject : Section 5.1 ========= - the initiator will reissue all outstanding commands with their original Initiator Task Tag and their original CmdRN if they are not acknowledged yet or a CmdRN of 0 (not-numbered) if they were acknowledged; the retry (X) flag in the command PDU will be set Section 1.2.2.1 =========== - The target MUST silently ignore any command outside this range or duplicates within the range not flagged with the retry bit (the X bit in the opcode). This, to me, means : - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. - ignore all duplicate commands within the range that are not flagged with the retry bit. << "duplicate" added this time >> Can you please clarify the intent of the text ? Matt Wakeley wrote: > Santosh Rao wrote: > > Alright (so the cut & paste did'nt work) ;-} > > > > The issue under discussion is a retried command [and thereby, a duplicate > > CmdSN] that is OUTSIDE the (ExpCmdSn, MaxCmdSN) range. That falls under the > > first criteria. > > And the point is? What am I missing? > > -Matt --------------0BB8A53F9DEE9BB4625AABB6 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------0BB8A53F9DEE9BB4625AABB6-- From owner-ips@ece.cmu.edu Thu Jan 25 20:38:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA13182 for ; Thu, 25 Jan 2001 20:38:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0PNgwD20882 for ips-outgoing; Thu, 25 Jan 2001 18:42:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0PNfv620847 for ; Thu, 25 Jan 2001 18:41:57 -0500 (EST) Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 25 Jan 2001 18:41:55 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101494@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: RE: iSCSI: retries and SCSI Date: Thu, 25 Jan 2001 18:41:54 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Please keep in mind that we're talking about optimizations available to an implementation that chooses not to retain state for retransmits. > > When the operation is not idempotent > > (doing it over has ill effects - tape reads and writes > > are examples), then this optimization is not applicable > > and if the target hasn't retained the stuff required > > to respond to the retry, it has to fail/reject the retry. > > Thanks for the clarification on this. This approach of > rejecting the retry, rather than not retrying for sequential > devices, poses a couple of problems : > > a) The responsibility of dealing with the reject will lie > with a iSCSI-pSCSI or iSCSI-FC bridge in most of > the tape connectivity cases. If one doesn't know that the optimization is safe, one shouldn't try it. One bit of configuration per target addressed through the bridge is sufficient to turn this on or off, and it could default to off. One could do better by looking at the iSCSI command and the embedded SCSI CDB, and may have to in any case because SCSI-ordered commands issued to a disk device are not always idempotent (although my understanding is that SCSI ordering is not generally used with disks). > b) The target iSCSI layer now needs to share SCSI state > information with its ULP on the device's class and abilities. > (something that would normally be exchanged b/n the peer > SCSI ULPs on either side.) "share SCSI state" is the wrong phrase. This is about configuring the iSCSI implementation's handling of retries. That configuration is based on the SCSI device's abilities, but I don't see a major issue here. > I believe the draft already intends to address the issue of retries > on a connection failure being made optional > [if the initiator detects ULP timeout and wishes not to retry]. > > I would like to suggest that the same policy be applied to digest > errors as well. (as initiators may not wish to retry on a digest error, > when it sees a ULP timeout.). > This allows initiators the flexibility to not retry the I/O to > non-idempotent target media. That seems reasonable, however this approach still requires targets to cope with (e.g., reject) retries issued by an initiator that exercised the flexibility to make the other choice. The initiator can choose not to retry as an optimization because it knows that retrying would be a waste of time for this target. Keep in mind that even non-idempotent targets may retain enough state to be able to respond to a retry without re-executing the command. I may have misread what Santosh wrote, but ULP timeout sounds to me like a SCSI-level timeout, something that should NEVER cause iSCSI to issue a retry - that's the responsibility of some sort of SCSI entity like a wedge driver. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Thu Jan 25 20:38:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA13193 for ; Thu, 25 Jan 2001 20:38:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0Q0R0422156 for ips-outgoing; Thu, 25 Jan 2001 19:27:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hotmail.com (oe29.law11.hotmail.com [64.4.16.86]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0Q0Qp622145 for ; Thu, 25 Jan 2001 19:26:51 -0500 (EST) Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Thu, 25 Jan 2001 16:26:45 -0800 X-Originating-IP: [24.218.3.65] From: "Eddy Quicksall" To: References: Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Date: Thu, 25 Jan 2001 15:43:32 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Message-ID: X-OriginalArrivalTime: 26 Jan 2001 00:26:45.0250 (UTC) FILETIME=[A9BF3620:01C0872E] Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Unfortunatly, I must agree with this issue. You should never rely on anything within the scope of a CRC error. So, there would be no way to verify the tag. Eddy ----- Original Message ----- From: To: Sent: Thursday, January 25, 2001 12:40 PM Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues > > > answers in text - Julo > > Santosh Rao on 25/01/2001 04:06:59 > > Please respond to Santosh Rao > > To: IPS Reflector > cc: > Subject: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues > > > > > Julian & All, > > There's a number of problems with the current iSCSI handling of digest > errors (Section 5.5) : > > "When an initiator receives an iSCSI data PDU with a digest error or any > other PDU with a header or data digest error, it must discard and > re-start the task -provided it can recognize the Initiator Task Tag". > > 1) The initiator task tag cannot be trusted when a header digest error > is seen. What does the phrase "provided it can recognize the initiator > task tag" mean ? > How can an initiator reliably claim that the initiator task tag is > trustworthy ? > > an initiator may choose to provide some redundancy in the tag itself > > [ rest of original message deleted here] > Regards, > Santosh > > - santoshr.vcf > > > > From owner-ips@ece.cmu.edu Thu Jan 25 22:24:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA15092 for ; Thu, 25 Jan 2001 22:24:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0Q19ij23331 for ips-outgoing; Thu, 25 Jan 2001 20:09:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0P58r608902 for ; Thu, 25 Jan 2001 00:08:53 -0500 (EST) Received: from divyaroot.India.Sun.COM ([129.158.225.35]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id VAA08795 for ; Wed, 24 Jan 2001 21:08:51 -0800 (PST) Received: from helix (helix [129.158.226.51]) by divyaroot.India.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1) with SMTP id KAA02522 for ; Thu, 25 Jan 2001 10:38:49 +0530 (IST) Message-Id: <200101250508.KAA02522@divyaroot.India.Sun.COM> Date: Thu, 25 Jan 2001 10:40:30 -0500 (GMT) From: Raghavendra Rao Reply-To: Raghavendra Rao Subject: Re: iSCSI: INQUIRY page 0x83 identifier To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: 4JduHLHeSjpQWLu7b07z8Q== Sender: owner-ips@ece.cmu.edu Precedence: bulk > > Since NDT has defined WWUIs for > DEVICES and these have a defined UTF-8 format, there is nothing to > *prevent* a vendor from using these strings as the starting point for LU > page 83 identifiers of the logical units within that device. This is a good starting point - I accept it. But wouldn't it help iSCSI if NDT recommends a LU identifier format ? I'm not asking for a new invention of LU identifier format here - I'm asking for adoption of one of the formats that SPC-2 currently defines. > 2) But..., iSCSI and NDT should not attempt to make a rule under which > these are used. FC's intervention in this was more as a strong suggestion, > and is certainly not a requirement. > Does this mean that "type" field in the page 83 LU identifier shouldn't be taken too seriously ? Thanks. -JP From owner-ips@ece.cmu.edu Thu Jan 25 22:59:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA16424 for ; Thu, 25 Jan 2001 22:59:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0Q2D3E25083 for ips-outgoing; Thu, 25 Jan 2001 21:13:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0Q2Ct625078 for ; Thu, 25 Jan 2001 21:12:55 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id B1826167 for ; Thu, 25 Jan 2001 18:12:54 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA12366 for ; Thu, 25 Jan 2001 18:12:49 -0800 (PST) Message-ID: <3A70DDB2.68F573CC@cup.hp.com> Date: Thu, 25 Jan 2001 18:15:15 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Command Ordering Proposal. Content-Type: multipart/mixed; boundary="------------4BAD780921B262B30E6FF7C2" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------4BAD780921B262B30E6FF7C2 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian & All, Proposal : ======= CmdSN should be used to enforce ordering only when the Ordered Task Tag attribute is set in the SCSI Command PDU. Reference : ========= Current iSCSI model for enforcing command ordering, based on striping commands across multiple TCP connections in an iSCSI Session and using a CmdSN to order the commands received out of order. Currently, CmdSN is used to enforce ordering for all commands. Objective of multi-connection sessions : ============================== Multiple TCP connections per session were defined in order to allow for : a) bandwidth aggregation by the creation of a fat pipe between the target and initiator. b) to optimize against single connection failures. c) to allow for iSCSI layer failover mechanisms within a session. Problems with the current ordering model : ================================ 1) When one TCP connection is faulty in an iSCSI session and is causing command transmission failures, all the commands being sent on other connections in the session will stall [due to the CmdSN ordering] at the receiving iSCSI layer of the target, until TCP timeouts + re-transmissions for the commands on the faulty connection are exhausted and a "link ping" of the faulty connection fails, thereby, detecting connection failure and initiating connection failure recovery. While operating under such conditions, the commands sent on other TCP connections within the session will be stalled until the initiator commences connection failure recovery and re-sends the commands with their original CmdRN on a new connection. This can result in a large number of stalled I/Os that can remain stalled for the period it takes to detect a faulty TCP connection and complete connection failure recovery. 2) This strict command ordering introduces a lot of complexity into error recovery conditions like connection failures, QUEUE FULL, BUSY & CHECK CONDITION handling due to the attempts iSCSI makes to enforce strict ordering of command delivery. 3) This model depends on the retry of ALL the failed commands on a new connection in order to bridge the missing CmdSNs at the target. If a subset of the failed commands were not to be retried [due to a retry policy that forbids this, such as non-idempotent media, or a ULP timeout], then, the CmdSNs at the target waiting for the out-of-order commands to arrive, need bridging which results in additional complexity. Requirements for Ordering : ===================== 1) SCSI ordering is only required on a per-LUN basis. 2) Strict SCSI ordering is only required when requested through the Ordered Task Tag attribute. For all the other types of tasks, no command ordering is required. Strict Command ordering is not enforced in other SCSI transport protocols. Command ordering in other SCSI transport protocols is violated when a QUEUE FULL, BUSY or CHECK CONDITION SCSI Status is returned. Command ordering can also be violated when the SCSI transport encounters transport failures or resource allocation failures on intermittent commands. 3) SAM Section 4.6.2 assumes the service delivery subsystem to provide in-order delivery for convinience. "This assumption is only made to simplify description of behaviour and does NOT constitute a requirement." Solution : ======== iSCSI should only enforce strict ordering when requested to do so by the SCSI ULP through the Ordered Task Tag attribute. Benefits : ======== 1) Optimizes iSCSI performance by avoiding stalls on other TCP connections within a session when one connection is faulty. 2) Simplifies error recovery on connection failure, digest error, CHECK CONDITION, QUEUE FULL and BUSY scsi status as well as initiator resource allocation errors. ------------------------------------------------------- Regards, Santosh --------------4BAD780921B262B30E6FF7C2 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------4BAD780921B262B30E6FF7C2-- From owner-ips@ece.cmu.edu Thu Jan 25 23:00:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA16469 for ; Thu, 25 Jan 2001 23:00:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0Q294l24977 for ips-outgoing; Thu, 25 Jan 2001 21:09:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0Q28e624964 for ; Thu, 25 Jan 2001 21:08:41 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id 0CB3BBC2 for ; Thu, 25 Jan 2001 18:08:40 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA06009; Thu, 25 Jan 2001 18:08:19 -0800 (PST) Message-ID: <3A70DC29.729EF4CE@hp.com> Date: Thu, 25 Jan 2001 18:08:41 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues References: <3A707BCC.7A322FD0@cup.hp.com> <3A708FAB.819018F3@agilent.com> <3A7098F3.4451FB97@cup.hp.com> <3A70A055.7123FE1E@agilent.com> <3A70B161.1990AAF2@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > Matt, > > (You may want to take another look at the thread.). Here's the details again : > > > 3) The policy of "discard and restart" is also subject some race > > conditions in the CmdSN sliding window. At the time the digest error was > > > > detected at the initiator, the ExpCmdSN may not yet have acknowledged > > that > > command. This causes the initiator to restart the command with the same > > Initiator Task Tag, CmdSN and "retry" bit set. The "retry" can work and works fine only in the case it "retry" a command that was issued on a failed connection that has been logged out. In the other cases you are in trouble because of the ghost IOs and "retry" MUST be avoided. > > > > > However, by the time the command gets to the target, the target may have > > > > updated its ExpCmdSN window having sent a later PDU which updated the > > ExpCmdSN. This results in the target discarding the received CmdSN since > > > > at command restart you never really rely on the CmdSN. you will want > > to check > > the Initiator Task Tag and accept it in the above case. > > The draft states the following on this subject : > > Section 5.1 > ========= > - the initiator will reissue all outstanding commands with their > original Initiator Task Tag and their original CmdRN if they > are not acknowledged yet or a CmdRN of 0 (not-numbered) if they > were acknowledged; the retry (X) flag in the command PDU will > be set > > Section 1.2.2.1 > =========== > - The target MUST silently ignore any command > outside this range or duplicates within the range not flagged with > the retry bit (the X bit in the opcode). > > This, to me, means : > - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. > - ignore all duplicate commands within the range that are not flagged with the > retry > bit. << "duplicate" added this time >> > > Can you please clarify the intent of the text ? Duplicate means a command with the same CmdSN. Regards, Pierre From owner-ips@ece.cmu.edu Thu Jan 25 23:04:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA16535 for ; Thu, 25 Jan 2001 23:04:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0Q220A24783 for ips-outgoing; Thu, 25 Jan 2001 21:02:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gordan.pl.gadzoox.com (i248.gadzoox.com [216.52.31.248]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0Q20r624746 for ; Thu, 25 Jan 2001 21:01:02 -0500 (EST) Received: by gordan.pl.gadzoox.com with Internet Mail Service (5.5.2650.21) id ; Thu, 25 Jan 2001 18:01:26 -0800 Message-ID: <312419998E3CD211A52900A0C991A47A3A21B7@gordan.pl.gadzoox.com> From: Gabriel Hecht To: ips@ece.cmu.edu Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Thu, 25 Jan 2001 18:01:26 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Currently, FCIP doesn't map TCP channels to FC streams. By stream, I mean, related FCP commands and data that may require 'in order delivery' by the SCSI application. Mapping TCP channels to FC addresses will guarantee that a FC-stream is fully contained within a TCP channel. FCIP leaves this mapping to the Gateway implementation. For example, assigning a TCP connection per remote D_ID address or group of address is relatively easy for the H/W to implement. If we don't want to use time-stamp for ordering, the only implementation limit will be to not put a FC stream on more than one TCP channel, to avoid out of order delivery. Gaby -----Original Message----- From: Douglas Otis [mailto:dotis@sanlight.net] Sent: Thursday, January 25, 2001 11:11 AM To: Black_David@emc.com; ips@ece.cmu.edu Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports David, For what it's worth, I am not making such fine distinctions. I was attempting to discuss a reason for including a time-stamp should such placement be onto FC media. (Looking to the day when there is a means to handle out of sequence frames.) Doug From owner-ips@ece.cmu.edu Fri Jan 26 04:09:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA03024 for ; Fri, 26 Jan 2001 04:09:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0Q7HAC01854 for ips-outgoing; Fri, 26 Jan 2001 02:17:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0Q7Gr601847 for ; Fri, 26 Jan 2001 02:16:53 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Thu, 25 Jan 2001 23:16:46 -0800 Message-ID: From: Charles Monia To: Santosh Rao Cc: IPS Reflector Subject: RE: iSCSI : Command Ordering Proposal. Date: Thu, 25 Jan 2001 23:16:46 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Santosh: FWIW: The ordering constraints in SAM were established before the prospect of deploying SCSI over relatively long latency, very high bandwidth interconnects was close to a reality. IMHO therefore, the decision as to the need for ordered delivery ought to be made on the basis of whether or not that requirement is appropriate for the specific interconnect environment. Anyhow (and with only a bit of irony), I'm not sure partial enforcement is worth the trouble. In my opinion, you'd be better off eliminating the requirement altogether since what's left may not be all that useful. For one thing, an ordered command acts like a barrier seperating the execution of queued commands received before and after it. So, there is an order dependancy between simple and ordered commands that would be hard, if not impossible, to enforce with the new ordering rules. For example, if I send commands A B |C| D E, where |C| is ordered, I expect that D and E won't complete before |C|. If A B D and E arrive followed by |C|, there's no way to obtain the correct behavior. Incidentally, similar constraints hold true for "head of queue" tasks. Also, to the extent that command ordering is useful, command sequence numbers have been added to the CDB payload. So devices would still have a limited way to enforce command sequentiality at the LU level (above the iSCSI layer). Another issue is the problem of insuring that an ABORT TASK function is processed in the correct order relative to the task being terminated. I believe that can be handled by insuring that the task management function is sent on the same TCP/IP connection that the task was bound to. Of course, there are other task management requests, such as ABORT TASK SET, for which that approach doesn't work. But those are edge conditions that may or may not be worth considering (and, frankly, I don't feel much like arguing the point). As far as queue full and the other conditions you mention below, I was under the impression that a SCSI change was being proposed to address the problems you describe. So those exceptions should not be among the reasons for implementing the change you propose. Charles > -----Original Message----- > From: Santosh Rao [mailto:santoshr@cup.hp.com] > Sent: Thursday, January 25, 2001 6:15 PM > To: IPS Reflector > Subject: iSCSI : Command Ordering Proposal. > > > Julian & All, > > Proposal : > ======= > CmdSN should be used to enforce ordering only when the > Ordered Task Tag > attribute is set in the SCSI Command PDU. > > Reference : > ========= > Current iSCSI model for enforcing command ordering, based on striping > commands across multiple TCP connections in an iSCSI Session > and using a > > CmdSN to order the commands received out of order. Currently, CmdSN is > used to enforce ordering for all commands. > > Objective of multi-connection sessions : > ============================== > Multiple TCP connections per session were defined in order to > allow for : > a) bandwidth aggregation by the creation of a fat pipe between the > target and initiator. > b) to optimize against single connection failures. > c) to allow for iSCSI layer failover mechanisms within a session. > > > Problems with the current ordering model : > ================================ > 1) When one TCP connection is faulty in an iSCSI session and > is causing > command transmission failures, all the commands being sent on other > connections in the session will stall [due to the CmdSN ordering] > at the receiving iSCSI layer of the target, until TCP > timeouts + re-transmissions for the commands on the faulty > connection are exhausted and a "link ping" of the faulty > connection fails, thereby, detecting connection failure and initiating > connection failure recovery. > > While operating under such conditions, the commands sent on > other TCP connections within the session will be stalled until the > initiator commences connection failure recovery and re-sends the > commands with their original CmdRN on a new connection. > > This can result in a large number of stalled I/Os that can remain > stalled for the period it takes to detect a faulty TCP connection and > complete connection failure recovery. > > 2) This strict command ordering introduces a lot of complexity into > error recovery conditions like connection failures, QUEUE FULL, BUSY & > CHECK CONDITION handling due to the attempts iSCSI makes to enforce > strict ordering of command delivery. > > 3) This model depends on the retry of ALL the failed commands on a new > connection in order to bridge the missing CmdSNs at the target. If a > subset of the failed commands were not to be retried [due to a retry > policy that forbids this, such as non-idempotent media, or a > ULP timeout], > then, the CmdSNs at the target waiting for the out-of-order > commands to > arrive, need bridging which results in additional complexity. > > Requirements for Ordering : > ===================== > 1) SCSI ordering is only required on a per-LUN basis. > > 2) Strict SCSI ordering is only required when requested through the > Ordered Task Tag attribute. For all the other types of tasks, > no command > > ordering is required. Strict Command ordering is not enforced in other > SCSI transport protocols. Command ordering in other SCSI transport > protocols is violated when a QUEUE FULL, BUSY or CHECK > CONDITION SCSI Status is returned. Command ordering can also > be violated when the SCSI transport encounters transport failures > or resource allocation failures on intermittent commands. > > 3) SAM Section 4.6.2 assumes the service delivery subsystem to provide > in-order delivery for convinience. "This assumption is only made to > simplify > description of behaviour and does NOT constitute a requirement." > > Solution : > ======== > iSCSI should only enforce strict ordering when requested to > do so by the > > SCSI ULP through the Ordered Task Tag attribute. > > Benefits : > ======== > 1) Optimizes iSCSI performance by avoiding stalls on other TCP > connections within a session when one connection is faulty. > > 2) Simplifies error recovery on connection failure, digest error, > CHECK CONDITION, QUEUE FULL and BUSY scsi status as > well as initiator resource allocation errors. > > ------------------------------------------------------- > > Regards, > Santosh > From owner-ips@ece.cmu.edu Fri Jan 26 05:09:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA03505 for ; Fri, 26 Jan 2001 05:09:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0Q8LCU03068 for ips-outgoing; Fri, 26 Jan 2001 03:21:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0Q8KC603030 for ; Fri, 26 Jan 2001 03:20:12 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Fri, 26 Jan 2001 00:20:09 -0800 Message-ID: From: Charles Monia To: "Ips (E-mail)" Cc: "John Hufferd (IBM) (E-mail)" Subject: SNIA Technical Working Group to Promote IP Storage Date: Fri, 26 Jan 2001 00:20:02 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi: This note is being posted in the interest of obtaining the widest possible distribution to potential participants. SNIA, the Storage Networking Industry Association, has authorized a marketing forum comprised of member companies who wish to promote IP-based storage technologies. In support of that effort, a companion technical working group is being formed within SNIA which will act as the Forum's interface to the IPS wg. The SNIA WG is expected to support the Marketing Forum by producing technical requirements based on Forum input. When appropriate, these will be brought to the IETF IPS Working Group as draft proposals or requirements. The goal of these two SNIA groups is to perform industry focussed efforts that complement the scope of IETF standardization activities. The initial formation meeting of the SNIA IP Storage Technical Working Group is scheduled for next week, within the overall SNIA meeting in Colorado Springs. The time, date and location are given below. Attendees must be affiliated with a SNIA member company. Date: Thursday, February 1, 2001 Time: 9 AM to 12 Noon Location: Colorado Springs Wyndham Hotel, Eagles Nest Conference Rooms 1 and 2. We anticipate one or two follow up meetings or telecons to complete the charter and organizational process. The temporary co-chairs for this meeting are Charles Monia, and John Hufferd For more information, visit the SNIA web site at http://www.snia.org Charles Monia Senior Technology Consultant Nishan Systems email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ece.cmu.edu Fri Jan 26 06:29:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA03970 for ; Fri, 26 Jan 2001 06:29:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0Q9TEv04357 for ips-outgoing; Fri, 26 Jan 2001 04:29:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (play-doh.Stanford.EDU [128.12.131.42]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0Q9Sa604346 for ; Fri, 26 Jan 2001 04:28:36 -0500 (EST) Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 6AB958D58; Fri, 26 Jan 2001 01:28:16 -0800 (PST) To: Santosh Rao Subject: Re: iSCSI : Command Ordering Proposal. References: <3A70DDB2.68F573CC@cup.hp.com> Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII From: csapuntz@cisco.com Date: 26 Jan 2001 01:28:16 -0800 In-Reply-To: Santosh Rao's message of "Thu, 25 Jan 2001 18:15:15 -0800" Message-ID: Lines: 51 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Capitol Reef" Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh Rao writes: > [1 ] > Julian & All, > > Proposal : > ======= > CmdSN should be used to enforce ordering only when the Ordered Task Tag > attribute is set in the SCSI Command PDU. Santosh, I believe you may be missing a subtlety to the SCSI task queueing model. I may also be missing a subtlety of your argument and, if so, please excuse this e-mail. Consider three commands sent down a FIFO SCSI transport: 1) Ordered Command 2) Simple Command 3) Simple command The ordered command is received first and then the two simple commands. The ordered command is guaranteed to be executed before either SIMPLE command. Relevant text from SAM-2 rev 14, sec 7.5.1 "The [simple] task shall not enter the Enabled state until all older Head of Queue and older Ordered tasks have ended." Your proposed change means that CmdSN no longer is able express that the Ordered task is "older" that the "simple" task. Thus, an initiator might need to stop-and-wait before issuing #2 and #3. I say "might" because iSCSI now transports SCSI Command Reference Numbers (CRNs) in the command PDU. The CRN is a SAM-2 rev. 15 (and FCP) construct that tells us which commands are older than others. So, in some cases, the CmdSN may no longer be needed at all to tell which commands are "older" than others. Given that we have CRNs, why do we need CmdSNs? Well, CmdSNs provide a couple extra features: - ordering on task management (SAM doesn't defined that CRNs are used with task management) - ordering when application doesn't support CRNs (though couldn't the iSCSI layer fabricate CRNs?) - flow control of the iSCSI target queue -Costa From owner-ips@ece.cmu.edu Fri Jan 26 12:15:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA15262 for ; Fri, 26 Jan 2001 12:15:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QFLDU26468 for ips-outgoing; Fri, 26 Jan 2001 10:21:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-1.cisco.com (sj-msg-core-1.cisco.com [171.71.163.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QFKiZ26456 for ; Fri, 26 Jan 2001 10:20:44 -0500 (EST) Received: from sponge.cisco.com (sponge.cisco.com [171.71.61.25]) by sj-msg-core-1.cisco.com (8.9.3/8.9.1) with ESMTP id HAA14573 for ; Fri, 26 Jan 2001 07:20:53 -0800 (PST) Received: from DAPW2K (dhcp-161-44-68-202.cisco.com [161.44.68.202]) by sponge.cisco.com (Mirapoint) with SMTP id AFU37701; Fri, 26 Jan 2001 09:20:36 -0600 (CST) From: "David Peterson" To: "Ips@Ece. Cmu. Edu" Subject: iSCSI : Command Ordering Proposal. Date: Fri, 26 Jan 2001 09:19:07 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Howdy Charles, The CmdRN has been renamed to CmdSN (Command Sequence Number). The previously reserved field (word 0, byte 2) is now the placeholder for SAM's CRN. Dave > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Charles Binford > Sent: Friday, January 26, 2001 9:10 AM > To: Ips (E-mail) > Subject: RE: iSCSI : Command Ordering Proposal. > > > I'm confused on a couple of points: > > - Are we talking about 'CmdSN' (a field that doesn't exist unless I missed > an update since iscsi-03) or 'CmdRN'? > > - Yes, CRN was added to SAM, but I see nothing in iSCSI that utilizes it. > Is this thread suggesting a new field be added? I certainly don't see how > the current CmdRN field could fill the roll of SAM's CRN and its current > roll concurrently (different rules on initial value, when to reset, etc.) > > Seems to me like adding a new CRN field for command ordering (if the > application so chooses) and re-defining the CmdRN field to only > provide the > flow control satisfies the requirements without overloading the > functionality of a single field. > > > Charles Binford > Blue Spruce Networks > office/cell: (316) 210-6404 > e-fax: (509) 756-4425 > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > csapuntz@cisco.com > Sent: Friday, January 26, 2001 3:28 AM > To: Santosh Rao > Cc: csapuntz@cisco.com > Subject: Re: iSCSI : Command Ordering Proposal. > > Santosh Rao writes: > > > [1 ] > > Julian & All, > > > > Proposal : > > ======= > > CmdSN should be used to enforce ordering only when the Ordered Task Tag > > attribute is set in the SCSI Command PDU. > > > Santosh, > > I believe you may be missing a subtlety to the SCSI task queueing model. > I may also be missing a subtlety of your argument and, if so, > please excuse > this e-mail. > > Consider three commands sent down a FIFO SCSI transport: > > 1) Ordered Command > 2) Simple Command > 3) Simple command > > The ordered command is received first and then the two simple commands. > The ordered command is guaranteed to be executed before either SIMPLE > command. > > Relevant text from SAM-2 rev 14, sec 7.5.1 > > "The [simple] task shall not enter the Enabled state until all > older Head of > Queue and older Ordered tasks have ended." > > Your proposed change means that CmdSN no longer is able express that > the Ordered task is "older" that the "simple" task. Thus, an initiator > might need to stop-and-wait before issuing #2 and #3. > > I say "might" because iSCSI now transports SCSI Command Reference > Numbers (CRNs) in the command PDU. The CRN is a SAM-2 rev. 15 (and > FCP) construct that tells us which commands are older than others. So, > in some cases, the CmdSN may no longer be needed at all to tell which > commands are "older" than others. > > Given that we have CRNs, why do we need CmdSNs? Well, CmdSNs provide > a couple extra features: > - ordering on task management (SAM doesn't defined > that CRNs are used with task management) > - ordering when application doesn't support CRNs > (though couldn't the iSCSI layer fabricate CRNs?) > - flow control of the iSCSI target queue > > -Costa > > From owner-ips@ece.cmu.edu Fri Jan 26 12:19:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA15355 for ; Fri, 26 Jan 2001 12:19:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QF8KM26084 for ips-outgoing; Fri, 26 Jan 2001 10:08:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from glatton.cnchost.com (glatton.cnchost.com [207.155.248.47]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QF7KZ26056 for ; Fri, 26 Jan 2001 10:07:20 -0500 (EST) Received: from BSNCBINFORD (service.recreationresource.com [208.189.17.82] (may be forged)) by glatton.cnchost.com id KAA15244; Fri, 26 Jan 2001 10:07:14 -0500 (EST) [ConcentricHost SMTP Relay 1.10] From: "Charles Binford" To: "Ips \(E-mail\)" Subject: RE: iSCSI : Command Ordering Proposal. Date: Fri, 26 Jan 2001 09:10:15 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I'm confused on a couple of points: - Are we talking about 'CmdSN' (a field that doesn't exist unless I missed an update since iscsi-03) or 'CmdRN'? - Yes, CRN was added to SAM, but I see nothing in iSCSI that utilizes it. Is this thread suggesting a new field be added? I certainly don't see how the current CmdRN field could fill the roll of SAM's CRN and its current roll concurrently (different rules on initial value, when to reset, etc.) Seems to me like adding a new CRN field for command ordering (if the application so chooses) and re-defining the CmdRN field to only provide the flow control satisfies the requirements without overloading the functionality of a single field. Charles Binford Blue Spruce Networks office/cell: (316) 210-6404 e-fax: (509) 756-4425 -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of csapuntz@cisco.com Sent: Friday, January 26, 2001 3:28 AM To: Santosh Rao Cc: csapuntz@cisco.com Subject: Re: iSCSI : Command Ordering Proposal. Santosh Rao writes: > [1 ] > Julian & All, > > Proposal : > ======= > CmdSN should be used to enforce ordering only when the Ordered Task Tag > attribute is set in the SCSI Command PDU. Santosh, I believe you may be missing a subtlety to the SCSI task queueing model. I may also be missing a subtlety of your argument and, if so, please excuse this e-mail. Consider three commands sent down a FIFO SCSI transport: 1) Ordered Command 2) Simple Command 3) Simple command The ordered command is received first and then the two simple commands. The ordered command is guaranteed to be executed before either SIMPLE command. Relevant text from SAM-2 rev 14, sec 7.5.1 "The [simple] task shall not enter the Enabled state until all older Head of Queue and older Ordered tasks have ended." Your proposed change means that CmdSN no longer is able express that the Ordered task is "older" that the "simple" task. Thus, an initiator might need to stop-and-wait before issuing #2 and #3. I say "might" because iSCSI now transports SCSI Command Reference Numbers (CRNs) in the command PDU. The CRN is a SAM-2 rev. 15 (and FCP) construct that tells us which commands are older than others. So, in some cases, the CmdSN may no longer be needed at all to tell which commands are "older" than others. Given that we have CRNs, why do we need CmdSNs? Well, CmdSNs provide a couple extra features: - ordering on task management (SAM doesn't defined that CRNs are used with task management) - ordering when application doesn't support CRNs (though couldn't the iSCSI layer fabricate CRNs?) - flow control of the iSCSI target queue -Costa From owner-ips@ece.cmu.edu Fri Jan 26 13:08:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA16580 for ; Fri, 26 Jan 2001 13:08:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QFuFT27768 for ips-outgoing; Fri, 26 Jan 2001 10:56:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QFtCZ27704 for ; Fri, 26 Jan 2001 10:55:13 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAA72138 for ; Fri, 26 Jan 2001 16:55:07 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA248888 for ; Fri, 26 Jan 2001 16:55:07 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E0.00576F90 ; Fri, 26 Jan 2001 16:55:01 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 26 Jan 2001 17:50:37 +0200 Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, I think that you are misreading my comments. When I say a may in one of my answers it does not mean that it has to appear as such in the draft. As for the interpretation of restart - once delivered a restarted command has to do just that restart. If for implementation reasons you want to take the abort path and then restart that is an implementation issue. In any case it is not equivalent with sending an explicit task abort then start the command as this is not an atomic sequence and may affect the command ordering. Julo N.B. I really appreciate you keeping the list so alive during you education session but I think that you can improve your hit ratio (i.e. getting to real problems) if you'll discuss your points with your powerful "home-team" (Randy, Pierre etc.). Santosh Rao on 25/01/2001 21:17:33 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Julian, My responses enclosed inline. Regards, Santosh julian_satran@il.ibm.com wrote: > 1) The initiator task tag cannot be trusted when a header digest error > is seen. What does the phrase "provided it can recognize the initiator > task tag" mean ? > How can an initiator reliably claim that the initiator task tag is > trustworthy ? > > an initiator may choose to provide some redundancy in the tag itself > All of these excessive "mays" only serve to complicate the standard and end up as un-used features. They are better off removed. > > 2) The use of a "discard and re-start" policy can cause some problems. > The target is un-aware that the initiator has discarded the task and > will then receive a 2nd command with the same Initiator Task Tag and > CmdSN for a task already in progress, which can cause transmission of > duplicate sets of data on the same I.T.T. > > Any policy of "discard and re-start" must be modified to "discard, abort > > and then, re-start, and the draft MUST specify connection allegiance of > the command and its Abort Task". > > a restart is recognized as such through the restart bit. Why would you > require abort? Where does the draft EXPLICITLY state that the "retry" bit implies an implicit abort of the command followed by re-start ? In the absence of such explicit clarifications, the draft is opening up opportunities for numerous creative mis-interpretations. Despite this, there is still a window b/n the command being re-started at the initiator and the target receiving the command with "retry" bit set, when stale frames continue to arrive at the initiator for that I.T.T. Therefore, in order to avoid stale frames from the previous command continuing to arrive at the initiator in this window, there needs to be an abort followed by a re-start. All of this complexity is only in the case of digest error handling [and not in the case of connection failures]. The draft can be rid of all this complexity if digest errors were to be recovered at a connection level, instead of this "discard & restart" policy which adds all these complexities. > > please explain > > 3) The policy of "discard and restart" is also subject some race > conditions in the CmdSN sliding window. At the time the digest error was > > detected at the initiator, the ExpCmdSN may not yet have acknowledged > that > command. This causes the initiator to restart the command with the same > Initiator Task Tag, CmdSN and "retry" bit set. > > However, by the time the command gets to the target, the target may have > > updated its ExpCmdSN window having sent a later PDU which updated the > ExpCmdSN. This results in the target discarding the received CmdSN since > > at command restart you never really rely on the CmdSN. you will want > to check > the Initiator Task Tag and accept it in the above case. The draft states the following on this subject : Section 5.1 ========= - the initiator will reissue all outstanding commands with their original Initiator Task Tag and their original CmdRN if they are not acknowledged yet or a CmdRN of 0 (not-numbered) if they were acknowledged; the retry (X) flag in the command PDU will be set Section 1.2.2.1 =========== - The target MUST silently ignore any command outside this range or duplicates within the range not flagged with the retry bit (the X bit in the opcode). This, to me, means : - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. - ignore all commands within the range that are not flagged with the retry bit. Can you please clarify the intent of the text ? - santoshr.vcf From owner-ips@ece.cmu.edu Fri Jan 26 13:09:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA16596 for ; Fri, 26 Jan 2001 13:09:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QGOGs28891 for ips-outgoing; Fri, 26 Jan 2001 11:24:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QGNDZ28828 for ; Fri, 26 Jan 2001 11:23:13 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id RAA63296; Fri, 26 Jan 2001 17:23:07 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA49442; Fri, 26 Jan 2001 17:23:02 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E0.0059FD74 ; Fri, 26 Jan 2001 17:22:55 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Laxmiprasad Nandipati cc: ips@ece.cmu.edu Message-ID: Date: Fri, 26 Jan 2001 18:18:31 +0200 Subject: Re: Basic question. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk 3 sources: http://www.ietf.org/internet-drafts/draft-ietf-ips-iSCSI-03.txt and other docs having "draft-ietf-ips" in their pathname. http://www.ece.cmu.edu/~ips http://www.haifa.il.ibm.com/satran/ips Julo From owner-ips@ece.cmu.edu Fri Jan 26 13:13:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA16689 for ; Fri, 26 Jan 2001 13:13:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QG5DX28118 for ips-outgoing; Fri, 26 Jan 2001 11:05:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QG4uZ28111 for ; Fri, 26 Jan 2001 11:04:57 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id RAA111432 for ; Fri, 26 Jan 2001 17:04:49 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA153690 for ; Fri, 26 Jan 2001 17:04:50 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E0.005851FA ; Fri, 26 Jan 2001 17:04:41 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 26 Jan 2001 18:00:13 +0200 Subject: Re: iSCSI : Bridging missing CmdSNs and Abort I/O Error recovery Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, You had a possible answer from Matt. However I agree that we might want to address this in text although a solution similar to that suggested by Matt should be by now obvious to every implementer - the target should leave a placeholder in the input queue until the command after gets delivered. Julo Santosh Rao on 25/01/2001 21:38:04 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iSCSI : Bridging missing CmdSNs and Abort I/O Error recovery Julian & All, The draft is currently lacking a section that addresses abort I/O error recovery. Specifically, how is CmdSN bridging issues to be handled in the case where an initiator chooses not to retry an I/O [that failed on a connection failure that affects the delivery of the command to the target or a digest error at the target] because its ULP timer may have expired. In such cases, the initiator can send an Abort Task to inform the target that the I.T.T is being aborted and its corresponding CmdSN can be bridged, instead of having the target stall infinitely in its attempt to enforce ordering and await the missing CmdSN [which is'nt going to arrive, because the initiator did not retry the command]. Regards, Santosh - santoshr.vcf From owner-ips@ece.cmu.edu Fri Jan 26 14:04:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA18382 for ; Fri, 26 Jan 2001 14:04:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QHUHw01515 for ips-outgoing; Fri, 26 Jan 2001 12:30:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hotmail.com (f233.law11.hotmail.com [64.4.17.233]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QHTIZ01476 for ; Fri, 26 Jan 2001 12:29:19 -0500 (EST) Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Fri, 26 Jan 2001 09:29:09 -0800 Received: from 24.218.3.65 by lw11fd.law11.hotmail.msn.com with HTTP; Fri, 26 Jan 2001 17:29:09 GMT X-Originating-IP: [24.218.3.65] From: "Eddy Quicksall" To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Date: Fri, 26 Jan 2001 12:29:09 -0500 Mime-Version: 1.0 Content-Type: text/plain; format=flowed Message-ID: X-OriginalArrivalTime: 26 Jan 2001 17:29:09.0322 (UTC) FILETIME=[7DA9C2A0:01C087BD] Sender: owner-ips@ece.cmu.edu Precedence: bulk Regarding number 1) which refers to section 5.5 ... why not just change the paragraph to say: When an initiator receives an iSCSI data PDU with a data payload digest error or any other iSCSI PDU with a header or payload digest error it MUST discard it, and restart the task - the later provided header does not have a digest error. If there is a header digest error the initiator may need to recognize and recover using an implementation specific scheme such as timing the request. Schemes may include loging out the connection and restarting it (including restarting all outstanding tasks). Eddy ----Original Message Follows---- From: julian_satran@il.ibm.com To: ips@ece.cmu.edu Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Date: Fri, 26 Jan 2001 17:50:37 +0200 Santosh, I think that you are misreading my comments. When I say a may in one of my answers it does not mean that it has to appear as such in the draft. As for the interpretation of restart - once delivered a restarted command has to do just that restart. If for implementation reasons you want to take the abort path and then restart that is an implementation issue. In any case it is not equivalent with sending an explicit task abort then start the command as this is not an atomic sequence and may affect the command ordering. Julo N.B. I really appreciate you keeping the list so alive during you education session but I think that you can improve your hit ratio (i.e. getting to real problems) if you'll discuss your points with your powerful "home-team" (Randy, Pierre etc.). Santosh Rao on 25/01/2001 21:17:33 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Julian, My responses enclosed inline. Regards, Santosh julian_satran@il.ibm.com wrote: > 1) The initiator task tag cannot be trusted when a header digest error > is seen. What does the phrase "provided it can recognize the initiator > task tag" mean ? > How can an initiator reliably claim that the initiator task tag is > trustworthy ? > > an initiator may choose to provide some redundancy in the tag itself > All of these excessive "mays" only serve to complicate the standard and end up as un-used features. They are better off removed. > > 2) The use of a "discard and re-start" policy can cause some problems. > The target is un-aware that the initiator has discarded the task and > will then receive a 2nd command with the same Initiator Task Tag and > CmdSN for a task already in progress, which can cause transmission of > duplicate sets of data on the same I.T.T. > > Any policy of "discard and re-start" must be modified to "discard, abort > > and then, re-start, and the draft MUST specify connection allegiance of > the command and its Abort Task". > > a restart is recognized as such through the restart bit. Why would you > require abort? Where does the draft EXPLICITLY state that the "retry" bit implies an implicit abort of the command followed by re-start ? In the absence of such explicit clarifications, the draft is opening up opportunities for numerous creative mis-interpretations. Despite this, there is still a window b/n the command being re-started at the initiator and the target receiving the command with "retry" bit set, when stale frames continue to arrive at the initiator for that I.T.T. Therefore, in order to avoid stale frames from the previous command continuing to arrive at the initiator in this window, there needs to be an abort followed by a re-start. All of this complexity is only in the case of digest error handling [and not in the case of connection failures]. The draft can be rid of all this complexity if digest errors were to be recovered at a connection level, instead of this "discard & restart" policy which adds all these complexities. > > please explain > > 3) The policy of "discard and restart" is also subject some race > conditions in the CmdSN sliding window. At the time the digest error was > > detected at the initiator, the ExpCmdSN may not yet have acknowledged > that > command. This causes the initiator to restart the command with the same > Initiator Task Tag, CmdSN and "retry" bit set. > > However, by the time the command gets to the target, the target may have > > updated its ExpCmdSN window having sent a later PDU which updated the > ExpCmdSN. This results in the target discarding the received CmdSN since > > at command restart you never really rely on the CmdSN. you will want > to check > the Initiator Task Tag and accept it in the above case. The draft states the following on this subject : Section 5.1 ========= - the initiator will reissue all outstanding commands with their original Initiator Task Tag and their original CmdRN if they are not acknowledged yet or a CmdRN of 0 (not-numbered) if they were acknowledged; the retry (X) flag in the command PDU will be set Section 1.2.2.1 =========== - The target MUST silently ignore any command outside this range or duplicates within the range not flagged with the retry bit (the X bit in the opcode). This, to me, means : - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. - ignore all commands within the range that are not flagged with the retry bit. Can you please clarify the intent of the text ? - santoshr.vcf _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com From owner-ips@ece.cmu.edu Fri Jan 26 14:05:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA18485 for ; Fri, 26 Jan 2001 14:05:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QGfG829648 for ips-outgoing; Fri, 26 Jan 2001 11:41:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QGemZ29638 for ; Fri, 26 Jan 2001 11:40:48 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id RAA17474 for ; Fri, 26 Jan 2001 17:40:42 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA42858 for ; Fri, 26 Jan 2001 17:40:42 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E0.005B9C42 ; Fri, 26 Jan 2001 17:40:37 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 26 Jan 2001 18:29:15 +0200 Subject: RE: new iSCSI PDU outline Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, You are right and we could do it but it will reduce the number of yet unallocated types in WN (and T10 is adding more stuff to the SCSI!) and will save us several bits in the command where we have plenty. I don't think it is worth it. Regards, Julo "Douglas Otis" on 26/01/2001 00:41:02 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: new iSCSI PDU outline Julian, Would you consider splitting headers. As was done with MTF, there was just a simple 32 bit simple XOR checksum for header fields. Here is an example with a CDB appended. The WN could expand the appended type into SCSI_CMND, SCSI_RSP, SCSI_R2T, iSCSI_Specific, etc. |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| WN | Reserved | AddCDB | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8| ->CmdSN or <-StatRN | +---------------+---------------+---------------+---------------+ 12| ->ExpStatSN or <-ExpCmdSN | +---------------+---------------+---------------+---------------+ 16| ->MaxStatRN? <-MaxCmdSN | +---------------+---------------+---------------+---------------+ 20| XOR Checksum | +---------------+---------------+---------------+---------------+ +---------------+---------------+---------------+---------------+ 24| SCSI_CMND ... | +- | 32| | +---------------+---------------+---------------+---------------+ 36| CDB ... | +---------------+---------------+---------------+---------------+ x | (Data ...) | +---------------+---------------+---------------+---------------+ y | CRC | +---------------+---------------+---------------+---------------+ Doug From owner-ips@ece.cmu.edu Fri Jan 26 14:07:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA18647 for ; Fri, 26 Jan 2001 14:07:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QHHHY01042 for ips-outgoing; Fri, 26 Jan 2001 12:17:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QHGsZ01031 for ; Fri, 26 Jan 2001 12:16:54 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA46844 for ; Fri, 26 Jan 2001 18:16:47 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA260880 for ; Fri, 26 Jan 2001 18:16:46 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E0.005EE990 ; Fri, 26 Jan 2001 18:16:41 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 26 Jan 2001 19:12:19 +0200 Subject: Re: iSCSI : Command Ordering Proposal. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Besides what Costa already stated vs. this will involve maintaining state per LU - an expensive thing we have rejected long ago. Julo Santosh Rao on 26/01/2001 04:15:15 Please respond to Santosh Rao To: IPS Reflector cc: Subject: iSCSI : Command Ordering Proposal. Julian & All, Proposal : ======= CmdSN should be used to enforce ordering only when the Ordered Task Tag attribute is set in the SCSI Command PDU. Reference : ========= Current iSCSI model for enforcing command ordering, based on striping commands across multiple TCP connections in an iSCSI Session and using a CmdSN to order the commands received out of order. Currently, CmdSN is used to enforce ordering for all commands. Objective of multi-connection sessions : ============================== Multiple TCP connections per session were defined in order to allow for : a) bandwidth aggregation by the creation of a fat pipe between the target and initiator. b) to optimize against single connection failures. c) to allow for iSCSI layer failover mechanisms within a session. Problems with the current ordering model : ================================ 1) When one TCP connection is faulty in an iSCSI session and is causing command transmission failures, all the commands being sent on other connections in the session will stall [due to the CmdSN ordering] at the receiving iSCSI layer of the target, until TCP timeouts + re-transmissions for the commands on the faulty connection are exhausted and a "link ping" of the faulty connection fails, thereby, detecting connection failure and initiating connection failure recovery. While operating under such conditions, the commands sent on other TCP connections within the session will be stalled until the initiator commences connection failure recovery and re-sends the commands with their original CmdRN on a new connection. This can result in a large number of stalled I/Os that can remain stalled for the period it takes to detect a faulty TCP connection and complete connection failure recovery. 2) This strict command ordering introduces a lot of complexity into error recovery conditions like connection failures, QUEUE FULL, BUSY & CHECK CONDITION handling due to the attempts iSCSI makes to enforce strict ordering of command delivery. 3) This model depends on the retry of ALL the failed commands on a new connection in order to bridge the missing CmdSNs at the target. If a subset of the failed commands were not to be retried [due to a retry policy that forbids this, such as non-idempotent media, or a ULP timeout], then, the CmdSNs at the target waiting for the out-of-order commands to arrive, need bridging which results in additional complexity. Requirements for Ordering : ===================== 1) SCSI ordering is only required on a per-LUN basis. 2) Strict SCSI ordering is only required when requested through the Ordered Task Tag attribute. For all the other types of tasks, no command ordering is required. Strict Command ordering is not enforced in other SCSI transport protocols. Command ordering in other SCSI transport protocols is violated when a QUEUE FULL, BUSY or CHECK CONDITION SCSI Status is returned. Command ordering can also be violated when the SCSI transport encounters transport failures or resource allocation failures on intermittent commands. 3) SAM Section 4.6.2 assumes the service delivery subsystem to provide in-order delivery for convinience. "This assumption is only made to simplify description of behaviour and does NOT constitute a requirement." Solution : ======== iSCSI should only enforce strict ordering when requested to do so by the SCSI ULP through the Ordered Task Tag attribute. Benefits : ======== 1) Optimizes iSCSI performance by avoiding stalls on other TCP connections within a session when one connection is faulty. 2) Simplifies error recovery on connection failure, digest error, CHECK CONDITION, QUEUE FULL and BUSY scsi status as well as initiator resource allocation errors. ------------------------------------------------------- Regards, Santosh - santoshr.vcf From owner-ips@ece.cmu.edu Fri Jan 26 14:11:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA18777 for ; Fri, 26 Jan 2001 14:11:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QGpF400035 for ips-outgoing; Fri, 26 Jan 2001 11:51:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QGohZ00014 for ; Fri, 26 Jan 2001 11:50:43 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id RAA55954 for ; Fri, 26 Jan 2001 17:50:35 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA274000 for ; Fri, 26 Jan 2001 17:50:36 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E0.005C84D0 ; Fri, 26 Jan 2001 17:50:33 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 26 Jan 2001 18:46:08 +0200 Subject: Re: I.T.T. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, You are right - I stand corrected. Julo Santosh Rao on 26/01/2001 03:41:24 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: I.T.T. Julian, As mentioned earlier, the below should state "to each task that it issues", since I.T.T. is also assigned to non-scsi PDUs. Regards, Santosh > 2.2.4.4 Initiator Task Tag > The initiator assigns a Task Tag to each SCSI task that it issues. - santoshr.vcf From owner-ips@ece.cmu.edu Fri Jan 26 14:54:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA19952 for ; Fri, 26 Jan 2001 14:54:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QIXQG03893 for ips-outgoing; Fri, 26 Jan 2001 13:33:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QIX4Z03876 for ; Fri, 26 Jan 2001 13:33:04 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 3A874796 for ; Fri, 26 Jan 2001 11:33:02 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id B8EBE69B for ; Fri, 26 Jan 2001 13:20:59 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id KAA27908 for ; Fri, 26 Jan 2001 10:20:58 -0800 (PST) Message-ID: <3A71C053.3BBD711A@agilent.com> Date: Fri, 26 Jan 2001 10:22:11 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh, I guess I don't understand what it is that you think needs clarification. -Matt Santosh Rao wrote: > > Matt, > > (You may want to take another look at the thread.). Here's the details again : > > > 3) The policy of "discard and restart" is also subject some race > > conditions in the CmdSN sliding window. At the time the digest error was > > > > detected at the initiator, the ExpCmdSN may not yet have acknowledged > > that > > command. This causes the initiator to restart the command with the same > > Initiator Task Tag, CmdSN and "retry" bit set. > > > > However, by the time the command gets to the target, the target may have > > > > updated its ExpCmdSN window having sent a later PDU which updated the > > ExpCmdSN. This results in the target discarding the received CmdSN since > > > > at command restart you never really rely on the CmdSN. you will want > > to check > > the Initiator Task Tag and accept it in the above case. > > The draft states the following on this subject : > > Section 5.1 > ========= > - the initiator will reissue all outstanding commands with their > original Initiator Task Tag and their original CmdRN if they > are not acknowledged yet or a CmdRN of 0 (not-numbered) if they > were acknowledged; the retry (X) flag in the command PDU will > be set > > Section 1.2.2.1 > =========== > - The target MUST silently ignore any command > outside this range or duplicates within the range not flagged with > the retry bit (the X bit in the opcode). > > This, to me, means : > - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. > - ignore all duplicate commands within the range that are not flagged with the > retry > bit. << "duplicate" added this time >> > > Can you please clarify the intent of the text ? > > Matt Wakeley wrote: > > > Santosh Rao wrote: > > > Alright (so the cut & paste did'nt work) ;-} > > > > > > The issue under discussion is a retried command [and thereby, a duplicate > > > CmdSN] that is OUTSIDE the (ExpCmdSn, MaxCmdSN) range. That falls under the > > > first criteria. > > > > And the point is? What am I missing? > > > > -Matt From owner-ips@ece.cmu.edu Fri Jan 26 14:54:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA19967 for ; Fri, 26 Jan 2001 14:54:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QHtIL02327 for ips-outgoing; Fri, 26 Jan 2001 12:55:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QHsIZ02291 for ; Fri, 26 Jan 2001 12:54:18 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA104830 for ; Fri, 26 Jan 2001 18:54:14 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA165302 for ; Fri, 26 Jan 2001 18:54:14 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E0.006257A1 ; Fri, 26 Jan 2001 18:54:09 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 26 Jan 2001 19:49:43 +0200 Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, Education was not meant to be insulting and I am sorry if did sound to you. However you are restarting discussion threads that where long settled without bringing new arguments. And I am not exactly thrilled in reliving the thread. Regards, Julo Santosh Rao on 26/01/2001 19:29:09 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: Black_David@emc.com, santoshr@hpcuhe.cup.hp.com (Santosh Rao) Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues > > Santosh, > > As for the interpretation of restart - once delivered a restarted command > has to do just that restart. Julian, I believe you are referring to the "Retry" bit , is that correct ? (There is nothing in the draft that I can find called a Restart bit). If you are referring to the retry bit, can you please point me to where the draft EXPLICITLY states that a "retry" shall cause the target to implicitly abort execution of the previous instance of that command ? In the absence of such explicit wording in the draft, how does one derive the conclusion you mention above ? Also, I did'nt see your response to the below 2 issues raised : > Section 1.2.2.1 > =========== > - The target MUST silently ignore any command > outside this range or duplicates within the range not flagged with > the retry bit (the X bit in the opcode). > > This, to me, means : > - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. > - ignore all duplicate commands within the range that are not > flagged with the retry bit. > Can you please clarify the intent of the text ? From the above it does'nt sound like the draft specifies that CmdSNs received out of the window are to be accepted if the Retry bit is set. > There is still a window b/n the command being re-started at > the initiator and the target receiving the command with "retry" > bit set, when stale frames continue to arrive at the initiator for > that I.T.T. Therefore, in order to avoid stale frames from the > previous command continuing to arrive at the initiator in this window, > there needs to be an abort followed by a re-start. How is the above to be avoided ? > N.B. I really appreciate you keeping the list so alive during you > education session but I think that you can improve your hit ratio Anyone would appreciate comments being restricted to technical responses instead of such remarks. Perhaps, you may want to check the threads in the past to see how many issues have been uncovered and how many times you stood corrected. (ex : faking of check conditions instead of using a service response, response data and response length, DataRN, target originated connection level error recovery thru the use of Async Messages, Abort Task response, retries to tapes result in possible data corruption, bridging CmdSNs, fragmentation & re-assembly issues, etc). Wonder who's getting educated here. (?) -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Fri Jan 26 14:55:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA19993 for ; Fri, 26 Jan 2001 14:55:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QI4Hj02652 for ips-outgoing; Fri, 26 Jan 2001 13:04:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from feral.com (IDENT:root@feral.com [192.67.166.1]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QI42Z02634 for ; Fri, 26 Jan 2001 13:04:03 -0500 (EST) Received: from zeppo.feral.com (IDENT:mjacob@zeppo [192.67.166.71]) by feral.com (8.9.3/8.9.3) with ESMTP id KAA27067; Fri, 26 Jan 2001 10:03:18 -0800 Date: Fri, 26 Jan 2001 10:03:14 -0800 (PST) From: Matthew Jacob Reply-To: mjacob@feral.com To: julian_satran@il.ibm.com cc: Laxmiprasad Nandipati , ips@ece.cmu.edu Subject: Re: Basic question. In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk On Fri, 26 Jan 2001 julian_satran@il.ibm.com wrote: > > > 3 sources: > > > http://www.ietf.org/internet-drafts/draft-ietf-ips-iSCSI-03.txt Bad URL. I can only fine version 1 online. Can you give a better link for -03? > > and other docs having "draft-ietf-ips" in their pathname. > > http://www.ece.cmu.edu/~ips > > http://www.haifa.il.ibm.com/satran/ips > > Julo > > From owner-ips@ece.cmu.edu Fri Jan 26 15:38:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA20923 for ; Fri, 26 Jan 2001 15:38:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QIYRR03942 for ips-outgoing; Fri, 26 Jan 2001 13:34:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QIXPZ03890 for ; Fri, 26 Jan 2001 13:33:25 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 20327765 for ; Fri, 26 Jan 2001 11:33:22 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 7A3AB239 for ; Fri, 26 Jan 2001 13:24:27 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id KAA28004 for ; Fri, 26 Jan 2001 10:24:26 -0800 (PST) Message-ID: <3A71C123.64AD97E5@agilent.com> Date: Fri, 26 Jan 2001 10:25:39 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI: I/O (command) recovery Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit There's been some confusion on how iSCSI commands that fail could be recovered at the iSCSI layer without involving SCSI, so I thought I'd write up (my understanding of) how this is done. For a Target Read operation: 1- The initiator sends a read command to the target. 2- The target iSCSI sends the command to the SCSI layer. 3- The SCSI layer performs the read command and hands the read data and status to the iSCSI layer. iSCSI then allocates resources and stores the read data and the I/O status in the allocated resources. 4- The target iSCSI sends the read data and status to the initiator, but does *not* deallocate the resources associated with the I/O. 5- When the initiator iSCSI successfully receives the data and status, it acknowledges the receipt of the status back to the target iSCSI via the ExpStatSN. 6- When the target iSCSI receives the ExpStatSN acknowledging the I/O, it then deallocates the resources for the I/O. If some error occurs such that the initiator iSCSI does not receive all the data and status, then the initiator iSCSI performs whatever link/connection/session recovery to re-establish communication with the target. At that point, the initiator iSCSI layer re-issues the I/O with the retry bit set. - At this point, the target iSCSI layer (*not* the SCSI layer) recognizes the command is a duplicate (because the retry bit is set), and instead of sending the command to the SCSI layer, it simply retransmits the buffered data and status [continue at #5 above]. For a Target Write operation: 1- The initiator sends a write command to the target. 2- The target iSCSI sends the command to the SCSI layer. 3- The SCSI layer prepares for the write and asks the iSCSI for the data. iSCSI then allocates resources for the I/O. 4- The target iSCSI sends R2T to the initiator. 5- The initiator sends the write data to the target. 6- The target iSCSI receives the data and sends it to the SCSI layer. 7- The SCSI layer performs the write and gives the status to the iSCSI layer. 8- The target iSCSI sends the status to the initiator (and saves it in the iSCSI allocated resources for the I/O), but does *not* deallocate the resources associated with the I/O. 9- When the initiator iSCSI successfully receives the status, it acknowledges the receipt of the status back to the target iSCSI via the ExpStatSN. 10- When the target iSCSI receives the ExpStatSN acknowledging the I/O, it then deallocates the resources for the I/O. If some error occurs such that the initiator iSCSI does not receive the status, then the initiator iSCSI performs whatever link/connection/session recovery to re-establish communication with the target. At that point, the initiator iSCSI layer re-issues the I/O with the retry bit set. - At this point, the target iSCSI layer (*not* the SCSI layer) recognizes the command is a duplicate. - If the target iSCSI has not received all the data for the R2T, then the iSCSI layer re-issues the R2T [continue at #5 above]. - If the target iSCSI layer has sent the SCSI status, it simply retransmits the status [continue at #9 above]. Now, this is just a basic overview on how it could be done. I'm sure there are variations on how it could be implemented in the target. Should this example be put into the iSCSI document? -Matt Wakeley Agilent Technologies From owner-ips@ece.cmu.edu Fri Jan 26 15:42:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA21068 for ; Fri, 26 Jan 2001 15:42:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QIXNm03888 for ips-outgoing; Fri, 26 Jan 2001 13:33:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QIX1Z03872 for ; Fri, 26 Jan 2001 13:33:01 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 9581E73D for ; Fri, 26 Jan 2001 11:32:58 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id C6E0B1C7 for ; Fri, 26 Jan 2001 13:19:41 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id KAA27753 for ; Fri, 26 Jan 2001 10:19:40 -0800 (PST) Message-ID: <3A71C006.2982B3FE@agilent.com> Date: Fri, 26 Jan 2001 10:20:54 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Command Ordering Proposal. Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit One feature that command numbering was used for was to avoid the queue full condition (because the initiator could not send commands past MaxCmdSN). If you don't care about ordering, then either use a CmdSN of zero, or open multiple "sessions" each with 1 TCP connection. Then you could scatter your commands across all the sessions. -Matt From owner-ips@ece.cmu.edu Fri Jan 26 16:22:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA22396 for ; Fri, 26 Jan 2001 16:22:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QJPXH06105 for ips-outgoing; Fri, 26 Jan 2001 14:25:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QJOYZ06044 for ; Fri, 26 Jan 2001 14:24:34 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0QKYo054378; Fri, 26 Jan 2001 12:34:54 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Gabriel Hecht" , Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Fri, 26 Jan 2001 11:22:59 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <312419998E3CD211A52900A0C991A47A3A21B7@gordan.pl.gadzoox.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Gaby, > Currently, FCIP doesn't map TCP channels to FC streams. By > stream, I mean, > related FCP commands and data that may require 'in order delivery' by the > SCSI application. Mapping TCP channels to FC addresses will > guarantee that > a FC-stream is fully contained within a TCP channel. Should the application be using multiple processors with multiple FC connections, then what you see as isolated is perhaps not. An advantage of using multiple TCP connections is lost if all traffic to a device is restricted to a connection. At what point in time would the gateway change to a different connection if there was a problem. All this activity is time related and to sift through the mess, a time-stamp still seems essential. > FCIP leaves this > mapping to the Gateway implementation. For example, assigning a TCP > connection per remote D_ID address or group of address is relatively easy > for the H/W to implement. If we don't want to use time-stamp for > ordering, > the only implementation limit will be to not put a FC stream on more than > one TCP channel, to avoid out of order delivery. This conclusion assumes no relationship between other sources and no fail-over will be made and all traffic will be held until a missing frame is recovered. A TCP interruption may recover within 20 seconds, but with respect to FC however, this is forever. If there are multiple working connections, and one is briefly lost, how does one allow a fail-over? TCP may attempt delivery for a very long period. Is it safe to assume no timing relationship between FC sources? As a TCP interruption may occur for periods longer than the R_A_TOV, is there a violation of the FC fabric to then release frames held for say 20 seconds? Doug > Gaby > > > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Thursday, January 25, 2001 11:11 AM > To: Black_David@emc.com; ips@ece.cmu.edu > Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports > > > David, > > For what it's worth, I am not making such fine distinctions. I was > attempting to discuss a reason for including a time-stamp should such > placement be onto FC media. (Looking to the day when there is a means to > handle out of sequence frames.) > > Doug > From owner-ips@ece.cmu.edu Fri Jan 26 16:22:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA22409 for ; Fri, 26 Jan 2001 16:22:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QJqOf07186 for ips-outgoing; Fri, 26 Jan 2001 14:52:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from feral.com (IDENT:root@feral.com [192.67.166.1]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QJpRZ07146 for ; Fri, 26 Jan 2001 14:51:28 -0500 (EST) Received: from zeppo.feral.com (IDENT:mjacob@zeppo [192.67.166.71]) by feral.com (8.9.3/8.9.3) with ESMTP id LAA27560; Fri, 26 Jan 2001 11:51:21 -0800 Date: Fri, 26 Jan 2001 11:51:20 -0800 (PST) From: Matthew Jacob Reply-To: mjacob@feral.com To: Joe Powell cc: julian_satran@il.ibm.com, Laxmiprasad Nandipati , ips@ece.cmu.edu Subject: RE: Basic question. In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Gronk. Thanks. > Matt, > > It's case-sensitive and all lower-case, so the link should be: > > http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-03.txt > > Joe Powell > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Matthew Jacob > Sent: Friday, January 26, 2001 10:03 AM > To: julian_satran@il.ibm.com > Cc: Laxmiprasad Nandipati; ips@ece.cmu.edu > Subject: Re: Basic question. > > > On Fri, 26 Jan 2001 julian_satran@il.ibm.com wrote: > > > > > > > 3 sources: > > > > > > http://www.ietf.org/internet-drafts/draft-ietf-ips-iSCSI-03.txt > > Bad URL. I can only fine version 1 online. Can you give a better link > for -03? > > > > > and other docs having "draft-ietf-ips" in their pathname. > > > > http://www.ece.cmu.edu/~ips > > > > http://www.haifa.il.ibm.com/satran/ips > > > > Julo > > > > > > From owner-ips@ece.cmu.edu Fri Jan 26 16:22:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA22431 for ; Fri, 26 Jan 2001 16:22:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QJPc206115 for ips-outgoing; Fri, 26 Jan 2001 14:25:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QJNLZ05992 for ; Fri, 26 Jan 2001 14:23:21 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0QKV9054370; Fri, 26 Jan 2001 12:31:14 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Charles Monia" , "Santosh Rao" Cc: "IPS Reflector" Subject: RE: iSCSI : Command Ordering Proposal. Date: Fri, 26 Jan 2001 11:19:17 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Charles, If you wish to use tape backup, you should reconsider a need for maintaining sequences. To allow a mixture of repeated command streams, this too is an area where keeping things in order, even on the same TCP connection is a challenge. On a single TCP connection, at what point do you allow a fail-over? Or do you simply fail the operation? As for task management, it seems once those commands are in play, things have already gone astray. For recovery, this interface is highly time sensitive but you could not tell from the proposal that time even matters. Life is easier if you assume a wait forever mode on any connection, but that is not the expectations of most implementations. Doug > Hi Santosh: > > FWIW: The ordering constraints in SAM were established before the prospect > of deploying SCSI over relatively long latency, very high bandwidth > interconnects was close to a reality. IMHO therefore, the > decision as to the > need for ordered delivery ought to be made on the basis of whether or not > that requirement is appropriate for the specific interconnect environment. > > Anyhow (and with only a bit of irony), I'm not sure partial enforcement is > worth the trouble. In my opinion, you'd be better off eliminating the > requirement altogether since what's left may not be all that useful. > > For one thing, an ordered command acts like a barrier seperating the > execution of queued commands received before and after it. So, > there is an > order dependancy between simple and ordered commands that would > be hard, if > not impossible, to enforce with the new ordering rules. For example, if I > send commands A B |C| D E, where |C| is ordered, I expect that D > and E won't > complete before |C|. If A B D and E arrive followed by |C|, > there's no way > to obtain the correct behavior. Incidentally, similar constraints > hold true > for "head of queue" tasks. > > Also, to the extent that command ordering is useful, command sequence > numbers have been added to the CDB payload. So devices would still have a > limited way to enforce command sequentiality at the LU level (above the > iSCSI layer). > > Another issue is the problem of insuring that an ABORT TASK function is > processed in the correct order relative to the task being terminated. I > believe that can be handled by insuring that the task management > function is > sent on the same TCP/IP connection that the task was bound to. Of course, > there are other task management requests, such as ABORT TASK SET, > for which > that approach doesn't work. But those are edge conditions that may or may > not be worth considering (and, frankly, I don't feel much like arguing the > point). > > As far as queue full and the other conditions you mention below, > I was under > the impression that a SCSI change was being proposed to address > the problems > you describe. So those exceptions should not be among the reasons for > implementing the change you propose. > > Charles > > -----Original Message----- > > From: Santosh Rao [mailto:santoshr@cup.hp.com] > > Sent: Thursday, January 25, 2001 6:15 PM > > To: IPS Reflector > > Subject: iSCSI : Command Ordering Proposal. > > > > > > Julian & All, > > > > Proposal : > > ======= > > CmdSN should be used to enforce ordering only when the > > Ordered Task Tag > > attribute is set in the SCSI Command PDU. > > > > Reference : > > ========= > > Current iSCSI model for enforcing command ordering, based on striping > > commands across multiple TCP connections in an iSCSI Session > > and using a > > > > CmdSN to order the commands received out of order. Currently, CmdSN is > > used to enforce ordering for all commands. > > > > Objective of multi-connection sessions : > > ============================== > > Multiple TCP connections per session were defined in order to > > allow for : > > a) bandwidth aggregation by the creation of a fat pipe between the > > target and initiator. > > b) to optimize against single connection failures. > > c) to allow for iSCSI layer failover mechanisms within a session. > > > > > > Problems with the current ordering model : > > ================================ > > 1) When one TCP connection is faulty in an iSCSI session and > > is causing > > command transmission failures, all the commands being sent on other > > connections in the session will stall [due to the CmdSN ordering] > > at the receiving iSCSI layer of the target, until TCP > > timeouts + re-transmissions for the commands on the faulty > > connection are exhausted and a "link ping" of the faulty > > connection fails, thereby, detecting connection failure and initiating > > connection failure recovery. > > > > While operating under such conditions, the commands sent on > > other TCP connections within the session will be stalled until the > > initiator commences connection failure recovery and re-sends the > > commands with their original CmdRN on a new connection. > > > > This can result in a large number of stalled I/Os that can remain > > stalled for the period it takes to detect a faulty TCP connection and > > complete connection failure recovery. > > > > 2) This strict command ordering introduces a lot of complexity into > > error recovery conditions like connection failures, QUEUE FULL, BUSY & > > CHECK CONDITION handling due to the attempts iSCSI makes to enforce > > strict ordering of command delivery. > > > > 3) This model depends on the retry of ALL the failed commands on a new > > connection in order to bridge the missing CmdSNs at the target. If a > > subset of the failed commands were not to be retried [due to a retry > > policy that forbids this, such as non-idempotent media, or a > > ULP timeout], > > then, the CmdSNs at the target waiting for the out-of-order > > commands to > > arrive, need bridging which results in additional complexity. > > > > Requirements for Ordering : > > ===================== > > 1) SCSI ordering is only required on a per-LUN basis. > > > > 2) Strict SCSI ordering is only required when requested through the > > Ordered Task Tag attribute. For all the other types of tasks, > > no command > > > > ordering is required. Strict Command ordering is not enforced in other > > SCSI transport protocols. Command ordering in other SCSI transport > > protocols is violated when a QUEUE FULL, BUSY or CHECK > > CONDITION SCSI Status is returned. Command ordering can also > > be violated when the SCSI transport encounters transport failures > > or resource allocation failures on intermittent commands. > > > > 3) SAM Section 4.6.2 assumes the service delivery subsystem to provide > > in-order delivery for convinience. "This assumption is only made to > > simplify > > description of behaviour and does NOT constitute a requirement." > > > > Solution : > > ======== > > iSCSI should only enforce strict ordering when requested to > > do so by the > > > > SCSI ULP through the Ordered Task Tag attribute. > > > > Benefits : > > ======== > > 1) Optimizes iSCSI performance by avoiding stalls on other TCP > > connections within a session when one connection is faulty. > > > > 2) Simplifies error recovery on connection failure, digest error, > > CHECK CONDITION, QUEUE FULL and BUSY scsi status as > > well as initiator resource allocation errors. > > > > ------------------------------------------------------- > > > > Regards, > > Santosh > > > From owner-ips@ece.cmu.edu Fri Jan 26 16:23:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA22445 for ; Fri, 26 Jan 2001 16:23:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QJPZv06109 for ips-outgoing; Fri, 26 Jan 2001 14:25:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QJOMZ06032 for ; Fri, 26 Jan 2001 14:24:24 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0QKYu054381 for ; Fri, 26 Jan 2001 12:34:59 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: Subject: RE: new iSCSI PDU outline Date: Fri, 26 Jan 2001 11:23:04 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Should there be software implementations that must deal with iSCSI on a real-time basis, splitting the headers as I described helps in allowing the transport to respond immediately in providing the transport related feed-back. The split header and simple XOR calculation was to make this possible with software implementations where these settings are made nano-seconds before being sent. To require a CRC to be done could introduce many micro-seconds into the operation and thus make the transport less current on its feed-back as this must be pipelined. You have a few other fields not defined and perhaps use those fields to expand the later content. I doubt there will be more than a handful of types but if you use 16 bits, there should never be a danger of running out of options. In addition, I would suggest retry flags be placed in this header for the same reasons of simplifying the transport. Once the SCSI structure is created, it becomes independent of the transport functions so that fiddling at the transport will not spoil this content. By keeping these values to but a few fields unrelated to the payload, a simple check should be all that is needed. Doug > Doug, > > You are right and we could do it but it will reduce the number of yet > unallocated types in WN (and T10 is adding more stuff to the SCSI!) and > will save us several bits in the command where we have plenty. I don't > think it is worth it. > > Regards, > Julo > > "Douglas Otis" on 26/01/2001 00:41:02 > > Please respond to "Douglas Otis" > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: new iSCSI PDU outline > > > > > Julian, > > Would you consider splitting headers. As was done with MTF, > there was just > a simple 32 bit simple XOR checksum for header fields. Here is an example > with a CDB appended. The WN could expand the appended type into > SCSI_CMND, > SCSI_RSP, SCSI_R2T, iSCSI_Specific, etc. > > |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| > +---------------+---------------+---------------+---------------+ > 0| WN | Reserved | AddCDB | > +---------------+---------------+---------------+---------------+ > 4| Length | > +---------------+---------------+---------------+---------------+ > 8| ->CmdSN or <-StatRN | > +---------------+---------------+---------------+---------------+ > 12| ->ExpStatSN or <-ExpCmdSN | > +---------------+---------------+---------------+---------------+ > 16| ->MaxStatRN? <-MaxCmdSN | > +---------------+---------------+---------------+---------------+ > 20| XOR Checksum | > +---------------+---------------+---------------+---------------+ > +---------------+---------------+---------------+---------------+ > 24| SCSI_CMND ... | > +- | > 32| | > +---------------+---------------+---------------+---------------+ > 36| CDB ... | > +---------------+---------------+---------------+---------------+ > x | (Data ...) | > +---------------+---------------+---------------+---------------+ > y | CRC | > +---------------+---------------+---------------+---------------+ > > Doug > > > > > > > From owner-ips@ece.cmu.edu Fri Jan 26 16:24:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA22480 for ; Fri, 26 Jan 2001 16:24:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QJnQe07054 for ips-outgoing; Fri, 26 Jan 2001 14:49:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from c007.snv.cp.net (c007-h011.c007.snv.cp.net [209.228.33.217]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f0QJmWZ07012 for ; Fri, 26 Jan 2001 14:48:32 -0500 (EST) Received: (cpmta 12611 invoked from network); 26 Jan 2001 11:48:26 -0800 Received: from unknown (HELO jia) (64.128.88.181) by smtp.telocity.com (209.228.33.217) with SMTP; 26 Jan 2001 11:48:26 -0800 X-Sent: 26 Jan 2001 19:48:26 GMT From: "Joe Powell" To: , Cc: "Laxmiprasad Nandipati" , Subject: RE: Basic question. Date: Fri, 26 Jan 2001 11:48:03 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt, It's case-sensitive and all lower-case, so the link should be: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-03.txt Joe Powell -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Matthew Jacob Sent: Friday, January 26, 2001 10:03 AM To: julian_satran@il.ibm.com Cc: Laxmiprasad Nandipati; ips@ece.cmu.edu Subject: Re: Basic question. On Fri, 26 Jan 2001 julian_satran@il.ibm.com wrote: > > > 3 sources: > > > http://www.ietf.org/internet-drafts/draft-ietf-ips-iSCSI-03.txt Bad URL. I can only fine version 1 online. Can you give a better link for -03? > > and other docs having "draft-ietf-ips" in their pathname. > > http://www.ece.cmu.edu/~ips > > http://www.haifa.il.ibm.com/satran/ips > > Julo > > From owner-ips@ece.cmu.edu Fri Jan 26 19:18:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA25550 for ; Fri, 26 Jan 2001 19:18:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QMMRA12728 for ips-outgoing; Fri, 26 Jan 2001 17:22:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QMM9Z12718 for ; Fri, 26 Jan 2001 17:22:09 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id E9F79E73; Fri, 26 Jan 2001 14:22:06 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id OAA25589; Fri, 26 Jan 2001 14:22:01 -0800 (PST) Message-ID: <3A71F91E.C2B62BFE@cup.hp.com> Date: Fri, 26 Jan 2001 14:24:30 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Matt Wakeley Cc: IPS Reflector Subject: Re: iSCSI: I/O (command) recovery References: <3A71C123.64AD97E5@agilent.com> Content-Type: multipart/mixed; boundary="------------F3EA194F1D357D215BA3587A" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------F3EA194F1D357D215BA3587A Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Matt, This is a good write-up and clearly outlines the behaviour. Just a couple of comments below. Regards, Santosh Matt Wakeley wrote: > 4- The target iSCSI sends the read data and status to the initiator, but does > *not* deallocate the resources associated with the I/O. The draft is ambiguous about (4). It states that the numbering MUST be supported, but the data and status recovery MAY be supported. This means initiators & targets MUST use StatSN/ExpStatSN. However, the target is allowed to de-allocate the resources associated with the I/O without waiting for ExpStatSN ack. > If some error occurs such that the initiator iSCSI does not receive all the > data and status, then the initiator iSCSI performs whatever > link/connection/session recovery to re-establish communication with the > target. At that point, the initiator iSCSI layer re-issues the I/O with the > retry bit set. > > - At this point, the target iSCSI layer (*not* the SCSI layer) recognizes the > command is a duplicate (because the retry bit is set), and instead of sending Again, the wording for the above in the draft could do with some clarity (Section 1.2.2.1). It is not clear from the draft whether CmdSNs that are outside the (ExpCmdSN, MaxCmdSN) window are to be accepted if the retry bit is set. My interpretation of : "- The target MUST silently ignore any command outside this range or duplicates within the range not flagged with the retry bit (the X bit in the opcode)." is : - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. - ignore all duplicate commands within the range that are not flagged with the retry So, if a command with the "retry" bit is set is received outside the (ExpCmdSN, MaxCmdSN) window, this causes the target to drop the received retry. The wording on this could do with further clarity. > > For a Target Write operation: > > 8- The target iSCSI sends the status to the initiator (and saves it in the > iSCSI allocated resources for the I/O), but does *not* deallocate the > resources associated with the I/O. Same comment as given for (4) under read operation. --------------F3EA194F1D357D215BA3587A Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------F3EA194F1D357D215BA3587A-- From owner-ips@ece.cmu.edu Fri Jan 26 19:19:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA25565 for ; Fri, 26 Jan 2001 19:19:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QMpT613679 for ips-outgoing; Fri, 26 Jan 2001 17:51:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QMpAZ13666 for ; Fri, 26 Jan 2001 17:51:10 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 77AF492E; Fri, 26 Jan 2001 14:51:09 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id OAA28324; Fri, 26 Jan 2001 14:50:57 -0800 (PST) Message-ID: <3A71FFE5.34FEC1B9@cup.hp.com> Date: Fri, 26 Jan 2001 14:53:26 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Charles Binford Cc: "Ips (E-mail)" Subject: Re: iSCSI : Command Ordering Proposal. References: Content-Type: multipart/mixed; boundary="------------D8293E2B7D132F66186806BD" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------D8293E2B7D132F66186806BD Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Charles Binford wrote: > Seems to me like adding a new CRN field for command ordering (if the > application so chooses) and re-defining the CmdRN field to only provide the > flow control satisfies the requirements without overloading the > functionality of a single field. > This is exactly my point. Strict ordering should only be enforced when driven by an application. (either through the use of Ordered Task Tags for task set ordering or the use of CRN for end-to-end ordering). The use of CmdSN to order all commands seems an over-kill when those applications that did need ordering would use the Ordered Task Tag or CRN, based on their need. Strict ordering is only required for the aborts that follow their commands. Also, it would be helpful to re-name the new 1-byte CmdRN in the new iSCSI PDU to CRN to avoid confusion with the old semantics of CmdRN. (which is now the CmdSN !!) Regards, Santosh --------------D8293E2B7D132F66186806BD Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------D8293E2B7D132F66186806BD-- From owner-ips@ece.cmu.edu Fri Jan 26 19:19:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA25576 for ; Fri, 26 Jan 2001 19:19:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QNFTR14473 for ips-outgoing; Fri, 26 Jan 2001 18:15:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gordan.pl.gadzoox.com (i248.gadzoox.com [216.52.31.248]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QNF9Z14445 for ; Fri, 26 Jan 2001 18:15:09 -0500 (EST) Received: by gordan.pl.gadzoox.com with Internet Mail Service (5.5.2650.21) id ; Fri, 26 Jan 2001 15:15:54 -0800 Message-ID: <312419998E3CD211A52900A0C991A47A3A21B9@gordan.pl.gadzoox.com> From: Gabriel Hecht To: "'Douglas Otis'" , ips@ece.cmu.edu Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Date: Fri, 26 Jan 2001 15:15:54 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, In Orlando, Ralph Weber suggested using time-stamp in FCIP and drop frames that exceed the R_A_TOV time so that the 20sec waiting period scenario will not happen. Framing can be used to prevent the other scenario. Framing will allow looking into particular FC streams that are not affected by a delivery problem that pauses other streams within a TCP channel. As discussed earlier, it may require a special TCP implementation. This can be done without using time stamps, although a time-stamp may help. Failover, load sharing, and alternate TCP channels are probably implementation specific. Time-stamp is not necessary, but if exist it may benefit these implementations. Gaby -----Original Message----- From: Douglas Otis [mailto:dotis@sanlight.net] Sent: Friday, January 26, 2001 11:23 AM To: Gabriel Hecht; ips@ece.cmu.edu Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports Gaby, > Currently, FCIP doesn't map TCP channels to FC streams. By > stream, I mean, > related FCP commands and data that may require 'in order delivery' by the > SCSI application. Mapping TCP channels to FC addresses will > guarantee that > a FC-stream is fully contained within a TCP channel. Should the application be using multiple processors with multiple FC connections, then what you see as isolated is perhaps not. An advantage of using multiple TCP connections is lost if all traffic to a device is restricted to a connection. At what point in time would the gateway change to a different connection if there was a problem. All this activity is time related and to sift through the mess, a time-stamp still seems essential. > FCIP leaves this > mapping to the Gateway implementation. For example, assigning a TCP > connection per remote D_ID address or group of address is relatively easy > for the H/W to implement. If we don't want to use time-stamp for > ordering, > the only implementation limit will be to not put a FC stream on more than > one TCP channel, to avoid out of order delivery. This conclusion assumes no relationship between other sources and no fail-over will be made and all traffic will be held until a missing frame is recovered. A TCP interruption may recover within 20 seconds, but with respect to FC however, this is forever. If there are multiple working connections, and one is briefly lost, how does one allow a fail-over? TCP may attempt delivery for a very long period. Is it safe to assume no timing relationship between FC sources? As a TCP interruption may occur for periods longer than the R_A_TOV, is there a violation of the FC fabric to then release frames held for say 20 seconds? Doug From owner-ips@ece.cmu.edu Fri Jan 26 19:19:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA25589 for ; Fri, 26 Jan 2001 19:19:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QNEbx14427 for ips-outgoing; Fri, 26 Jan 2001 18:14:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QNDrZ14405 for ; Fri, 26 Jan 2001 18:13:53 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id 68098408 for ; Fri, 26 Jan 2001 15:13:52 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id PAA14052; Fri, 26 Jan 2001 15:13:50 -0800 (PST) Message-ID: <3A7204C5.46DB9C54@hp.com> Date: Fri, 26 Jan 2001 15:14:13 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues References: <3A71C053.3BBD711A@agilent.com> <3A71FA9B.A197C500@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > Matt, > > The description of CmdSN window mangement in Section 1.2.2.1 reads : > > "- The target MUST silently ignore any command > outside this range or duplicates within the range not flagged with > the retry bit (the X bit in the opcode)." > > Does this imply : > - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. > - ignore all duplicate commands within the range that are not flagged with the > retry > ? > > If so, a command with the "retry" bit set and received outside the > (ExpCmdSN, MaxCmdSN) window causes the target to drop the > received retry. Is this a valid interpretation of the draft ? I think so, normally an initiator before issuing a "retry" does a Logout. In the Logout response it receives an ExpCmdSN. Form this value the initiator decides to retry the command not numbered if the original CmdSN is less than ExpCmdSN. It avoids the "retry" to be rejected. Regards, Pierre > > > Regards, > Santosh > > Matt Wakeley wrote: > > > Santosh, > > > > I guess I don't understand what it is that you think needs clarification. > > > > -Matt > > > > Santosh Rao wrote: > > > > > > Matt, > > > > > > (You may want to take another look at the thread.). Here's the details again : > > > > > > > 3) The policy of "discard and restart" is also subject some race > > > > conditions in the CmdSN sliding window. At the time the digest error was > > > > > > > > detected at the initiator, the ExpCmdSN may not yet have acknowledged > > > > that > > > > command. This causes the initiator to restart the command with the same > > > > Initiator Task Tag, CmdSN and "retry" bit set. > > > > > > > > However, by the time the command gets to the target, the target may have > > > > > > > > updated its ExpCmdSN window having sent a later PDU which updated the > > > > ExpCmdSN. This results in the target discarding the received CmdSN since > > > > > > > > at command restart you never really rely on the CmdSN. you will want > > > > to check > > > > the Initiator Task Tag and accept it in the above case. > > > > > > The draft states the following on this subject : > > > > > > Section 5.1 > > > ========= > > > - the initiator will reissue all outstanding commands with their > > > original Initiator Task Tag and their original CmdRN if they > > > are not acknowledged yet or a CmdRN of 0 (not-numbered) if they > > > were acknowledged; the retry (X) flag in the command PDU will > > > be set > > > > > > Section 1.2.2.1 > > > =========== > > > - The target MUST silently ignore any command > > > outside this range or duplicates within the range not flagged with > > > the retry bit (the X bit in the opcode). > > > > > > This, to me, means : > > > - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. > > > - ignore all duplicate commands within the range that are not flagged with the > > > retry > > > bit. << "duplicate" added this time >> > > > > > > Can you please clarify the intent of the text ? > > > > From owner-ips@ece.cmu.edu Fri Jan 26 19:19:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA25634 for ; Fri, 26 Jan 2001 19:19:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QMxVZ13928 for ips-outgoing; Fri, 26 Jan 2001 17:59:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QMxAZ13917 for ; Fri, 26 Jan 2001 17:59:10 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 76465707 for ; Fri, 26 Jan 2001 15:59:09 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 152F8104 for ; Fri, 26 Jan 2001 17:59:06 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id OAA05293 for ; Fri, 26 Jan 2001 14:59:04 -0800 (PST) Message-ID: <3A72019C.D5CC61BE@agilent.com> Date: Fri, 26 Jan 2001 15:00:44 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI: I/O (command) recovery References: <3A71C123.64AD97E5@agilent.com> <3A71F91E.C2B62BFE@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > > Matt, > > This is a good write-up and clearly outlines the behaviour. Just a couple of > comments below. > > Regards, > Santosh > > Matt Wakeley wrote: > > > 4- The target iSCSI sends the read data and status to the initiator, but does > > *not* deallocate the resources associated with the I/O. > > The draft is ambiguous about (4). It states that the numbering MUST be supported, > but the data and status recovery MAY be supported. This means initiators > & targets MUST use StatSN/ExpStatSN. However, the target is allowed to > de-allocate the resources associated with the I/O without waiting for ExpStatSN > ack. That's correct. There is nothing in the document that states that retry must be supported. There should probably be a negotiation key that a target uses to indicate if it supports retry or not. > > If some error occurs such that the initiator iSCSI does not receive all the > > data and status, then the initiator iSCSI performs whatever > > link/connection/session recovery to re-establish communication with the > > target. At that point, the initiator iSCSI layer re-issues the I/O with the > > retry bit set. > > > > - At this point, the target iSCSI layer (*not* the SCSI layer) recognizes the > > command is a duplicate (because the retry bit is set), and instead of sending > > Again, the wording for the above in the draft could do with some clarity (Section > 1.2.2.1). It is not clear from the draft whether CmdSNs that are outside the > (ExpCmdSN, MaxCmdSN) window are to be accepted if the retry bit is set. Well, in my mind, the retried command will either have a CmdSN of zero, or a (new/different) valid value within the current range (the next available CmdSN). I don't think there should ever be duplicates. > My interpretation of : > "- The target MUST silently ignore any command > outside this range or duplicates within the range not flagged with > the retry bit (the X bit in the opcode)." > > is : > - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. > - ignore all duplicate commands within the range that are not flagged with the > retry > > So, if a command with the "retry" bit is set is received outside the (ExpCmdSN, > MaxCmdSN) window, this causes the target to drop the received retry. The wording > on this could do with further clarity. > > > > > For a Target Write operation: > > > > 8- The target iSCSI sends the status to the initiator (and saves it in the > > iSCSI allocated resources for the I/O), but does *not* deallocate the > > resources associated with the I/O. > > Same comment as given for (4) under read operation. Same reply as for (4) under read.... -Matt From owner-ips@ece.cmu.edu Fri Jan 26 19:19:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA25646 for ; Fri, 26 Jan 2001 19:19:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QMkSF13500 for ips-outgoing; Fri, 26 Jan 2001 17:46:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stoneflynetworks.net ([207.38.33.135]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QMjTZ13477 for ; Fri, 26 Jan 2001 17:45:29 -0500 (EST) Received: from [24.8.168.143] (HELO MAYURI) by stoneflynetworks.net (CommuniGate Pro SMTP 3.3.2) with SMTP id 79671 for ips@ece.cmu.edu; Fri, 26 Jan 2001 14:45:26 -0800 From: "Satish M" To: Subject: List of current iSCSI drafts Date: Fri, 26 Jan 2001 14:47:21 -0800 Message-ID: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0005_01C087A6.E3693E20" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------=_NextPart_000_0005_01C087A6.E3693E20 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Hi, Please find a list of drafts for the iSCSI related area. Thanks - Satish ------=_NextPart_000_0005_01C087A6.E3693E20 Content-Type: application/octet-stream; name="Shortcut to draft-csapuntz-ips-iscsivi-00.txt.URL" Content-Disposition: attachment; filename="Shortcut to draft-csapuntz-ips-iscsivi-00.txt.URL" Content-Transfer-Encoding: quoted-printable [InternetShortcut]=0A= URL=3Dhttp://www.ietf.org/internet-drafts/draft-csapuntz-ips-iscsivi-00.t= xt=00 ------=_NextPart_000_0005_01C087A6.E3693E20 Content-Type: application/octet-stream; name="Shortcut to draft-haagens-ips-iscsireqs-00.txt.URL" Content-Disposition: attachment; filename="Shortcut to draft-haagens-ips-iscsireqs-00.txt.URL" Content-Transfer-Encoding: quoted-printable [InternetShortcut]=0A= URL=3Dhttp://www.ietf.org/internet-drafts/draft-haagens-ips-iscsireqs-00.= txt=00 ------=_NextPart_000_0005_01C087A6.E3693E20 Content-Type: application/octet-stream; name="Shortcut to draft-ietf-ips-fcovertcpip-01.txt.URL" Content-Disposition: attachment; filename="Shortcut to draft-ietf-ips-fcovertcpip-01.txt.URL" Content-Transfer-Encoding: quoted-printable [InternetShortcut]=0A= URL=3Dhttp://www.ietf.org/internet-drafts/draft-ietf-ips-fcovertcpip-01.t= xt=00 ------=_NextPart_000_0005_01C087A6.E3693E20 Content-Type: application/octet-stream; name="Shortcut to draft-ietf-ips-framework-00.txt.URL" Content-Disposition: attachment; filename="Shortcut to draft-ietf-ips-framework-00.txt.URL" Content-Transfer-Encoding: quoted-printable [InternetShortcut]=0A= URL=3Dhttp://www.ietf.org/internet-drafts/draft-ietf-ips-framework-00.txt= =00 ------=_NextPart_000_0005_01C087A6.E3693E20 Content-Type: application/octet-stream; name="Shortcut to draft-ietf-ips-iscsi-03.txt.URL" Content-Disposition: attachment; filename="Shortcut to draft-ietf-ips-iscsi-03.txt.URL" Content-Transfer-Encoding: quoted-printable [InternetShortcut]=0A= URL=3Dhttp://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-03.txt=00 ------=_NextPart_000_0005_01C087A6.E3693E20 Content-Type: application/octet-stream; name="Shortcut to draft-ietf-ips-iscsi-boot-01.txt.URL" Content-Disposition: attachment; filename="Shortcut to draft-ietf-ips-iscsi-boot-01.txt.URL" Content-Transfer-Encoding: quoted-printable [InternetShortcut]=0A= URL=3Dhttp://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-boot-01.tx= t=00 ------=_NextPart_000_0005_01C087A6.E3693E20 Content-Type: application/octet-stream; name="Shortcut to draft-ietf-ips-iscsi-disc-reqts-01.txt.URL" Content-Disposition: attachment; filename="Shortcut to draft-ietf-ips-iscsi-disc-reqts-01.txt.URL" Content-Transfer-Encoding: quoted-printable [InternetShortcut]=0A= URL=3Dhttp://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-disc-reqts= -01.txt=00 ------=_NextPart_000_0005_01C087A6.E3693E20 Content-Type: application/octet-stream; name="Shortcut to draft-ietf-ips-iscsi-reqmts-00.txt.URL" Content-Disposition: attachment; filename="Shortcut to draft-ietf-ips-iscsi-reqmts-00.txt.URL" Content-Transfer-Encoding: quoted-printable [InternetShortcut]=0A= URL=3Dhttp://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-reqmts-00.= txt=00 ------=_NextPart_000_0005_01C087A6.E3693E20 Content-Type: application/octet-stream; name="Shortcut to draft-ietf-ips-isns-00.txt.URL" Content-Disposition: attachment; filename="Shortcut to draft-ietf-ips-isns-00.txt.URL" Content-Transfer-Encoding: quoted-printable [InternetShortcut]=0A= URL=3Dhttp://www.ietf.org/internet-drafts/draft-ietf-ips-isns-00.txt=00 ------=_NextPart_000_0005_01C087A6.E3693E20 Content-Type: application/octet-stream; name="Shortcut to draft-klein-ips-virt-00.txt.URL" Content-Disposition: attachment; filename="Shortcut to draft-klein-ips-virt-00.txt.URL" Content-Transfer-Encoding: quoted-printable [InternetShortcut]=0A= URL=3Dhttp://www.ietf.org/internet-drafts/draft-klein-ips-virt-00.txt=00 ------=_NextPart_000_0005_01C087A6.E3693E20 Content-Type: application/octet-stream; name="Shortcut to draft-monia-ips-ifcp-01.txt.URL" Content-Disposition: attachment; filename="Shortcut to draft-monia-ips-ifcp-01.txt.URL" Content-Transfer-Encoding: quoted-printable [InternetShortcut]=0A= URL=3Dhttp://www.ietf.org/internet-drafts/draft-monia-ips-ifcp-01.txt=00 ------=_NextPart_000_0005_01C087A6.E3693E20 Content-Type: application/octet-stream; name="Shortcut to draft-monia-ips-ifcparch-00.txt.URL" Content-Disposition: attachment; filename="Shortcut to draft-monia-ips-ifcparch-00.txt.URL" Content-Transfer-Encoding: quoted-printable [InternetShortcut]=0A= URL=3Dhttp://www.ietf.org/internet-drafts/draft-monia-ips-ifcparch-00.txt= =00 ------=_NextPart_000_0005_01C087A6.E3693E20 Content-Type: application/octet-stream; name="Shortcut to draft-monia-ips-mfcp-00.txt.URL" Content-Disposition: attachment; filename="Shortcut to draft-monia-ips-mfcp-00.txt.URL" Content-Transfer-Encoding: quoted-printable [InternetShortcut]=0A= URL=3Dhttp://www.ietf.org/internet-drafts/draft-monia-ips-mfcp-00.txt=00 ------=_NextPart_000_0005_01C087A6.E3693E20 Content-Type: application/octet-stream; name="Shortcut to draft-tseng-ips-isns-01.txt.URL" Content-Disposition: attachment; filename="Shortcut to draft-tseng-ips-isns-01.txt.URL" Content-Transfer-Encoding: quoted-printable [InternetShortcut]=0A= URL=3Dhttp://www.ietf.org/internet-drafts/draft-tseng-ips-isns-01.txt=00 ------=_NextPart_000_0005_01C087A6.E3693E20-- From owner-ips@ece.cmu.edu Fri Jan 26 19:20:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA25660 for ; Fri, 26 Jan 2001 19:20:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QNEZu14424 for ips-outgoing; Fri, 26 Jan 2001 18:14:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QNECZ14412 for ; Fri, 26 Jan 2001 18:14:12 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 24565B53; Fri, 26 Jan 2001 15:14:12 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id PAA00109; Fri, 26 Jan 2001 15:14:07 -0800 (PST) Message-ID: <3A720553.C85A423A@cup.hp.com> Date: Fri, 26 Jan 2001 15:16:36 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Charles Monia Cc: IPS Reflector Subject: Re: iSCSI : Command Ordering Proposal. References: Content-Type: multipart/mixed; boundary="------------425ED8DEF8926858B77F3908" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------425ED8DEF8926858B77F3908 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Charles Monia wrote: > For example, if I > send commands A B |C| D E, where |C| is ordered, I expect that D and E won't > complete before |C|. If A B D and E arrive followed by |C|, there's no way > to obtain the correct behavior. Incidentally, similar constraints hold true > for "head of queue" tasks. > Charles, SAM-2 Section 7 states : "The rules for task set management apply only after the task is entered into the task set." This implies that the enforcement of the ordering of task tags in your above example is NOT based on when the initiator sent the commands, but on when these tasks enter the task set at the target. i.e. The initiator may send A, B, |C|, D, E, where |C| is ordered. The commands may arrive at the target in the order : A, E, |C|, B, D in a multi-connection session. The commands A and E are not subject to any form of ordering. (being simple tag commands and having no ordered tag commands ahead in the task set.) The commands |C|, B & D are subject to the ordering that B & D cannot be executed until C is first executed. Thereafter, B & D are subject to simple tag rules. I believe the confusion lies in whether ordered task tags imply end-to-end ordering or ordering within the received task set at the target. the former requires strict ordering on the link or re-ordering at the target, in the absence of strict link ordering. The latter does NOT require any link ordering or re-ordering at the target, since it only enforces ordering within the received task set. Comments ? Regards, Santosh --------------425ED8DEF8926858B77F3908 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------425ED8DEF8926858B77F3908-- From owner-ips@ece.cmu.edu Fri Jan 26 19:20:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA25672 for ; Fri, 26 Jan 2001 19:20:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QN6Up14163 for ips-outgoing; Fri, 26 Jan 2001 18:06:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QN6BZ14156 for ; Fri, 26 Jan 2001 18:06:11 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 241A5A22; Fri, 26 Jan 2001 15:06:09 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id PAA29408; Fri, 26 Jan 2001 15:06:04 -0800 (PST) Message-ID: <3A720371.F64A5749@cup.hp.com> Date: Fri, 26 Jan 2001 15:08:33 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: csapuntz@cisco.com Cc: IPS Reflector Subject: Re: iSCSI : Command Ordering Proposal. References: <3A70DDB2.68F573CC@cup.hp.com> Content-Type: multipart/mixed; boundary="------------9E2241FD98D2A46A8B38BDD8" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------9E2241FD98D2A46A8B38BDD8 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit csapuntz@cisco.com wrote: > Santosh, > > I believe you may be missing a subtlety to the SCSI task queueing model. > I may also be missing a subtlety of your argument and, if so, please excuse > this e-mail. > > Consider three commands sent down a FIFO SCSI transport: > > 1) Ordered Command > 2) Simple Command > 3) Simple command > > The ordered command is received first and then the two simple commands. > The ordered command is guaranteed to be executed before either SIMPLE command. > > Relevant text from SAM-2 rev 14, sec 7.5.1 > > "The [simple] task shall not enter the Enabled state until all older Head of Queue and older Ordered tasks have ended." > > Your proposed change means that CmdSN no longer is able express that > the Ordered task is "older" that the "simple" task. Thus, an initiator > might need to stop-and-wait before issuing #2 and #3. Costa, I need to add some more clarity on the discussion here. It was not my intent to suggest that CmdSN NOT be used for simple task tags. My suggestion was that the targets use CmdSN to enforce ordering only when an ordered task tag enters the task set and for the tasks that are older than the ordered task tag in the task set. I see some confusion regarding the semantics of ordered task tag. SAM-2 Section 7 states : "The rules for task set management (including those that govern the ordered task tags) apply only after the task is entered into the task set." This implies that ordering is enforced not from an end-to-end perspective, but from the perspective of the tasks entering the task set at the target. When end-to-end ordering is required, CRN should be used. Ordered Task Tags do not imply end-to-end ordering (unless someone pulled that above statement out of SAM-2). Given the above, enforcing ordering of interleaved ordered tags and simple tags within a task set becomes a 2-queue implementation along the following lines. Consider 2 queues, unordered_q & ordered_q at the target. A received task shall be placed at the tail of : a) unordered_q if there are no pending tasks in the ordered_q and it has a simple task tag. b) ordered_q if it has an ordered task tag or (has a simple task tag and there are pending tasks on the ordered_q). If all older ordered task tag commands in the ordered_q have been processed, the contiguous set of simple task tag commands that follow on the ordered_q shall be processed as per the rules of simple task tag mgmt. Any Head of Queue Tag or ACA Task Tag command shall be immediately processed by the SCSI ULP. This model ensures that a stream of "simple task tag" traffic only (which is the majority of the cases toay, if not all) will not encounter stalled commands on other TCP connections when one connection is faulty. It also is in line with the spirit of the SAM ordering of task tags. Anyone desiring end-to-end ordering uses CRN. Since I'm not proposing that CmdSN NOT be initialized for simple task tag commands (only that they not be used to enforce ordering), they can still be used for flow control that complements the current QUEUE FULL based SCSI flow control mechanisms. Regards, Santosh --------------9E2241FD98D2A46A8B38BDD8 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------9E2241FD98D2A46A8B38BDD8-- From owner-ips@ece.cmu.edu Fri Jan 26 19:20:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA25684 for ; Fri, 26 Jan 2001 19:20:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QMTUm12928 for ips-outgoing; Fri, 26 Jan 2001 17:29:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QMSSZ12898 for ; Fri, 26 Jan 2001 17:28:28 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 1E506DE0; Fri, 26 Jan 2001 14:28:28 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id OAA26193; Fri, 26 Jan 2001 14:28:23 -0800 (PST) Message-ID: <3A71FA9B.A197C500@cup.hp.com> Date: Fri, 26 Jan 2001 14:30:52 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Matt Wakeley Cc: IPS Reflector Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues References: <3A71C053.3BBD711A@agilent.com> Content-Type: multipart/mixed; boundary="------------C841AAA1F3B21DC66F82B901" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------C841AAA1F3B21DC66F82B901 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Matt, The description of CmdSN window mangement in Section 1.2.2.1 reads : "- The target MUST silently ignore any command outside this range or duplicates within the range not flagged with the retry bit (the X bit in the opcode)." Does this imply : - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. - ignore all duplicate commands within the range that are not flagged with the retry ? If so, a command with the "retry" bit set and received outside the (ExpCmdSN, MaxCmdSN) window causes the target to drop the received retry. Is this a valid interpretation of the draft ? Regards, Santosh Matt Wakeley wrote: > Santosh, > > I guess I don't understand what it is that you think needs clarification. > > -Matt > > Santosh Rao wrote: > > > > Matt, > > > > (You may want to take another look at the thread.). Here's the details again : > > > > > 3) The policy of "discard and restart" is also subject some race > > > conditions in the CmdSN sliding window. At the time the digest error was > > > > > > detected at the initiator, the ExpCmdSN may not yet have acknowledged > > > that > > > command. This causes the initiator to restart the command with the same > > > Initiator Task Tag, CmdSN and "retry" bit set. > > > > > > However, by the time the command gets to the target, the target may have > > > > > > updated its ExpCmdSN window having sent a later PDU which updated the > > > ExpCmdSN. This results in the target discarding the received CmdSN since > > > > > > at command restart you never really rely on the CmdSN. you will want > > > to check > > > the Initiator Task Tag and accept it in the above case. > > > > The draft states the following on this subject : > > > > Section 5.1 > > ========= > > - the initiator will reissue all outstanding commands with their > > original Initiator Task Tag and their original CmdRN if they > > are not acknowledged yet or a CmdRN of 0 (not-numbered) if they > > were acknowledged; the retry (X) flag in the command PDU will > > be set > > > > Section 1.2.2.1 > > =========== > > - The target MUST silently ignore any command > > outside this range or duplicates within the range not flagged with > > the retry bit (the X bit in the opcode). > > > > This, to me, means : > > - ignore all commands outside the (ExpCmdSN, MaxCmdSN) range. > > - ignore all duplicate commands within the range that are not flagged with the > > retry > > bit. << "duplicate" added this time >> > > > > Can you please clarify the intent of the text ? > > --------------C841AAA1F3B21DC66F82B901 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------C841AAA1F3B21DC66F82B901-- From owner-ips@ece.cmu.edu Fri Jan 26 20:34:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA26729 for ; Fri, 26 Jan 2001 20:34:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0QNjU715460 for ips-outgoing; Fri, 26 Jan 2001 18:45:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.brocade.com (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0QLVFZ10976 for ; Fri, 26 Jan 2001 16:31:15 -0500 (EST) Received: from thor.brocade.com (thor [192.168.126.45]) by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id NAA09017; Fri, 26 Jan 2001 13:31:08 -0800 (PST) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Fri, 26 Jan 2001 13:30:47 -0800 Message-ID: From: Robert Snively To: "'Jim Hafner'" , ips@ece.cmu.edu Subject: RE: iSCSI: INQUIRY page 0x83 identifier Date: Fri, 26 Jan 2001 13:30:43 -0800 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk > As for an iSCSI type or format for logical unit identifiers: > 1) SCSI already allows for an ascii formatted identifier (typically this > includes vendor, model, serial number). Since NDT has defined WWUIs for > DEVICES and these have a defined UTF-8 format, there is nothing to > *prevent* a vendor from using these strings as the starting point for LU > page 83 identifiers of the logical units within that device. > 2) But..., iSCSI and NDT should not attempt to make a rule under which > these are used. FC's intervention in this was more as a strong suggestion, > and is certainly not a requirement. Jim, I generally agree with the text of your comments, but there are a couple of details I would like to address. Any name not based on a registered entity and in a registered format is not guaranteed to be world-wide unique, and is therefore inappropriate. That is why numbers like OUIs and SCSI Vendor IDs in specified parts of the identifier are a requirement. EUI-64 and FC identifiers have that property. FC identifiers have the added benefit of having a regular extension capability for dynamic creation of logical units. As you point out, iSCSI need not use FC identifiers, but a bunch of people like them for their constant defined length and guaranteed world-wide unique identifiers. However, I feel that iSCSI should be a lot more hard nosed than SAM-2 and require that VPD page "83" contain a mandatory world-wide unique logical unit identifier in an appropriate invariant format. For compatibility with other SCSI drivers, it should be limited to 128 bits of length. This would not be an identifier set by the user (there are other SCSI identifiers that can be set by the user), but one that is invariant from the time of manufacture (for physical devices) or creation (for logical devices) of the logical unit. From owner-ips@ece.cmu.edu Fri Jan 26 22:22:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA28093 for ; Fri, 26 Jan 2001 22:22:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0R1dZ518308 for ips-outgoing; Fri, 26 Jan 2001 20:39:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0R1cYZ18290 for ; Fri, 26 Jan 2001 20:38:34 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0R2kh054675; Fri, 26 Jan 2001 18:46:44 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Santosh Rao" , "Charles Binford" Cc: "Ips \(E-mail\)" Subject: RE: iSCSI : Command Ordering Proposal. Date: Fri, 26 Jan 2001 17:34:52 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <3A71FFE5.34FEC1B9@cup.hp.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh, If there is a task attribute, the complexity is when you allow CmdSN = 0; Perhaps CmdSN = 0 should not be allowed or treated as just another in the sequence. One could view CmdSN as Client Sequence Number and StatSN as Server Sequence Number where these change to CSN and SSN. :) This gets away from the duplexed meaning of the number as defined for CRN. With the task attribute and to be sure of the sequence number, there is little gained from the use of CmdSN = 0 as an exception. If a retry flag is set, the CmdSN below the ExpCmdSN should be viewed as an exception much as Head of Queue should be an exception for MaxCmdSN. Both of these cases imply exceptional behavior by the initiator in response to a problem. Doug > Charles Binford wrote: > > > Seems to me like adding a new CRN field for command ordering (if the > > application so chooses) and re-defining the CmdRN field to only > provide the > > flow control satisfies the requirements without overloading the > > functionality of a single field. > > > > This is exactly my point. Strict ordering should only be enforced when > driven by an application. (either through the use of Ordered Task Tags for > task set ordering or the use of CRN for end-to-end ordering). > > The use of CmdSN to order all commands seems an over-kill when those > applications that did need ordering would use the Ordered Task Tag or > CRN, based on their need. > > Strict ordering is only required for the aborts that follow their > commands. > > Also, it would be helpful to re-name the new 1-byte CmdRN in the > new iSCSI PDU to CRN to avoid confusion with the old semantics of > CmdRN. (which is now the CmdSN !!) > > Regards, > Santosh > From owner-ips@ece.cmu.edu Fri Jan 26 22:23:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA28104 for ; Fri, 26 Jan 2001 22:23:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0R1saj18625 for ips-outgoing; Fri, 26 Jan 2001 20:54:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0R1rrZ18615 for ; Fri, 26 Jan 2001 20:53:53 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 0BC4AE2A; Fri, 26 Jan 2001 17:53:53 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA18941; Fri, 26 Jan 2001 17:53:44 -0800 (PST) Message-ID: <3A722ABD.AE763CA2@cup.hp.com> Date: Fri, 26 Jan 2001 17:56:13 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Douglas Otis Cc: "Ips (E-mail)" Subject: Re: iSCSI : Command Ordering Proposal. References: Content-Type: multipart/mixed; boundary="------------2B4B2FA5DD3B59100779E5AE" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------2B4B2FA5DD3B59100779E5AE Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Douglas Otis wrote: > Santosh, > > If there is a task attribute, the complexity is when you allow CmdSN = 0; > > Perhaps CmdSN = 0 should not be allowed or treated as just another in the > sequence. The (CmdSN==0) does have its uses. One example is the use of NOP-OUT for a "connection ping." > If a retry flag is set, the CmdSN below the > ExpCmdSN should be viewed as an exception much as Head of Queue should be an > exception for MaxCmdSN. Both of these cases imply exceptional behavior by > the initiator in response to a problem. Accepting a command outside the (ExpCmdSN, MaxCmdSN) window with the retry bit set then opens up windows for other types of stale and duplicate commands to be processed. Here's the deal (as I view it) : Commands are sent with the "retry" bit under 2 conditions : - Connection Failures - Digest Errors In both these cases, an explicit handshake is required regarding the (ExpCmdSN, MaxCmdSN) window, prior to starting to send "retry" commands. This ensures that genuine "retry" commands do not slip behind the (ExpCmdSN, MaxCmdSN) window. In the case of a connection failure, the Logout Response provides this handshake. For digest errors, there is no such hand-shake. Hence, a command that encountered a digest error and was retried could likely slip behind the (ExpCmdSN, MaxCmdSN) window and then be never processed. Regards, Santosh --------------2B4B2FA5DD3B59100779E5AE Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------2B4B2FA5DD3B59100779E5AE-- From owner-ips@ece.cmu.edu Fri Jan 26 22:23:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA28115 for ; Fri, 26 Jan 2001 22:23:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0R1NZV17976 for ips-outgoing; Fri, 26 Jan 2001 20:23:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0R1MaZ17953 for ; Fri, 26 Jan 2001 20:22:36 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 39B7CEA1 for ; Fri, 26 Jan 2001 17:22:36 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA17112 for ; Fri, 26 Jan 2001 17:22:31 -0800 (PST) Message-ID: <3A72236C.44881B25@cup.hp.com> Date: Fri, 26 Jan 2001 17:25:00 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Command Ordering Proposal. References: <3A720553.C85A423A@cup.hp.com> Content-Type: multipart/mixed; boundary="------------95041571AA328425F0C0C934" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------95041571AA328425F0C0C934 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit I'd also like to point out that all this discussion on ordered task tags is mostly of academic interest. How many implementations today use Ordered Task Tags ? I have'nt seen this in use in the implementations I've worked with. Does anybody know of how widely used this ordered task tag feature is ? If it is a totally un-used feature, we're adding all of this complexity to address something that is un-used. Regards, Santosh Santosh Rao wrote: > Charles Monia wrote: > > > For example, if I > > send commands A B |C| D E, where |C| is ordered, I expect that D and E won't > > complete before |C|. If A B D and E arrive followed by |C|, there's no way > > to obtain the correct behavior. Incidentally, similar constraints hold true > > for "head of queue" tasks. > > > > Charles, > > SAM-2 Section 7 states : > > "The rules for task set management apply only after the task is entered > into the task set." > > This implies that the enforcement of the ordering of task tags in your above > example is NOT based on when the initiator sent the commands, but on when > these tasks enter the task set at the target. > > i.e. The initiator may send A, B, |C|, D, E, where |C| is ordered. > > The commands may arrive at the target in the order : > A, E, |C|, B, D in a multi-connection session. > > The commands A and E are not subject to any form of ordering. > (being simple tag commands and having no ordered tag commands > ahead in the task set.) > > The commands |C|, B & D are subject to the ordering that B & D cannot > be executed until C is first executed. Thereafter, B & D are subject to > simple tag rules. > > I believe the confusion lies in whether ordered task tags imply end-to-end > ordering or ordering within the received task set at the target. the former > requires strict ordering on the link or re-ordering at the target, in the > absence > of strict link ordering. The latter does NOT require any link ordering or > re-ordering > at the target, since it only enforces ordering within the received task set. > > Comments ? > > Regards, > Santosh --------------95041571AA328425F0C0C934 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------95041571AA328425F0C0C934-- From owner-ips@ece.cmu.edu Fri Jan 26 22:23:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA28126 for ; Fri, 26 Jan 2001 22:23:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0R1eZu18347 for ips-outgoing; Fri, 26 Jan 2001 20:40:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0R1eXZ18343 for ; Fri, 26 Jan 2001 20:40:33 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 0B854962 for ; Fri, 26 Jan 2001 17:40:32 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA18167 for ; Fri, 26 Jan 2001 17:40:27 -0800 (PST) Message-ID: <3A72279F.9F6352F9@cup.hp.com> Date: Fri, 26 Jan 2001 17:42:56 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Command Ordering Proposal. References: <3A70DDB2.68F573CC@cup.hp.com> <3A720371.F64A5749@cup.hp.com> Content-Type: multipart/mixed; boundary="------------CBF0685633CA67A74D940611" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------CBF0685633CA67A74D940611 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Santosh Rao wrote: > Costa, > > I need to add some more clarity on the discussion here. It was not my intent > to suggest that CmdSN NOT be used for simple task tags. My suggestion > was that the targets use CmdSN to enforce ordering only when an ordered > task tag enters the task set and for the tasks that are older than the ordered > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The above should read "when an ordered task tag enters the task set and for simple task tags that arrive after an ordered task tag in the task set, until the completion of the ordered task tag." - Santosh --------------CBF0685633CA67A74D940611 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------CBF0685633CA67A74D940611-- From owner-ips@ece.cmu.edu Fri Jan 26 23:33:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA29972 for ; Fri, 26 Jan 2001 23:33:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0R3PeJ20526 for ips-outgoing; Fri, 26 Jan 2001 22:25:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0R3PUZ20514 for ; Fri, 26 Jan 2001 22:25:30 -0500 (EST) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id 8376C4B9; Fri, 26 Jan 2001 19:25:29 -0800 (PST) Received: from mk731913.cup.hp.com (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id TAA13313; Fri, 26 Jan 2001 19:28:00 -0800 (PST) Message-Id: <5.0.0.25.2.20010126185412.076d8e80@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.0 Date: Fri, 26 Jan 2001 18:59:12 -0800 To: julian_satran@il.ibm.com From: Michael Krause Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Cc: ips@ece.cmu.edu In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 07:40 PM 1/25/2001 +0200, julian_satran@il.ibm.com wrote: >1) The initiator task tag cannot be trusted when a header digest error >is seen. What does the phrase "provided it can recognize the initiator >task tag" mean ? >How can an initiator reliably claim that the initiator task tag is >trustworthy ? > > an initiator may choose to provide some redundancy in the tag itself > I'm aware of some techniques for inserting redundant information in tags which limits the potential error exposure when a multi-bit error occurs, however these are not fail-safe leading to potential incorrect operation - perhaps benign in many cases; perhaps not in others. As such, if a header digest error occurs, the PDU should be silently discarded and recovery should be left to the ULP. There is little to no value having two mechanisms to solve the same problem. Mike From owner-ips@ece.cmu.edu Fri Jan 26 23:34:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA29996 for ; Fri, 26 Jan 2001 23:34:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0R3PhO20532 for ips-outgoing; Fri, 26 Jan 2001 22:25:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0R3PXZ20518 for ; Fri, 26 Jan 2001 22:25:33 -0500 (EST) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id 6BCB4E72; Fri, 26 Jan 2001 19:25:32 -0800 (PST) Received: from mk731913.cup.hp.com (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id TAA13316; Fri, 26 Jan 2001 19:28:03 -0800 (PST) Message-Id: <5.0.0.25.2.20010126190554.076ca0c0@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.0 Date: Fri, 26 Jan 2001 19:07:41 -0800 To: Robert Snively From: Michael Krause Subject: RE: iSCSI: INQUIRY page 0x83 identifier Cc: ips@ece.cmu.edu In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 01:30 PM 1/26/2001 -0800, Robert Snively wrote: > > As for an iSCSI type or format for logical unit identifiers: > > 1) SCSI already allows for an ascii formatted identifier (typically this > > includes vendor, model, serial number). Since NDT has defined WWUIs for > > DEVICES and these have a defined UTF-8 format, there is nothing to > > *prevent* a vendor from using these strings as the starting point for LU > > page 83 identifiers of the logical units within that device. > > 2) But..., iSCSI and NDT should not attempt to make a rule under which > > these are used. FC's intervention in this was more as a strong suggestion, > > and is certainly not a requirement. > >Jim, > >I generally agree with the text of your comments, but there are >a couple of details I would like to address. > >Any name not based on a registered entity and in a registered format >is not guaranteed to be world-wide unique, and is therefore >inappropriate. That is why numbers like OUIs and SCSI Vendor IDs >in specified parts of the identifier are a requirement. EUI-64 >and FC identifiers have that property. FC identifiers have the added >benefit of having a regular extension capability for dynamic creation >of logical units. As you point out, iSCSI need not use FC >identifiers, but a bunch of people like them for their constant >defined length and guaranteed world-wide unique identifiers. > >However, I feel that iSCSI should be a lot more hard nosed than >SAM-2 and require that VPD page "83" contain a mandatory world-wide >unique logical unit identifier in an appropriate invariant format. >For compatibility with other SCSI drivers, it should be limited >to 128 bits of length. This would not be an identifier set by >the user (there are other SCSI identifiers that can be set by the >user), but one that is invariant from the time of manufacture (for >physical devices) or creation (for logical devices) of the logical unit. Why not just require EUI-64 which is a very large name space and is / will be used in many hardware solutions today / future? This would also fit in well with other I/O standards that use EUI-64 for identifying all I/O components / controllers. Having a 128-bit value seems like overkill and provides little benefit. Mike From owner-ips@ece.cmu.edu Fri Jan 26 23:35:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA00015 for ; Fri, 26 Jan 2001 23:35:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0R2hcK19622 for ips-outgoing; Fri, 26 Jan 2001 21:43:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0R2h9Z19614 for ; Fri, 26 Jan 2001 21:43:09 -0500 (EST) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 6F9151CAE for ; Fri, 26 Jan 2001 18:43:08 -0800 (PST) Received: from mk731913.cup.hp.com (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA12940 for ; Fri, 26 Jan 2001 18:45:39 -0800 (PST) Message-Id: <5.0.0.25.2.20010126165115.00a73a80@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.0 Date: Fri, 26 Jan 2001 18:31:31 -0800 To: ips@ece.cmu.edu From: Michael Krause Subject: iSCSI Data Integrity - Digests Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="=====================_104576853==_.ALT" Sender: owner-ips@ece.cmu.edu Precedence: bulk --=====================_104576853==_.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed The following describes three alternatives for iSCSI data integrity. Mike Krause HP Requirements: * Use existing / proven CRC algorithms and techniques to provide fast market enablement while avoiding a "reinvention of the wheel" exercise. * Provide strong end-to-end data integrity for both iSCSI PDU header and data payload * CRC is required for all implementations, i.e. strong end-to-end data integrity is not an option as customers will not adopt solutions without such guarantees. Alternative 1 (Highest preference): * An iSCSI PDU shall be restricted to a single TCP segment. Multiple iSCSI PDU may be present within the same TCP segment but none shall span multiple segments. * Each iSCSI PDU is protected by a trailing 32-bit CRC (Ethernet polynomial), i.e. a single CRC covers the entire iSCSI header and data. Assumption: * iSCSI PDU header is not modified during transmission. While there has been some discussion of a desire to provide such capabilities in the future, there are no current requirements to support requiring the specification to take this into account at this time. Should this become a requirement in the future, an intermediate endnode supporting iSCSI header modification would need to guarantee strong data integrity within its implementation using any of the well-known / deployed techniques. Benefits: * Strong end-to-end data integrity using a well-known, proven technology. * Low-cost, high-speed hardware implementations with readily available hardware cores can be created with minimal design complexity. * Only one CRC which can be implemented in software mitigating the performance impacts this iSCSI data integrity would impose. * Ability to accelerate software iSCSI implementations using a slightly modified NIC to perform the CRC calculation / verification for both inbound and outbound data streams. This modified NIC would only require minor understanding of the iSCSI header, i.e. to identify it and locate the CRC within the data stream. The CRC can be verified while coming in off the "wire" or inserted while being placed on the "wire". This technique is well understood since it is very similar to what is implemented by TCP checksum off-load implementations in use today. * Note: A NIC implementing this functionality could combine the verification of the TCP checksum into a "one-stop" verification operation and silently drop invalid packets or tag them as "bad" for ULP processing. * Solves the framing problem while eliminating the need for future support of "chunking" / RDMA technology. Each PDU header contains sufficient information required for direct data placement providing the same benefits attributed to chunking / RDMA. This will also allow simplified "bridge" solutions to be constructed, e.g. iSCSI-to-InfiniBand, iSCSI-to-SRP, etc. * Eliminates the need to maintain intermediate CRC results (both inbound and outbound) reducing implementation cost / complexity. * Eliminates bandwidth waste by reducing the number of bytes required to guarantee end-to-end data integrity while supporting multiple small PDU per segment (compaction) * Provides improved QoS arbitration control / management - if a PDU were allowed to span multiple segments, then an implementation would need to transmit segments back-to-back (or very close) to deliver strong end-to-end performance / transaction throughput. This may be implementation-specific but is still a tangible benefit for customers. * If an intermediate endnode performs re-segmentation, a PDU may be span multiple segments. This would be detected by a PDU CRC error providing a simple detection mechanism allowing implementations to recover either at the connection or session level. Constraints: * iSCSI implementations must be able to determine each connection's MSS and create iSCSI PDU that fit within the MSS. Such functionality is available in a variety of TCP implementations today and for hardware implementations. * For the send-side retransmission problem (i.e. how to delineate packets within a byte stream), a hardware implementation is straight-forward to support since it provide the PDU-segment correlation. * For a software implementation, the mbuf / mblk encompassing the iSCSI PDU would be marked to indicate whether the associated buffer should be sent within a separate segment or not. This is not common to any TCP implementations to date but is not difficult to implement. It should also be noted that this is an implementation not a TCP protocol issue. * If a layer 4 intermediate endnode glues together two TCP streams and is not iSCSI aware, the send-side retransmission is a problem. However, it is unclear whether this usage model must be transparently supported by iSCSI, i.e. such an intermediate endnode should be required to be iSCSI aware. This is not unreasonable as most layer 4 intermediate endnodes are providing some value-add service as a function of layer 4; why wouldn't such an endnode provide iSCSI value-add and thus be layer 5 aware. Alternative 2 (middle preference) * An iSCSI PDU shall be restricted to a single TCP segment. Multiple iSCSI PDU may be present within the same TCP segment but none shall span multiple segments. * Each iSCSI PDU is protected by two CRCs - one invariant and one variant. The invariant CRC (ICRC) is a 32-bit CRC covering the PDU data and invariant header fields (e.g. address). The variant CRC (VCRC) is either a 16 or 32-bit CRC that covers the entire PDU header, data, and invariant CRC. PDU layout would be: header, data, ICRC, VCRC. * Note: This scheme is conceptually the same as what is used in InfiniBand providing customers and the industry with a single paradigm and improved technology integration for both compute and storage endnodes. Benefits relative to Alternative 1: * Supports an intermediated endnode updating iSCSI header fields while supporting strong end-to-end data integrity of all invariant header fields and data. It is critical that all invariant header fields such as target address be protected at all times to avoid silent data corruption / illegal memory access since these fields are used to DMA the data into / from target memory. * Note: This problem does not exist in IP-based applications today since such implementations do not expose addresses across the wire but use look-up techniques as a function of the header. iSCSI implementations may choose to use a similar technique but at the cost of increased resources / complexity. * Limits the complexity / overhead required to support a separate header CRC - e.g. intermediate byte-stream CRC injection / verification. This simplifies the hardware implementation for full off-load solution as well as provides the ability to create simplified CRC acceleration as described in alternative 1 for software-based iSCSI implementations. * Use of two trailer CRCs does not impact overall end-to-end performance or endnode hardware resources. Implementations are gated more by the memory subsystems / cache coherency overheads than by external wire speed transmission, i.e .the packet will, in general, arrive before one could complete the first few cache line fetch operations. As such, given the single-segment operation, the data can be verified as it comes in off the wire and the memory operations initiated with minimal latency (most operations will be pipeline operations within a few cycles). * An intermediate endnode can provide data integrity checks while data is in-flight and stomp the CRC should it detect an error. This allows packet flow-through to be supported while providing fault isolation and a single for subsequent endnodes to drop invalid packets if they desire. Constraints: * Invariant header fields must be identified and included within the ICRC calculation adding minor complexity to the overall implementation. Alternative 3 (least preferred): * Allow a PDU to span multiple TCP segments. * Implement two CRC: a header CRC and a data CRC. * Do not allow intermediate endnodes to modify the iSCSI header. Constraints / Disadvantages: * Increased implementation complexity and overhead. The header CRC must occur following the header requiring injection / removal within the endnodes. This complexity is compounded for variable header protocols such as iSCSI and is why such a solution has been rejected in other high-speed technologies. * Requires intermediate CRC state to be maintained for both inbound and outbound requests. * Increased QoS scheduling complexity for strong end-to-end application throughput. * Does not solve the framing problem perhaps necessitates the need for a chunking / RDMA solution. This increases solution complexity and creates interoperability / support issues for customers, i.e. options are bad for developers; bad for customers. * Severely limits creating high performance iSCSI software-based implementations perhaps making them impractical as a general purpose implementation. This will limit the potential market for iSCSI solutions. * Note: If an intermediate endnode is allowed to modify the PDU header, then there exists a possibility of silent data corruption since the invariant portions no longer have end-to-end data integrity. This will be a major issue for customers in terms of their ability to adopt iSCSI across a variety of solution spaces, i.e. if there is the potential for silent data corruption, then customers will not deploy iSCSI and will turn to alternatives that provide stronger end-to-end data integrity. --=====================_104576853==_.ALT Content-Type: text/html; charset="us-ascii"

The following describes three alternatives for iSCSI data integrity.

Mike Krause
HP


Requirements:
  • Use existing / proven CRC algorithms and techniques to provide fast market enablement while avoiding a "reinvention of the wheel" exercise.
  • Provide strong end-to-end data integrity for both iSCSI PDU header and data payload
  • CRC is required for all implementations,  i.e. strong end-to-end data integrity is not an option as customers will not adopt solutions without such guarantees.
Alternative 1 (Highest preference):
  • An iSCSI PDU shall be restricted to a single TCP segment.  Multiple iSCSI PDU may be present within the same TCP segment but none shall span multiple segments.
  • Each iSCSI PDU is protected by a trailing 32-bit CRC (Ethernet polynomial), i.e. a single CRC covers the entire iSCSI header and data.
Assumption:
  • iSCSI PDU header is not modified during transmission.  While there has been some discussion of a desire to provide such capabilities in the future, there are no current requirements to support requiring the specification to take this into account at this time. Should this become a requirement in the future, an intermediate endnode supporting iSCSI header modification would need to guarantee strong data integrity within its implementation using any of the well-known / deployed techniques.
Benefits:
  • Strong end-to-end data integrity using a well-known, proven technology.
  • Low-cost, high-speed hardware implementations with readily available hardware cores can be created with minimal design complexity.
  • Only one CRC which can be implemented in software mitigating the performance impacts this iSCSI data integrity would impose.
  • Ability to accelerate software iSCSI implementations using a slightly modified NIC to perform the CRC calculation / verification for both inbound and outbound data streams.  This modified NIC would only require minor understanding of the iSCSI header, i.e. to identify it and locate the CRC within the data stream. The CRC can be verified while coming in off the "wire" or inserted while being placed on the "wire".   This technique is well understood since it is very similar to what is implemented by TCP checksum off-load implementations in use today.
    • Note: A NIC implementing this functionality could combine the verification of the TCP checksum into a "one-stop" verification operation and silently drop invalid packets or tag them as "bad" for ULP processing.
  • Solves the framing problem while eliminating the need for future support of "chunking" / RDMA technology.  Each PDU header contains sufficient information required for direct data placement providing the same benefits attributed to chunking / RDMA.  This will also allow simplified "bridge" solutions to be constructed, e.g. iSCSI-to-InfiniBand, iSCSI-to-SRP, etc.
  • Eliminates the need to maintain intermediate CRC results (both inbound and outbound) reducing implementation cost / complexity.
  • Eliminates bandwidth waste by reducing the number of bytes required to guarantee end-to-end data integrity while supporting multiple small PDU per segment (compaction)
  • Provides improved QoS arbitration control / management - if a PDU were allowed to span multiple segments, then an implementation would need to transmit segments back-to-back (or very close) to deliver strong end-to-end performance / transaction throughput.  This may be implementation-specific but is still a tangible benefit for customers.
  • If an intermediate endnode performs re-segmentation, a PDU may be span multiple segments.  This would be detected by a PDU CRC error providing a simple detection mechanism allowing implementations to recover either at the connection or session level. 
Constraints:
  • iSCSI implementations must be able to determine each connection's MSS and create iSCSI PDU that fit within the MSS.  Such functionality is available in a variety of TCP implementations today and for hardware implementations. 
    • For the send-side retransmission problem (i.e. how to delineate packets within a byte stream), a hardware implementation is straight-forward to support since it provide the PDU-segment correlation. 
    • For a software implementation, the mbuf / mblk encompassing the iSCSI PDU would be marked to indicate whether the associated buffer should be sent within a separate segment or not.  This is not common to any TCP implementations to date but is not difficult to implement.  It should also be noted that this is an implementation not a TCP protocol issue. 
    • If a layer 4 intermediate endnode glues together two TCP streams and is not iSCSI aware, the send-side retransmission is a problem.  However, it is unclear whether this usage model must be transparently supported by iSCSI, i.e. such an intermediate endnode should be required to be iSCSI aware.  This is not unreasonable as most layer 4 intermediate endnodes are providing some value-add service as a function of layer 4; why wouldn't such an endnode provide iSCSI value-add and thus be layer 5 aware.


Alternative 2 (middle preference)
  • An iSCSI PDU shall be restricted to a single TCP segment.  Multiple iSCSI PDU may be present within the same TCP segment but none shall span multiple segments.
  • Each iSCSI PDU is protected by two CRCs - one invariant and one variant.  The invariant CRC (ICRC) is a 32-bit CRC covering the PDU data and invariant header fields (e.g. address).  The variant CRC (VCRC) is either a 16 or 32-bit CRC that covers the entire PDU header, data, and invariant CRC.  PDU layout would be: header, data, ICRC, VCRC. 
    • Note: This scheme is conceptually the same as what is used in InfiniBand providing customers and the industry with a single paradigm and improved technology integration for both compute and storage endnodes.



Benefits relative to Alternative 1:
  • Supports an intermediated endnode updating iSCSI header fields while supporting strong end-to-end data integrity of all invariant header fields and data.  It is critical that all invariant header fields such as target address be protected at all times to avoid silent data corruption / illegal memory access since these fields are used to DMA the data into / from target memory. 
    • Note: This problem does not exist in IP-based applications today since such implementations do not expose addresses across the wire but use look-up techniques as a function of the header.  iSCSI implementations may choose to use a similar technique but at the cost of increased resources / complexity.
  • Limits the complexity / overhead required to support a separate header CRC - e.g. intermediate byte-stream CRC injection / verification.  This simplifies the hardware implementation for full off-load solution as well as provides the ability to create simplified CRC acceleration as described in alternative 1 for software-based iSCSI implementations.
  • Use of two trailer CRCs does not impact overall end-to-end performance or endnode hardware resources.  Implementations are gated more by the memory subsystems / cache coherency overheads than by external wire speed transmission, i.e .the packet will, in general, arrive before one could complete the first few cache line fetch operations.   As such, given the single-segment operation, the data can be verified as it comes in off the wire and the memory operations initiated with minimal latency (most operations will be pipeline operations within a few cycles).
  • An intermediate endnode can provide data integrity checks while data is in-flight and stomp the CRC should it detect an error.  This allows packet flow-through to be supported while providing fault isolation and a single for subsequent endnodes to drop invalid packets if they desire.
Constraints:
  • Invariant header fields must be identified and included within the ICRC calculation adding minor complexity to the overall implementation.



Alternative 3 (least preferred):
  • Allow a PDU to span multiple TCP segments.
  • Implement two CRC: a header CRC and a data CRC.
  • Do not allow intermediate endnodes to modify the iSCSI header.
Constraints / Disadvantages:
  • Increased implementation complexity and overhead.  The header CRC must occur following the header requiring injection / removal within the endnodes.  This complexity is compounded for variable header protocols such as iSCSI and is why such a solution has been rejected in other high-speed technologies.
  • Requires intermediate CRC state to be maintained for both inbound and outbound requests.
  • Increased QoS scheduling complexity for strong end-to-end application throughput.
  • Does not solve the framing problem perhaps necessitates the need for a chunking / RDMA solution.  This increases solution complexity and creates interoperability / support issues for customers, i.e. options are bad for developers; bad for customers.
  • Severely limits creating high performance iSCSI software-based implementations perhaps making them impractical as a general purpose implementation.  This will limit the potential market for iSCSI solutions.
  • Note: If an intermediate endnode is allowed to modify the PDU header, then there exists a possibility of silent data corruption since the invariant portions no longer have end-to-end data integrity.  This will be a major issue for customers in terms of their ability to adopt iSCSI across a variety of solution spaces, i.e. if there is the potential for silent data corruption, then customers will not deploy iSCSI and will turn to alternatives that provide stronger end-to-end data integrity.
--=====================_104576853==_.ALT-- From owner-ips@ece.cmu.edu Fri Jan 26 23:47:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA00131 for ; Fri, 26 Jan 2001 23:47:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0R3qf221120 for ips-outgoing; Fri, 26 Jan 2001 22:52:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0R3qCZ21107 for ; Fri, 26 Jan 2001 22:52:13 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Fri, 26 Jan 2001 19:52:06 -0800 Message-ID: From: Charles Monia To: Santosh Rao Cc: IPS Reflector Subject: RE: iSCSI : Command Ordering Proposal. Date: Fri, 26 Jan 2001 19:52:06 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Santosh: See comments below. > -----Original Message----- > From: Santosh Rao [mailto:santoshr@cup.hp.com] > Sent: Friday, January 26, 2001 3:17 PM > To: Charles Monia > Cc: IPS Reflector > Subject: Re: iSCSI : Command Ordering Proposal. > > > Charles Monia wrote: > > > For example, if I > > send commands A B |C| D E, where |C| is ordered, I expect > that D and E won't > > complete before |C|. If A B D and E arrive followed by > |C|, there's no way > > to obtain the correct behavior. Incidentally, similar > constraints hold true > > for "head of queue" tasks. > > > > Charles, > > SAM-2 Section 7 states : > > "The rules for task set management apply only after the task > is entered > into the task set." > > This implies that the enforcement of the ordering of task > tags in your above > example is NOT based on when the initiator sent the commands, > but on when > these tasks enter the task set at the target. > > i.e. The initiator may send A, B, |C|, D, E, where |C| is ordered. > > The commands may arrive at the target in the order : > A, E, |C|, B, D in a multi-connection session. > > The commands A and E are not subject to any form of ordering. > (being simple tag commands and having no ordered tag commands > ahead in the task set.) > > The commands |C|, B & D are subject to the ordering that B & D cannot > be executed until C is first executed. Thereafter, B & D are > subject to > simple tag rules. > > I believe the confusion lies in whether ordered task tags > imply end-to-end > ordering or ordering within the received task set at the > target. the former > requires strict ordering on the link or re-ordering at the > target, in the > absence > of strict link ordering. The latter does NOT require any link > ordering or > re-ordering > at the target, since it only enforces ordering within the > received task set. > > Comments ? Aside from the fact that the ordering semantics make no sense without this property, the initiator's real-world expectation is that end-to-end ordering is preserved. Also, as I mentioned earlier, without end-to-end ordering, there is a potential for other undesirable side effects due to the misordering of task management requests relative to commands in flight. Charles From owner-ips@ece.cmu.edu Sat Jan 27 00:19:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA00377 for ; Sat, 27 Jan 2001 00:19:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0R41fA21338 for ips-outgoing; Fri, 26 Jan 2001 23:01:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0R40jZ21316 for ; Fri, 26 Jan 2001 23:00:45 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Fri, 26 Jan 2001 20:00:39 -0800 Message-ID: From: Charles Monia To: Santosh Rao , IPS Reflector Subject: RE: iSCSI : Command Ordering Proposal. Date: Fri, 26 Jan 2001 20:00:38 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Santosh Rao [mailto:santoshr@cup.hp.com] > Sent: Friday, January 26, 2001 5:25 PM > To: IPS Reflector > Subject: Re: iSCSI : Command Ordering Proposal. > > > I'd also like to point out that all this discussion on > ordered task tags is > mostly of academic interest. How many implementations today > use Ordered Task Tags > ? I have'nt seen this in use in the implementations I've worked with. > > Does anybody know of how widely used this ordered task tag > feature is ? If it is > a totally un-used feature, we're adding all of this > complexity to address > something that is un-used. > The point isn't whether or not ordered task tags are used but whether or not the ordering property is required. From owner-ips@ece.cmu.edu Sat Jan 27 00:19:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA00389 for ; Sat, 27 Jan 2001 00:19:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0R4WfV22054 for ips-outgoing; Fri, 26 Jan 2001 23:32:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from bacchus-int.veritas.com (bacchus.veritas.com [204.177.156.37]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0R4VlZ22040 for ; Fri, 26 Jan 2001 23:31:47 -0500 (EST) Received: from megami.veritas.com (urd.veritas.com [192.203.47.101]) by bacchus-int.veritas.com (8.11.0/8.9.1) with SMTP id f0R4VkH06108; Fri, 26 Jan 2001 20:31:46 -0800 (PST) Received: from localhost (2635 bytes) by megami.veritas.com via sendmail with P:esmtp/R:smart_host/T:smtp (sender: ) id for ; Fri, 26 Jan 2001 20:31:46 -0800 (PST) (Smail-3.2.0.101 1997-Dec-17 #4 built 1999-Aug-24) Date: Fri, 26 Jan 2001 20:31:46 -0800 (PST) From: Patrick Stirling To: Michael Krause cc: Robert Snively , ips@ece.cmu.edu Subject: RE: iSCSI: INQUIRY page 0x83 identifier In-Reply-To: <5.0.0.25.2.20010126190554.076ca0c0@hpindlm.cup.hp.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk On Fri, 26 Jan 2001, Michael Krause wrote: > At 01:30 PM 1/26/2001 -0800, Robert Snively wrote: > > >However, I feel that iSCSI should be a lot more hard nosed than > >SAM-2 and require that VPD page "83" contain a mandatory world-wide > >unique logical unit identifier in an appropriate invariant format. > >For compatibility with other SCSI drivers, it should be limited > >to 128 bits of length. This would not be an identifier set by > >the user (there are other SCSI identifiers that can be set by the > >user), but one that is invariant from the time of manufacture (for > >physical devices) or creation (for logical devices) of the logical unit. > > Why not just require EUI-64 which is a very large name space and is / will > be used in many hardware solutions today / future? This would also fit in > well with other I/O standards that use EUI-64 for identifying all I/O > components / controllers. Having a 128-bit value seems like overkill and > provides little benefit. Well, I'd prefer to stick with the current scheme of allowing the device to specify its identifier within certain limits (table 181 in spc2r18). I'm working on a virtual storage server, which serves up virtual disks on the fibre-channel SAN (with iSCSI coming soon I expect). We set just this identifier in VPD page 0x83, using a type 3 identifier (FC-PH) - actually a Registered Extended IEEE Name, 128bits. It's convenient to use this length as it allows me to set the upper 64bits to our server's WWN (which is also used as the World-Wide Node Name shared by all target HBAs on the server), and the lower 64bits to a simple number identifying the virtual disk on this server. This makes it easy to set an identifier which is guaranteed unique world-wide, although it does have the drawback of making it hard to keep the id if you move the virtual disk's data to another server. With the formatting of the various identifiers, 64 bits doesn't go very far at all. patrick Patrick Stirling VERITAS Software Corporation pms@veritas.com From owner-ips@ece.cmu.edu Sat Jan 27 12:38:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA17079 for ; Sat, 27 Jan 2001 12:38:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0RFZ9P14007 for ips-outgoing; Sat, 27 Jan 2001 10:35:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraac.compuserve.com (ds-img-rel-3.compuserve.com [149.174.206.154]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0RFYFZ13991 for ; Sat, 27 Jan 2001 10:34:15 -0500 (EST) Received: (from mailgate@localhost) by spdmraac.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id KAA23599 for ips@ece.cmu.edu; Sat, 27 Jan 2001 10:34:09 -0500 (EST) Received: from compuserve.com (dal-tgn-tvn-vty20.as.wcom.net [206.175.228.20]) by spdmraac.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id KAA23509; Sat, 27 Jan 2001 10:34:01 -0500 (EST) Message-ID: <3A72EAFA.290DEB7E@compuserve.com> Date: Sat, 27 Jan 2001 09:36:27 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: "Ips (E-mail)" CC: Santosh Rao Subject: Re: iSCSI : Command Ordering Proposal. References: <3A722ABD.AE763CA2@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit One of the masters from whom I learned programming repeatedly advised, "Never under estimate the creativity of those who use your programs." In this case, there are two architected capabilities: 1) At the client-server level, there's Ordered Tags, and 2) At the transport level, there's CRN ordering. True, most applications will want to coordinate the usage of the two. In my opinion, it is less likely that applications will want to disable CRN for Simple tags since I perceive the benefits of transport ordering as having broader link reliability implications. I can more readily agree that use of an Ordered tag strongly suggests use of CRN. However, I once helped design an application where only the Ordered Tag guarantee was needed. Our cluster application sought only to obtain the following guarantee from the target: "When status is returned for this (ordered) command, all commands received prior to this (ordered) command from all initiators also have been completed." By design, our application took care of all the other ordering details and the our application had to do that because it was dealing with multiple initiators. Based on my experience, I'd ask you to ensure that the standards contain all the tools application clients need but to refrain from dictating how those tools are used. Thanks. Ralph... From owner-ips@ece.cmu.edu Sat Jan 27 14:11:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA17667 for ; Sat, 27 Jan 2001 14:11:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0RHOCx16326 for ips-outgoing; Sat, 27 Jan 2001 12:24:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d06lmsgate-3.uk.ibm.com (d06lmsgate-3.uk.ibm.com [195.212.29.3]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0RHO3Z16320 for ; Sat, 27 Jan 2001 12:24:03 -0500 (EST) Received: from d06relay02.portsmouth.uk.ibm.com (d06relay02.portsmouth.uk.ibm.com [9.166.84.148]) by d06lmsgate-3.uk.ibm.com (1.0.0) with ESMTP id RAA40818; Sat, 27 Jan 2001 17:15:21 GMT From: julian_satran@il.ibm.com Received: from d06mta03.portsmouth.uk.ibm.com (d06mta03_cs0 [9.180.35.1]) by d06relay02.portsmouth.uk.ibm.com (8.8.8m3/NCO v4.95) with SMTP id RAA14112; Sat, 27 Jan 2001 17:23:55 GMT Received: by d06mta03.portsmouth.uk.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 802569E1.005F91B3 ; Sat, 27 Jan 2001 17:23:52 +0000 X-Lotus-FromDomain: IBMIL@IBMDE@IBMGB To: mjacob@feral.com cc: Laxmiprasad Nandipati , ips@ece.cmu.edu Message-ID: <802569E1.005F9065.00@d06mta03.portsmouth.uk.ibm.com> Date: Sat, 27 Jan 2001 18:53:25 +0200 Subject: Re: Basic question. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Try the last one (it has all versions). Julo Matthew Jacob on 26/01/2001 20:03:14 Please respond to mjacob@feral.com To: Julian Satran/Haifa/IBM@IBMIL cc: Laxmiprasad Nandipati , ips@ece.cmu.edu Subject: Re: Basic question. On Fri, 26 Jan 2001 julian_satran@il.ibm.com wrote: > > > 3 sources: > > > http://www.ietf.org/internet-drafts/draft-ietf-ips-iSCSI-03.txt Bad URL. I can only fine version 1 online. Can you give a better link for -03? > > and other docs having "draft-ietf-ips" in their pathname. > > http://www.ece.cmu.edu/~ips > > http://www.haifa.il.ibm.com/satran/ips > > Julo > > From owner-ips@ece.cmu.edu Sat Jan 27 14:11:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA17678 for ; Sat, 27 Jan 2001 14:11:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0RHOGA16331 for ips-outgoing; Sat, 27 Jan 2001 12:24:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0RHNtZ16318 for ; Sat, 27 Jan 2001 12:23:55 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA25552 for ; Sat, 27 Jan 2001 18:23:49 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA42878 for ; Sat, 27 Jan 2001 18:23:48 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E1.005F8F23 ; Sat, 27 Jan 2001 18:23:45 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 27 Jan 2001 18:37:53 +0200 Subject: RE: new iSCSI PDU outline Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, CRC is not mandatory. If you are using a local net (real time) on a reliable media you probably don't need anything. In any case a simple XOR will add nothing to what TCP already has (checksum) in the way of protection. And given that most of the time you have to deal with SCSI commands and responses and data blocks a basic block that takes care of those header types (the 48 byte) is probably optimal. The extended CDB is for those cases in which SCSI uses a CDB longer than 16 bytes (16 bytes of CDB are still provided for in the BHS). Julo "Douglas Otis" on 26/01/2001 21:23:04 Please respond to "Douglas Otis" To: ips@ece.cmu.edu cc: Subject: RE: new iSCSI PDU outline Julian, Should there be software implementations that must deal with iSCSI on a real-time basis, splitting the headers as I described helps in allowing the transport to respond immediately in providing the transport related feed-back. The split header and simple XOR calculation was to make this possible with software implementations where these settings are made nano-seconds before being sent. To require a CRC to be done could introduce many micro-seconds into the operation and thus make the transport less current on its feed-back as this must be pipelined. You have a few other fields not defined and perhaps use those fields to expand the later content. I doubt there will be more than a handful of types but if you use 16 bits, there should never be a danger of running out of options. In addition, I would suggest retry flags be placed in this header for the same reasons of simplifying the transport. Once the SCSI structure is created, it becomes independent of the transport functions so that fiddling at the transport will not spoil this content. By keeping these values to but a few fields unrelated to the payload, a simple check should be all that is needed. Doug > Doug, > > You are right and we could do it but it will reduce the number of yet > unallocated types in WN (and T10 is adding more stuff to the SCSI!) and > will save us several bits in the command where we have plenty. I don't > think it is worth it. > > Regards, > Julo > > "Douglas Otis" on 26/01/2001 00:41:02 > > Please respond to "Douglas Otis" > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: new iSCSI PDU outline > > > > > Julian, > > Would you consider splitting headers. As was done with MTF, > there was just > a simple 32 bit simple XOR checksum for header fields. Here is an example > with a CDB appended. The WN could expand the appended type into > SCSI_CMND, > SCSI_RSP, SCSI_R2T, iSCSI_Specific, etc. > > |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| > +---------------+---------------+---------------+---------------+ > 0| WN | Reserved | AddCDB | > +---------------+---------------+---------------+---------------+ > 4| Length | > +---------------+---------------+---------------+---------------+ > 8| ->CmdSN or <-StatRN | > +---------------+---------------+---------------+---------------+ > 12| ->ExpStatSN or <-ExpCmdSN | > +---------------+---------------+---------------+---------------+ > 16| ->MaxStatRN? <-MaxCmdSN | > +---------------+---------------+---------------+---------------+ > 20| XOR Checksum | > +---------------+---------------+---------------+---------------+ > +---------------+---------------+---------------+---------------+ > 24| SCSI_CMND ... | > +- | > 32| | > +---------------+---------------+---------------+---------------+ > 36| CDB ... | > +---------------+---------------+---------------+---------------+ > x | (Data ...) | > +---------------+---------------+---------------+---------------+ > y | CRC | > +---------------+---------------+---------------+---------------+ > > Doug > > > > > > > From owner-ips@ece.cmu.edu Sat Jan 27 17:48:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA18588 for ; Sat, 27 Jan 2001 17:48:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0RKTKw20243 for ips-outgoing; Sat, 27 Jan 2001 15:29:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0RKT4Z20231 for ; Sat, 27 Jan 2001 15:29:04 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id VAA60100 for ; Sat, 27 Jan 2001 21:28:57 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id VAA283016 for ; Sat, 27 Jan 2001 21:28:57 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E1.007081A5 ; Sat, 27 Jan 2001 21:28:52 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 27 Jan 2001 22:24:31 +0200 Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk If the header is a data header we can hardly trust the ULP to recognize the error (he might be unaware of a missing packet). With data numbering this situation could have been discovered at "status time". The only thing we could do is restart all commands but this is equivalent to a connection restart for all practical purposes. Dropping data numbering might have some more "side-effects" like this. As the combination of values - tag, address, offset may stil let some implementations to assume that they have a correct task identifier I don't see a point in mandating a recovery behavior and the implementer may choose to: -retry/restart command -logout drop and rebuild connection login and restart/retry -abort all task sets (practically reset the target!) and report for all commands a "delivery system failure" (kick-in the ULP recovery) and if you suspect the link quality rebuild it; this later behavior means also that you have to stop delivering anything on any link to the target to avoid out of order execution until you have finished the cleanup - pretty drastic With data numbering recovery could have stayed within the confines of a command even if a header was bad. Perhaps we should leave the DataSN only as a sequencer so that at status-time the initiator should be able to find if a data packet was dropped (no ExpDataSN on a NOP). Regards, Julo Michael Krause on 27/01/2001 04:59:12 Please respond to Michael Krause To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues At 07:40 PM 1/25/2001 +0200, julian_satran@il.ibm.com wrote: >1) The initiator task tag cannot be trusted when a header digest error >is seen. What does the phrase "provided it can recognize the initiator >task tag" mean ? >How can an initiator reliably claim that the initiator task tag is >trustworthy ? > > an initiator may choose to provide some redundancy in the tag itself > I'm aware of some techniques for inserting redundant information in tags which limits the potential error exposure when a multi-bit error occurs, however these are not fail-safe leading to potential incorrect operation - perhaps benign in many cases; perhaps not in others. As such, if a header digest error occurs, the PDU should be silently discarded and recovery should be left to the ULP. There is little to no value having two mechanisms to solve the same problem. Mike From owner-ips@ece.cmu.edu Sat Jan 27 18:51:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA18806 for ; Sat, 27 Jan 2001 18:51:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0RMKPk22624 for ips-outgoing; Sat, 27 Jan 2001 17:20:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hotmail.com (f137.law11.hotmail.com [64.4.17.137]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0RMJVZ22590 for ; Sat, 27 Jan 2001 17:19:31 -0500 (EST) Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Sat, 27 Jan 2001 14:19:25 -0800 Received: from 24.218.3.65 by lw11fd.law11.hotmail.msn.com with HTTP; Sat, 27 Jan 2001 22:19:25 GMT X-Originating-IP: [24.218.3.65] From: "Eddy Quicksall" To: ips@ece.cmu.edu Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Date: Sat, 27 Jan 2001 17:19:25 -0500 Mime-Version: 1.0 Content-Type: text/plain; format=flowed Message-ID: X-OriginalArrivalTime: 27 Jan 2001 22:19:25.0495 (UTC) FILETIME=[34EC9C70:01C088AF] Sender: owner-ips@ece.cmu.edu Precedence: bulk Actually, its not that bad. Parallel SCSI logic already considers a similar case. That is the case where one of the tagged commands never completes. What happens is that one of the upper layers (sometimes the application and sometimes the operating system) times the I/O. The I/O associated with this tag will timeout and the layer that is responsible takes corrective action. With a disk it is very simple ... just abort the I/O and re-issue it. With something like a tape, the application has to determine if the tape needs to be backspaced or not and then re-issues the I/O. In parallel SCSI, the driver or one of the upper layers will usually issue a Target Reset to get rid of all the tags (because it is assumed that the device has malfunctioned). This is not considered drastic in many systems and is very natural when testing RAID systems. So ... for us, all we need to do is drop the packet. In the iSCSI spec, we are basically saying you can rely on corrupted data and I just don't think that is a good idea. Eddy ----Original Message Follows---- From: julian_satran@il.ibm.com To: ips@ece.cmu.edu Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Date: Sat, 27 Jan 2001 22:24:31 +0200 If the header is a data header we can hardly trust the ULP to recognize the error (he might be unaware of a missing packet). With data numbering this situation could have been discovered at "status time". The only thing we could do is restart all commands but this is equivalent to a connection restart for all practical purposes. Dropping data numbering might have some more "side-effects" like this. As the combination of values - tag, address, offset may stil let some implementations to assume that they have a correct task identifier I don't see a point in mandating a recovery behavior and the implementer may choose to: -retry/restart command -logout drop and rebuild connection login and restart/retry -abort all task sets (practically reset the target!) and report for all commands a "delivery system failure" (kick-in the ULP recovery) and if you suspect the link quality rebuild it; this later behavior means also that you have to stop delivering anything on any link to the target to avoid out of order execution until you have finished the cleanup - pretty drastic With data numbering recovery could have stayed within the confines of a command even if a header was bad. Perhaps we should leave the DataSN only as a sequencer so that at status-time the initiator should be able to find if a data packet was dropped (no ExpDataSN on a NOP). Regards, Julo Michael Krause on 27/01/2001 04:59:12 Please respond to Michael Krause To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues At 07:40 PM 1/25/2001 +0200, julian_satran@il.ibm.com wrote: >1) The initiator task tag cannot be trusted when a header digest error >is seen. What does the phrase "provided it can recognize the initiator >task tag" mean ? >How can an initiator reliably claim that the initiator task tag is >trustworthy ? > > an initiator may choose to provide some redundancy in the tag itself > I'm aware of some techniques for inserting redundant information in tags which limits the potential error exposure when a multi-bit error occurs, however these are not fail-safe leading to potential incorrect operation - perhaps benign in many cases; perhaps not in others. As such, if a header digest error occurs, the PDU should be silently discarded and recovery should be left to the ULP. There is little to no value having two mechanisms to solve the same problem. Mike _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com From owner-ips@ece.cmu.edu Sat Jan 27 18:51:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA18817 for ; Sat, 27 Jan 2001 18:51:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0RM7OD22315 for ips-outgoing; Sat, 27 Jan 2001 17:07:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0RM7EZ22309 for ; Sat, 27 Jan 2001 17:07:14 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 6AD331BF0; Sat, 27 Jan 2001 14:07:13 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id OAA13704; Sat, 27 Jan 2001 14:07:08 -0800 (PST) From: Santosh Rao Message-Id: <200101272207.OAA13704@hpcuhe.cup.hp.com> Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues To: julian_satran@il.ibm.com Date: Sat, 27 Jan 2001 14:07:08 -0800 (PST) Cc: ips@ece.cmu.edu In-Reply-To: from "julian_satran@il.ibm.com" at Jan 27, 2001 10:24:31 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, The missing Data PDU could be detected if the initiator were to perform a count check operation upon receiving SCSI Response PDU, along the lines of : no. of bytes xfer'ed = (Expected Data Xfer Length) - (Basic Residual Count) where, Expected Data Xfer Length -> as specified in SCSI Command PDU Basic Residual Count -> as specified in SCSI Response PDU However, this is currently not possible due to overlapped data transfers being allowed by iSCSI. If iSCSI were to dis-allow overlapping data xfer's and initiators used a count check [as is done in FC], this would also address the problem. Regards, Santosh > > > > If the header is a data header we can hardly trust the ULP to recognize the > error (he might be unaware > of a missing packet). With data numbering this situation could have been > discovered at "status time". > The only thing we could do is restart all commands but this is equivalent > to a connection restart for all practical purposes. Dropping data > numbering might have some more "side-effects" like this. > As the combination of values - tag, address, offset may stil let some > implementations to assume that they have > a correct task identifier I don't see a point in mandating a recovery > behavior and the implementer may choose to: > > -retry/restart command > -logout drop and rebuild connection login and restart/retry > -abort all task sets (practically reset the target!) and report for all > commands a "delivery system failure" (kick-in the ULP recovery) and if you > suspect the link quality rebuild it; this later behavior means also that > you have to stop delivering anything on any link to the target to avoid > out of order execution until you have finished the cleanup - pretty drastic > > With data numbering recovery could have stayed within the confines of a > command even if a header was bad. > Perhaps we should leave the DataSN only as a sequencer so that at > status-time the initiator should be able to find if a data packet was > dropped (no ExpDataSN on a NOP). > > Regards, > Julo > > > > > Michael Krause on 27/01/2001 04:59:12 > > Please respond to Michael Krause > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues > > > > > At 07:40 PM 1/25/2001 +0200, julian_satran@il.ibm.com wrote: > > > >1) The initiator task tag cannot be trusted when a header digest error > >is seen. What does the phrase "provided it can recognize the initiator > >task tag" mean ? > >How can an initiator reliably claim that the initiator task tag is > >trustworthy ? > > > > an initiator may choose to provide some redundancy in the tag itself > > > > I'm aware of some techniques for inserting redundant information in tags > which limits the potential error exposure when a multi-bit error occurs, > however these are not fail-safe leading to potential incorrect operation - > perhaps benign in many cases; perhaps not in others. As such, if a header > digest error occurs, the PDU should be silently discarded and recovery > should be left to the ULP. There is little to no value having two > mechanisms to solve the same problem. > > Mike > > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Sat Jan 27 23:55:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA22125 for ; Sat, 27 Jan 2001 23:55:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0S2KYO27505 for ips-outgoing; Sat, 27 Jan 2001 21:20:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0S2JkZ27467 for ; Sat, 27 Jan 2001 21:19:46 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 3503215C2 for ; Sat, 27 Jan 2001 18:19:44 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA24302 for ; Sat, 27 Jan 2001 18:19:40 -0800 (PST) Message-ID: <3A738253.AA6C1C32@cup.hp.com> Date: Sat, 27 Jan 2001 18:22:12 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Bridging missing CmdSNs and Abort I/O Error recovery References: Content-Type: multipart/mixed; boundary="------------44DDC4FEE2DAD04568DB5195" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------44DDC4FEE2DAD04568DB5195 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, 2 issues are worth addressing in the I/O Error recovery descriptions : 1) How I/O error recovery is performed on a SCSI ULP timeout detected by the initiator. Section 9 of the FC-PLDA document is a good example of the type of error recovery description that would be helpful. I'm assuming that iSCSI expects initiators to use the Abort Task or a higher level error recovery mechanism on a ULP timeout. [unlike FC which defines a seperate ABTS-LS protocol for I/O Error Recovery]. 2) The technique[s] that should be used to bridge missing CmdSNs when an initiator chooses not to "retry" a command on a connection failure or a digest error, in order to prevent other CmdSNs ahead in the sequence from stalling, awaiting receipt of the missing CmdSN. Regards, Santosh julian_satran@il.ibm.com wrote: > Santosh, > > You had a possible answer from Matt. However I agree that we might want to > address this in text although > a solution similar to that suggested by Matt should be by now obvious to > every implementer - the target should leave a placeholder in the input > queue until the command after gets delivered. > > Julo > > Santosh Rao on 25/01/2001 21:38:04 > > Please respond to Santosh Rao > > To: IPS Reflector > cc: > Subject: iSCSI : Bridging missing CmdSNs and Abort I/O Error recovery > > Julian & All, > > The draft is currently lacking a section that addresses abort I/O error > recovery. Specifically, how is CmdSN bridging issues to be handled in > the case where an initiator chooses not to retry an I/O [that failed on > a connection failure that affects the delivery of the command to the > target or a digest error at the target] because its ULP timer may have > expired. > > In such cases, the initiator can send an Abort Task to inform the target > that the I.T.T is being aborted and its corresponding CmdSN can be > bridged, instead of having the target stall infinitely in its attempt to > enforce ordering and await the missing CmdSN [which is'nt going to > arrive, because the initiator did not retry the command]. > > Regards, > Santosh > > - santoshr.vcf --------------44DDC4FEE2DAD04568DB5195 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------44DDC4FEE2DAD04568DB5195-- From owner-ips@ece.cmu.edu Sun Jan 28 09:48:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id JAA08718 for ; Sun, 28 Jan 2001 09:48:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0SD2w019318 for ips-outgoing; Sun, 28 Jan 2001 08:02:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0SD1uZ19301 for ; Sun, 28 Jan 2001 08:01:56 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id OAA98088 for ; Sun, 28 Jan 2001 14:01:52 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id OAA247698 for ; Sun, 28 Jan 2001 14:01:52 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E2.004791CF ; Sun, 28 Jan 2001 14:01:43 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 28 Jan 2001 08:07:54 +0200 Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, The trouble with forbidding a certain behavior is that you have to enforce it (i.e., check and signal errors for units that do not behave). Besides - the whole philosophy of the SCSI set of protocols is that the target is the master and the initiator should let the target decide how to fulfill the command. That is why we chose not to impose restrictions above those imposed by SCSI. The whole set of issues is also raised only because we provide also for storage proxies - otherwise a stronger checksum at TCP level and recovery at TCP level would have done what we wanted and recovery of the type we are dealing now with would have been done at TCP level. I am confident that we can reinstate DataSN a simple mean to sequence (not ack) data packets and considerably simplify recovery. And do not forget that raising the error up to ULP with a service response will make the recovery far more expensive (as Prasenjit has already stated) - far more than current wedge drivers do as these rarely consider commands in flight and the need to keep order in a target that is not yet aware that something went wrong. Julo Santosh Rao on 28/01/2001 00:07:08 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Julian, The missing Data PDU could be detected if the initiator were to perform a count check operation upon receiving SCSI Response PDU, along the lines of : no. of bytes xfer'ed = (Expected Data Xfer Length) - (Basic Residual Count) where, Expected Data Xfer Length -> as specified in SCSI Command PDU Basic Residual Count -> as specified in SCSI Response PDU However, this is currently not possible due to overlapped data transfers being allowed by iSCSI. If iSCSI were to dis-allow overlapping data xfer's and initiators used a count check [as is done in FC], this would also address the problem. Regards, Santosh > > > > If the header is a data header we can hardly trust the ULP to recognize the > error (he might be unaware > of a missing packet). With data numbering this situation could have been > discovered at "status time". > The only thing we could do is restart all commands but this is equivalent > to a connection restart for all practical purposes. Dropping data > numbering might have some more "side-effects" like this. > As the combination of values - tag, address, offset may stil let some > implementations to assume that they have > a correct task identifier I don't see a point in mandating a recovery > behavior and the implementer may choose to: > > -retry/restart command > -logout drop and rebuild connection login and restart/retry > -abort all task sets (practically reset the target!) and report for all > commands a "delivery system failure" (kick-in the ULP recovery) and if you > suspect the link quality rebuild it; this later behavior means also that > you have to stop delivering anything on any link to the target to avoid > out of order execution until you have finished the cleanup - pretty drastic > > With data numbering recovery could have stayed within the confines of a > command even if a header was bad. > Perhaps we should leave the DataSN only as a sequencer so that at > status-time the initiator should be able to find if a data packet was > dropped (no ExpDataSN on a NOP). > > Regards, > Julo > > > > > Michael Krause on 27/01/2001 04:59:12 > > Please respond to Michael Krause > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues > > > > > At 07:40 PM 1/25/2001 +0200, julian_satran@il.ibm.com wrote: > > > >1) The initiator task tag cannot be trusted when a header digest error > >is seen. What does the phrase "provided it can recognize the initiator > >task tag" mean ? > >How can an initiator reliably claim that the initiator task tag is > >trustworthy ? > > > > an initiator may choose to provide some redundancy in the tag itself > > > > I'm aware of some techniques for inserting redundant information in tags > which limits the potential error exposure when a multi-bit error occurs, > however these are not fail-safe leading to potential incorrect operation - > perhaps benign in many cases; perhaps not in others. As such, if a header > digest error occurs, the PDU should be silently discarded and recovery > should be left to the ULP. There is little to no value having two > mechanisms to solve the same problem. > > Mike > > > > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Sun Jan 28 17:58:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA10734 for ; Sun, 28 Jan 2001 17:58:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0SK7K327653 for ips-outgoing; Sun, 28 Jan 2001 15:07:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pluto.he.net (pluto.he.net [216.218.167.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0SK6dZ27636 for ; Sun, 28 Jan 2001 15:06:44 -0500 (EST) Received: from vaio (ACB46FD9.ipt.aol.com [172.180.111.217]) by pluto.he.net (8.8.6/8.8.2) with SMTP id MAA17690 for ; Sun, 28 Jan 2001 12:06:35 -0800 From: "Sriram Rupanagunta" To: Subject: New proposal for FCIP ? Date: Sun, 28 Jan 2001 14:04:37 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <312419998E3CD211A52900A0C991A47A3A21B9@gordan.pl.gadzoox.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I understand that there was a new proposal at the interim meeting at Orlando, adding timestamps for ordered delivery in FCIP. Could someone post a pointer to this proposal (if it exists as a ID) ? If not, could someone post the minutes of this discussion ? I apologize if this has been posted already - I could not find this on the archives either. Thanks, -Sriram From owner-ips@ece.cmu.edu Mon Jan 29 12:11:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA06040 for ; Mon, 29 Jan 2001 12:11:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TEaoJ28786 for ips-outgoing; Mon, 29 Jan 2001 09:36:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.brocade.com (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TCbhZ25878 for ; Mon, 29 Jan 2001 07:37:43 -0500 (EST) Received: from thor.brocade.com (thor [192.168.126.45]) by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id EAA03400 for ; Mon, 29 Jan 2001 04:34:48 -0800 (PST) Received: by thor.brocadecomm.com with Internet Mail Service (5.5.2650.21) id ; Mon, 29 Jan 2001 04:34:44 -0800 Message-ID: From: Ralph Weber To: "'ips@ece.cmu.edu '" Cc: Robert Snively Subject: RE: New proposal for FCIP ? Date: Mon, 29 Jan 2001 04:34:42 -0800 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk } I understand that there was a new proposal at the interim meeting } at Orlando, adding timestamps for ordered delivery in FCIP. } } Could someone post a pointer to this proposal (if it exists as a } ID) ? If not, could someone post the minutes of this discussion ? } } I apologize if this has been posted already - I could not find } this on the archives either. } } Thanks, } } -Sriram Regrettably, the proposal is still under development, promised for delivery by Valentines Day. It was discussed in overview at the Orlando meeting. The addition of time stamps is a part of the proposal. However, they have nothing to do with ordered delivery because ordered delivery is guaranteed by TCP/IP upon which FCIP is layered. The time stamps are to assist in the handling of R_A_TOV and E_D_TOV, two Fibre Channel timeout values. I can't help you with the minutes. Ralph Weber Representing Brocade Communications From owner-ips@ece.cmu.edu Mon Jan 29 12:18:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA06169 for ; Mon, 29 Jan 2001 12:18:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TEaqp28789 for ips-outgoing; Mon, 29 Jan 2001 09:36:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TBiDZ24930 for ; Mon, 29 Jan 2001 06:44:13 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA29115; Mon, 29 Jan 2001 06:44:12 -0500 (EST) Message-Id: <200101291144.GAA29115@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-iscsi-disc-reqts-00.txt Date: Mon, 29 Jan 2001 06:44:12 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iSCSI Naming and Discovery Requirements Author(s) : M. Bakke et al. Filename : draft-ietf-ips-iscsi-disc-reqts-00.txt Pages : 18 Date : 26-Jan-01 This document describes the iSCSI [7] naming and discovery requirements. The requirements presented in this document have been agreed to by the members of the iSCSI naming and discovery team. This document complements the iSCSI IETF draft. Flexibility is the key guiding principle behind this requirements document. That is, an effort has been made to satisfy the needs of both small isolated environments, as well as large environments requiring secure/scalable solutions. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-disc-reqts-00.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-disc-reqts-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-disc-reqts-00.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20010126081949.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-iscsi-disc-reqts-00.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-iscsi-disc-reqts-00.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20010126081949.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Mon Jan 29 13:59:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA07476 for ; Mon, 29 Jan 2001 13:59:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TH9vH04728 for ips-outgoing; Mon, 29 Jan 2001 12:09:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TH8xZ04696 for ; Mon, 29 Jan 2001 12:08:59 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id MAA63668; Mon, 29 Jan 2001 12:02:01 -0500 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id KAA152846; Mon, 29 Jan 2001 10:08:46 -0700 Importance: Normal Subject: RE: iSCSI: INQUIRY page 0x83 identifier To: ips@ece.cmu.edu, Robert Snively From: "Jim Hafner" Date: Mon, 29 Jan 2001 09:08:44 -0800 Message-ID: X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.6 |December 14, 2000) at 01/29/2001 09:08:45 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, I strongly agree with the requirement that these identifiers be WWUnique. But I also agree with Charles that whatever identifier that is presented should be the same regardless of the interconnect, so binding them to FC or to some defined iSCSI mechanism seems inappropriate. So, I think I'm agreeing the Rob Elliot. Namely, specify a 128 format rooted in EUI-64 for the device. Back away from anything FC-specific in the format of the EUI-64. A vendor of a storage controller will probably have an EUI-64 for the device, though that might not be FC-based. Use this as the basis for some vendor-specific algorithm to generate the LU identifier. Leave it to the vendor to define an algorithm that will provide the uniqueness. Also, I'd like to keep iSCSI from specifying any requirements on logical units (that's not the right space). SPC-2/3 can spec a general format and the requirement for uniqueness. Then leave it to the vendors to implement correctly. Does that sound reasonable? Jim Hafner Robert Snively on 01-26-2001 01:30:43 PM To: Jim Hafner/Almaden/IBM@IBMUS, ips@ece.cmu.edu cc: Subject: RE: iSCSI: INQUIRY page 0x83 identifier > As for an iSCSI type or format for logical unit identifiers: > 1) SCSI already allows for an ascii formatted identifier (typically this > includes vendor, model, serial number). Since NDT has defined WWUIs for > DEVICES and these have a defined UTF-8 format, there is nothing to > *prevent* a vendor from using these strings as the starting point for LU > page 83 identifiers of the logical units within that device. > 2) But..., iSCSI and NDT should not attempt to make a rule under which > these are used. FC's intervention in this was more as a strong suggestion, > and is certainly not a requirement. Jim, I generally agree with the text of your comments, but there are a couple of details I would like to address. Any name not based on a registered entity and in a registered format is not guaranteed to be world-wide unique, and is therefore inappropriate. That is why numbers like OUIs and SCSI Vendor IDs in specified parts of the identifier are a requirement. EUI-64 and FC identifiers have that property. FC identifiers have the added benefit of having a regular extension capability for dynamic creation of logical units. As you point out, iSCSI need not use FC identifiers, but a bunch of people like them for their constant defined length and guaranteed world-wide unique identifiers. However, I feel that iSCSI should be a lot more hard nosed than SAM-2 and require that VPD page "83" contain a mandatory world-wide unique logical unit identifier in an appropriate invariant format. For compatibility with other SCSI drivers, it should be limited to 128 bits of length. This would not be an identifier set by the user (there are other SCSI identifiers that can be set by the user), but one that is invariant from the time of manufacture (for physical devices) or creation (for logical devices) of the logical unit. From owner-ips@ece.cmu.edu Mon Jan 29 13:59:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA07492 for ; Mon, 29 Jan 2001 13:59:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TGZq503319 for ips-outgoing; Mon, 29 Jan 2001 11:35:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zmamail04.zma.compaq.com (zmamail04.zma.compaq.com [161.114.64.104]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TGZVZ03299 for ; Mon, 29 Jan 2001 11:35:31 -0500 (EST) Received: by zmamail04.zma.compaq.com (Postfix, from userid 12345) id A3723547D; Mon, 29 Jan 2001 11:35:25 -0500 (EST) Received: from exchou-gh02.cca.cpqcorp.net (exchou-gh02.cca.cpqcorp.net [16.110.248.202]) by zmamail04.zma.compaq.com (Postfix) with ESMTP id 537E85532 for ; Mon, 29 Jan 2001 11:35:25 -0500 (EST) Received: by exchou-gh02.cca.cpqcorp.net with Internet Mail Service (5.5.2652.78) id ; Mon, 29 Jan 2001 10:35:24 -0600 Message-ID: <78AF3C342AEAEF4BA33B35A8A15668C659C0BC@cceexc17.americas.cpqcorp.net> From: "Elliott, Robert" To: ips@ece.cmu.edu Subject: RE: iSCSI: INQUIRY page 0x83 identifier Date: Mon, 29 Jan 2001 10:38:37 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2652.78) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Michael Krause [mailto:krause@cup.hp.com] > Sent: Friday, January 26, 2001 9:08 PM > To: Robert Snively > Cc: ips@ece.cmu.edu > Subject: RE: iSCSI: INQUIRY page 0x83 identifier > ... > Why not just require EUI-64 which is a very large name space > and is / will be used in many hardware solutions today / future? > This would also fit in well with other I/O standards that use > EUI-64 for identifying all I/O components / controllers. Having > a 128-bit value seems like overkill and provides little benefit. A RAID controller needs to construct unique identifiers for its virtual volumes. The hardware typically has a single EUI-64. It's difficult for software to construct unique 64-bit identifiers that don't interfere with the EUI-64 namespace. The hardware EUI-64 would have to represent a range rather than just one ID. It's much simpler to use the EUI-64 as a base and append bits to the end. That's how the 128-bit Registered Extended format is used. I wouldn't mind only documenting the 64-bit IEEE Registered and 128-bit IEEE Registered Extended formats in type 3h. However, this is an SPC-3 issue, not an iSCSI issue. The difference between the type 2h EUI-64 and the type 3h IEEE Registered type is the latter includes a 4-bit Naming Address Authority (NAA) field, restricting the vendor-specific portion to 36 bits. Type 2 (EIU-64): 24-bit company ID + 40 bit vendor-specific Type 3 (FC-FS): 4 bit NAA + 60 bit NAA-specific Type 3 NAA=IEEE Registered: 24-bit company ID + 36-bit vendor-specific Type 3 NAA=IEEE Registered Extended: 24-bit company ID + 36-bit vendor-specific + 64-bit vendor-specific --- PC: Robert.Elliott@compaq.com UNIX: relliott@unixmail.compaq.com From owner-ips@ece.cmu.edu Mon Jan 29 14:00:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA07531 for ; Mon, 29 Jan 2001 14:00:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0THA3N04732 for ips-outgoing; Mon, 29 Jan 2001 12:10:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TH9eZ04717 for ; Mon, 29 Jan 2001 12:09:42 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA116134; Mon, 29 Jan 2001 18:09:28 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id SAA63934; Mon, 29 Jan 2001 18:09:27 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E3.005E3EF6 ; Mon, 29 Jan 2001 18:09:25 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: "Barry Reinhold - Lamprey" cc: ips@ece.cmu.edu Message-ID: Date: Mon, 29 Jan 2001 19:00:45 +0200 Subject: Re: Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Barry, Immediate data is "attached to the command". If your unsolicited data (the so called initial burst) is larger that what you are ready to commit to a single PDU you can send several PDUs with the last having the F bit. After that the target can decide that it wants more and send an R2T or send status. You are no supposed to send both immediate and separate PDU. I admit that the text is bit confusing (in 1.2.5) and there is an error also in the appendix. 2.7.1 talk about unsolicited data (not immediate) (the context is data PDUs). The new version of 1.2.5 now reads: Unsolicited data can be sent as part of an iSCSI command PDU ("immediate data") or an in separate iSCSI data PDUs. An initiator may send unsolicited data either as immediate (up to the negotiated maximum PDU size - DataPDULength - disconnect-reconnect mode page) or in a separate PDU sequence (up to the negotiated limit - FirstBurstSize - disconnect-reconnect mode page). All subsequent data have to be solicited. The maximum size of an individual data PDU or the immediate-part of the initial unsolicited burst as well as the initial burst size MAY be negotiated at login. Thanks, Julo "Barry Reinhold - Lamprey" on 29/01/2001 16:34:30 Please respond to "Barry Reinhold - Lamprey" To: Julian Satran/Haifa/IBM@IBMIL cc: "Jon Sreekanth" , "James Smart" Subject: Issue: I am unclear on how to respond to an iSCSI SCSI command PDU when there is immediate data but not enough to meet the requirements of the IO operation as given in the expected data transfer length field. Should an R2T be issued or not? Background: Clause 2.7.1 in 03 reads: "2.7.1 F (Final) bit This bit is 1 for the last PDU of immediate data or the last PDU of a sequence answering a R2T." This suggests that immediate data can extend beyond the iSCSI SCSI CMD PDU (there is no final bit on the command PDU)but the closet thing we have to a definition for immediate data, which is in clause 1.2.5 below, suggests that immediate data is limitted to the command PDU: "Outgoing SCSI data (initiator to target - user data or command parameters) will be sent as either solicited data or unsolicited data. Solicited data are sent in response to Ready To Transfer (R2T) PDUs. Unsolicited data can be part of an iSCSI command PDU ("immediate data") or an iSCSI data PDU. An initiator may send unsolicited data (immediate or in a separate PDU) up to the negotiated limit (initial burst size - mode page 02h). All subsequent data have to be solicited. The maximum size of an individual data PDU or the immediate-part of the initial unsolicited burst MAY be negotiated at login (maximum connect size - mode page 02h)." Unsolicited data is bounded in a number of ways, but most significant to this issue is in the description of the login key useR2t which states: "Note than only the first outgoing data item (either immediate data or a separate PDU) can be sent unsolicited by a R2T." Given the above it is unclear to me if the concept of immediate data is: 1. Write data that can be sent without an R2T (unsolicited write data)and starts in the command PDU. 2. Write data that is entirely contained within the command PDU. (my initial concept of immediate data) 3. The non-zero portion of data, in a possible multi-PDU write operation, that is contained in the iSCSI SCSI command PDU. Where the remaining PDUs of the write operation must be solicited with an R2T. Given the current definition of the Final bit and the current limits on unsolicited data it is unclear to me what a target should do when receiving an iSCSI SCSI command PDU that has immediate data in it, but not sufficient data to meet the expected data transfer length specificied in the command. Does an R2T go out of not? Barry Reinhold 603-659-0885 From owner-ips@ece.cmu.edu Mon Jan 29 15:22:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA09353 for ; Mon, 29 Jan 2001 15:22:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TIIsY07773 for ips-outgoing; Mon, 29 Jan 2001 13:18:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TIIdZ07763 for ; Mon, 29 Jan 2001 13:18:40 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0TJPM060814; Mon, 29 Jan 2001 11:25:22 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Michael Krause" , Subject: RE: iSCSI Data Integrity - Digests Date: Mon, 29 Jan 2001 10:13:34 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <5.0.0.25.2.20010126165115.00a73a80@hpindlm.cup.hp.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Michael, Your highest and second highest preference represents a change to the TCP protocol in that it mandates segment alignment of the PDU with the Ethernet frame. The impact of changing TCP into a datagram protocol may not be well received. If such a requirement exists, use SCTP. If you wish to see two frame checks, one for the iSCSI header and one again for the SCSI payload, I would suggest that there are variables used as handshakes within the iSCSI header that could use even a simpler scheme than a 16 bit CRC, a 32 bit XOR. If done in software, both the TCP checksum and this header checksum will require updating. A single CRC field will represent a time constraint in that a pipeline delay of iSCSI feedback will exist. Allowing these calculations to take place prior to being placed for delivery within an interrupt routine is the advantage of allowing a simple feedback update. Doug The following describes three alternatives for iSCSI data integrity. Mike Krause HP Requirements: Use existing / proven CRC algorithms and techniques to provide fast market enablement while avoiding a "reinvention of the wheel" exercise. Provide strong end-to-end data integrity for both iSCSI PDU header and data payload CRC is required for all implementations, i.e. strong end-to-end data integrity is not an option as customers will not adopt solutions without such guarantees. Alternative 1 (Highest preference): An iSCSI PDU shall be restricted to a single TCP segment. Multiple iSCSI PDU may be present within the same TCP segment but none shall span multiple segments. Each iSCSI PDU is protected by a trailing 32-bit CRC (Ethernet polynomial), i.e. a single CRC covers the entire iSCSI header and data. Assumption: iSCSI PDU header is not modified during transmission. While there has been some discussion of a desire to provide such capabilities in the future, there are no current requirements to support requiring the specification to take this into account at this time. Should this become a requirement in the future, an intermediate endnode supporting iSCSI header modification would need to guarantee strong data integrity within its implementation using any of the well-known / deployed techniques. Benefits: Strong end-to-end data integrity using a well-known, proven technology. Low-cost, high-speed hardware implementations with readily available hardware cores can be created with minimal design complexity. Only one CRC which can be implemented in software mitigating the performance impacts this iSCSI data integrity would impose. Ability to accelerate software iSCSI implementations using a slightly modified NIC to perform the CRC calculation / verification for both inbound and outbound data streams. This modified NIC would only require minor understanding of the iSCSI header, i.e. to identify it and locate the CRC within the data stream. The CRC can be verified while coming in off the "wire" or inserted while being placed on the "wire". This technique is well understood since it is very similar to what is implemented by TCP checksum off-load implementations in use today. Note: A NIC implementing this functionality could combine the verification of the TCP checksum into a "one-stop" verification operation and silently drop invalid packets or tag them as "bad" for ULP processing. Solves the framing problem while eliminating the need for future support of "chunking" / RDMA technology. Each PDU header contains sufficient information required for direct data placement providing the same benefits attributed to chunking / RDMA. This will also allow simplified "bridge" solutions to be constructed, e.g. iSCSI-to-InfiniBand, iSCSI-to-SRP, etc. Eliminates the need to maintain intermediate CRC results (both inbound and outbound) reducing implementation cost / complexity. Eliminates bandwidth waste by reducing the number of bytes required to guarantee end-to-end data integrity while supporting multiple small PDU per segment (compaction) Provides improved QoS arbitration control / management - if a PDU were allowed to span multiple segments, then an implementation would need to transmit segments back-to-back (or very close) to deliver strong end-to-end performance / transaction throughput. This may be implementation-specific but is still a tangible benefit for customers. If an intermediate endnode performs re-segmentation, a PDU may be span multiple segments. This would be detected by a PDU CRC error providing a simple detection mechanism allowing implementations to recover either at the connection or session level. Constraints: iSCSI implementations must be able to determine each connection's MSS and create iSCSI PDU that fit within the MSS. Such functionality is available in a variety of TCP implementations today and for hardware implementations. For the send-side retransmission problem (i.e. how to delineate packets within a byte stream), a hardware implementation is straight-forward to support since it provide the PDU-segment correlation. For a software implementation, the mbuf / mblk encompassing the iSCSI PDU would be marked to indicate whether the associated buffer should be sent within a separate segment or not. This is not common to any TCP implementations to date but is not difficult to implement. It should also be noted that this is an implementation not a TCP protocol issue. If a layer 4 intermediate endnode glues together two TCP streams and is not iSCSI aware, the send-side retransmission is a problem. However, it is unclear whether this usage model must be transparently supported by iSCSI, i.e. such an intermediate endnode should be required to be iSCSI aware. This is not unreasonable as most layer 4 intermediate endnodes are providing some value-add service as a function of layer 4; why wouldn't such an endnode provide iSCSI value-add and thus be layer 5 aware. Alternative 2 (middle preference) An iSCSI PDU shall be restricted to a single TCP segment. Multiple iSCSI PDU may be present within the same TCP segment but none shall span multiple segments. Each iSCSI PDU is protected by two CRCs - one invariant and one variant. The invariant CRC (ICRC) is a 32-bit CRC covering the PDU data and invariant header fields (e.g. address). The variant CRC (VCRC) is either a 16 or 32-bit CRC that covers the entire PDU header, data, and invariant CRC. PDU layout would be: header, data, ICRC, VCRC. Note: This scheme is conceptually the same as what is used in InfiniBand providing customers and the industry with a single paradigm and improved technology integration for both compute and storage endnodes. Benefits relative to Alternative 1: Supports an intermediated endnode updating iSCSI header fields while supporting strong end-to-end data integrity of all invariant header fields and data. It is critical that all invariant header fields such as target address be protected at all times to avoid silent data corruption / illegal memory access since these fields are used to DMA the data into / from target memory. Note: This problem does not exist in IP-based applications today since such implementations do not expose addresses across the wire but use look-up techniques as a function of the header. iSCSI implementations may choose to use a similar technique but at the cost of increased resources / complexity. Limits the complexity / overhead required to support a separate header CRC - e.g. intermediate byte-stream CRC injection / verification. This simplifies the hardware implementation for full off-load solution as well as provides the ability to create simplified CRC acceleration as described in alternative 1 for software-based iSCSI implementations. Use of two trailer CRCs does not impact overall end-to-end performance or endnode hardware resources. Implementations are gated more by the memory subsystems / cache coherency overheads than by external wire speed transmission, i.e .the packet will, in general, arrive before one could complete the first few cache line fetch operations. As such, given the single-segment operation, the data can be verified as it comes in off the wire and the memory operations initiated with minimal latency (most operations will be pipeline operations within a few cycles). An intermediate endnode can provide data integrity checks while data is in-flight and stomp the CRC should it detect an error. This allows packet flow-through to be supported while providing fault isolation and a single for subsequent endnodes to drop invalid packets if they desire. Constraints: Invariant header fields must be identified and included within the ICRC calculation adding minor complexity to the overall implementation. Alternative 3 (least preferred): Allow a PDU to span multiple TCP segments. Implement two CRC: a header CRC and a data CRC. Do not allow intermediate endnodes to modify the iSCSI header. Constraints / Disadvantages: Increased implementation complexity and overhead. The header CRC must occur following the header requiring injection / removal within the endnodes. This complexity is compounded for variable header protocols such as iSCSI and is why such a solution has been rejected in other high-speed technologies. Requires intermediate CRC state to be maintained for both inbound and outbound requests. Increased QoS scheduling complexity for strong end-to-end application throughput. Does not solve the framing problem perhaps necessitates the need for a chunking / RDMA solution. This increases solution complexity and creates interoperability / support issues for customers, i.e. options are bad for developers; bad for customers. Severely limits creating high performance iSCSI software-based implementations perhaps making them impractical as a general purpose implementation. This will limit the potential market for iSCSI solutions. Note: If an intermediate endnode is allowed to modify the PDU header, then there exists a possibility of silent data corruption since the invariant portions no longer have end-to-end data integrity. This will be a major issue for customers in terms of their ability to adopt iSCSI across a variety of solution spaces, i.e. if there is the potential for silent data corruption, then customers will not deploy iSCSI and will turn to alternatives that provide stronger end-to-end data integrity. From owner-ips@ece.cmu.edu Mon Jan 29 15:22:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA09367 for ; Mon, 29 Jan 2001 15:22:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TILtr07931 for ips-outgoing; Mon, 29 Jan 2001 13:21:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TILSZ07891 for ; Mon, 29 Jan 2001 13:21:28 -0500 (EST) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 7C3EE27AA; Mon, 29 Jan 2001 10:19:00 -0800 (PST) Received: from mk731913.cup.hp.com (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id KAA05723; Mon, 29 Jan 2001 10:21:21 -0800 (PST) Message-Id: <5.0.0.25.2.20010129100829.022edfe0@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.0 Date: Mon, 29 Jan 2001 10:13:35 -0800 To: julian_satran@il.ibm.com From: Michael Krause Subject: RE: new iSCSI PDU outline Cc: ips@ece.cmu.edu In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 06:37 PM 1/27/2001 +0200, julian_satran@il.ibm.com wrote: >Doug, > >CRC is not mandatory. If you are using a local net (real time) on a >reliable media you probably don't need anything. Unclear how iSCSI will be able to determine whether the target is local or not given IP subnets can already span large geographic distances and the movement to 10 GbE translates to large Ethernet fabrics. Also as transparent fail-over is delivered by multiple vendors, what started out as local may become remote in some way. I strongly disagree with making CRC protection optional and I suspect customers will given the various studies on error rate escapes associated with TCP checksum already discussed on this reflector. Mike From owner-ips@ece.cmu.edu Mon Jan 29 15:23:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA09384 for ; Mon, 29 Jan 2001 15:23:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TIHs907722 for ips-outgoing; Mon, 29 Jan 2001 13:17:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TIHQZ07704 for ; Mon, 29 Jan 2001 13:17:26 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0TJPR060820; Mon, 29 Jan 2001 11:25:27 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: Cc: "Ips \(E-mail\)" Subject: RE: iSCSI : Command Ordering Proposal. Date: Mon, 29 Jan 2001 10:13:40 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <3A722ABD.AE763CA2@cup.hp.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh, Rather than being clever with a special case setting the sequence number to zero, the limits imposed on the sequence for things like Head of Queue or NOP could be ignored and not be treated as violations. This would allow all such activity between the initiator and the target to remain in sequence. Allowing this exception to a basic scheme of ensuring sequential communication has several drawbacks such as lack of acknowledgement. What drawback is there in having the target making allowances for some commands such as Head of Queue or NOP to exceed a maximum? Doug > Douglas Otis wrote: > > > Santosh, > > > > If there is a task attribute, the complexity is when you allow > CmdSN = 0; > > > > Perhaps CmdSN = 0 should not be allowed or treated as just > another in the > > sequence. > > The (CmdSN==0) does have its uses. One example is the use of NOP-OUT for > a "connection ping." > > > > If a retry flag is set, the CmdSN below the > > ExpCmdSN should be viewed as an exception much as Head of Queue > should be an > > exception for MaxCmdSN. Both of these cases imply exceptional > behavior by > > the initiator in response to a problem. > > Accepting a command outside the (ExpCmdSN, MaxCmdSN) window with the retry > bit set then opens up windows for other types of stale and > duplicate commands > to be > processed. > > Here's the deal (as I view it) : > > Commands are sent with the "retry" bit under 2 conditions : > - Connection Failures > - Digest Errors > > In both these cases, an explicit handshake is required regarding the > (ExpCmdSN, MaxCmdSN) window, prior to starting to send "retry" > commands. This ensures that genuine "retry" commands do not slip > behind the (ExpCmdSN, MaxCmdSN) window. > > In the case of a connection failure, the Logout Response provides this > handshake. For digest errors, there is no such hand-shake. > Hence, a command that encountered a digest error and was retried could > likely slip behind the (ExpCmdSN, MaxCmdSN) window and then be never > processed. > > Regards, > Santosh > From owner-ips@ece.cmu.edu Mon Jan 29 15:26:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA09436 for ; Mon, 29 Jan 2001 15:26:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TIJuc07816 for ips-outgoing; Mon, 29 Jan 2001 13:19:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TIJ2Z07780 for ; Mon, 29 Jan 2001 13:19:02 -0500 (EST) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id 0E7EE10B9; Mon, 29 Jan 2001 10:19:01 -0800 (PST) Received: from mk731913.cup.hp.com (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id KAA05731; Mon, 29 Jan 2001 10:21:32 -0800 (PST) Message-Id: <5.0.0.25.2.20010129101457.022ceda0@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.0 Date: Mon, 29 Jan 2001 10:16:15 -0800 To: Patrick Stirling From: Michael Krause Subject: RE: iSCSI: INQUIRY page 0x83 identifier Cc: ips@ece.cmu.edu In-Reply-To: References: <5.0.0.25.2.20010126190554.076ca0c0@hpindlm.cup.hp.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 08:31 PM 1/26/2001 -0800, Patrick Stirling wrote: >On Fri, 26 Jan 2001, Michael Krause wrote: > > > At 01:30 PM 1/26/2001 -0800, Robert Snively wrote: > > > > >However, I feel that iSCSI should be a lot more hard nosed than > > >SAM-2 and require that VPD page "83" contain a mandatory world-wide > > >unique logical unit identifier in an appropriate invariant format. > > >For compatibility with other SCSI drivers, it should be limited > > >to 128 bits of length. This would not be an identifier set by > > >the user (there are other SCSI identifiers that can be set by the > > >user), but one that is invariant from the time of manufacture (for > > >physical devices) or creation (for logical devices) of the logical unit. > > > > Why not just require EUI-64 which is a very large name space and is / will > > be used in many hardware solutions today / future? This would also fit in > > well with other I/O standards that use EUI-64 for identifying all I/O > > components / controllers. Having a 128-bit value seems like overkill and > > provides little benefit. > >Well, I'd prefer to stick with the current scheme of allowing the device >to specify its identifier within certain limits (table 181 in spc2r18). > >I'm working on a virtual storage server, which serves up virtual disks on >the fibre-channel SAN (with iSCSI coming soon I expect). We set just this >identifier in VPD page 0x83, using a type 3 identifier (FC-PH) - actually >a Registered Extended IEEE Name, 128bits. It's convenient to use this >length as it allows me to set the upper 64bits to our server's WWN (which >is also used as the World-Wide Node Name shared by all target HBAs on the >server), and the lower 64bits to a simple number identifying the virtual >disk on this server. Sounds like an IPv6 address. InfiniBand uses the same technique where an EUI64 is combined with an IPv6 subnet prefix. This provides the flexibility you desire with a manufacturer unique GUID. It also makes attachment to the Internet much simpler as well as leverages existing / new IPv6-based tools / services. BTW, wireless seems to be moving towards IPv6 as well so why not just use the same paradigm for everything: server I/O interconnect, wireless, storage, etc. Mike From owner-ips@ece.cmu.edu Mon Jan 29 15:27:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA09470 for ; Mon, 29 Jan 2001 15:27:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TIGEK07663 for ips-outgoing; Mon, 29 Jan 2001 13:16:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TIF0Z07586 for ; Mon, 29 Jan 2001 13:15:00 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0TJPS060823; Mon, 29 Jan 2001 11:25:28 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: , Subject: RE: new iSCSI PDU outline Date: Mon, 29 Jan 2001 10:13:41 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, You will still have the 16-bit summation within the TCP IP providing an effective error rate of one in 64k. A 32-bit XOR in conjunction with this checksum is not the same odds of missing an error. As you add bytes covered by this XOR (not the entire frame), the reliability is reduced. If to evaluate a simple scheme, the fewer words the better and if information such as tags and static command related data can be provided better protection, why not? Doug > Doug, > > CRC is not mandatory. If you are using a local net (real time) on a > reliable media you probably don't need > anything. In any case a simple XOR will add nothing to what TCP already > has (checksum) in the way of protection. > And given that most of the time you have to deal with SCSI commands and > responses and data blocks a basic block that takes care of those header > types (the 48 byte) is probably optimal. The extended CDB is for those > cases in which SCSI uses a CDB longer than 16 bytes (16 bytes of CDB are > still provided for in the BHS). > > > Julo > > > "Douglas Otis" on 26/01/2001 21:23:04 > > Please respond to "Douglas Otis" > > To: ips@ece.cmu.edu > cc: > Subject: RE: new iSCSI PDU outline > > > > > Julian, > > Should there be software implementations that must deal with iSCSI on a > real-time basis, splitting the headers as I described helps in > allowing the > transport to respond immediately in providing the transport related > feed-back. The split header and simple XOR calculation was to make this > possible with software implementations where these settings are made > nano-seconds before being sent. To require a CRC to be done could > introduce > many micro-seconds into the operation and thus make the transport less > current on its feed-back as this must be pipelined. You have a few other > fields not defined and perhaps use those fields to expand the later > content. > I doubt there will be more than a handful of types but if you use 16 bits, > there should never be a danger of running out of options. In addition, I > would suggest retry flags be placed in this header for the same reasons of > simplifying the transport. Once the SCSI structure is created, it becomes > independent of the transport functions so that fiddling at the transport > will not spoil this content. By keeping these values to but a few fields > unrelated to the payload, a simple check should be all that is needed. > > Doug > > Doug, > > > > You are right and we could do it but it will reduce the number of yet > > unallocated types in WN (and T10 is adding more stuff to the SCSI!) and > > will save us several bits in the command where we have plenty. I don't > > think it is worth it. > > > > Regards, > > Julo > > > > "Douglas Otis" on 26/01/2001 00:41:02 > > > > Please respond to "Douglas Otis" > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: new iSCSI PDU outline > > > > > > > > > > Julian, > > > > Would you consider splitting headers. As was done with MTF, > > there was just > > a simple 32 bit simple XOR checksum for header fields. Here is an > example > > with a CDB appended. The WN could expand the appended type into > > SCSI_CMND, > > SCSI_RSP, SCSI_R2T, iSCSI_Specific, etc. > > > > |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| > > +---------------+---------------+---------------+---------------+ > > 0| WN | Reserved | AddCDB | > > +---------------+---------------+---------------+---------------+ > > 4| Length | > > +---------------+---------------+---------------+---------------+ > > 8| ->CmdSN or <-StatRN | > > +---------------+---------------+---------------+---------------+ > > 12| ->ExpStatSN or <-ExpCmdSN | > > +---------------+---------------+---------------+---------------+ > > 16| ->MaxStatRN? <-MaxCmdSN | > > +---------------+---------------+---------------+---------------+ > > 20| XOR Checksum | > > +---------------+---------------+---------------+---------------+ > > +---------------+---------------+---------------+---------------+ > > 24| SCSI_CMND ... | > > +- | > > 32| | > > +---------------+---------------+---------------+---------------+ > > 36| CDB ... | > > +---------------+---------------+---------------+---------------+ > > x | (Data ...) | > > +---------------+---------------+---------------+---------------+ > > y | CRC | > > +---------------+---------------+---------------+---------------+ > > > > Doug > > > > > > > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Mon Jan 29 15:28:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA09496 for ; Mon, 29 Jan 2001 15:28:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TIG4507650 for ips-outgoing; Mon, 29 Jan 2001 13:16:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TIF0Z07587 for ; Mon, 29 Jan 2001 13:15:01 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0TJPS060826; Mon, 29 Jan 2001 11:25:29 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Ralph Weber" , Cc: "Robert Snively" Subject: RE: New proposal for FCIP ? Date: Mon, 29 Jan 2001 10:13:42 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Ralph, There was a suggestion of allowing multiple TCP connections. A time-stamp that resolves to the packet rate could be used as a means of ordering frames on multiple streams assuming a hold period in the area of RTT. The advantage of such a scheme would be from the freedom found in allowing a fail-over within a short time. Frames outside the time window are dropped. Doug > } I understand that there was a new proposal at the interim meeting > } at Orlando, adding timestamps for ordered delivery in FCIP. > } > } Could someone post a pointer to this proposal (if it exists as a > } ID) ? If not, could someone post the minutes of this discussion ? > } > } I apologize if this has been posted already - I could not find > } this on the archives either. > } > } Thanks, > } > } -Sriram > > Regrettably, the proposal is still under development, promised for > delivery by Valentines Day. It was discussed in overview at the > Orlando meeting. > > The addition of time stamps is a part of the proposal. However, > they have nothing to do with ordered delivery because ordered > delivery is guaranteed by TCP/IP upon which FCIP is layered. > The time stamps are to assist in the handling of R_A_TOV and > E_D_TOV, two Fibre Channel timeout values. > > I can't help you with the minutes. > > Ralph Weber > Representing > Brocade Communications > From owner-ips@ece.cmu.edu Mon Jan 29 17:28:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA10883 for ; Mon, 29 Jan 2001 17:28:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TJfwW11684 for ips-outgoing; Mon, 29 Jan 2001 14:41:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from aegis.indstorage.com (www.diigroup.com [208.132.17.2] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f0TJeOZ11610 for ; Mon, 29 Jan 2001 14:40:25 -0500 (EST) Received: from aegis ([192.168.1.106]) by aegis.indstorage.com (8.9.3/8.9.2) with SMTP id MAA26581 for ; Mon, 29 Jan 2001 12:38:30 -0700 From: "Mark Bradley" To: Subject: RE: iSCSI: INQUIRY page 0x83 identifier Date: Mon, 29 Jan 2001 12:32:27 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit 128 bits will not last as unique for very long. It should probably be 256 bits. Have a look at CORBA, OLE IDs for ideas on how these can be generated. There's a significant body of work there. -- markb > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Jim Hafner > Sent: Monday, January 29, 2001 10:09 AM > To: ips@ece.cmu.edu; Robert Snively > Subject: RE: iSCSI: INQUIRY page 0x83 identifier > > > > Bob, > > I strongly agree with the requirement that these identifiers be WWUnique. > But I also agree with Charles that whatever identifier that is presented > should be the same regardless of the interconnect, so binding > them to FC or > to some defined iSCSI mechanism seems inappropriate. > > So, I think I'm agreeing the Rob Elliot. Namely, specify a 128 format > rooted in EUI-64 for the device. Back away from anything > FC-specific in the > format of the EUI-64. A vendor of a storage controller will probably have > an EUI-64 for the device, though that might not be FC-based. Use this as > the basis for some vendor-specific algorithm to generate the LU > identifier. > Leave it to the vendor to define an algorithm that will provide the > uniqueness. > > Also, I'd like to keep iSCSI from specifying any requirements on logical > units (that's not the right space). SPC-2/3 can spec a general format and > the requirement for uniqueness. Then leave it to the vendors to implement > correctly. > > Does that sound reasonable? > Jim Hafner > > > Robert Snively on 01-26-2001 01:30:43 PM > > To: Jim Hafner/Almaden/IBM@IBMUS, ips@ece.cmu.edu > cc: > Subject: RE: iSCSI: INQUIRY page 0x83 identifier > > > > > As for an iSCSI type or format for logical unit identifiers: > > 1) SCSI already allows for an ascii formatted identifier (typically this > > includes vendor, model, serial number). Since NDT has defined WWUIs for > > DEVICES and these have a defined UTF-8 format, there is nothing to > > *prevent* a vendor from using these strings as the starting point for LU > > page 83 identifiers of the logical units within that device. > > 2) But..., iSCSI and NDT should not attempt to make a rule under which > > these are used. FC's intervention in this was more as a strong > suggestion, > > and is certainly not a requirement. > > Jim, > > I generally agree with the text of your comments, but there are > a couple of details I would like to address. > > Any name not based on a registered entity and in a registered format > is not guaranteed to be world-wide unique, and is therefore > inappropriate. That is why numbers like OUIs and SCSI Vendor IDs > in specified parts of the identifier are a requirement. EUI-64 > and FC identifiers have that property. FC identifiers have the added > benefit of having a regular extension capability for dynamic creation > of logical units. As you point out, iSCSI need not use FC > identifiers, but a bunch of people like them for their constant > defined length and guaranteed world-wide unique identifiers. > > However, I feel that iSCSI should be a lot more hard nosed than > SAM-2 and require that VPD page "83" contain a mandatory world-wide > unique logical unit identifier in an appropriate invariant format. > For compatibility with other SCSI drivers, it should be limited > to 128 bits of length. This would not be an identifier set by > the user (there are other SCSI identifiers that can be set by the > user), but one that is invariant from the time of manufacture (for > physical devices) or creation (for logical devices) of the logical unit. > > > > From owner-ips@ece.cmu.edu Mon Jan 29 17:33:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA10942 for ; Mon, 29 Jan 2001 17:33:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TL00015276 for ips-outgoing; Mon, 29 Jan 2001 16:00:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TKxNZ15254 for ; Mon, 29 Jan 2001 15:59:23 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id A73477CC for ; Mon, 29 Jan 2001 13:59:22 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 8CF85134 for ; Mon, 29 Jan 2001 15:59:21 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id MAA15298 for ; Mon, 29 Jan 2001 12:59:19 -0800 (PST) Message-ID: <3A75DA22.833FA40C@agilent.com> Date: Mon, 29 Jan 2001 13:01:22 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Command Ordering Proposal. References: <3A722ABD.AE763CA2@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Retried commands should have already been queued and/or executed. The goal of the retry is simply to replay back to the initiator the results of the execution. Therefore, there is no ordering requirement for "retried" commands and CmdSN should always be zero. -Matt Santosh Rao wrote: > > Douglas Otis wrote: > > > Santosh, > > > > If there is a task attribute, the complexity is when you allow CmdSN = 0; > > > > Perhaps CmdSN = 0 should not be allowed or treated as just another in the > > sequence. > > The (CmdSN==0) does have its uses. One example is the use of NOP-OUT for > a "connection ping." > > > If a retry flag is set, the CmdSN below the > > ExpCmdSN should be viewed as an exception much as Head of Queue should be an > > exception for MaxCmdSN. Both of these cases imply exceptional behavior by > > the initiator in response to a problem. > > Accepting a command outside the (ExpCmdSN, MaxCmdSN) window with the retry > bit set then opens up windows for other types of stale and duplicate commands > to be > processed. > > Here's the deal (as I view it) : > > Commands are sent with the "retry" bit under 2 conditions : > - Connection Failures > - Digest Errors > > In both these cases, an explicit handshake is required regarding the > (ExpCmdSN, MaxCmdSN) window, prior to starting to send "retry" > commands. This ensures that genuine "retry" commands do not slip > behind the (ExpCmdSN, MaxCmdSN) window. > > In the case of a connection failure, the Logout Response provides this > handshake. For digest errors, there is no such hand-shake. > Hence, a command that encountered a digest error and was retried could > likely slip behind the (ExpCmdSN, MaxCmdSN) window and then be never > processed. > > Regards, > Santosh From owner-ips@ece.cmu.edu Mon Jan 29 17:41:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA11183 for ; Mon, 29 Jan 2001 17:41:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TKk0J14635 for ips-outgoing; Mon, 29 Jan 2001 15:46:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TKjIZ14600 for ; Mon, 29 Jan 2001 15:45:18 -0500 (EST) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id A882FD3; Mon, 29 Jan 2001 12:45:16 -0800 (PST) Received: from mk731913.cup.hp.com (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id MAA09119; Mon, 29 Jan 2001 12:47:51 -0800 (PST) Message-Id: <5.0.0.25.2.20010129124010.023ce6d0@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.0 Date: Mon, 29 Jan 2001 12:45:07 -0800 To: "Douglas Otis" From: Michael Krause Subject: RE: iSCSI Data Integrity - Digests Cc: In-Reply-To: References: <5.0.0.25.2.20010126165115.00a73a80@hpindlm.cup.hp.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 10:13 AM 1/29/2001 -0800, Douglas Otis wrote: >Michael, > >Your highest and second highest preference represents a change to the TCP >protocol in that it mandates segment alignment of the PDU with the Ethernet >frame. This is not a change to the TCP protocol - strictly implementation specific. >The impact of changing TCP into a datagram protocol may not be well >received. It is still byte streaming but iSCSI PDUs don't span multiple segments. This is an application-specific issue, i.e. it is the unit of how one sends data and has nothing to do with the TCP protocol itself. This really isn't any different than what applications do today - a peek() to determine how to cast the byte stream into a message - which is very common for most sockets applications. > If you wish to see two frame checks, one for the iSCSI header and one > again for the SCSI payload, I would suggest that there are variables used > as handshakes within the iSCSI >header that could use even a simpler scheme than a 16 bit CRC, a 32 bit XOR. >If done in software, both the TCP checksum and this header checksum will >require updating. A single CRC field will represent a time constraint in >that a pipeline delay of iSCSI feedback will exist. Allowing these >calculations to take place prior to being placed for delivery within an >interrupt routine is the advantage of allowing a simple feedback update. Disagree with your recommendations and analysis. There is no change required for the TCP checksum this proposal is strictly above the TCP layer, i.e. iSCSI. Mike From owner-ips@ece.cmu.edu Mon Jan 29 17:54:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA11450 for ; Mon, 29 Jan 2001 17:54:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TL49315485 for ips-outgoing; Mon, 29 Jan 2001 16:04:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TL3PZ15451 for ; Mon, 29 Jan 2001 16:03:25 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 868F0659 for ; Mon, 29 Jan 2001 14:03:19 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 6105F191 for ; Mon, 29 Jan 2001 16:03:18 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id NAA15703 for ; Mon, 29 Jan 2001 13:03:16 -0800 (PST) Message-ID: <3A75DB10.70771F96@agilent.com> Date: Mon, 29 Jan 2001 13:05:20 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Command Ordering Proposal. References: <3A70DDB2.68F573CC@cup.hp.com> <3A720371.F64A5749@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > I need to add some more clarity on the discussion here. It was not my intent > to suggest that CmdSN NOT be used for simple task tags. My suggestion > was that the targets use CmdSN to enforce ordering only when an ordered > task tag enters the task set and for the tasks that are older than the ordered > task tag in the task set. Without CmdSN or CRN you could not guarantee what simple tasks get into the queue before or after the ordered task. CRN is new and in an unfinished standard (SAM-2 r15). Therefore, I believe you need CmdSN ordering to handle "legacy" SCSI stacks that don't know this new feature. -Matt From owner-ips@ece.cmu.edu Mon Jan 29 18:02:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA11521 for ; Mon, 29 Jan 2001 18:02:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TLa3x16993 for ips-outgoing; Mon, 29 Jan 2001 16:36:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TLZsZ16987 for ; Mon, 29 Jan 2001 16:35:55 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 532016F8 for ; Mon, 29 Jan 2001 13:35:50 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id NAA28065 for ; Mon, 29 Jan 2001 13:35:45 -0800 (PST) Message-ID: <3A75E2CE.A399174D@cup.hp.com> Date: Mon, 29 Jan 2001 13:38:23 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 Cc: IPS Reflector Subject: Re: iSCSI : Command Ordering Proposal. References: <3A722ABD.AE763CA2@cup.hp.com> <3A75DA22.833FA40C@agilent.com> Content-Type: multipart/mixed; boundary="------------3B10EBF8291143F533B21CFA" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------3B10EBF8291143F533B21CFA Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Matt Wakeley wrote: > Retried commands should have already been queued and/or executed. Not necessarily. A TCP connection failure can affect the command PDU. Similarly, the target may have detected a header or data digest error on the SCSI Command PDU, sent a REJECT and the initiator would have "discarded and retried" the task. In both of the above cases, the command is not queued and/or being executed at the target. Such commands, when retried, need to enforce ordering. Hence, the need to specify CmdSN in the "retry", if the CmdSN is yet to be acknowledged. The difficulty lies in the establishment of a "sync point" with the ExpCmdSN, prior to the initiation of these "retry" commands, so as to ensure that the "retry" will not be discarded by the target, because the CmdSN slid behind the (ExpCmdSN, MaxCmdSN) window. The ExpCmdSN in the login response provides this "sync point" during connection failure recovery. The problem is only with digest error "retries". If one were to NOT apply the "discard and retry" policy on a header digest error, [under the assumption that the I.T.T cannot be trusted on a header digest error], the problem further reduces to the following cases : a) data digest error detected by a target on a command PDU. (immediate data). b) data digest error detected by an initiator on inbound Data PDUs of a READ I/O, and the CmdSN for the command is yet to be acknowledged, but an ACK may be in-flight behind the Data PDU. Regards, Santosh > The goal of > the retry is simply to replay back to the initiator the results of the > execution. Therefore, there is no ordering requirement for "retried" commands > and CmdSN should always be zero. > > -Matt > > Santosh Rao wrote: > > > > > Accepting a command outside the (ExpCmdSN, MaxCmdSN) window with the retry > > bit set then opens up windows for other types of stale and duplicate commands > > to be > > processed. > > > > Here's the deal (as I view it) : > > > > Commands are sent with the "retry" bit under 2 conditions : > > - Connection Failures > > - Digest Errors > > > > In both these cases, an explicit handshake is required regarding the > > (ExpCmdSN, MaxCmdSN) window, prior to starting to send "retry" > > commands. This ensures that genuine "retry" commands do not slip > > behind the (ExpCmdSN, MaxCmdSN) window. > > > > In the case of a connection failure, the Logout Response provides this > > handshake. For digest errors, there is no such hand-shake. > > Hence, a command that encountered a digest error and was retried could > > likely slip behind the (ExpCmdSN, MaxCmdSN) window and then be never > > processed. > > > > Regards, > > Santosh --------------3B10EBF8291143F533B21CFA Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------3B10EBF8291143F533B21CFA-- From owner-ips@ece.cmu.edu Mon Jan 29 18:03:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA11532 for ; Mon, 29 Jan 2001 18:03:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TLfCm17238 for ips-outgoing; Mon, 29 Jan 2001 16:41:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.brocade.com (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TL0tZ15333 for ; Mon, 29 Jan 2001 16:00:55 -0500 (EST) Received: from thor.brocade.com (thor [192.168.126.45]) by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id MAA00095; Mon, 29 Jan 2001 12:50:48 -0800 (PST) Received: by thor.brocadecomm.com with Internet Mail Service (5.5.2650.21) id ; Mon, 29 Jan 2001 12:50:45 -0800 Message-ID: From: Robert Snively To: "'Michael Krause'" , Robert Snively Cc: ips@ece.cmu.edu Subject: RE: iSCSI: INQUIRY page 0x83 identifier Date: Mon, 29 Jan 2001 12:50:43 -0800 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk Michael, In answer to your question: > Why not just require EUI-64 which is a very large name space > and is / will > be used in many hardware solutions today / future? This > would also fit in > well with other I/O standards that use EUI-64 for > identifying all I/O > components / controllers. Having a 128-bit value seems like > overkill and > provides little benefit. The EUI-64 is fine for devices coming from a factory. It is less satisfactory for logical units that are created in the field, as RAID logical units typically are. Those cases also require unique names which are never recreated when a logical unit is destroyed and a new one created. There is also no architected limit to the number of logical units that can be created either at one time or over the life of a RAID box. As a result, the 128-bit format 6 Fibre Channel format is very popular for logical units because the 64-bit core can be taken from the identifier of the creating controller (which may change over the life of the RAID) and the 64-bit extension can be created in a controller specific manner (using combinations of time stamps and other parameters) that still guarantees world-wide uniqueness. Bob > -----Original Message----- > From: Michael Krause [mailto:krause@cup.hp.com] > Sent: Friday, January 26, 2001 7:08 PM > To: Robert Snively > Cc: ips@ece.cmu.edu > Subject: RE: iSCSI: INQUIRY page 0x83 identifier > > > At 01:30 PM 1/26/2001 -0800, Robert Snively wrote: > > > As for an iSCSI type or format for logical unit identifiers: > > > 1) SCSI already allows for an ascii formatted identifier > (typically this > > > includes vendor, model, serial number). Since NDT has > defined WWUIs for > > > DEVICES and these have a defined UTF-8 format, there is > nothing to > > > *prevent* a vendor from using these strings as the > starting point for LU > > > page 83 identifiers of the logical units within that device. > > > 2) But..., iSCSI and NDT should not attempt to make a > rule under which > > > these are used. FC's intervention in this was more as a > strong suggestion, > > > and is certainly not a requirement. > > > >Jim, > > > >I generally agree with the text of your comments, but there are > >a couple of details I would like to address. > > > >Any name not based on a registered entity and in a registered format > >is not guaranteed to be world-wide unique, and is therefore > >inappropriate. That is why numbers like OUIs and SCSI Vendor IDs > >in specified parts of the identifier are a requirement. EUI-64 > >and FC identifiers have that property. FC identifiers have > the added > >benefit of having a regular extension capability for > dynamic creation > >of logical units. As you point out, iSCSI need not use FC > >identifiers, but a bunch of people like them for their constant > >defined length and guaranteed world-wide unique identifiers. > > > >However, I feel that iSCSI should be a lot more hard nosed than > >SAM-2 and require that VPD page "83" contain a mandatory world-wide > >unique logical unit identifier in an appropriate invariant format. > >For compatibility with other SCSI drivers, it should be limited > >to 128 bits of length. This would not be an identifier set by > >the user (there are other SCSI identifiers that can be set by the > >user), but one that is invariant from the time of manufacture (for > >physical devices) or creation (for logical devices) of the > logical unit. > > Why not just require EUI-64 which is a very large name space > and is / will > be used in many hardware solutions today / future? This > would also fit in > well with other I/O standards that use EUI-64 for > identifying all I/O > components / controllers. Having a 128-bit value seems like > overkill and > provides little benefit. > > Mike > > > From owner-ips@ece.cmu.edu Mon Jan 29 18:03:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA11554 for ; Mon, 29 Jan 2001 18:03:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TLJXC16230 for ips-outgoing; Mon, 29 Jan 2001 16:19:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.brocade.com (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TL0tZ15333 for ; Mon, 29 Jan 2001 16:00:55 -0500 (EST) Received: from thor.brocade.com (thor [192.168.126.45]) by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id MAA00095; Mon, 29 Jan 2001 12:50:48 -0800 (PST) Received: by thor.brocadecomm.com with Internet Mail Service (5.5.2650.21) id ; Mon, 29 Jan 2001 12:50:45 -0800 Message-ID: From: Robert Snively To: "'Michael Krause'" , Robert Snively Cc: ips@ece.cmu.edu Subject: RE: iSCSI: INQUIRY page 0x83 identifier Date: Mon, 29 Jan 2001 12:50:43 -0800 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk Michael, In answer to your question: > Why not just require EUI-64 which is a very large name space > and is / will > be used in many hardware solutions today / future? This > would also fit in > well with other I/O standards that use EUI-64 for > identifying all I/O > components / controllers. Having a 128-bit value seems like > overkill and > provides little benefit. The EUI-64 is fine for devices coming from a factory. It is less satisfactory for logical units that are created in the field, as RAID logical units typically are. Those cases also require unique names which are never recreated when a logical unit is destroyed and a new one created. There is also no architected limit to the number of logical units that can be created either at one time or over the life of a RAID box. As a result, the 128-bit format 6 Fibre Channel format is very popular for logical units because the 64-bit core can be taken from the identifier of the creating controller (which may change over the life of the RAID) and the 64-bit extension can be created in a controller specific manner (using combinations of time stamps and other parameters) that still guarantees world-wide uniqueness. Bob > -----Original Message----- > From: Michael Krause [mailto:krause@cup.hp.com] > Sent: Friday, January 26, 2001 7:08 PM > To: Robert Snively > Cc: ips@ece.cmu.edu > Subject: RE: iSCSI: INQUIRY page 0x83 identifier > > > At 01:30 PM 1/26/2001 -0800, Robert Snively wrote: > > > As for an iSCSI type or format for logical unit identifiers: > > > 1) SCSI already allows for an ascii formatted identifier > (typically this > > > includes vendor, model, serial number). Since NDT has > defined WWUIs for > > > DEVICES and these have a defined UTF-8 format, there is > nothing to > > > *prevent* a vendor from using these strings as the > starting point for LU > > > page 83 identifiers of the logical units within that device. > > > 2) But..., iSCSI and NDT should not attempt to make a > rule under which > > > these are used. FC's intervention in this was more as a > strong suggestion, > > > and is certainly not a requirement. > > > >Jim, > > > >I generally agree with the text of your comments, but there are > >a couple of details I would like to address. > > > >Any name not based on a registered entity and in a registered format > >is not guaranteed to be world-wide unique, and is therefore > >inappropriate. That is why numbers like OUIs and SCSI Vendor IDs > >in specified parts of the identifier are a requirement. EUI-64 > >and FC identifiers have that property. FC identifiers have > the added > >benefit of having a regular extension capability for > dynamic creation > >of logical units. As you point out, iSCSI need not use FC > >identifiers, but a bunch of people like them for their constant > >defined length and guaranteed world-wide unique identifiers. > > > >However, I feel that iSCSI should be a lot more hard nosed than > >SAM-2 and require that VPD page "83" contain a mandatory world-wide > >unique logical unit identifier in an appropriate invariant format. > >For compatibility with other SCSI drivers, it should be limited > >to 128 bits of length. This would not be an identifier set by > >the user (there are other SCSI identifiers that can be set by the > >user), but one that is invariant from the time of manufacture (for > >physical devices) or creation (for logical devices) of the > logical unit. > > Why not just require EUI-64 which is a very large name space > and is / will > be used in many hardware solutions today / future? This > would also fit in > well with other I/O standards that use EUI-64 for > identifying all I/O > components / controllers. Having a 128-bit value seems like > overkill and > provides little benefit. > > Mike > > > From owner-ips@ece.cmu.edu Mon Jan 29 20:19:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA13342 for ; Mon, 29 Jan 2001 20:19:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TNP7f21479 for ips-outgoing; Mon, 29 Jan 2001 18:25:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TNOqZ21469 for ; Mon, 29 Jan 2001 18:24:52 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0U0ZP061046; Mon, 29 Jan 2001 16:35:26 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Matt Wakeley" , "IPS Reflector" Subject: RE: iSCSI : Command Ordering Proposal. Date: Mon, 29 Jan 2001 15:23:39 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <3A75DA22.833FA40C@agilent.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt, Whether you use the CmdSN or the tag to associate the retry, the CmdSN is still used as a means to handshake freeing of resources and not the tag. It seems like a cleaner solution not to include exceptions on maintaining the sequence just to allow this type of non-sequential use. Clearly there is an alternative that has benefits. Simply have the target allow exceptions based on the command or task attribute and not drop the ability to handshake. I am aware of the present concept, but you did not speak about any downside to this alternative. Doug > Retried commands should have already been queued and/or executed. > The goal of > the retry is simply to replay back to the initiator the results of the > execution. Therefore, there is no ordering requirement for > "retried" commands > and CmdSN should always be zero. > > -Matt > > Santosh Rao wrote: > > > > Douglas Otis wrote: > > > > > Santosh, > > > > > > If there is a task attribute, the complexity is when you > allow CmdSN = 0; > > > > > > Perhaps CmdSN = 0 should not be allowed or treated as just > another in the > > > sequence. > > > > The (CmdSN==0) does have its uses. One example is the use of NOP-OUT for > > a "connection ping." > > > > > If a retry flag is set, the CmdSN below the > > > ExpCmdSN should be viewed as an exception much as Head of > Queue should be an > > > exception for MaxCmdSN. Both of these cases imply > exceptional behavior by > > > the initiator in response to a problem. > > > > Accepting a command outside the (ExpCmdSN, MaxCmdSN) window > with the retry > > bit set then opens up windows for other types of stale and > duplicate commands > > to be > > processed. > > > > Here's the deal (as I view it) : > > > > Commands are sent with the "retry" bit under 2 conditions : > > - Connection Failures > > - Digest Errors > > > > In both these cases, an explicit handshake is required regarding the > > (ExpCmdSN, MaxCmdSN) window, prior to starting to send "retry" > > commands. This ensures that genuine "retry" commands do not slip > > behind the (ExpCmdSN, MaxCmdSN) window. > > > > In the case of a connection failure, the Logout Response provides this > > handshake. For digest errors, there is no such hand-shake. > > Hence, a command that encountered a digest error and was retried could > > likely slip behind the (ExpCmdSN, MaxCmdSN) window and then be never > > processed. > > > > Regards, > > Santosh > From owner-ips@ece.cmu.edu Mon Jan 29 20:19:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA13343 for ; Mon, 29 Jan 2001 20:19:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0TND4a20999 for ips-outgoing; Mon, 29 Jan 2001 18:13:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0TNCvZ20989 for ; Mon, 29 Jan 2001 18:12:58 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0U0Ji061029; Mon, 29 Jan 2001 16:19:44 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Michael Krause" Cc: Subject: RE: iSCSI Data Integrity - Digests Date: Mon, 29 Jan 2001 15:07:58 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <5.0.0.25.2.20010129124010.023ce6d0@hpindlm.cup.hp.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Michael, > At 10:13 AM 1/29/2001 -0800, Douglas Otis wrote: > >Michael, > > > >Your highest and second highest preference represents a change to the TCP > >protocol in that it mandates segment alignment of the PDU with > the Ethernet > >frame. > > This is not a change to the TCP protocol - strictly > implementation specific. > > >The impact of changing TCP into a datagram protocol may not be well > >received. > > It is still byte streaming but iSCSI PDUs don't span multiple > segments. This is an application-specific issue, i.e. it is the unit of > how one sends data and has nothing to do with the TCP protocol > itself. This really isn't any different than what applications > do today - > a peek() to determine how to cast the byte stream into a message > - which is > very common for most sockets applications. Are you saying the network adapter stack becomes starved to the point of only seeing one potential Ethernet frame at a time? > > If you wish to see two frame checks, one for the iSCSI header and one > > again for the SCSI payload, I would suggest that there are > variables used > > as handshakes within the iSCSI > >header that could use even a simpler scheme than a 16 bit CRC, a > 32 bit XOR. > >If done in software, both the TCP checksum and this header checksum will > >require updating. A single CRC field will represent a time constraint in > >that a pipeline delay of iSCSI feedback will exist. Allowing these > >calculations to take place prior to being placed for delivery within an > >interrupt routine is the advantage of allowing a simple feedback update. > > Disagree with your recommendations and analysis. There is no change > required for the TCP checksum this proposal is strictly above the TCP > layer, i.e. iSCSI. Perhaps I should have not also included feedback on your suggestion of splitting of headers in conjunction with objecting to your datagram proposal. I was not saying to change TCP, only that an error checking supplement for a small handshake field could be kept separate from the static SCSI payload. A simple check scheme may be effective if applied over a few words of feedback. Perhaps a SCSI interface driver encapsulates SCSI payload in a pipeline fashion. As the stack is ready, feedback is added to the constructed frame near when the TCP is checksum is updated. A natural place to divide these headers checks would be with respect to dynamic feedback. The rest of the payload may have already had CRC and partial checksums calculated in place waiting for placement on the network. If these segments are prepared within an interrupt routine, the overhead to include feedback is important. A handful of instructions to calculate CRC could be replace by a single instruction as in the MTF standard. Wanting to indicate a possible consideration of where to delineate a header split to include the static portion of the header together with data as one check and the dynamic portions of the header as another. Doug > Mike > > From owner-ips@ece.cmu.edu Tue Jan 30 01:09:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA18670 for ; Tue, 30 Jan 2001 01:09:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U3oDq00219 for ips-outgoing; Mon, 29 Jan 2001 22:50:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from plan9.cs.bell-labs.com (plan9.bell-labs.com [204.178.31.2]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f0U3noZ00204 for ; Mon, 29 Jan 2001 22:49:50 -0500 (EST) Received: from bell-labs.com ([135.104.50.40]) by plan9; Mon Jan 29 22:49:32 EST 2001 Message-ID: <3A763A5B.B11ADDA6@bell-labs.com> Date: Mon, 29 Jan 2001 22:51:55 -0500 From: Sean Quinlan X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Michael Krause CC: ips@ece.cmu.edu Subject: Re: iSCSI Data Integrity - Digests References: <5.0.0.25.2.20010126165115.00a73a80@hpindlm.cup.hp.com> <5.0.0.25.2.20010129124010.023ce6d0@hpindlm.cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I would suggest that if iSCSI requires that a PDU does not span a TCP segment then iSCSI is indeed changing TCP. A valid implementations of TCP has considerable freedom in deciding how to pack data into segments and this is not under the control of the application layer. Such a requirement in iSCSI would mean it is not feasible to implement iSCSI on many existing TCP stacks. If you change the requirements of for a valid TCP implementation I believe you have effectively changed the protocol. I agree with Douglas Otis that such a change may not be well received. What would you suggest for situations such a iSCSI over a protocol such as TLS(SSL). TLS uses an internal framing mechanism that is not under the applications control and is also not related to the underlying TCP segments. It would seem to me a little unwise to mandate end-to-end data integrity within iSCSI especially using a CRC algorithm. Rather a lot of data goes over protocols such as HTTP, FTP, NFS and CIFS without any additional end-to-end data integrity over what is provided by TCP and link level integrity - I think it is a little strong to say strong end-to-end data integrity is not an option as customers will not adopt solutions without such guarantees. If TLS or IPsec are used in conjunction with iSCSI, then strong end-to-end integrity is already provided and duplicating this in iSCSI seem rather a waste. It may be the case that some people are not satisfied with the integrity provided by TCP and yet do not want to use TLS or IPsec to ensure integrity/authenticity, but I suspect that such people are not the majority and thus mandating a data integrity mechanism within iSCSI seems like a mistake - perhaps as an option or perhaps leave it to other layers in the protocol stack. If data integrity is provided by iSCSI, CRC algorithms are not particularly ideal from a software implementation point of view - surely there are better alternatives than CRC that are easy to implement in hardware and can take advantage of 32bit wide data ops. Michael Krause wrote: > > At 10:13 AM 1/29/2001 -0800, Douglas Otis wrote: > >Michael, > > > >Your highest and second highest preference represents a change to the TCP > >protocol in that it mandates segment alignment of the PDU with the Ethernet > >frame. > > This is not a change to the TCP protocol - strictly implementation specific. > > >The impact of changing TCP into a datagram protocol may not be well > >received. > > It is still byte streaming but iSCSI PDUs don't span multiple > segments. This is an application-specific issue, i.e. it is the unit of > how one sends data and has nothing to do with the TCP protocol > itself. This really isn't any different than what applications do today - > a peek() to determine how to cast the byte stream into a message - which is > very common for most sockets applications. > > > If you wish to see two frame checks, one for the iSCSI header and one > > again for the SCSI payload, I would suggest that there are variables used > > as handshakes within the iSCSI > >header that could use even a simpler scheme than a 16 bit CRC, a 32 bit XOR. > >If done in software, both the TCP checksum and this header checksum will > >require updating. A single CRC field will represent a time constraint in > >that a pipeline delay of iSCSI feedback will exist. Allowing these > >calculations to take place prior to being placed for delivery within an > >interrupt routine is the advantage of allowing a simple feedback update. > > Disagree with your recommendations and analysis. There is no change > required for the TCP checksum this proposal is strictly above the TCP > layer, i.e. iSCSI. > > Mike From owner-ips@ece.cmu.edu Tue Jan 30 01:45:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA23304 for ; Tue, 30 Jan 2001 01:45:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U4QFK01302 for ips-outgoing; Mon, 29 Jan 2001 23:26:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0U4Q2Z01294 for ; Mon, 29 Jan 2001 23:26:02 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 4146D106D for ; Mon, 29 Jan 2001 20:26:01 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id UAA28557 for ; Mon, 29 Jan 2001 20:25:56 -0800 (PST) Message-ID: <3A7642F2.3CC5C079@cup.hp.com> Date: Mon, 29 Jan 2001 20:28:35 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI : Digest Error recovery causes data corruption Content-Type: multipart/mixed; boundary="------------9083715174E84FD9DEE9EBE2" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------9083715174E84FD9DEE9EBE2 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian & All, Section 5.5 on digest errors states that an initiator MUST "discard and re-start" a task when it encounters a header or data digest error, provided it can recognize the initiator task tag. I assume the above reference to re-start is to the use of the "retry" bit. (?) If so, there is a possibility of this error recovery mechanism leading to data corruption. The probability is reduced with the removal of partial data recovery (based on DataRN/SN). However, data corruption can still occur as follows : - Last Data PDU on a READ I/O is returned from target to initiator. - Initiator detects a header or data digest error on this last Data PDU and discards the PDU. - Initiator re-starts the task (using the "retry" bit). - Target sends in the Status PDU on the previous instance of the command. (It is not clear from the spec what the initiator does with stale frames that continue to arrive on the previous instance of the I/O. For now, I assume the initiator will, by some mechanism discard such frames.) - When the target receives the "retry" of this command, it thinks it has sent back all the data and so, it only sends back Status for this "retry". - Initiator has no count based checks and so, depends (trusts !) the target with its status, based on which it reports a successful I/O completion to the initiator's SCSI ULP, [indicating no residual count, since target thought it sent all the data]. - SCSI ULP assumes a completed I/O and notifies application, [since it depends on the initiator notifying it with an appropriate service response on an underflow, which the initiator in this case did not detect]. - Application encounters data corruption, due to the missing Data PDU which was discarded by the initiator on a digest error, and which was never re-sent by the target, since it does partial recovery by only sending the status. The StatSN based partial status recovery can lead to such dangerous corner cases causing possible data corruption scenarios. Regards, Santosh --------------9083715174E84FD9DEE9EBE2 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------9083715174E84FD9DEE9EBE2-- From owner-ips@ece.cmu.edu Tue Jan 30 01:46:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA23421 for ; Tue, 30 Jan 2001 01:46:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U4sEi02096 for ips-outgoing; Mon, 29 Jan 2001 23:54:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0U4rlZ02084 for ; Mon, 29 Jan 2001 23:53:47 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id XAA114258 for ; Mon, 29 Jan 2001 23:46:54 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.95) with ESMTP id VAA76140 for ; Mon, 29 Jan 2001 21:53:45 -0700 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI: INQUIRY page 0x83 identifier To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Mon, 29 Jan 2001 20:48:47 -0800 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.6 |December 14, 2000) at 01/29/2001 09:53:45 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, I think we have strayed from iSCSIness. The things you are talking about here is best handled with T10, I think. As the attached note from Jim Hafner, below states "I'd like to keep iSCSI from specifying any requirements on logical units (that's not the right space)". It was an interesting seg-way getting into this, however, it does not apply to the SCSI Transport called iSCSI. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Internet address: hufferd@us.ibm.com Jim Hafner/Almaden/IBM@IBMUS@ece.cmu.edu on 01/29/2001 09:08:44 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu, Robert Snively cc: Subject: RE: iSCSI: INQUIRY page 0x83 identifier Bob, I strongly agree with the requirement that these identifiers be WWUnique. But I also agree with Charles that whatever identifier that is presented should be the same regardless of the interconnect, so binding them to FC or to some defined iSCSI mechanism seems inappropriate. So, I think I'm agreeing the Rob Elliot. Namely, specify a 128 format rooted in EUI-64 for the device. Back away from anything FC-specific in the format of the EUI-64. A vendor of a storage controller will probably have an EUI-64 for the device, though that might not be FC-based. Use this as the basis for some vendor-specific algorithm to generate the LU identifier. Leave it to the vendor to define an algorithm that will provide the uniqueness. Also, I'd like to keep iSCSI from specifying any requirements on logical units (that's not the right space). SPC-2/3 can spec a general format and the requirement for uniqueness. Then leave it to the vendors to implement correctly. Does that sound reasonable? Jim Hafner Robert Snively on 01-26-2001 01:30:43 PM To: Jim Hafner/Almaden/IBM@IBMUS, ips@ece.cmu.edu cc: Subject: RE: iSCSI: INQUIRY page 0x83 identifier > As for an iSCSI type or format for logical unit identifiers: > 1) SCSI already allows for an ascii formatted identifier (typically this > includes vendor, model, serial number). Since NDT has defined WWUIs for > DEVICES and these have a defined UTF-8 format, there is nothing to > *prevent* a vendor from using these strings as the starting point for LU > page 83 identifiers of the logical units within that device. > 2) But..., iSCSI and NDT should not attempt to make a rule under which > these are used. FC's intervention in this was more as a strong suggestion, > and is certainly not a requirement. Jim, I generally agree with the text of your comments, but there are a couple of details I would like to address. Any name not based on a registered entity and in a registered format is not guaranteed to be world-wide unique, and is therefore inappropriate. That is why numbers like OUIs and SCSI Vendor IDs in specified parts of the identifier are a requirement. EUI-64 and FC identifiers have that property. FC identifiers have the added benefit of having a regular extension capability for dynamic creation of logical units. As you point out, iSCSI need not use FC identifiers, but a bunch of people like them for their constant defined length and guaranteed world-wide unique identifiers. However, I feel that iSCSI should be a lot more hard nosed than SAM-2 and require that VPD page "83" contain a mandatory world-wide unique logical unit identifier in an appropriate invariant format. For compatibility with other SCSI drivers, it should be limited to 128 bits of length. This would not be an identifier set by the user (there are other SCSI identifiers that can be set by the user), but one that is invariant from the time of manufacture (for physical devices) or creation (for logical devices) of the logical unit. From owner-ips@ece.cmu.edu Tue Jan 30 02:39:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA01810 for ; Tue, 30 Jan 2001 02:39:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U62Gj04074 for ips-outgoing; Tue, 30 Jan 2001 01:02:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0U61kZ04058 for ; Tue, 30 Jan 2001 01:01:46 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id D5C573CF for ; Mon, 29 Jan 2001 23:01:45 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id F27291EA; Tue, 30 Jan 2001 01:01:44 -0500 (EST) Received: from agilent.com (cos1nai135105.cos.agilent.com [141.184.135.105]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id WAA11159; Mon, 29 Jan 2001 22:01:42 -0800 (PST) Message-ID: <3A7658C5.940929CE@agilent.com> Date: Mon, 29 Jan 2001 22:01:41 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Michael Krause , IPS Reflector Subject: Re: iSCSI Data Integrity - Digests References: <5.0.0.25.2.20010126165115.00a73a80@hpindlm.cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Michael Krause wrote: > > The following describes three alternatives for iSCSI data integrity. > Alternative 1 (Highest preference): > > * An iSCSI PDU shall be restricted to a single TCP segment. Multiple > iSCSI PDU may be present within the same TCP segment but none shall > span multiple segments. > Alternative 2 (middle preference) > > * An iSCSI PDU shall be restricted to a single TCP segment. Multiple > iSCSI PDU may be present within the same TCP segment but none shall > span multiple segments. Mike, While your suggestions do not require a change to the way TCP looks on the wire, it requires a change in the TCP API. You are going down a rat hole that we've already attempted, and were repudiated by the TCP gods. -Matt From owner-ips@ece.cmu.edu Tue Jan 30 02:39:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA01821 for ; Tue, 30 Jan 2001 02:39:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U5lHC03675 for ips-outgoing; Tue, 30 Jan 2001 00:47:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0U5kPZ03655 for ; Tue, 30 Jan 2001 00:46:25 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id EE12B751 for ; Mon, 29 Jan 2001 22:46:24 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 269AA17D; Tue, 30 Jan 2001 00:46:24 -0500 (EST) Received: from agilent.com (cos1nai135105.cos.agilent.com [141.184.135.105]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id VAA09654; Mon, 29 Jan 2001 21:46:21 -0800 (PST) Message-ID: <3A76552C.2388C23C@agilent.com> Date: Mon, 29 Jan 2001 21:46:20 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Michael Krause , ips@ece.cmu.edu Subject: Re: new iSCSI PDU outline References: <5.0.0.25.2.20010129100829.022edfe0@hpindlm.cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mike, If an iSCSI implementation is put together, without using routers or other middle boxes that muck with the TCP checksum, then the TCP checksum should be adequate. The first implementations of iSCSI are going to be software based. In these early software based implementations, performance could probably be increased if CRC was disabled. -Matt Michael Krause wrote: > > At 06:37 PM 1/27/2001 +0200, julian_satran@il.ibm.com wrote: > > >Doug, > > > >CRC is not mandatory. If you are using a local net (real time) on a > >reliable media you probably don't need anything. > > Unclear how iSCSI will be able to determine whether the target is local or > not given IP subnets can already span large geographic distances and the > movement to 10 GbE translates to large Ethernet fabrics. Also as > transparent fail-over is delivered by multiple vendors, what started out as > local may become remote in some way. > > I strongly disagree with making CRC protection optional and I suspect > customers will given the various studies on error rate escapes associated > with TCP checksum already discussed on this reflector. > > Mike From owner-ips@ece.cmu.edu Tue Jan 30 02:40:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA01836 for ; Tue, 30 Jan 2001 02:40:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U5dG003448 for ips-outgoing; Tue, 30 Jan 2001 00:39:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0U5cbZ03428 for ; Tue, 30 Jan 2001 00:38:37 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id BA6F26F8 for ; Mon, 29 Jan 2001 22:38:36 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 029271AF for ; Tue, 30 Jan 2001 00:38:35 -0500 (EST) Received: from agilent.com (cos1nai135105.cos.agilent.com [141.184.135.105]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id VAA08930 for ; Mon, 29 Jan 2001 21:38:33 -0800 (PST) Message-ID: <3A765358.9D4A23F@agilent.com> Date: Mon, 29 Jan 2001 21:38:32 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > > Santosh, > > The trouble with forbidding a certain behavior is that you have to enforce > it (i.e., check and signal errors for units that do not behave). No you don't have to "enforce" it at all. Using Santosh's example, if a target broke the rules and performed data overlay, then the initiator will always mark the I/O as bad, and the market forces will take over (customer will buy a compliant target). Besides > - the whole philosophy of the SCSI set of protocols is that the target is > the master and the initiator should let the target decide how to fulfill > the command. That is why we chose not to impose restrictions above those > imposed by SCSI. The whole set of issues is also raised only because we > provide also for storage proxies - otherwise a stronger checksum at TCP > level and recovery at TCP level would have done what we wanted and recovery > of the type we are dealing now with would have been done at TCP level. > I am confident that we can reinstate DataSN a simple mean to sequence (not > ack) data packets and considerably simplify recovery. If there is a data digest failure, the iSCSI PDU is discarded, and the test that Santosh describes will fail. At that point, the command is "retried", and using my example of the retry implementation in the thread "iSCSI: I/O (command) recovery" error recovery is performed. No need for DataSN... > And do not forget that raising the error up to ULP with a service response > will make the recovery far more expensive (as Prasenjit has already stated) > - far more than current wedge drivers do as these rarely consider commands > in flight and the need to keep order in a target that is not yet aware that > something went wrong. I agree that erorrs due to the transport should not be propagated to the ULP. In the case of a digest failure, this means that the TCP checksum indicated the segment was good, meaning that a middle box corrupted the TCP segment and sent it out with a "fixed" TCP checksum. The simple iSCSI error recovery using the retry should handle this corner case very well. -Matt > > Julo > > Santosh Rao on 28/01/2001 00:07:08 > > Please respond to Santosh Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues > > Julian, > > The missing Data PDU could be detected if the initiator were to > perform a count check operation upon receiving SCSI Response PDU, > along the lines of : > > no. of bytes xfer'ed = > (Expected Data Xfer Length) - (Basic Residual Count) > > where, > Expected Data Xfer Length -> as specified in SCSI Command PDU > Basic Residual Count -> as specified in SCSI Response PDU > > However, this is currently not possible due to overlapped data > transfers being allowed by iSCSI. If iSCSI were to dis-allow > overlapping data xfer's and initiators used a count check > [as is done in FC], this would also address the problem. > > Regards, > Santosh > > > > > > > > > If the header is a data header we can hardly trust the ULP to recognize > the > > error (he might be unaware > > of a missing packet). With data numbering this situation could have been > > discovered at "status time". > > The only thing we could do is restart all commands but this is equivalent > > to a connection restart for all practical purposes. Dropping data > > numbering might have some more "side-effects" like this. > > As the combination of values - tag, address, offset may stil let some > > implementations to assume that they have > > a correct task identifier I don't see a point in mandating a recovery > > behavior and the implementer may choose to: > > > > -retry/restart command > > -logout drop and rebuild connection login and restart/retry > > -abort all task sets (practically reset the target!) and report for all > > commands a "delivery system failure" (kick-in the ULP recovery) and if > you > > suspect the link quality rebuild it; this later behavior means also that > > you have to stop delivering anything on any link to the target to avoid > > out of order execution until you have finished the cleanup - pretty > drastic > > > > With data numbering recovery could have stayed within the confines of a > > command even if a header was bad. > > Perhaps we should leave the DataSN only as a sequencer so that at > > status-time the initiator should be able to find if a data packet was > > dropped (no ExpDataSN on a NOP). > > > > Regards, > > Julo > > > > > > > > > > Michael Krause on 27/01/2001 04:59:12 > > > > Please respond to Michael Krause > > > > To: Julian Satran/Haifa/IBM@IBMIL > > cc: ips@ece.cmu.edu > > Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window > issues > > > > > > > > > > At 07:40 PM 1/25/2001 +0200, julian_satran@il.ibm.com wrote: > > > > > > >1) The initiator task tag cannot be trusted when a header digest error > > >is seen. What does the phrase "provided it can recognize the initiator > > >task tag" mean ? > > >How can an initiator reliably claim that the initiator task tag is > > >trustworthy ? > > > > > > an initiator may choose to provide some redundancy in the tag > itself > > > > > > > I'm aware of some techniques for inserting redundant information in tags > > which limits the potential error exposure when a multi-bit error occurs, > > however these are not fail-safe leading to potential incorrect operation > - > > perhaps benign in many cases; perhaps not in others. As such, if a header > > digest error occurs, the PDU should be silently discarded and recovery > > should be left to the ULP. There is little to no value having two > > mechanisms to solve the same problem. > > > > Mike > > > > > > > > > > > > -- > ################################# > Santosh Rao > Software Design Engineer, > HP, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################# From owner-ips@ece.cmu.edu Tue Jan 30 02:42:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA01862 for ; Tue, 30 Jan 2001 02:42:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U6MH404581 for ips-outgoing; Tue, 30 Jan 2001 01:22:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0U6LsZ04569 for ; Tue, 30 Jan 2001 01:21:54 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id E58AB5A0; Mon, 29 Jan 2001 23:21:53 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id B87391E3; Tue, 30 Jan 2001 01:18:27 -0500 (EST) Received: from agilent.com (cos1nai135105.cos.agilent.com [141.184.135.105]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id WAA12701; Mon, 29 Jan 2001 22:18:25 -0800 (PST) Message-ID: <3A765CB0.99483D2D@agilent.com> Date: Mon, 29 Jan 2001 22:18:24 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: David Black , ips@ece.cmu.edu Subject: Re: I-D ACTION:draft-ietf-ips-iscsi-disc-reqts-00.txt References: <200101291144.GAA29115@ietf.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Well, this is interesting: At http://www.ietf.org/internet-drafts/ there are the following two files: draft-ietf-ips-iscsi-disc-reqts-00.txt 26-Jan-2001 08:08 46k draft-ietf-ips-iscsi-disc-reqts-01.txt 12-Jan-2001 13:56 44k Notice how 01 predates 00? The later was really but I suspect the name was too long so it was changed. David, can you cause this to be fixed? -Matt Internet-Drafts@ietf.org wrote: > > A New Internet-Draft is available from the on-line Internet-Drafts directories. > This draft is a work item of the IP Storage Working Group of the IETF. > > Title : iSCSI Naming and Discovery Requirements > Author(s) : M. Bakke et al. > Filename : draft-ietf-ips-iscsi-disc-reqts-00.txt > Pages : 18 > Date : 26-Jan-01 > > This document describes the iSCSI [7] naming and discovery > requirements. The requirements presented in this document have been > agreed to by the members of the iSCSI naming and discovery team. This > document complements the iSCSI IETF draft. Flexibility is the key > guiding principle behind this requirements document. That is, an > effort has been made to satisfy the needs of both small isolated > environments, as well as large environments requiring secure/scalable > solutions. > > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-disc-reqts-00.txt > > Internet-Drafts are also available by anonymous FTP. Login with the username > "anonymous" and a password of your e-mail address. After logging in, > type "cd internet-drafts" and then > "get draft-ietf-ips-iscsi-disc-reqts-00.txt". > > A list of Internet-Drafts directories can be found in > http://www.ietf.org/shadow.html > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > > Internet-Drafts can also be obtained by e-mail. > > Send a message to: > mailserv@ietf.org. > In the body type: > "FILE /internet-drafts/draft-ietf-ips-iscsi-disc-reqts-00.txt". > > NOTE: The mail server at ietf.org can return the document in > MIME-encoded form by using the "mpack" utility. To use this > feature, insert the command "ENCODING mime" before the "FILE" > command. To decode the response(s), you will need "munpack" or > a MIME-compliant mail reader. Different MIME-compliant mail readers > exhibit different behavior, especially when dealing with > "multipart" MIME messages (i.e. documents which have been split > up into multiple messages), so check your local documentation on > how to manipulate these messages. > > > Below is the data which will enable a MIME compliant mail reader > implementation to automatically retrieve the ASCII version of the > Internet-Draft. > > ------------------------------------------------------------------------------ > Content-Type: text/plain > Content-ID: <20010126081949.I-D@ietf.org> From owner-ips@ece.cmu.edu Tue Jan 30 04:22:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA02429 for ; Tue, 30 Jan 2001 04:22:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U75IZ05745 for ips-outgoing; Tue, 30 Jan 2001 02:05:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0U752Z05728 for ; Tue, 30 Jan 2001 02:05:02 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id A4C63A0D for ; Tue, 30 Jan 2001 00:05:01 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id EE366183 for ; Tue, 30 Jan 2001 02:05:00 -0500 (EST) Received: from agilent.com (cos1nai135105.cos.agilent.com [141.184.135.105]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id XAA16427 for ; Mon, 29 Jan 2001 23:04:58 -0800 (PST) Message-ID: <3A76679A.7FDB8C04@agilent.com> Date: Mon, 29 Jan 2001 23:04:58 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Command Ordering Proposal. References: <3A722ABD.AE763CA2@cup.hp.com> <3A75DA22.833FA40C@agilent.com> <3A75E2CE.A399174D@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > > Matt Wakeley wrote: > > > Retried commands should have already been queued and/or executed. > > Not necessarily. A TCP connection failure can affect the command PDU. In which case the target will have "silently discarded" the PDU. Then, it will not acknowledge via ExpCmdSN any commands past the err'd command, so the initiator can resend the command with the same CmdSN and enabling command delivery to SCSI to continue. > Similarly, the target may have detected a header or data digest error > on the SCSI Command PDU, sent a REJECT I thought any digest errors are silently discarded - therefore, there is no reject. Nevertheless, the ExpCmdSN is not incremented (as stated above), and the initiator can then resend that command with the same CmdSN. -Matt > and the initiator would have "discarded and retried" the task. > > In both of the above cases, the command is not queued and/or being executed > at the target. > > Such commands, when retried, need to enforce ordering. Hence, the need to > specify CmdSN in the "retry", if the CmdSN is yet to be acknowledged. > The difficulty lies in the establishment of a "sync point" with the ExpCmdSN, > prior to the initiation of these "retry" commands, so as to ensure that the "retry" > will not be discarded by the target, because the CmdSN slid behind the > (ExpCmdSN, MaxCmdSN) window. > > The ExpCmdSN in the login response provides this "sync point" during connection > failure recovery. The problem is only with digest error "retries". > > If one were to NOT apply the "discard and retry" policy on a header digest error, > [under the assumption that the I.T.T cannot be trusted on a header digest error], > the problem further reduces to the following cases : > > a) data digest error detected by a target on a command PDU. (immediate data). > b) data digest error detected by an initiator on inbound Data PDUs of a READ > I/O, and the CmdSN for the command is yet to be acknowledged, but an ACK > may be in-flight behind the Data PDU. > > Regards, > Santosh > > > The goal of > > the retry is simply to replay back to the initiator the results of the > > execution. Therefore, there is no ordering requirement for "retried" commands > > and CmdSN should always be zero. > > > > -Matt > > > > Santosh Rao wrote: > > > > > > > > Accepting a command outside the (ExpCmdSN, MaxCmdSN) window with the retry > > > bit set then opens up windows for other types of stale and duplicate commands > > > to be > > > processed. > > > > > > Here's the deal (as I view it) : > > > > > > Commands are sent with the "retry" bit under 2 conditions : > > > - Connection Failures > > > - Digest Errors > > > > > > In both these cases, an explicit handshake is required regarding the > > > (ExpCmdSN, MaxCmdSN) window, prior to starting to send "retry" > > > commands. This ensures that genuine "retry" commands do not slip > > > behind the (ExpCmdSN, MaxCmdSN) window. > > > > > > In the case of a connection failure, the Logout Response provides this > > > handshake. For digest errors, there is no such hand-shake. > > > Hence, a command that encountered a digest error and was retried could > > > likely slip behind the (ExpCmdSN, MaxCmdSN) window and then be never > > > processed. > > > > > > Regards, > > > Santosh From owner-ips@ece.cmu.edu Tue Jan 30 04:22:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA02442 for ; Tue, 30 Jan 2001 04:22:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U79IO05901 for ips-outgoing; Tue, 30 Jan 2001 02:09:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0U78vZ05885 for ; Tue, 30 Jan 2001 02:08:57 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 359429F7 for ; Tue, 30 Jan 2001 00:08:57 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id D23F717F for ; Tue, 30 Jan 2001 02:08:25 -0500 (EST) Received: from agilent.com (cos1nai135105.cos.agilent.com [141.184.135.105]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id XAA16665 for ; Mon, 29 Jan 2001 23:08:23 -0800 (PST) Message-ID: <3A766866.53684AB8@agilent.com> Date: Mon, 29 Jan 2001 23:08:22 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Command Ordering Proposal. References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Douglas Otis wrote: > > Matt, > > Whether you use the CmdSN or the tag to associate the retry, the CmdSN is > still used as a means to handshake freeing of resources and not the tag. You are wrong. It specifically states in the document in 1.2.2.1: "CmdRNs are significant only during command delivery to the target. Once the device serving part of the target SCSI has received a command, CmdRN ceases to be significant." Any resources the target allocates must be based on the task tag. -Matt > It > seems like a cleaner solution not to include exceptions on maintaining the > sequence just to allow this type of non-sequential use. Clearly there is an > alternative that has benefits. Simply have the target allow exceptions based > on the command or task attribute and not drop the ability to handshake. I > am aware of the present concept, but you did not speak about any downside to > this alternative. > > Doug > > > Retried commands should have already been queued and/or executed. > > The goal of > > the retry is simply to replay back to the initiator the results of the > > execution. Therefore, there is no ordering requirement for > > "retried" commands > > and CmdSN should always be zero. > > > > -Matt > > > > Santosh Rao wrote: > > > > > > Douglas Otis wrote: > > > > > > > Santosh, > > > > > > > > If there is a task attribute, the complexity is when you > > allow CmdSN = 0; > > > > > > > > Perhaps CmdSN = 0 should not be allowed or treated as just > > another in the > > > > sequence. > > > > > > The (CmdSN==0) does have its uses. One example is the use of NOP-OUT for > > > a "connection ping." > > > > > > > If a retry flag is set, the CmdSN below the > > > > ExpCmdSN should be viewed as an exception much as Head of > > Queue should be an > > > > exception for MaxCmdSN. Both of these cases imply > > exceptional behavior by > > > > the initiator in response to a problem. > > > > > > Accepting a command outside the (ExpCmdSN, MaxCmdSN) window > > with the retry > > > bit set then opens up windows for other types of stale and > > duplicate commands > > > to be > > > processed. > > > > > > Here's the deal (as I view it) : > > > > > > Commands are sent with the "retry" bit under 2 conditions : > > > - Connection Failures > > > - Digest Errors > > > > > > In both these cases, an explicit handshake is required regarding the > > > (ExpCmdSN, MaxCmdSN) window, prior to starting to send "retry" > > > commands. This ensures that genuine "retry" commands do not slip > > > behind the (ExpCmdSN, MaxCmdSN) window. > > > > > > In the case of a connection failure, the Logout Response provides this > > > handshake. For digest errors, there is no such hand-shake. > > > Hence, a command that encountered a digest error and was retried could > > > likely slip behind the (ExpCmdSN, MaxCmdSN) window and then be never > > > processed. > > > > > > Regards, > > > Santosh > > From owner-ips@ece.cmu.edu Tue Jan 30 04:25:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA02473 for ; Tue, 30 Jan 2001 04:25:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U8CKF07629 for ips-outgoing; Tue, 30 Jan 2001 03:12:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0U8BlZ07617 for ; Tue, 30 Jan 2001 03:11:48 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id ECEC5D26 for ; Tue, 30 Jan 2001 00:11:46 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id AAA10230 for ips@ece.cmu.edu; Tue, 30 Jan 2001 00:11:42 -0800 (PST) From: Santosh Rao Message-Id: <200101300811.AAA10230@hpcuhe.cup.hp.com> Subject: Re: iSCSI : Command Ordering Proposal. To: ips@ece.cmu.edu (ips) Date: Tue, 30 Jan 2001 00:11:42 -0800 (PST) In-Reply-To: <3A766866.53684AB8@agilent.com> from "Matt Wakeley" at Jan 29, 2001 11:08:22 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Doug, I'm not sure if we are speaking the same issue here. Can you please elaborate further on your concerns. The handshake being discussed by me does NOT have to do with freeing resources. It is a handshake b/n the target & initiator on the CmdSN/ExpCmdSN, so that "retry" commands from initiator to target remain with the (ExpCmdSN, MaxCmdSN) window. Resources at the target end are de-allocated when it receives an ExpStatSN for the StatSN. (IOW, resource de-allocation has to do with StatSN ACKs from the initiator, not CmdSN). If a target were not to implement status recovery, resource de-allocation can occur once a TCP ACK is received for the SCSI Response PDU. (if a mechanism were to exist to identify that all the octets that constitute the Status PDU have been ACK'ed.) This also brings up an interesting issue which is : If the target needs to hold onto its status PDU until at least the TCP ACK is received for all of the bytes that constitute the PDU and there is no easy mechanism to associate b/n a TCP ACK and all bytes of the Status PDU being ACK'ed, the simplest approach becomes de-allocation of status & I/O resources based on ExpStatSN. If that is the case, the spec should change "targets MAY implement status recovery" to "targets MUST hold onto status PDU until ExpStatSN" update is received. Regards, Santosh > > Douglas Otis wrote: > > > > Matt, > > > > Whether you use the CmdSN or the tag to associate the retry, the CmdSN is > > still used as a means to handshake freeing of resources and not the tag. > > You are wrong. It specifically states in the document in 1.2.2.1: > > "CmdRNs are significant only during command delivery to the target. > Once the device serving part of the target SCSI has received a > command, CmdRN ceases to be significant." > > Any resources the target allocates must be based on the task tag. > > -Matt > > > > > It > > seems like a cleaner solution not to include exceptions on maintaining the > > sequence just to allow this type of non-sequential use. Clearly there is an > > alternative that has benefits. Simply have the target allow exceptions based > > on the command or task attribute and not drop the ability to handshake. I > > am aware of the present concept, but you did not speak about any downside to > > this alternative. > > > > Doug > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Tue Jan 30 04:27:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA02484 for ; Tue, 30 Jan 2001 04:27:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U8QKF07985 for ips-outgoing; Tue, 30 Jan 2001 03:26:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0U8PqZ07972 for ; Tue, 30 Jan 2001 03:25:52 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 7D553244E for ; Tue, 30 Jan 2001 00:25:50 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id AAA10849 for ips@ece.cmu.edu; Tue, 30 Jan 2001 00:25:45 -0800 (PST) From: Santosh Rao Message-Id: <200101300825.AAA10849@hpcuhe.cup.hp.com> Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues To: ips@ece.cmu.edu (ips) Date: Tue, 30 Jan 2001 00:25:45 -0800 (PST) In-Reply-To: <3A765358.9D4A23F@agilent.com> from "Matt Wakeley" at Jan 29, 2001 09:38:32 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > julian_satran@il.ibm.com wrote: > > > > Santosh, > > > > The trouble with forbidding a certain behavior is that you have to enforce > > it (i.e., check and signal errors for units that do not behave). > > No you don't have to "enforce" it at all. Using Santosh's example, if a > target broke the rules and performed data overlay, then the initiator will > always mark the I/O as bad, and the market forces will take over (customer > will buy a compliant target). Julian, Removing support for overlapped data xfer's has multiple benefits: 1) It provides initiators a reliable way of ensuring that the I/O did complete without any underrun. 2) It simplifies SCSI Assist implementations that no longer need to deal with overlapped data xfer conditions. 3) It is simpler than performing book-keeping on DataSN to ensure that all DataSNs have been received. (IOW, score-boarding at a DataSN level, instead of at a byte level.) I'd say the count based solution is preferrable, given the above. All protocols inherently enforce certain behaviour by mandating features (the use of MUST, shall). I don't think that's a strong enough reason to reject this proposal. To summarize : o Dis-allow overlapped data xfer's. o Initiators to perform a count check as is done in FC. o On detecting an underrun, the command may be retried BUT WITHOUT SETTING the "retry" bit. This is particularly important because targets that implement status recovery may be ignorant of the fact that the initiator encountered a digest error [which caused the underrun] and so, they just send back a Status PDU under the belief that the command is complete, whereas, the initiator wants all the Data PDUs to be re-sent. Regards, Santosh > > Besides > > - the whole philosophy of the SCSI set of protocols is that the target is > > the master and the initiator should let the target decide how to fulfill > > the command. That is why we chose not to impose restrictions above those > > imposed by SCSI. The whole set of issues is also raised only because we > > provide also for storage proxies - otherwise a stronger checksum at TCP > > level and recovery at TCP level would have done what we wanted and recovery > > of the type we are dealing now with would have been done at TCP level. > > I am confident that we can reinstate DataSN a simple mean to sequence (not > > ack) data packets and considerably simplify recovery. > > If there is a data digest failure, the iSCSI PDU is discarded, and the test > that Santosh describes will fail. At that point, the command is "retried", > and using my example of the retry implementation in the thread "iSCSI: I/O > (command) recovery" error recovery is performed. No need for DataSN... > > > And do not forget that raising the error up to ULP with a service response > > will make the recovery far more expensive (as Prasenjit has already stated) > > - far more than current wedge drivers do as these rarely consider commands > > in flight and the need to keep order in a target that is not yet aware that > > something went wrong. > > I agree that erorrs due to the transport should not be propagated to the ULP. > In the case of a digest failure, this means that the TCP checksum indicated > the segment was good, meaning that a middle box corrupted the TCP segment and > sent it out with a "fixed" TCP checksum. > > The simple iSCSI error recovery using the retry should handle this corner case > very well. > > -Matt > > > > > > Julo > > > > Santosh Rao on 28/01/2001 00:07:08 > > > > Please respond to Santosh Rao > > > > To: Julian Satran/Haifa/IBM@IBMIL > > cc: ips@ece.cmu.edu > > Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues > > > > Julian, > > > > The missing Data PDU could be detected if the initiator were to > > perform a count check operation upon receiving SCSI Response PDU, > > along the lines of : > > > > no. of bytes xfer'ed = > > (Expected Data Xfer Length) - (Basic Residual Count) > > > > where, > > Expected Data Xfer Length -> as specified in SCSI Command PDU > > Basic Residual Count -> as specified in SCSI Response PDU > > > > However, this is currently not possible due to overlapped data > > transfers being allowed by iSCSI. If iSCSI were to dis-allow > > overlapping data xfer's and initiators used a count check > > [as is done in FC], this would also address the problem. > > > > Regards, > > Santosh > > > > > > > > > > > > > > If the header is a data header we can hardly trust the ULP to recognize > > the > > > error (he might be unaware > > > of a missing packet). With data numbering this situation could have been > > > discovered at "status time". > > > The only thing we could do is restart all commands but this is equivalent > > > to a connection restart for all practical purposes. Dropping data > > > numbering might have some more "side-effects" like this. > > > As the combination of values - tag, address, offset may stil let some > > > implementations to assume that they have > > > a correct task identifier I don't see a point in mandating a recovery > > > behavior and the implementer may choose to: > > > > > > -retry/restart command > > > -logout drop and rebuild connection login and restart/retry > > > -abort all task sets (practically reset the target!) and report for all > > > commands a "delivery system failure" (kick-in the ULP recovery) and if > > you > > > suspect the link quality rebuild it; this later behavior means also that > > > you have to stop delivering anything on any link to the target to avoid > > > out of order execution until you have finished the cleanup - pretty > > drastic > > > > > > With data numbering recovery could have stayed within the confines of a > > > command even if a header was bad. > > > Perhaps we should leave the DataSN only as a sequencer so that at > > > status-time the initiator should be able to find if a data packet was > > > dropped (no ExpDataSN on a NOP). > > > > > > Regards, > > > Julo > > > > > > > > > > > > > > > Michael Krause on 27/01/2001 04:59:12 > > > > > > Please respond to Michael Krause > > > > > > To: Julian Satran/Haifa/IBM@IBMIL > > > cc: ips@ece.cmu.edu > > > Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window > > issues > > > > > > > > > > > > > > > At 07:40 PM 1/25/2001 +0200, julian_satran@il.ibm.com wrote: > > > > > > > > > >1) The initiator task tag cannot be trusted when a header digest error > > > >is seen. What does the phrase "provided it can recognize the initiator > > > >task tag" mean ? > > > >How can an initiator reliably claim that the initiator task tag is > > > >trustworthy ? > > > > > > > > an initiator may choose to provide some redundancy in the tag > > itself > > > > > > > > > > I'm aware of some techniques for inserting redundant information in tags > > > which limits the potential error exposure when a multi-bit error occurs, > > > however these are not fail-safe leading to potential incorrect operation > > - > > > perhaps benign in many cases; perhaps not in others. As such, if a header > > > digest error occurs, the PDU should be silently discarded and recovery > > > should be left to the ULP. There is little to no value having two > > > mechanisms to solve the same problem. > > > > > > Mike > > > > > > > > > > > > > > > > > > > -- > > ################################# > > Santosh Rao > > Software Design Engineer, > > HP, Cupertino. > > email : santoshr@cup.hp.com > > Phone : 408-447-3751 > > ################################# > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Tue Jan 30 04:28:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA02513 for ; Tue, 30 Jan 2001 04:28:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U8PKf07949 for ips-outgoing; Tue, 30 Jan 2001 03:25:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0U8OLZ07915 for ; Tue, 30 Jan 2001 03:24:21 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id B6BEC46C for ; Tue, 30 Jan 2001 01:24:20 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 1179B17F for ; Tue, 30 Jan 2001 03:24:12 -0500 (EST) Received: from agilent.com (cos1nai135105.cos.agilent.com [141.184.135.105]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id AAA21995 for ; Tue, 30 Jan 2001 00:24:09 -0800 (PST) Message-ID: <3A767A28.A632BA23@agilent.com> Date: Tue, 30 Jan 2001 00:24:08 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI : Digest Error recovery causes data corruption References: <3A7642F2.3CC5C079@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > - When the target receives the "retry" of this command, it thinks it > has sent back all the data and so, it only sends back Status for > this "retry". The retry means, retry the whole thing, not just the status. The target must resend all the data, and the status. > The StatSN based partial status recovery can lead to such dangerous > corner cases causing possible data corruption scenarios. Where do you get this notion of "StatSN based partial status recovery"? > > Regards, > Santosh -Matt From owner-ips@ece.cmu.edu Tue Jan 30 04:32:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA02550 for ; Tue, 30 Jan 2001 04:32:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U7xKT07313 for ips-outgoing; Tue, 30 Jan 2001 02:59:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0U7wsZ07299 for ; Tue, 30 Jan 2001 02:58:54 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id E2C1B8D0; Mon, 29 Jan 2001 23:58:53 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id XAA09569; Mon, 29 Jan 2001 23:58:47 -0800 (PST) From: Santosh Rao Message-Id: <200101300758.XAA09569@hpcuhe.cup.hp.com> Subject: Re: iSCSI : Command Ordering Proposal. To: matt_wakeley@agilent.com Date: Mon, 29 Jan 2001 23:58:47 -0800 (PST) Cc: ips@ece.cmu.edu (IPS Reflector) In-Reply-To: <3A76679A.7FDB8C04@agilent.com> from "Matt Wakeley" at Jan 29, 2001 11:04:58 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt, Comments in-line. Regards, Santosh > > Santosh Rao wrote: > > > > Matt Wakeley wrote: > > > > > Retried commands should have already been queued and/or executed. > > > > Not necessarily. A TCP connection failure can affect the command PDU. > > In which case the target will have "silently discarded" the PDU. Then, it > will not acknowledge via ExpCmdSN any commands past the err'd command, so the > initiator can resend the command with the same CmdSN and enabling command > delivery to SCSI to continue. The above was stated to explain why "retry" commands need to use CmdSN [and cannot use a CmdSN of 0 as you were proposing]. The problem being originally described can occur when the initiator applies a "discard & retry" policy on detecting a digest error at its end. In such a case, it would "retry" the command, and if the CmdSN was not yet acknowledged, would send the "retry" with the original CmdSN. Since the target continues to send PDUs to the initiator, it can update the ExpCmdSN causing the "retry" to be discarded by the target, because its CmdSN has slid behind the window. The spec does not mandate that an update to ExpCmdSN MUST accompany the Data PDUs for a command. IOW, a target may lag behind in updates to ExpCmdSN while continuing to send Data PDUs for the received command. > > > Similarly, the target may have detected a header or data digest error > > on the SCSI Command PDU, sent a REJECT > > I thought any digest errors are silently discarded - therefore, there is no > reject. Not when targets detect a digest error. Section 5.5 mandates targets to send a REJECT with a reason of digest error. The "discard & retry" policy is for initiators. > > b) data digest error detected by an initiator on inbound Data PDUs of a READ > > I/O, and the CmdSN for the command is yet to be acknowledged, but an ACK > > may be in-flight behind the Data PDU. > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Tue Jan 30 05:12:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA02841 for ; Tue, 30 Jan 2001 05:12:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U8tLN08671 for ips-outgoing; Tue, 30 Jan 2001 03:55:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0U8t6Z08653 for ; Tue, 30 Jan 2001 03:55:06 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 7D8A6179 for ; Tue, 30 Jan 2001 00:55:05 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id AAA12269 for ips@ece.cmu.edu; Tue, 30 Jan 2001 00:54:59 -0800 (PST) From: Santosh Rao Message-Id: <200101300854.AAA12269@hpcuhe.cup.hp.com> Subject: Re: iSCSI : Digest Error recovery causes data corruption To: ips@ece.cmu.edu (ips) Date: Tue, 30 Jan 2001 00:54:58 -0800 (PST) In-Reply-To: <3A767A28.A632BA23@agilent.com> from "Matt Wakeley" at Jan 30, 2001 12:24:08 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > Where do you get this notion of "StatSN based partial status recovery"? > > -Matt From Section 5.1 : "-upon receiving the new/retry commands the target will resume command execution; ... <> data and for "terminated" commands retransmitting the ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Status & Sense while retaining the original StatRN." ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The above seems to indicate that StatSN did allow for re-transmission of just the status [and sense, if present]. Regards, Santosh ps : (Just the fact that 2 independent implementors could interpret "retry" handling at the target in such different ways goes to prove that the "retry" handling description in the draft needs a lot more clarity and a lot less ambiguity.) -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Tue Jan 30 05:13:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA02859 for ; Tue, 30 Jan 2001 05:13:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U8gLm08378 for ips-outgoing; Tue, 30 Jan 2001 03:42:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0U8fqZ08368 for ; Tue, 30 Jan 2001 03:41:53 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 66215463 for ; Tue, 30 Jan 2001 01:41:52 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id A294B196 for ; Tue, 30 Jan 2001 03:41:51 -0500 (EST) Received: from agilent.com (cos1nai135105.cos.agilent.com [141.184.135.105]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id AAA23709 for ; Tue, 30 Jan 2001 00:41:49 -0800 (PST) Message-ID: <3A767E4C.A1E14AC9@agilent.com> Date: Tue, 30 Jan 2001 00:41:48 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues References: <200101300825.AAA10849@hpcuhe.cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh Rao wrote: > > To summarize : > o Dis-allow overlapped data xfer's. > > o Initiators to perform a count check as is done in FC. > > o On detecting an underrun, the command may be retried > BUT WITHOUT SETTING the "retry" bit. This is > particularly important because targets that implement > status recovery may be ignorant of the fact that the > initiator encountered a digest error [which caused > the underrun] and so, they just send back a Status > PDU under the belief that the command is complete, > whereas, the initiator wants all the Data PDUs > to be re-sent. A retry MUST resend all the data as well as the status. -Matt From owner-ips@ece.cmu.edu Tue Jan 30 06:56:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA03374 for ; Tue, 30 Jan 2001 06:56:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0UA8O910442 for ips-outgoing; Tue, 30 Jan 2001 05:08:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0UA7cZ10425 for ; Tue, 30 Jan 2001 05:07:38 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 150142C2 for ; Tue, 30 Jan 2001 03:07:38 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 710511CE for ; Tue, 30 Jan 2001 05:07:37 -0500 (EST) Received: from agilent.com (cos1nai135105.cos.agilent.com [141.184.135.105]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id CAA01643 for ; Tue, 30 Jan 2001 02:07:35 -0800 (PST) Message-ID: <3A769265.2C73333A@agilent.com> Date: Tue, 30 Jan 2001 02:07:33 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: new iSCSI PDU outline References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Section 2.2.1, bits 6-4: I think you forgot the BHS structure in the encodings Also, why do we (you) want the long data header??? we already have 32 bits of length, what in the world would we want 64 bits of length for??? Section 2.3 - I don't see how you made it 44 bytes from 48 without eliminating any fields...? looks like you cut the CDB from 16 to 12 bytes... I don't understand 2.2.1, bit 7: 0=another header, and 1=no header, only data? You should specify that bits 1-0 only are valid if bit 7 indicates no header In other words, within each WN, you are specifying if the there is another WN, if not, then (implies data) you specify the digest of the data. -Matt From owner-ips@ece.cmu.edu Tue Jan 30 06:59:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA03396 for ; Tue, 30 Jan 2001 06:59:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0U9nic09992 for ips-outgoing; Tue, 30 Jan 2001 04:49:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0U9mhZ09971 for ; Tue, 30 Jan 2001 04:48:43 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id KAA66646; Tue, 30 Jan 2001 10:48:36 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id KAA232966; Tue, 30 Jan 2001 10:48:35 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E4.0035DFD9 ; Tue, 30 Jan 2001 10:48:27 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: "Charles Binford" cc: ips@ece.cmu.edu Message-ID: Date: Tue, 30 Jan 2001 11:44:01 +0200 Subject: Re: Comments on iSCSI -03 Mime-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t" Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline Comments in text. Thanks, Julo "Charles Binford" on 29/01/2001 23:09:46 Please respond to "Charles Binford" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Comments on iSCSI -03 Julian, Below are several comments on iSCSI-03. I --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6m fairly new to the iSCSI discussion (but have been involved in T10/T11 for several years) = so I apologize if I= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6m rehashing already made decisions. I= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6ve sent this directly to you instead of the general reflector because many of the items are trivial editing issues. Please feel free to cut and paste to the refle= ctor any issue(s) you feel justify a wider discussion. (Of course I reserve= the right to do the same if I strongly disagree with your answer :-) ) Before I get into specifics, let me start with an overall comment. Com= ing from T10/T11 work, it bothers me that many of the PDU descriptions in section 2 are not complete. It seems to me an attitude was taken that unless a field has new meaning for this PDU, it doesn= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6t need any text. I disagree. I believe that every field in every PDU needs a description.= In many cases that description may only spell out the acronym and then say= = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6See x.y= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 but at least the implementer using the spec for reference can quick= ly get a pointer to the authoritative text on the given field. Thanks for your time. Charles Binford Blue Spruce Networks ************************************************* [cb-01] pages 8 and 9 In the following two places = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6CDB= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 is referred to as = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6Command Data Block= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6. It= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6s real definition (from SAM) is Command Descriptor Block. ************************************************* will change 1. Overview 1.1 SCSI Concepts . . . Command Data Blocks (CDB) are the data structures used to contain th= e ^^^^^^^^^^^^^^^^^^^^^^^^^ command parameters to be handed by an initiator to a target. The CDB= content and structure is defined by [SAM] and device-type specific SCSI standards. 1.2.1 Layers & Sessions The following conceptual layering model is used in this document to specify initiator and target actions and how those relate to transmitted and received Protocol Data Units: -the SCSI layer builds/receives SCSI CDB (Command Data Blocks) ^^^^^^^^^^^^^^^^^^^ and relays/receives them with the remaining command execute parameters (cf. SAM-2) to/from the -the iSCSI layer that builds/receives iSCSI PDUs and relays/receives them to/from - one or more TCP connections that form an initiator-target "session". ************************************************* [cb-02] page 11 It seems to read better if you change = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6target SCSI= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 to = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6SCSI target= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6. ************************************************* 1.2.2.1 Command numbering . . . CmdRNs are significant only during command delivery to the target. Once the device serving part of the target SCSI has received a ^^^^^^^^^^^ command, CmdRN ceases to be significant. During command delivery to= the target, the allocated numbers are unique session wide. will fix ************************************************* [cb-03] page 14 Change = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6an= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 to = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6and= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6. ************************************************* 1.2.5 iSCSI Full Feature Phase . . . that was used to deliver the SCSI command. If an initiator issues a= WRITE command, the initiator must send the data, if any, for that command and the target MUST return R2T, if any, an the status over ^^^ the same TCP connection that was used to deliver the SCSI command. will fix ************************************************* [cb-04] page 22 Change = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6Response bit (bit 6)= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 to = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6Response bit (bit 7)= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6. ************************************************* 2.2.1 Opcode The Opcode indicates what type of iSCSI PDU the header encapsulates.= The Opcode is further encoded as follows: b7 Response b6-0 Operation The opcodes are divided into two categories: initiator opcodes and target opcodes. Initiator opcodes are in PDUs sent by the initiators= , and target opcodes are in PDUs sent by the target. The initiator MUS= T NOT send target opcodes and the target MUST NOT send initiator opcodes. Target opcodes are also called responses and are distinguished by having the Response bit (bit 6) set to 1. ^^^ will fix ************************************************* [cb-05] page 30 Again, I= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6m having a hard time finding a concise set of rules of how this parameter (StatRN) is used. It is incremented by 1 for every response,= but what is the starting value; 0? 1, any non-zero? What conditions reset = it = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =FB any? Does is roll over back to 0, or does it explicitly skip 0 when rolling over from 0xffffffff? ************************************************* 2.4.7 StatRN - Status Reference Number StatRN is a reference number that the target iSCSI layer generates per connection and that in turn enables the initiator to acknowledge= status reception. StatRN is incremented by 1 for every response/status sent on a connection. There is an InitStatSN for every connection in the login response. = The value 0 has no special significance so you wrap around. As usuall if th= e difference between expected and sent is larger than 2**31-1 a recovery action MUST be taken. ************************************************* [cb-06] page 32 This section implies ACA handing is mandatory if AE reporting is suppor= ted. This seems to be an unnecessary linking of two independent concepts. At least part of what I think is being solved with this auto ACA requirement is handled better by the new Task Aborted Status described in SAM-2. (= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6better= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 in my biased opinion anyway = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =FB I was the author of the Task Aborted Status proposal.) Was utilizing Task Aborted considered and rejected, = or was it not even known/understood since it is so new? ************************************************* For the , if SCSI control mode enables AE reporting,= the target MUST send an Asynchronous Event to all other attached initiators to inform them that all pending tasks are cancelled and then enter the ACA state for any initiator for which it had pending ^^^^^^^^^^^^^^^^^^^ tasks. For the and functions, the target cancels all pending operations and are both equivalent to the= Target Reset as specified by SAM-2. Provided that SCSI control mode= enables AE reporting, the target MUST send an Asynchronous Event to all attached initiators notifying them that the target is being reset. In addition, for the the target will enter the ACA state on all sessions and all LUs on which an AE was sent. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ We heard about task aborted but had little understanding about how= to use it. If like ACA it will stay on untill cleared and will thus serve us to dr= op all commands in flight the we could use it. If it does not we will have= to stay with ACA. Again the issue where the commands in flight ************************************************* [cb-07] page 46 Need to reword sentence = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =FB the word = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6but= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 doesn= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6t fit. ************************************************* 2.11.2 Version-active/lowest Indicates the version supported (the highest supported by the target= and initiator). If the target is not supporting a version within the= Satran, J. Standards-Track, August 2001 45 iSCSI January 10, 2000 range of the initiator it will reject the login but and this field ^^^^ will indicate the lowest version supported by the target. will fix ************************************************* [cb-08] page 50 Sentence organization: In the description of the P bit, the last sente= nce the = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6IF= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 is the final phrase, making it hard to understand on the first reading. If the sentence is turned around (IF ... THEN style) I believ= e it is much clearer. ************************************************* 2.13.1 P bit A target may issue a NOP-In by its own to test connection and the state of the initiator. In this case the Initiator Task Tag MUST be = 0 and the Target Tag MUST be set (not x'ffffffff') only if the P bit i= s ^^^^^^^^^^^^^^^^^^^= ^ 1. ^^ will fix ************************************************* [cb-09] page 57 Two problems with the following list; 5 & 6 should be 4 & 5, and item number 3 is not in SAM-2 r15 (the latest version I could find). Was something= recently added? Again, I believe the new Task Aborted Status solves th= e problem. ************************************************* 2.17.2 SCSI Event Indicator The following values are defined. (See [SAM2] for details): 1 An error condition was encountered after command completion. 2 A newly initialized device is available to this initiator. 3 All Task Sets are being Reset by another Initiator 5 Some other type of unit attention condition has occurred. 6 An asynchronous event has occurred. numbering will be fixed. I will be glad to change/elliminate 3 to t= ask aborted if I get a good pointer and it is good for commands in flight -= meanwhile would it be fair to say that it is subsumed by 6 ************************************************* [cb-10] page 59 At the end of section 2, and there is no description of Markers. I bel= ieve the standard needs a format diagram just like any of the other PDUs = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =FB maybe not in section 2, but somewhere. ************************************************* markers will move to an appendix and I'll add a descriptor ************************************************* [cb-11] page 60 Why redefine the mode page??? The Max Burst Size ad First Burst Size h= ave been defined as 512 byte chunks for a long time. A SCSI target doesn= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6t want to have to translate fields based on the particular transport. The 512= definition give a range of up to just under 32MB = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =FB seems like a big enough = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6first burst= --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 to me. ************************************************* 3.1.2 Maximum Burst Size field (16 bit) This field is used by iSCSI to define the maximum data payload in iSCSI data PDUs or as immediate data in command PDUs in units of 409= 6 ^^^^^^^^^^^^^= ^ bytes. This value can also be set by a text-mode key:value pair (DataPDULength). 3.1.3 First Burst Size field (16 bit) This field is used by iSCSI to define the maximum of unsolicited dat= a an iSCSI initiator is allowed to send to the target in units of 4096= ^^^^^^^^^^^^^= bytes. This value can also be set by a text-mode key:value pair (InitialBurstLength). We where told that those fields are protocol specific and we have= to define their values. If a multiple of 512 is more a better fit for SCS= I I will change it Charles Binford Blue Spruce Networks office/cell: (316) 210-6404 e-fax: (509) 756-4425 = --0__=RGdVuYZjQaDlYK3YXzuyxkS2hjOVjrpg2G1Veltf1ewXz5keOuEdmB2t-- From owner-ips@ece.cmu.edu Tue Jan 30 08:25:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id IAA05165 for ; Tue, 30 Jan 2001 08:25:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0UBSRG22384 for ips-outgoing; Tue, 30 Jan 2001 06:28:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0UBRjZ22367 for ; Tue, 30 Jan 2001 06:27:45 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id MAA32348 for ; Tue, 30 Jan 2001 12:27:38 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id MAA260906 for ; Tue, 30 Jan 2001 12:27:38 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E4.003EF180 ; Tue, 30 Jan 2001 12:27:30 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 30 Jan 2001 13:23:06 +0200 Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, By enforce I meant - enforce like in the legalese - i.e., police. If you have a MUST that you are never going to check better find a better solution. Checking it entails scoreboarding that no other SCSI protocol does or needs. Sequencing is simple (and that is what FC does) and lets the target master the transfer the way it usually does for all SCSI protocols. I feel we have spent already too much time on this single issue -:) Julo Santosh Rao on 30/01/2001 10:25:45 Please respond to Santosh Rao To: ips@ece.cmu.edu (ips) cc: Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues > julian_satran@il.ibm.com wrote: > > > > Santosh, > > > > The trouble with forbidding a certain behavior is that you have to enforce > > it (i.e., check and signal errors for units that do not behave). > > No you don't have to "enforce" it at all. Using Santosh's example, if a > target broke the rules and performed data overlay, then the initiator will > always mark the I/O as bad, and the market forces will take over (customer > will buy a compliant target). Julian, Removing support for overlapped data xfer's has multiple benefits: 1) It provides initiators a reliable way of ensuring that the I/O did complete without any underrun. 2) It simplifies SCSI Assist implementations that no longer need to deal with overlapped data xfer conditions. 3) It is simpler than performing book-keeping on DataSN to ensure that all DataSNs have been received. (IOW, score-boarding at a DataSN level, instead of at a byte level.) I'd say the count based solution is preferrable, given the above. All protocols inherently enforce certain behaviour by mandating features (the use of MUST, shall). I don't think that's a strong enough reason to reject this proposal. To summarize : o Dis-allow overlapped data xfer's. o Initiators to perform a count check as is done in FC. o On detecting an underrun, the command may be retried BUT WITHOUT SETTING the "retry" bit. This is particularly important because targets that implement status recovery may be ignorant of the fact that the initiator encountered a digest error [which caused the underrun] and so, they just send back a Status PDU under the belief that the command is complete, whereas, the initiator wants all the Data PDUs to be re-sent. Regards, Santosh > > Besides > > - the whole philosophy of the SCSI set of protocols is that the target is > > the master and the initiator should let the target decide how to fulfill > > the command. That is why we chose not to impose restrictions above those > > imposed by SCSI. The whole set of issues is also raised only because we > > provide also for storage proxies - otherwise a stronger checksum at TCP > > level and recovery at TCP level would have done what we wanted and recovery > > of the type we are dealing now with would have been done at TCP level. > > I am confident that we can reinstate DataSN a simple mean to sequence (not > > ack) data packets and considerably simplify recovery. > > If there is a data digest failure, the iSCSI PDU is discarded, and the test > that Santosh describes will fail. At that point, the command is "retried", > and using my example of the retry implementation in the thread "iSCSI: I/O > (command) recovery" error recovery is performed. No need for DataSN... > > > And do not forget that raising the error up to ULP with a service response > > will make the recovery far more expensive (as Prasenjit has already stated) > > - far more than current wedge drivers do as these rarely consider commands > > in flight and the need to keep order in a target that is not yet aware that > > something went wrong. > > I agree that erorrs due to the transport should not be propagated to the ULP. > In the case of a digest failure, this means that the TCP checksum indicated > the segment was good, meaning that a middle box corrupted the TCP segment and > sent it out with a "fixed" TCP checksum. > > The simple iSCSI error recovery using the retry should handle this corner case > very well. > > -Matt > > > > > > Julo > > > > Santosh Rao on 28/01/2001 00:07:08 > > > > Please respond to Santosh Rao > > > > To: Julian Satran/Haifa/IBM@IBMIL > > cc: ips@ece.cmu.edu > > Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues > > > > Julian, > > > > The missing Data PDU could be detected if the initiator were to > > perform a count check operation upon receiving SCSI Response PDU, > > along the lines of : > > > > no. of bytes xfer'ed = > > (Expected Data Xfer Length) - (Basic Residual Count) > > > > where, > > Expected Data Xfer Length -> as specified in SCSI Command PDU > > Basic Residual Count -> as specified in SCSI Response PDU > > > > However, this is currently not possible due to overlapped data > > transfers being allowed by iSCSI. If iSCSI were to dis-allow > > overlapping data xfer's and initiators used a count check > > [as is done in FC], this would also address the problem. > > > > Regards, > > Santosh > > > > > > > > > > > > > > If the header is a data header we can hardly trust the ULP to recognize > > the > > > error (he might be unaware > > > of a missing packet). With data numbering this situation could have been > > > discovered at "status time". > > > The only thing we could do is restart all commands but this is equivalent > > > to a connection restart for all practical purposes. Dropping data > > > numbering might have some more "side-effects" like this. > > > As the combination of values - tag, address, offset may stil let some > > > implementations to assume that they have > > > a correct task identifier I don't see a point in mandating a recovery > > > behavior and the implementer may choose to: > > > > > > -retry/restart command > > > -logout drop and rebuild connection login and restart/retry > > > -abort all task sets (practically reset the target!) and report for all > > > commands a "delivery system failure" (kick-in the ULP recovery) and if > > you > > > suspect the link quality rebuild it; this later behavior means also that > > > you have to stop delivering anything on any link to the target to avoid > > > out of order execution until you have finished the cleanup - pretty > > drastic > > > > > > With data numbering recovery could have stayed within the confines of a > > > command even if a header was bad. > > > Perhaps we should leave the DataSN only as a sequencer so that at > > > status-time the initiator should be able to find if a data packet was > > > dropped (no ExpDataSN on a NOP). > > > > > > Regards, > > > Julo > > > > > > > > > > > > > > > Michael Krause on 27/01/2001 04:59:12 > > > > > > Please respond to Michael Krause > > > > > > To: Julian Satran/Haifa/IBM@IBMIL > > > cc: ips@ece.cmu.edu > > > Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window > > issues > > > > > > > > > > > > > > > At 07:40 PM 1/25/2001 +0200, julian_satran@il.ibm.com wrote: > > > > > > > > > >1) The initiator task tag cannot be trusted when a header digest error > > > >is seen. What does the phrase "provided it can recognize the initiator > > > >task tag" mean ? > > > >How can an initiator reliably claim that the initiator task tag is > > > >trustworthy ? > > > > > > > > an initiator may choose to provide some redundancy in the tag > > itself > > > > > > > > > > I'm aware of some techniques for inserting redundant information in tags > > > which limits the potential error exposure when a multi-bit error occurs, > > > however these are not fail-safe leading to potential incorrect operation > > - > > > perhaps benign in many cases; perhaps not in others. As such, if a header > > > digest error occurs, the PDU should be silently discarded and recovery > > > should be left to the ULP. There is little to no value having two > > > mechanisms to solve the same problem. > > > > > > Mike > > > > > > > > > > > > > > > > > > > -- > > ################################# > > Santosh Rao > > Software Design Engineer, > > HP, Cupertino. > > email : santoshr@cup.hp.com > > Phone : 408-447-3751 > > ################################# > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Tue Jan 30 11:27:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA12123 for ; Tue, 30 Jan 2001 11:27:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0UEGX127108 for ips-outgoing; Tue, 30 Jan 2001 09:16:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0UEFvZ27095 for ; Tue, 30 Jan 2001 09:15:57 -0500 (EST) Received: from cisco.com (mbakke@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id JAA19761; Tue, 30 Jan 2001 09:15:37 -0500 (EST) Message-ID: <3A76CCFA.DED52322@cisco.com> Date: Tue, 30 Jan 2001 08:17:30 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Matt Wakeley CC: David Black , ips@ece.cmu.edu Subject: Re: I-D ACTION:draft-ietf-ips-iscsi-disc-reqts-00.txt References: <200101291144.GAA29115@ietf.org> <3A765CB0.99483D2D@agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I just took a look through this; it's the draft we had filed for the interim meeting. We made several decisions at the interim meeting that will cause a newer version of this draft to be re-written. Please stay tuned for an even newer version. -- Mark Bakke Matt Wakeley wrote: > > Well, this is interesting: > > At http://www.ietf.org/internet-drafts/ there are the following two files: > > draft-ietf-ips-iscsi-disc-reqts-00.txt 26-Jan-2001 > 08:08 46k > draft-ietf-ips-iscsi-disc-reqts-01.txt 12-Jan-2001 > 13:56 44k > > Notice how 01 predates 00? > > The later was really but I > suspect the name was too long so it was changed. > > David, can you cause this to be fixed? > > -Matt > > Internet-Drafts@ietf.org wrote: > > > > A New Internet-Draft is available from the on-line Internet-Drafts directories. > > This draft is a work item of the IP Storage Working Group of the IETF. > > > > Title : iSCSI Naming and Discovery Requirements > > Author(s) : M. Bakke et al. > > Filename : draft-ietf-ips-iscsi-disc-reqts-00.txt > > Pages : 18 > > Date : 26-Jan-01 > > > > This document describes the iSCSI [7] naming and discovery > > requirements. The requirements presented in this document have been > > agreed to by the members of the iSCSI naming and discovery team. This > > document complements the iSCSI IETF draft. Flexibility is the key > > guiding principle behind this requirements document. That is, an > > effort has been made to satisfy the needs of both small isolated > > environments, as well as large environments requiring secure/scalable > > solutions. > > > > A URL for this Internet-Draft is: > > http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-disc-reqts-00.txt > > > > Internet-Drafts are also available by anonymous FTP. Login with the username > > "anonymous" and a password of your e-mail address. After logging in, > > type "cd internet-drafts" and then > > "get draft-ietf-ips-iscsi-disc-reqts-00.txt". > > > > A list of Internet-Drafts directories can be found in > > http://www.ietf.org/shadow.html > > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > > > > Internet-Drafts can also be obtained by e-mail. > > > > Send a message to: > > mailserv@ietf.org. > > In the body type: > > "FILE /internet-drafts/draft-ietf-ips-iscsi-disc-reqts-00.txt". > > > > NOTE: The mail server at ietf.org can return the document in > > MIME-encoded form by using the "mpack" utility. To use this > > feature, insert the command "ENCODING mime" before the "FILE" > > command. To decode the response(s), you will need "munpack" or > > a MIME-compliant mail reader. Different MIME-compliant mail readers > > exhibit different behavior, especially when dealing with > > "multipart" MIME messages (i.e. documents which have been split > > up into multiple messages), so check your local documentation on > > how to manipulate these messages. > > > > > > Below is the data which will enable a MIME compliant mail reader > > implementation to automatically retrieve the ASCII version of the > > Internet-Draft. > > > > ------------------------------------------------------------------------------ > > Content-Type: text/plain > > Content-ID: <20010126081949.I-D@ietf.org> -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Tue Jan 30 15:06:26 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA16782 for ; Tue, 30 Jan 2001 15:05:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0UFdeo00218 for ips-outgoing; Tue, 30 Jan 2001 10:39:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ztxmail04.ztx.compaq.com (ztxmail04.ztx.compaq.com [161.114.1.208]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0UFdEZ00200 for ; Tue, 30 Jan 2001 10:39:14 -0500 (EST) Received: by ztxmail04.ztx.compaq.com (Postfix, from userid 12345) id 4010F224; Tue, 30 Jan 2001 09:39:08 -0600 (CST) Received: from exchou-gh02.cca.cpqcorp.net (exchou-gh02.cca.cpqcorp.net [16.110.248.202]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id 71970266; Tue, 30 Jan 2001 09:39:08 -0600 (CST) Received: by exchou-gh02.cca.cpqcorp.net with Internet Mail Service (5.5.2652.78) id ; Tue, 30 Jan 2001 09:39:08 -0600 Message-ID: <78AF3C342AEAEF4BA33B35A8A15668C659C0C1@cceexc17.americas.cpqcorp.net> From: "Elliott, Robert" To: "'julian_satran@il.ibm.com'" , Charles Binford , ips@ece.cmu.edu Subject: RE: Comments on iSCSI -03 Date: Tue, 30 Jan 2001 09:42:06 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2652.78) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > [cb-09] page 57 > Two problems with the following list; 5 & 6 should be 4 & 5, > and item number 3 is not in SAM-2 r15 (the latest version > I could find). Was something recently added? Again, I > believe the new Task Aborted Status solves the problem. > > 2.17.2 SCSI Event Indicator > > The following values are defined. (See [SAM2] for details): > > 1 An error condition was encountered after command > completion. > 2 A newly initialized device is available to this initiator. > 3 All Task Sets are being Reset by another Initiator > 5 Some other type of unit attention condition has occurred. > 6 An asynchronous event has occurred. >numbering will be fixed. I will be glad to change/elliminate > 3 to task aborted if I get a good pointer and it is good for > commands in flight -= > > meanwhile would it be fair to say that it is subsumed by 6 Julian, according to your message on 4 Jan copied below, the SCSI Event reasons are going to be eliminated altogether. The sense data contains the most accurate information about which event occurred. --- PC: Robert.Elliott@compaq.com UNIX: relliott@unixmail.compaq.com Previous message: To: "Elliott, Robert" Subject: Re: iSCSI: SCSI Event Indicator From: julian_satran@il.ibm.com Date: Thu, 4 Jan 2001 09:55:06 +0200 cc: ips@ece.cmu.edu Sender: owner-ips@ece.cmu.edu Will do, Thanks, Julo "Elliott, Robert" on 03/01/2001 19:00:58 Please respond to "Elliott, Robert" To: "'ips@ece.cmu.edu'" cc: Subject: iSCSI: SCSI Event Indicator The Asynchronous Event data structure includes an "iSCSI Event Indicator" field and a "SCSI Event Indicator" field. The iSCSI field indicates iSCSI-specific causes, which is fine. The SCSI Event field indicates generic SCSI causes. I suggest removing the SCSI Event Indicator values. The information is contained within the sense data itself. The SEND command in SPC-2, the current mechanism to report Asynchronous Events, does not include an Event field like this. I suggest simplifying the field to: 0 = not a SCSI Event 1 = SCSI Event This is the text in iSCSI revision 02, with comments listing where in the sense data that information is communicated: 2.17.2 SCSI Event Indicator The following values are defined. (See [SAM2] for details): 1 An error condition was encountered after command completion. [Sense data RESPONSE CODE field is DEFERRED ERRORS - 71h] 2 A newly initialized device is available to this initiator. [Sense data ASC/ASCQ is REPORTED LUNS DATA HAS CHANGED] 3 All Task Sets are being Reset by another Initiator [Sense data ASC/ASCQ is COMMANDS CLEARED BY ANOTHER INITIATOR] [where is 4?] 5 Some other type of unit attention condition has occurred. [Sense data SENSE KEY field is UNIT ATTENTION - 6h] 6 An asynchronous event has occurred. [Any other sense data] Sense Data accompanying the report identifies the condition. The Length parameter is set to the length of the Sense Data. For new device identification an iSCSI target MUST support the Device Identification page. The list was probably taken from this list in SAM-2; however, it's not an exact match: Asynchronous Event Reporting is used to signal a device that one of the four events listed below has occurred: a) an error condition was encountered after command completion; b) a newly initialized device is available; c) some other type of unit attention condition has occurred; or d) an asynchronous event has occurred. --- PC: Robert.Elliott@compaq.com UNIX: relliott@unixmail.compaq.com From owner-ips@ece.cmu.edu Tue Jan 30 15:06:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA16816 for ; Tue, 30 Jan 2001 15:06:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0UFEZk29211 for ips-outgoing; Tue, 30 Jan 2001 10:14:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.brocade.com (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0UEuGZ28526 for ; Tue, 30 Jan 2001 09:56:16 -0500 (EST) Received: from thor.brocade.com (thor [192.168.126.45]) by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id GAA10933; Tue, 30 Jan 2001 06:56:10 -0800 (PST) Received: by thor.brocadecomm.com with Internet Mail Service (5.5.2650.21) id ; Tue, 30 Jan 2001 06:56:06 -0800 Message-ID: From: Robert Snively To: Jim Hafner , ips@ece.cmu.edu, Robert Snively Subject: RE: iSCSI: INQUIRY page 0x83 identifier Date: Tue, 30 Jan 2001 06:56:03 -0800 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk > > I strongly agree with the requirement that these identifiers > be WWUnique. > But I also agree with Charles that whatever identifier that > is presented > should be the same regardless of the interconnect, so > binding them to FC or > to some defined iSCSI mechanism seems inappropriate. > Jim, There is actually nothing "FC" about the number defined except the fact that the format is defined in an FC document. Everything else about it is generic and is in fact used in all SCSI transport families. > So, I think I'm agreeing the Rob Elliot. Namely, specify a > 128 format > rooted in EUI-64 for the device. > Jim and Rob, There is no standard 128 bit format rooted in EUI-64. You will have to invent one, define it in a standard document, route it up through the IEEE tutorial definition process, and change SPC to get it identified as a standard. Why not use the already defined transport independent format defined in FC-FS, SPC, and the IEEE tutorials. Bob From owner-ips@ece.cmu.edu Tue Jan 30 17:28:26 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA18435 for ; Tue, 30 Jan 2001 17:28:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0UKKjl12494 for ips-outgoing; Tue, 30 Jan 2001 15:20:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0UKKdZ12488 for ; Tue, 30 Jan 2001 15:20:39 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 8425A3EA for ; Tue, 30 Jan 2001 13:20:38 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id F2E4B198 for ; Tue, 30 Jan 2001 15:20:36 -0500 (EST) Received: from agilent.com (cos1nai133125.cos.agilent.com [141.184.133.125]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id MAA10189 for ; Tue, 30 Jan 2001 12:20:33 -0800 (PST) Message-ID: <3A77220F.81A7E31F@agilent.com> Date: Tue, 30 Jan 2001 12:20:31 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: new iSCSI PDU outline References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > > Matt, > > answers in text. > > Julo > > Matt Wakeley on 30/01/2001 12:07:33 > > Please respond to Matt Wakeley > > To: ips@ece.cmu.edu > cc: > Subject: Re: new iSCSI PDU outline > > Julian, > > Section 2.2.1, bits 6-4: > > I think you forgot the BHS structure in the encodings > > BHS is allways there and WN is what's next :)- So the concept is, "I'm telling you what is coming after the segment following this WN descriptor (which you should already know what it is)". Using the following example (I'm numbering the sections for discussion purposes): 1 - WN 2 - BHS 3 - WN 4 - AHS 5 - Data When a new iscsi PDU arrives, I "know" that #2 is BHS, so #1 tells me what #4 is. #3 tells me what #5 is. I think you need to add more descriptive text describing this. [I think it would be easier to understand if #1 describes #2, and #3 describes #4. #3 could also indicate that following #4 is data] > Also, why do we (you) want the long data header??? we already have 32 bits > of length, what in the world would we want 64 bits of length for??? > > academic I suppose - the length of a single PDU data part is limited > to 24 bits now Well we are already talking about limiting a PDU to a (small number of) TCP segment size in order to preserve data integrity, so I don't think we need additional length... > Section 2.3 - I don't see how you made it 44 bytes from 48 without > eliminating > any fields...? looks like you cut the CDB from 16 to 12 bytes... > I took out the length - it is always anyhow the length of what comes > after No you didn't. In the email you sent out, the length field is still there, starting at byte 4. All you did was change the "48" after the CDB to "44". > I don't understand 2.2.1, bit 7: 0=another header, and 1=no header, only > data? > > You should specify that bits 1-0 only are valid if bit 7 indicates no > header > > In other words, within each WN, you are specifying if the there is another > WN, > if not, then (implies data) you specify the digest of the data. > correct we specify both the digest of the data and the header - we can > restrict it to the last header segment > -Matt From owner-ips@ece.cmu.edu Tue Jan 30 17:29:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA18451 for ; Tue, 30 Jan 2001 17:29:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0UKcrJ13283 for ips-outgoing; Tue, 30 Jan 2001 15:38:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0UKbjZ13236 for ; Tue, 30 Jan 2001 15:37:45 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 9401858D for ; Tue, 30 Jan 2001 13:37:39 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 0C5E7226 for ; Tue, 30 Jan 2001 15:37:38 -0500 (EST) Received: from agilent.com (cos1nai133125.cos.agilent.com [141.184.133.125]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id MAA11720 for ; Tue, 30 Jan 2001 12:37:34 -0800 (PST) Message-ID: <3A77260C.2CDF4CA@agilent.com> Date: Tue, 30 Jan 2001 12:37:32 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, I do not beleive that a MUST requirement of a sender "MUST" be verfied by the receiver. For example, if reserved fields MUST be zero, I am certainly not going to verify that reserved fields are zero - I don't care about them - they're reserved! You are trying to detect a misbehaving target. Well, I could build a misbehaving target that sends your sequence numbers, but not enough of them, and indicate no residual in the status, and you still wouldn't be able to detect an error... -Matt julian_satran@il.ibm.com wrote: > > Santosh, > > By enforce I meant - enforce like in the legalese - i.e., police. > If you have a MUST that you are never going to check better find a better > solution. > Checking it entails scoreboarding that no other SCSI protocol does or > needs. > > Sequencing is simple (and that is what FC does) and lets the target master > the transfer > the way it usually does for all SCSI protocols. > > I feel we have spent already too much time on this single issue -:) > > Julo > > Santosh Rao on 30/01/2001 10:25:45 > > Please respond to Santosh Rao > > To: ips@ece.cmu.edu (ips) > cc: > Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues > > > julian_satran@il.ibm.com wrote: > > > > > > Santosh, > > > > > > The trouble with forbidding a certain behavior is that you have to > enforce > > > it (i.e., check and signal errors for units that do not behave). > > > > No you don't have to "enforce" it at all. Using Santosh's example, if a > > target broke the rules and performed data overlay, then the initiator > will > > always mark the I/O as bad, and the market forces will take over > (customer > > will buy a compliant target). > > Julian, > > Removing support for overlapped data xfer's has multiple benefits: > 1) It provides initiators a reliable way of ensuring > that the I/O did complete without any underrun. > > 2) It simplifies SCSI Assist implementations that no longer > need to deal with overlapped data xfer conditions. > > 3) It is simpler than performing book-keeping > on DataSN to ensure that all DataSNs have been > received. (IOW, score-boarding at a DataSN level, > instead of at a byte level.) > > I'd say the count based solution is preferrable, given the above. > All protocols inherently enforce certain behaviour by mandating > features (the use of MUST, shall). I don't think that's a > strong enough reason to reject this proposal. > > To summarize : > o Dis-allow overlapped data xfer's. > > o Initiators to perform a count check as is done in FC. > > o On detecting an underrun, the command may be retried > BUT WITHOUT SETTING the "retry" bit. This is > particularly important because targets that implement > status recovery may be ignorant of the fact that the > initiator encountered a digest error [which caused > the underrun] and so, they just send back a Status > PDU under the belief that the command is complete, > whereas, the initiator wants all the Data PDUs > to be re-sent. > > Regards, > Santosh > > > > > Besides > > > - the whole philosophy of the SCSI set of protocols is that the target > is > > > the master and the initiator should let the target decide how to > fulfill > > > the command. That is why we chose not to impose restrictions above > those > > > imposed by SCSI. The whole set of issues is also raised only because > we > > > provide also for storage proxies - otherwise a stronger checksum at TCP > > > level and recovery at TCP level would have done what we wanted and > recovery > > > of the type we are dealing now with would have been done at TCP level. > > > I am confident that we can reinstate DataSN a simple mean to sequence > (not > > > ack) data packets and considerably simplify recovery. > > > > If there is a data digest failure, the iSCSI PDU is discarded, and the > test > > that Santosh describes will fail. At that point, the command is > "retried", > > and using my example of the retry implementation in the thread "iSCSI: > I/O > > (command) recovery" error recovery is performed. No need for DataSN... > > > > > And do not forget that raising the error up to ULP with a service > response > > > will make the recovery far more expensive (as Prasenjit has already > stated) > > > - far more than current wedge drivers do as these rarely consider > commands > > > in flight and the need to keep order in a target that is not yet aware > that > > > something went wrong. > > > > I agree that erorrs due to the transport should not be propagated to the > ULP. > > In the case of a digest failure, this means that the TCP checksum > indicated > > the segment was good, meaning that a middle box corrupted the TCP segment > and > > sent it out with a "fixed" TCP checksum. > > > > The simple iSCSI error recovery using the retry should handle this corner > case > > very well. > > > > -Matt > > > > > > > > > > Julo > > > > > > Santosh Rao on 28/01/2001 00:07:08 > > > > > > Please respond to Santosh Rao > > > > > > To: Julian Satran/Haifa/IBM@IBMIL > > > cc: ips@ece.cmu.edu > > > Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window > issues > > > > > > Julian, > > > > > > The missing Data PDU could be detected if the initiator were to > > > perform a count check operation upon receiving SCSI Response PDU, > > > along the lines of : > > > > > > no. of bytes xfer'ed = > > > (Expected Data Xfer Length) - (Basic Residual Count) > > > > > > where, > > > Expected Data Xfer Length -> as specified in SCSI Command PDU > > > Basic Residual Count -> as specified in SCSI Response PDU > > > > > > However, this is currently not possible due to overlapped data > > > transfers being allowed by iSCSI. If iSCSI were to dis-allow > > > overlapping data xfer's and initiators used a count check > > > [as is done in FC], this would also address the problem. > > > > > > Regards, > > > Santosh > > > > > > > > > > > > > > > > > > > If the header is a data header we can hardly trust the ULP to > recognize > > > the > > > > error (he might be unaware > > > > of a missing packet). With data numbering this situation could have > been > > > > discovered at "status time". > > > > The only thing we could do is restart all commands but this is > equivalent > > > > to a connection restart for all practical purposes. Dropping data > > > > numbering might have some more "side-effects" like this. > > > > As the combination of values - tag, address, offset may stil let some > > > > implementations to assume that they have > > > > a correct task identifier I don't see a point in mandating a recovery > > > > behavior and the implementer may choose to: > > > > > > > > -retry/restart command > > > > -logout drop and rebuild connection login and restart/retry > > > > -abort all task sets (practically reset the target!) and report for > all > > > > commands a "delivery system failure" (kick-in the ULP recovery) and > if > > > you > > > > suspect the link quality rebuild it; this later behavior means also > that > > > > you have to stop delivering anything on any link to the target to > avoid > > > > out of order execution until you have finished the cleanup - pretty > > > drastic > > > > > > > > With data numbering recovery could have stayed within the confines of > a > > > > command even if a header was bad. > > > > Perhaps we should leave the DataSN only as a sequencer so that at > > > > status-time the initiator should be able to find if a data packet was > > > > dropped (no ExpDataSN on a NOP). > > > > > > > > Regards, > > > > Julo > > > > > > > > > > > > > > > > > > > > Michael Krause on 27/01/2001 04:59:12 > > > > > > > > Please respond to Michael Krause > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL > > > > cc: ips@ece.cmu.edu > > > > Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window > > > issues > > > > > > > > > > > > > > > > > > > > At 07:40 PM 1/25/2001 +0200, julian_satran@il.ibm.com wrote: > > > > > > > > > > > > >1) The initiator task tag cannot be trusted when a header digest > error > > > > >is seen. What does the phrase "provided it can recognize the > initiator > > > > >task tag" mean ? > > > > >How can an initiator reliably claim that the initiator task tag is > > > > >trustworthy ? > > > > > > > > > > an initiator may choose to provide some redundancy in the tag > > > itself > > > > > > > > > > > > > I'm aware of some techniques for inserting redundant information in > tags > > > > which limits the potential error exposure when a multi-bit error > occurs, > > > > however these are not fail-safe leading to potential incorrect > operation > > > - > > > > perhaps benign in many cases; perhaps not in others. As such, if a > header > > > > digest error occurs, the PDU should be silently discarded and > recovery > > > > should be left to the ULP. There is little to no value having two > > > > mechanisms to solve the same problem. > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > ################################# > > > Santosh Rao > > > Software Design Engineer, > > > HP, Cupertino. > > > email : santoshr@cup.hp.com > > > Phone : 408-447-3751 > > > ################################# > > > > -- > ################################# > Santosh Rao > Software Design Engineer, > HP, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################# From owner-ips@ece.cmu.edu Tue Jan 30 17:29:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA18514 for ; Tue, 30 Jan 2001 17:29:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0UJwGT11507 for ips-outgoing; Tue, 30 Jan 2001 14:58:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0UJtnZ11398 for ; Tue, 30 Jan 2001 14:55:49 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 96F6F20C for ; Tue, 30 Jan 2001 12:55:48 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 57BF51AA for ; Tue, 30 Jan 2001 14:55:47 -0500 (EST) Received: from agilent.com (cos1nai133125.cos.agilent.com [141.184.133.125]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id LAA07233 for ; Tue, 30 Jan 2001 11:55:44 -0800 (PST) Message-ID: <3A771C3E.80CDF549@agilent.com> Date: Tue, 30 Jan 2001 11:55:42 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: [Fwd: Re: new iSCSI PDU outline] Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit -------- Original Message -------- From: julian_satran@il.ibm.com Subject: Re: new iSCSI PDU outline To: Matt Wakeley Matt, answers in text. Julo Matt Wakeley on 30/01/2001 12:07:33 Please respond to Matt Wakeley To: ips@ece.cmu.edu cc: Subject: Re: new iSCSI PDU outline Julian, Section 2.2.1, bits 6-4: I think you forgot the BHS structure in the encodings BHS is allways there and WN is what's next :)- Also, why do we (you) want the long data header??? we already have 32 bits of length, what in the world would we want 64 bits of length for??? academic I suppose - the length of a single PDU data part is limited to 24 bits now Section 2.3 - I don't see how you made it 44 bytes from 48 without eliminating any fields...? looks like you cut the CDB from 16 to 12 bytes... I took out the length - it is always anyhow the length of what comes after I don't understand 2.2.1, bit 7: 0=another header, and 1=no header, only data? You should specify that bits 1-0 only are valid if bit 7 indicates no header In other words, within each WN, you are specifying if the there is another WN, if not, then (implies data) you specify the digest of the data. correct we specify both the digest of the data and the header - we can restrict it to the last header segment -Matt From owner-ips@ece.cmu.edu Tue Jan 30 17:31:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA18561 for ; Tue, 30 Jan 2001 17:31:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0UJxl911578 for ips-outgoing; Tue, 30 Jan 2001 14:59:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0UJwdZ11520 for ; Tue, 30 Jan 2001 14:58:40 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id UAA55964 for ; Tue, 30 Jan 2001 20:58:14 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id UAA236020 for ; Tue, 30 Jan 2001 20:58:14 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E4.006DAF15 ; Tue, 30 Jan 2001 20:58:02 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 30 Jan 2001 17:58:22 +0200 Subject: Re: iSCSI : Command Ordering Proposal. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, The only puzzling question is what kind of digest error would you detect for a command that the target has not yet started acting upon? This assumes out of order delivery to SCSI right? And even then if the task is recognized you could use a CmdSN of 0 as the command is clearly active. Only for commands that never gave a sign "of life" you have to use the original CmdSN in case the command did not make it to the delivery queue. Julo Santosh Rao on 30/01/2001 09:58:47 Please respond to Santosh Rao To: matt_wakeley@agilent.com cc: ips@ece.cmu.edu (IPS Reflector) Subject: Re: iSCSI : Command Ordering Proposal. Matt, Comments in-line. Regards, Santosh > > Santosh Rao wrote: > > > > Matt Wakeley wrote: > > > > > Retried commands should have already been queued and/or executed. > > > > Not necessarily. A TCP connection failure can affect the command PDU. > > In which case the target will have "silently discarded" the PDU. Then, it > will not acknowledge via ExpCmdSN any commands past the err'd command, so the > initiator can resend the command with the same CmdSN and enabling command > delivery to SCSI to continue. The above was stated to explain why "retry" commands need to use CmdSN [and cannot use a CmdSN of 0 as you were proposing]. The problem being originally described can occur when the initiator applies a "discard & retry" policy on detecting a digest error at its end. In such a case, it would "retry" the command, and if the CmdSN was not yet acknowledged, would send the "retry" with the original CmdSN. Since the target continues to send PDUs to the initiator, it can update the ExpCmdSN causing the "retry" to be discarded by the target, because its CmdSN has slid behind the window. The spec does not mandate that an update to ExpCmdSN MUST accompany the Data PDUs for a command. IOW, a target may lag behind in updates to ExpCmdSN while continuing to send Data PDUs for the received command. > > > Similarly, the target may have detected a header or data digest error > > on the SCSI Command PDU, sent a REJECT > > I thought any digest errors are silently discarded - therefore, there is no > reject. Not when targets detect a digest error. Section 5.5 mandates targets to send a REJECT with a reason of digest error. The "discard & retry" policy is for initiators. > > b) data digest error detected by an initiator on inbound Data PDUs of a READ > > I/O, and the CmdSN for the command is yet to be acknowledged, but an ACK > > may be in-flight behind the Data PDU. > > -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Tue Jan 30 18:39:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA19270 for ; Tue, 30 Jan 2001 18:39:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0ULhvC16256 for ips-outgoing; Tue, 30 Jan 2001 16:43:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0ULhMZ16229 for ; Tue, 30 Jan 2001 16:43:22 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0UMrq062431; Tue, 30 Jan 2001 14:53:53 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Matt Wakeley" , "IPS Reflector" Subject: RE: iSCSI : Command Ordering Proposal. Date: Tue, 30 Jan 2001 13:42:10 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <3A766866.53684AB8@agilent.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Matt, View iSCSI as isolated from either SCSI or TCP. Mixing SCSI tags or TCP Acks with Client/Server sequencing is confusing. By resources, I was focused only on the iSCSI function and not either TCP or SCSI. From the client's perspective, its PDU is sent only if acknowledged. This is where you run into definitions of when is a PDU delivered to the target. The same should be true for the server and this send resource is my focus. Rather than fully using this handshake, a restart of the SCSI command is seen as recovery but then we lose sense of sequence nor is there a reliable link. Doug > Douglas Otis wrote: > > > > Matt, > > > > Whether you use the CmdSN or the tag to associate the retry, > the CmdSN is > > still used as a means to handshake freeing of resources and not the tag. > > You are wrong. It specifically states in the document in 1.2.2.1: > > "CmdRNs are significant only during command delivery to the target. > Once the device serving part of the target SCSI has received a > command, CmdRN ceases to be significant." > > Any resources the target allocates must be based on the task tag. > > -Matt From owner-ips@ece.cmu.edu Tue Jan 30 18:40:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA19294 for ; Tue, 30 Jan 2001 18:40:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0ULlnb16406 for ips-outgoing; Tue, 30 Jan 2001 16:47:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0ULl7Z16388 for ; Tue, 30 Jan 2001 16:47:07 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0UMrl062428; Tue, 30 Jan 2001 14:53:51 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Santosh Rao" , "ips" Subject: RE: iSCSI : Command Ordering Proposal. Date: Tue, 30 Jan 2001 13:42:04 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <200101300811.AAA10230@hpcuhe.cup.hp.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh, I was trying to view iSCSI as isolated from either SCSI or TCP. The mixing of SCSI tags or TCP Acks together with Client/Server sequencing clouds this conversation. By resources, I was focused only on the iSCSI function and not either TCP or SCSI. SCSI uses tags, TCP uses its sequence number. iSCSI uses CmdSN and StatSN. An iSCSI PDU may require repeating if ExpCmdSN does not acknowledge reception due to digest failure perhaps. This is a thorny issue as there is no sense of time to allow recovery. If there is a digest error, should the server tell the client via a type of selective acknowledgement or by means of reporting a digest error on who knows what? iSCSI v3: "It may happen that a target receives a message with a format error (inconsistent fields, reserved fields not 0, inexistent LUN etc.) or a digest error (invalid payload or header). The target returns the header of the message in error as the data of the response." "When a target receives an iSCSI data PDU with a data payload digest error, it MUST discard it and request retransmission with a R2T. When a target receives an iSCSI PDU with a header digest error or a payload digest error in anything but a data iSCSI PDU it MUST answer with a Reject iSCSI PDU with a Reject iSCSI PDU with a Reason-code of Digest-error." Here is where a sequence is lost. "When an initiator receives an iSCSI data PDU with a data payload digest error or any other iSCSI PDU with a header or payload digest error it MUST discard it, and restart the task - the later provided it could recognize the Initiator Task Tag. If the initiator can't recognize the Initiator Task Tag, (e.g., a header digest error) the initiators MUST logout the connection and restart it (including restarting all outstanding tasks)." From the client's perspective, its PDU was sent only if acknowledged. The same should be true for the server. Rather than using this level of handshake, a restart of the SCSI command is seen as recovery but then we lose any sense of sequence nor have these handshakes been fully utilized in establishing a reliable link. Using bad data to restart a command? That does not seem a means of ensuring reliability. View iSCSI as a transport independent of both TCP and SCSI and then we can make progress on understanding how this should work. A brew of all three is where we are now. Doug > Doug, > > I'm not sure if we are speaking the same issue here. Can you > please elaborate further on your concerns. > > The handshake being discussed by me does NOT have to do with > freeing resources. It is a handshake b/n the target & > initiator on the CmdSN/ExpCmdSN, so that "retry" commands > from initiator to target remain with the (ExpCmdSN, MaxCmdSN) > window. > > Resources at the target end are de-allocated when it receives > an ExpStatSN for the StatSN. (IOW, resource de-allocation has > to do with StatSN ACKs from the initiator, not CmdSN). > If a target were not to implement status recovery, resource > de-allocation can occur once a TCP ACK is received for the > SCSI Response PDU. (if a mechanism were to exist to identify > that all the octets that constitute the Status PDU have been > ACK'ed.) > > This also brings up an interesting issue which is : > > If the target needs to hold onto its status PDU until at least > the TCP ACK is received for all of the bytes that constitute > the PDU and there is no easy mechanism to associate b/n > a TCP ACK and all bytes of the Status PDU being ACK'ed, > the simplest approach becomes de-allocation of status & > I/O resources based on ExpStatSN. > > If that is the case, the spec should change > "targets MAY implement status recovery" > to "targets MUST hold onto status PDU until ExpStatSN" update > is received. > > Regards, > Santosh > > > > > Douglas Otis wrote: > > > > > > Matt, > > > > > > Whether you use the CmdSN or the tag to associate the retry, > the CmdSN is > > > still used as a means to handshake freeing of resources and > not the tag. > > > > You are wrong. It specifically states in the document in 1.2.2.1: > > > > "CmdRNs are significant only during command delivery to the target. > > Once the device serving part of the target SCSI has received a > > command, CmdRN ceases to be significant." > > > > Any resources the target allocates must be based on the task tag. > > > > -Matt > > > > > > > > > It > > > seems like a cleaner solution not to include exceptions on > maintaining the > > > sequence just to allow this type of non-sequential use. > Clearly there is an > > > alternative that has benefits. Simply have the target allow > exceptions based > > > on the command or task attribute and not drop the ability to > handshake. I > > > am aware of the present concept, but you did not speak about > any downside to > > > this alternative. > > > > > > Doug > > > > > > -- > ################################# > Santosh Rao > Software Design Engineer, > HP, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################# > From owner-ips@ece.cmu.edu Tue Jan 30 21:44:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA21569 for ; Tue, 30 Jan 2001 21:44:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0V0ctY22997 for ips-outgoing; Tue, 30 Jan 2001 19:38:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from tiny-teddy.aarnet.edu.au (IDENT:root@tiny-teddy.aarnet.edu.au [203.21.37.30]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0V0TPZ22686 for ; Tue, 30 Jan 2001 19:29:25 -0500 (EST) Received: from aarnet.edu.au (wired-usr88.wired-238.uhnet.net [128.171.238.88]) by tiny-teddy.aarnet.edu.au (8.9.3/8.9.3) with ESMTP id KAA21444; Wed, 31 Jan 2001 10:59:10 +1030 Message-ID: <3A775C8B.BF315CA5@aarnet.edu.au> Date: Wed, 31 Jan 2001 11:00:03 +1030 From: Glen Turner Organization: Australian Academic and Research Network X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.4.0 i686) X-Accept-Language: en MIME-Version: 1.0 To: Sean Quinlan , IPS Reflector Subject: Re: iSCSI Data Integrity - Digests References: <5.0.0.25.2.20010126165115.00a73a80@hpindlm.cup.hp.com> <5.0.0.25.2.20010129124010.023ce6d0@hpindlm.cup.hp.com> <3A763A5B.B11ADDA6@bell-labs.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Sean Quinlan wrote: > If TLS or IPsec are used in conjunction with iSCSI, then strong > end-to-end integrity is already provided and duplicating this in iSCSI seem > rather a waste. Um, this doesn't match the paper presented at SIGCOMM2000 showing an uncomfortably high level of errors *above* the TCP layer due to faulty computer hardware. [1] The paper strongly recommended application-layer checksums. Also, your analogy with HTTP is a bit misleading. iSCSI errors are cumulative, as disks are usually attached to iSCSI sessions and a write corruption is stored to disk for later reading. > If data integrity is provided by iSCSI, CRC algorithms are not particularly > ideal from a software implementation point of view - surely there > are better alternatives than CRC that are easy to implement in hardware and > can take advantage of 32bit wide data ops. There is unlikely to be an algorithm better than CRC that is also faster than CRC. Essentially, to handle data in words it appears that you need to trade off the level of protection. I know of no paper that explores this assumption, but math is not my strong point. Glen [1] Sorry that I can't give a reference, I'm in the terminal room at the Internet2 Joint Techs meeting. -- Glen Turner Network Engineer (08) 8303 3936 Australian Academic and Research Network glen.turner@aarnet.edu.au http://www.aarnet.edu.au/ -- The revolution will not be televised, it will be digitised From owner-ips@ece.cmu.edu Tue Jan 30 23:13:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA23047 for ; Tue, 30 Jan 2001 23:13:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0V28xw26168 for ips-outgoing; Tue, 30 Jan 2001 21:08:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0V28rZ26159 for ; Tue, 30 Jan 2001 21:08:53 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id E4ED2C67 for ; Tue, 30 Jan 2001 18:08:52 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA25293 for ; Tue, 30 Jan 2001 18:08:48 -0800 (PST) Message-ID: <3A777451.F5BBB1D8@cup.hp.com> Date: Tue, 30 Jan 2001 18:11:29 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI : Command Ordering Proposal. References: Content-Type: multipart/mixed; boundary="------------62021682850A7FF0625CAA67" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------62021682850A7FF0625CAA67 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > Santosh, > > The only puzzling question is what kind of digest error would you detect > for a command that the target has not yet started acting upon? Julian, The target has started acting on the command. However, there is nothing in the standard that requires a ExpCmdSN ACK to accompany the inbound Data PDUs for a READ command. In the absence of this, targets are free to send in delayed ExpCmdSN updates and can commence transmission of Data PDUs [which are not yet ACKing the CmdSN for the command in response to which the Data PDU is being sent.] Either : a) An inbound Data PDU should be considered an implicit ack of the CmdSN which was sent. Or : b) When initiators detect digest errors on an inbound Data PDUs (header or digest), they should send the "retry" with a CmdSN of 0. > And even then if the task is recognized you could use a CmdSN of 0 > as the command is clearly active. The above is sufficient. The draft should state either (a) or (b) in its policy for initializing CmdSN on "retry" commands. Regards, Santosh --------------62021682850A7FF0625CAA67 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------62021682850A7FF0625CAA67-- From owner-ips@ece.cmu.edu Wed Jan 31 00:33:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA23914 for ; Wed, 31 Jan 2001 00:33:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0V3E1C28173 for ips-outgoing; Tue, 30 Jan 2001 22:14:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from plan9.cs.bell-labs.com (plan9.bell-labs.com [204.178.31.2]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id f0V37dZ27977 for ; Tue, 30 Jan 2001 22:07:39 -0500 (EST) Received: from research.bell-labs.com ([135.104.9.225]) by plan9; Tue Jan 30 22:07:36 EST 2001 Message-ID: <3A778275.9B41AEDD@research.bell-labs.com> Date: Tue, 30 Jan 2001 22:11:49 -0500 From: Sean Quinlan X-Mailer: Mozilla 4.72 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Glen Turner CC: Sean Quinlan , IPS Reflector Subject: Re: iSCSI Data Integrity - Digests References: <5.0.0.25.2.20010126165115.00a73a80@hpindlm.cup.hp.com> <5.0.0.25.2.20010129124010.023ce6d0@hpindlm.cup.hp.com> <3A763A5B.B11ADDA6@bell-labs.com> <3A775C8B.BF315CA5@aarnet.edu.au> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Glen, Glen Turner wrote: > > Sean Quinlan wrote: > > > If TLS or IPsec are used in conjunction with iSCSI, then strong > > end-to-end integrity is already provided and duplicating this in iSCSI seem > > rather a waste. > > Um, this doesn't match the paper presented at SIGCOMM2000 showing an > uncomfortably high level of errors *above* the TCP layer due to faulty > computer hardware. [1] > > The paper strongly recommended application-layer checksums. I think the paper you mention is "When the CRC and TCP checksum disagree" by Stone and Partridge http://www.acm.org/sigcomm/sigcomm2000/conf/techprog.htm I had not seen this paper and it is certainly an interesting read. One of the conclusions of the paper is that if you really care about your data you should do a checksum before the data has passed through a DMA engine. They mention SSL (also know as TLS) as one way to do this, but claim that IPsec is not a good option since it is often done by hardware assist in the NIC after the DMA - Then again if you really believe this, then you need to do checksums before passing data to your plain old SCSI card, since this also uses a DMA engine. This seems a little impractical! Also, considering much of the iSCSI discussions appears to expect that iSCSI will be offloaded onto hardware on an adapter card, a checksum at the iSCSI layer would probably occur after the DMA engine as well... The paper states In the past, one of the authors has periodically recommended improving transmission performance by doing the checksum as part of the DMA process or in the transmission path in the network interface. Base on this study, we can now say that advice is wrong because it leave data too exposed to hardware errors. This sort of goes against the current trend in high speed NICs which provide offloading of TCP checksums... My guess is people want the speed provided such offloading and if the DMA is really causing high error rates, then the solution is to build more reliable DMA/PCI interfaces. As a side note, I had a look at the number of TCP checksum errors on out outside HTTP server and on an internal cpu server. The outside machine an error rate of about 1 in 15,000 while the internal machine had an error rate of 1 in 1,000,000. Both these numbers where higher than I would have expected! Thus, the paper certainly has a point! > > Also, your analogy with HTTP is a bit misleading. iSCSI errors > are cumulative, as disks are usually attached to iSCSI sessions > and a write corruption is stored to disk for later reading. > HTTP might not be a perfect analogy, but I believe NFS and CIFS are pretty close. I believe almost every byte written to a NetApp server goes over one of these two protocols and does not include more integrity checking than provided by TCP and the link layer. > > If data integrity is provided by iSCSI, CRC algorithms are not particularly > > ideal from a software implementation point of view - surely there > > are better alternatives than CRC that are easy to implement in hardware and > > can take advantage of 32bit wide data ops. > > There is unlikely to be an algorithm better than CRC that is also > faster than CRC. Essentially, to handle data in words it appears > that you need to trade off the level of protection. I know of no > paper that explores this assumption, but math is not my strong point. > I am certainly no expert in checksums, but I believe the adler32 checksum used in zip (rfc1950) and SCTP (rfc2960) is claimed to be as good as a 32 bit CRC and it is more friendly to software implementation on a 32 bit machine, though given that it uses the mod operator it is more difficult than CRC to implement in hardware. Secure hashes such as md5 are also fairly reasonable in software with the added bonus that they can resist tampering - these are also probably difficult to do in hardware. From owner-ips@ece.cmu.edu Wed Jan 31 02:24:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA07624 for ; Wed, 31 Jan 2001 02:24:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0V5Q4802088 for ips-outgoing; Wed, 31 Jan 2001 00:26:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0V5PuZ02080 for ; Wed, 31 Jan 2001 00:25:56 -0500 (EST) Received: from yp_portable (slip-32-102-64-143.ca.us.prserv.net [32.102.64.143]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id 1AACLSSH; Tue, 30 Jan 2001 21:25:53 -0800 From: "Y P Cheng" To: "'Ips@Ece. Cmu. Edu'" Subject: iSCSI: CmdSN and Retry Date: Tue, 30 Jan 2001 21:24:02 -0800 Message-ID: <001101c08b46$0582be80$8f406620@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit A few points herein would help reduce the confusion of ordered delivery using CmdSN and error retry. If the initiator sends both commands A and B to a target and requires B to follow A. After completing A with an error condition, nothing prevents the target to start B immediately. Therefore, what do we accomplishing by retrying A? In every SCSI implementation I know of, initiator is always ultimately responsible of understanding the ordered execution as well as retry. Therefore, the retry corner cases pointed by Santosh simply will not exist with properly behaved initiator, unless an iSCSI initiator will behave differently. Since TCP ensures ordered delivery, for an iSCSI session with multiple TCP connections, all we need to do is to ensure the sequentiality of CmdSN from multiple connections. The real question is what is the timeout value of TCP before starting retransmit? Even if we yank a TCP connection, it does not change the requirement of enforcing sequential CmdSN. I am not aware of any SCSI target allocates resource for status phase. Once the status is sent, all resources are released. If the initiator times out the status, it retries the whole command. A target certainly can timeout the ACK of the status before releasing all resources. But, the target timeout should be shorter than that of the initiator. But, it gets really confusing if the target TCP has a shorter timeout value than the initiator TCP. A header digest error is same as a missing PDU except it is detect by iSCSI, not TCP. Because TCP has delivered the segment, it is possible for the receiver to quickly notify the sender to resend the erroneous header. Again, for a target if non-zero CmdSN is enforced, it must wait for the resend. In conclusion, if CmdSN is enforced, a target must take the TCP transport delivery sequentially whether there is one or more TCP connections because the missing one could just be an ordered-queue or head-of-queue. For a header digest error, a target can't proceed until it gets the missing header as long as CmdSN is non-zero. Since a target will always move to next command as soon as it completes one with or without error, all application software and initiators should know better not to send a SCSI request which depends on the success of a previous request. The exception to the above discussion is the tape I/O. This is because the tape devices use "lying writes". Once write data gets into device buffer, the device takes full responsibility of writing the data to media. All read data are buffered. Another read follows an erroneous one reads the same data. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ece.cmu.edu Wed Jan 31 02:25:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA07652 for ; Wed, 31 Jan 2001 02:25:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0V6F4f03447 for ips-outgoing; Wed, 31 Jan 2001 01:15:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (ns1.advansys.com [204.247.22.3] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0V6EOZ03431 for ; Wed, 31 Jan 2001 01:14:25 -0500 (EST) Received: from yp_portable (slip-32-102-109-117.ca.us.prserv.net [32.102.109.117]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id 1AACLST2; Tue, 30 Jan 2001 22:14:22 -0800 From: "Y P Cheng" To: "'Ips@Ece. Cmu. Edu'" Subject: RE: iSCSI: CmdSN and Retry Date: Tue, 30 Jan 2001 22:12:29 -0800 Message-ID: <000001c08b4c$cae5b140$756d6620@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > All read data are buffered. Another > read follows an erroneous one reads the same data. My mistake. For tape read retry, one needs a BKSP, Back-Space, command before sending a read again. A SCSI initiator with help from application software issues the BKSP. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ece.cmu.edu Wed Jan 31 02:26:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA07663 for ; Wed, 31 Jan 2001 02:26:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0V5w4702947 for ips-outgoing; Wed, 31 Jan 2001 00:58:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0V5vPZ02930 for ; Wed, 31 Jan 2001 00:57:26 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id GAA27376; Wed, 31 Jan 2001 06:57:17 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id GAA180114; Wed, 31 Jan 2001 06:57:13 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E5.0020B35C ; Wed, 31 Jan 2001 06:57:10 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu cc: ralphoweber@compuserve.com, Charles Binford , ips@ece.cmu.edu Message-ID: Date: Wed, 31 Jan 2001 07:33:46 +0200 Subject: RE: Comments on iSCSI -03 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Elliott, There seems to be a requirement in SAM to send the SCSI event indicator and as you point out a discrepancy with SPC. I would appreciate if one of the T10 luminaries would comment. Julo "Elliott, Robert" on 30/01/2001 17:42:06 Please respond to "Elliott, Robert" To: Julian Satran/Haifa/IBM@IBMIL, Charles Binford , ips@ece.cmu.edu cc: Subject: RE: Comments on iSCSI -03 > [cb-09] page 57 > Two problems with the following list; 5 & 6 should be 4 & 5, > and item number 3 is not in SAM-2 r15 (the latest version > I could find). Was something recently added? Again, I > believe the new Task Aborted Status solves the problem. > > 2.17.2 SCSI Event Indicator > > The following values are defined. (See [SAM2] for details): > > 1 An error condition was encountered after command > completion. > 2 A newly initialized device is available to this initiator. > 3 All Task Sets are being Reset by another Initiator > 5 Some other type of unit attention condition has occurred. > 6 An asynchronous event has occurred. >numbering will be fixed. I will be glad to change/elliminate > 3 to task aborted if I get a good pointer and it is good for > commands in flight -= > > meanwhile would it be fair to say that it is subsumed by 6 Julian, according to your message on 4 Jan copied below, the SCSI Event reasons are going to be eliminated altogether. The sense data contains the most accurate information about which event occurred. --- PC: Robert.Elliott@compaq.com UNIX: relliott@unixmail.compaq.com Previous message: To: "Elliott, Robert" Subject: Re: iSCSI: SCSI Event Indicator From: julian_satran@il.ibm.com Date: Thu, 4 Jan 2001 09:55:06 +0200 cc: ips@ece.cmu.edu Sender: owner-ips@ece.cmu.edu Will do, Thanks, Julo "Elliott, Robert" on 03/01/2001 19:00:58 Please respond to "Elliott, Robert" To: "'ips@ece.cmu.edu'" cc: Subject: iSCSI: SCSI Event Indicator The Asynchronous Event data structure includes an "iSCSI Event Indicator" field and a "SCSI Event Indicator" field. The iSCSI field indicates iSCSI-specific causes, which is fine. The SCSI Event field indicates generic SCSI causes. I suggest removing the SCSI Event Indicator values. The information is contained within the sense data itself. The SEND command in SPC-2, the current mechanism to report Asynchronous Events, does not include an Event field like this. I suggest simplifying the field to: 0 = not a SCSI Event 1 = SCSI Event This is the text in iSCSI revision 02, with comments listing where in the sense data that information is communicated: 2.17.2 SCSI Event Indicator The following values are defined. (See [SAM2] for details): 1 An error condition was encountered after command completion. [Sense data RESPONSE CODE field is DEFERRED ERRORS - 71h] 2 A newly initialized device is available to this initiator. [Sense data ASC/ASCQ is REPORTED LUNS DATA HAS CHANGED] 3 All Task Sets are being Reset by another Initiator [Sense data ASC/ASCQ is COMMANDS CLEARED BY ANOTHER INITIATOR] [where is 4?] 5 Some other type of unit attention condition has occurred. [Sense data SENSE KEY field is UNIT ATTENTION - 6h] 6 An asynchronous event has occurred. [Any other sense data] Sense Data accompanying the report identifies the condition. The Length parameter is set to the length of the Sense Data. For new device identification an iSCSI target MUST support the Device Identification page. The list was probably taken from this list in SAM-2; however, it's not an exact match: Asynchronous Event Reporting is used to signal a device that one of the four events listed below has occurred: a) an error condition was encountered after command completion; b) a newly initialized device is available; c) some other type of unit attention condition has occurred; or d) an asynchronous event has occurred. --- PC: Robert.Elliott@compaq.com UNIX: relliott@unixmail.compaq.com From owner-ips@ece.cmu.edu Wed Jan 31 02:26:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA07674 for ; Wed, 31 Jan 2001 02:26:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0V5w6r02950 for ips-outgoing; Wed, 31 Jan 2001 00:58:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0V5vUZ02931 for ; Wed, 31 Jan 2001 00:57:30 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id GAA93628; Wed, 31 Jan 2001 06:57:24 +0100 From: julian_satran@il.ibm.com Received: from d12mta05.de.ibm.com (d12mta05_cs0 [9.165.222.239]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id GAA232748; Wed, 31 Jan 2001 06:57:24 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E5.0020B474 ; Wed, 31 Jan 2001 06:57:13 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu cc: ralphoweber@compuserve.com, Charles Binford , ips@ece.cmu.edu Message-ID: Date: Wed, 31 Jan 2001 07:33:46 +0200 Subject: RE: Comments on iSCSI -03 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Elliott, There seems to be a requirement in SAM to send the SCSI event indicator and as you point out a discrepancy with SPC. I would appreciate if one of the T10 luminaries would comment. Julo "Elliott, Robert" on 30/01/2001 17:42:06 Please respond to "Elliott, Robert" To: Julian Satran/Haifa/IBM@IBMIL, Charles Binford , ips@ece.cmu.edu cc: Subject: RE: Comments on iSCSI -03 > [cb-09] page 57 > Two problems with the following list; 5 & 6 should be 4 & 5, > and item number 3 is not in SAM-2 r15 (the latest version > I could find). Was something recently added? Again, I > believe the new Task Aborted Status solves the problem. > > 2.17.2 SCSI Event Indicator > > The following values are defined. (See [SAM2] for details): > > 1 An error condition was encountered after command > completion. > 2 A newly initialized device is available to this initiator. > 3 All Task Sets are being Reset by another Initiator > 5 Some other type of unit attention condition has occurred. > 6 An asynchronous event has occurred. >numbering will be fixed. I will be glad to change/elliminate > 3 to task aborted if I get a good pointer and it is good for > commands in flight -= > > meanwhile would it be fair to say that it is subsumed by 6 Julian, according to your message on 4 Jan copied below, the SCSI Event reasons are going to be eliminated altogether. The sense data contains the most accurate information about which event occurred. --- PC: Robert.Elliott@compaq.com UNIX: relliott@unixmail.compaq.com Previous message: To: "Elliott, Robert" Subject: Re: iSCSI: SCSI Event Indicator From: julian_satran@il.ibm.com Date: Thu, 4 Jan 2001 09:55:06 +0200 cc: ips@ece.cmu.edu Sender: owner-ips@ece.cmu.edu Will do, Thanks, Julo "Elliott, Robert" on 03/01/2001 19:00:58 Please respond to "Elliott, Robert" To: "'ips@ece.cmu.edu'" cc: Subject: iSCSI: SCSI Event Indicator The Asynchronous Event data structure includes an "iSCSI Event Indicator" field and a "SCSI Event Indicator" field. The iSCSI field indicates iSCSI-specific causes, which is fine. The SCSI Event field indicates generic SCSI causes. I suggest removing the SCSI Event Indicator values. The information is contained within the sense data itself. The SEND command in SPC-2, the current mechanism to report Asynchronous Events, does not include an Event field like this. I suggest simplifying the field to: 0 = not a SCSI Event 1 = SCSI Event This is the text in iSCSI revision 02, with comments listing where in the sense data that information is communicated: 2.17.2 SCSI Event Indicator The following values are defined. (See [SAM2] for details): 1 An error condition was encountered after command completion. [Sense data RESPONSE CODE field is DEFERRED ERRORS - 71h] 2 A newly initialized device is available to this initiator. [Sense data ASC/ASCQ is REPORTED LUNS DATA HAS CHANGED] 3 All Task Sets are being Reset by another Initiator [Sense data ASC/ASCQ is COMMANDS CLEARED BY ANOTHER INITIATOR] [where is 4?] 5 Some other type of unit attention condition has occurred. [Sense data SENSE KEY field is UNIT ATTENTION - 6h] 6 An asynchronous event has occurred. [Any other sense data] Sense Data accompanying the report identifies the condition. The Length parameter is set to the length of the Sense Data. For new device identification an iSCSI target MUST support the Device Identification page. The list was probably taken from this list in SAM-2; however, it's not an exact match: Asynchronous Event Reporting is used to signal a device that one of the four events listed below has occurred: a) an error condition was encountered after command completion; b) a newly initialized device is available; c) some other type of unit attention condition has occurred; or d) an asynchronous event has occurred. --- PC: Robert.Elliott@compaq.com UNIX: relliott@unixmail.compaq.com From owner-ips@ece.cmu.edu Wed Jan 31 02:27:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA07690 for ; Wed, 31 Jan 2001 02:27:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0V5w8C02953 for ips-outgoing; Wed, 31 Jan 2001 00:58:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0V5vEZ02917 for ; Wed, 31 Jan 2001 00:57:15 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id GAA26890 for ; Wed, 31 Jan 2001 06:57:08 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id GAA265312 for ; Wed, 31 Jan 2001 06:57:08 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E5.0020B0F6 ; Wed, 31 Jan 2001 06:57:04 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 31 Jan 2001 07:28:20 +0200 Subject: Re: Comments on iSCSI -03 Mime-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh" Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline This is a resend - please let me know if it is still broken. Julo ---------------------- Forwarded by Julian Satran/Haifa/IBM on 31/01/2001 07:27 --------------------------- Julian Satran 30/01/2001 11:44 To: "Charles Binford" cc: ips@ece.cmu.edu From: Julian Satran/Haifa/IBM@Haifa/IBM@IBMIL Subject: Re: Comments on iSCSI -03 (Document link: Julian Satran - Mail) Comments in text. Thanks, Julo "Charles Binford" on 29/01/2001 23:09:46 Please respond to "Charles Binford" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Comments on iSCSI -03 Julian, Below are several comments on iSCSI-03. I --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6m fairly new to the iSCSI discussion (but have been involved in T10/T11 for several years) = so I apologize if I= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6m rehashing already made decisions. I= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6ve sent this directly to you instead of the general reflector because many of the items are trivial editing issues. Please feel free to cut and paste to the refle= ctor any issue(s) you feel justify a wider discussion. (Of course I reserve= the right to do the same if I strongly disagree with your answer :-) ) Before I get into specifics, let me start with an overall comment. Com= ing from T10/T11 work, it bothers me that many of the PDU descriptions in section 2 are not complete. It seems to me an attitude was taken that unless a field has new meaning for this PDU, it doesn= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6t need any text. I disagree. I believe that every field in every PDU needs a description.= In many cases that description may only spell out the acronym and then say= = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6See x.y= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 but at least the implementer using the spec for reference can quick= ly get a pointer to the authoritative text on the given field. Thanks for your time. Charles Binford Blue Spruce Networks ************************************************* [cb-01] pages 8 and 9 In the following two places = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6CDB= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 is referred to as = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6Command Data Block= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6. It= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6s real definition (from SAM) is Command Descriptor Block. ************************************************* will change 1. Overview 1.1 SCSI Concepts . . . Command Data Blocks (CDB) are the data structures used to contain th= e ^^^^^^^^^^^^^^^^^^^^^^^^^ command parameters to be handed by an initiator to a target. The CDB= content and structure is defined by [SAM] and device-type specific SCSI standards. 1.2.1 Layers & Sessions The following conceptual layering model is used in this document to specify initiator and target actions and how those relate to transmitted and received Protocol Data Units: -the SCSI layer builds/receives SCSI CDB (Command Data Blocks) ^^^^^^^^^^^^^^^^^^^ and relays/receives them with the remaining command execute parameters (cf. SAM-2) to/from the -the iSCSI layer that builds/receives iSCSI PDUs and relays/receives them to/from - one or more TCP connections that form an initiator-target "session". ************************************************* [cb-02] page 11 It seems to read better if you change = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6target SCSI= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 to = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6SCSI target= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6. ************************************************* 1.2.2.1 Command numbering . . . CmdRNs are significant only during command delivery to the target. Once the device serving part of the target SCSI has received a ^^^^^^^^^^^ command, CmdRN ceases to be significant. During command delivery to= the target, the allocated numbers are unique session wide. will fix ************************************************* [cb-03] page 14 Change = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6an= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 to = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6and= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6. ************************************************* 1.2.5 iSCSI Full Feature Phase . . . that was used to deliver the SCSI command. If an initiator issues a= WRITE command, the initiator must send the data, if any, for that command and the target MUST return R2T, if any, an the status over ^^^ the same TCP connection that was used to deliver the SCSI command. will fix ************************************************* [cb-04] page 22 Change = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6Response bit (bit 6)= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 to = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6Response bit (bit 7)= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6. ************************************************* 2.2.1 Opcode The Opcode indicates what type of iSCSI PDU the header encapsulates.= The Opcode is further encoded as follows: b7 Response b6-0 Operation The opcodes are divided into two categories: initiator opcodes and target opcodes. Initiator opcodes are in PDUs sent by the initiators= , and target opcodes are in PDUs sent by the target. The initiator MUS= T NOT send target opcodes and the target MUST NOT send initiator opcodes. Target opcodes are also called responses and are distinguished by having the Response bit (bit 6) set to 1. ^^^ will fix ************************************************* [cb-05] page 30 Again, I= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6m having a hard time finding a concise set of rules of how this parameter (StatRN) is used. It is incremented by 1 for every response,= but what is the starting value; 0? 1, any non-zero? What conditions reset = it = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =FB any? Does is roll over back to 0, or does it explicitly skip 0 when rolling over from 0xffffffff? ************************************************* 2.4.7 StatRN - Status Reference Number StatRN is a reference number that the target iSCSI layer generates per connection and that in turn enables the initiator to acknowledge= status reception. StatRN is incremented by 1 for every response/status sent on a connection. There is an InitStatSN for every connection in the login response. = The value 0 has no special significance so you wrap around. As usuall if th= e difference between expected and sent is larger than 2**31-1 a recovery action MUST be taken. ************************************************* [cb-06] page 32 This section implies ACA handing is mandatory if AE reporting is suppor= ted. This seems to be an unnecessary linking of two independent concepts. At least part of what I think is being solved with this auto ACA requirement is handled better by the new Task Aborted Status described in SAM-2. (= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6better= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 in my biased opinion anyway = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =FB I was the author of the Task Aborted Status proposal.) Was utilizing Task Aborted considered and rejected, = or was it not even known/understood since it is so new? ************************************************* For the , if SCSI control mode enables AE reporting,= the target MUST send an Asynchronous Event to all other attached initiators to inform them that all pending tasks are cancelled and then enter the ACA state for any initiator for which it had pending ^^^^^^^^^^^^^^^^^^^ tasks. For the and functions, the target cancels all pending operations and are both equivalent to the= Target Reset as specified by SAM-2. Provided that SCSI control mode= enables AE reporting, the target MUST send an Asynchronous Event to all attached initiators notifying them that the target is being reset. In addition, for the the target will enter the ACA state on all sessions and all LUs on which an AE was sent. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ We heard about task aborted but had little understanding about how= to use it. If like ACA it will stay on untill cleared and will thus serve us to dr= op all commands in flight the we could use it. If it does not we will have= to stay with ACA. Again the issue where the commands in flight ************************************************* [cb-07] page 46 Need to reword sentence = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =FB the word = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6but= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 doesn= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6t fit. ************************************************* 2.11.2 Version-active/lowest Indicates the version supported (the highest supported by the target= and initiator). If the target is not supporting a version within the= Satran, J. Standards-Track, August 2001 45 iSCSI January 10, 2000 range of the initiator it will reject the login but and this field ^^^^ will indicate the lowest version supported by the target. will fix ************************************************* [cb-08] page 50 Sentence organization: In the description of the P bit, the last sente= nce the = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6IF= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 is the final phrase, making it hard to understand on the first reading. If the sentence is turned around (IF ... THEN style) I believ= e it is much clearer. ************************************************* 2.13.1 P bit A target may issue a NOP-In by its own to test connection and the state of the initiator. In this case the Initiator Task Tag MUST be = 0 and the Target Tag MUST be set (not x'ffffffff') only if the P bit i= s ^^^^^^^^^^^^^^^^^^^= ^ 1. ^^ will fix ************************************************* [cb-09] page 57 Two problems with the following list; 5 & 6 should be 4 & 5, and item number 3 is not in SAM-2 r15 (the latest version I could find). Was something= recently added? Again, I believe the new Task Aborted Status solves th= e problem. ************************************************* 2.17.2 SCSI Event Indicator The following values are defined. (See [SAM2] for details): 1 An error condition was encountered after command completion. 2 A newly initialized device is available to this initiator. 3 All Task Sets are being Reset by another Initiator 5 Some other type of unit attention condition has occurred. 6 An asynchronous event has occurred. numbering will be fixed. I will be glad to change/elliminate 3 to t= ask aborted if I get a good pointer and it is good for commands in flight -= meanwhile would it be fair to say that it is subsumed by 6 ************************************************* [cb-10] page 59 At the end of section 2, and there is no description of Markers. I bel= ieve the standard needs a format diagram just like any of the other PDUs = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =FB maybe not in section 2, but somewhere. ************************************************* markers will move to an appendix and I'll add a descriptor ************************************************* [cb-11] page 60 Why redefine the mode page??? The Max Burst Size ad First Burst Size h= ave been defined as 512 byte chunks for a long time. A SCSI target doesn= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6t want to have to translate fields based on the particular transport. The 512= definition give a range of up to just under 32MB = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =FB seems like a big enough = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =E6first burst= --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=us-ascii Content-Disposition: inline --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable =C6 to me. ************************************************* 3.1.2 Maximum Burst Size field (16 bit) This field is used by iSCSI to define the maximum data payload in iSCSI data PDUs or as immediate data in command PDUs in units of 409= 6 ^^^^^^^^^^^^^= ^ bytes. This value can also be set by a text-mode key:value pair (DataPDULength). 3.1.3 First Burst Size field (16 bit) This field is used by iSCSI to define the maximum of unsolicited dat= a an iSCSI initiator is allowed to send to the target in units of 4096= ^^^^^^^^^^^^^= bytes. This value can also be set by a text-mode key:value pair (InitialBurstLength). We where told that those fields are protocol specific and we have= to define their values. If a multiple of 512 is more a better fit for SCS= I I will change it Charles Binford Blue Spruce Networks office/cell: (316) 210-6404 e-fax: (509) 756-4425 = --0__=UlT9g8BImtKMjbZw74I7x5AhWTp8nzBHin3An1TS3on6lzLIM8RBTbLh-- From owner-ips@ece.cmu.edu Wed Jan 31 04:28:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA08410 for ; Wed, 31 Jan 2001 04:28:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0V7Z6Z05466 for ips-outgoing; Wed, 31 Jan 2001 02:35:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0V7Y4Z05433 for ; Wed, 31 Jan 2001 02:34:04 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA39844 for ; Wed, 31 Jan 2001 08:33:54 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA188056 for ; Wed, 31 Jan 2001 08:33:54 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E5.00298B4C ; Wed, 31 Jan 2001 08:33:46 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 31 Jan 2001 08:52:46 +0200 Subject: Re: iSCSI: CmdSN and Retry Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Except for sending the status - an executing command helds-up the LU queue and makes the "local" recovery simpler than clearing the LU queue and resending the commands. Add to this the fact that you don't have ACA for service responses not seen by the target (and even for those seen) and you have a perfect "appetite enhancer" for command recovery. The issue in my mind is to make this a mandatory behaviour that can only be excplicitily called off by the target or initiatior (by clearing the LU) or make it negotiable at login. Julo "Y P Cheng" on 31/01/2001 07:24:02 Please respond to "Y P Cheng" To: "'Ips@Ece. Cmu. Edu'" cc: Subject: iSCSI: CmdSN and Retry A few points herein would help reduce the confusion of ordered delivery using CmdSN and error retry. If the initiator sends both commands A and B to a target and requires B to follow A. After completing A with an error condition, nothing prevents the target to start B immediately. Therefore, what do we accomplishing by retrying A? In every SCSI implementation I know of, initiator is always ultimately responsible of understanding the ordered execution as well as retry. Therefore, the retry corner cases pointed by Santosh simply will not exist with properly behaved initiator, unless an iSCSI initiator will behave differently. Since TCP ensures ordered delivery, for an iSCSI session with multiple TCP connections, all we need to do is to ensure the sequentiality of CmdSN from multiple connections. The real question is what is the timeout value of TCP before starting retransmit? Even if we yank a TCP connection, it does not change the requirement of enforcing sequential CmdSN. I am not aware of any SCSI target allocates resource for status phase. Once the status is sent, all resources are released. If the initiator times out the status, it retries the whole command. A target certainly can timeout the ACK of the status before releasing all resources. But, the target timeout should be shorter than that of the initiator. But, it gets really confusing if the target TCP has a shorter timeout value than the initiator TCP. A header digest error is same as a missing PDU except it is detect by iSCSI, not TCP. Because TCP has delivered the segment, it is possible for the receiver to quickly notify the sender to resend the erroneous header. Again, for a target if non-zero CmdSN is enforced, it must wait for the resend. In conclusion, if CmdSN is enforced, a target must take the TCP transport delivery sequentially whether there is one or more TCP connections because the missing one could just be an ordered-queue or head-of-queue. For a header digest error, a target can't proceed until it gets the missing header as long as CmdSN is non-zero. Since a target will always move to next command as soon as it completes one with or without error, all application software and initiators should know better not to send a SCSI request which depends on the success of a previous request. The exception to the above discussion is the tape I/O. This is because the tape devices use "lying writes". Once write data gets into device buffer, the device takes full responsibility of writing the data to media. All read data are buffered. Another read follows an erroneous one reads the same data. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ece.cmu.edu Wed Jan 31 04:29:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id EAA08421 for ; Wed, 31 Jan 2001 04:29:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0V7Y6M05437 for ips-outgoing; Wed, 31 Jan 2001 02:34:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0V7XoZ05427 for ; Wed, 31 Jan 2001 02:33:50 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA26714 for ; Wed, 31 Jan 2001 08:33:44 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id IAA269514 for ; Wed, 31 Jan 2001 08:33:43 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E5.00298888 ; Wed, 31 Jan 2001 08:33:39 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 31 Jan 2001 08:37:36 +0200 Subject: Re: iSCSI : Command Ordering Proposal. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I will spell-out more recovery scenarios in the next draft. Julo Santosh Rao on 31/01/2001 04:11:29 Please respond to Santosh Rao To: ips@ece.cmu.edu cc: Subject: Re: iSCSI : Command Ordering Proposal. julian_satran@il.ibm.com wrote: > Santosh, > > The only puzzling question is what kind of digest error would you detect > for a command that the target has not yet started acting upon? Julian, The target has started acting on the command. However, there is nothing in the standard that requires a ExpCmdSN ACK to accompany the inbound Data PDUs for a READ command. In the absence of this, targets are free to send in delayed ExpCmdSN updates and can commence transmission of Data PDUs [which are not yet ACKing the CmdSN for the command in response to which the Data PDU is being sent.] Either : a) An inbound Data PDU should be considered an implicit ack of the CmdSN which was sent. Or : b) When initiators detect digest errors on an inbound Data PDUs (header or digest), they should send the "retry" with a CmdSN of 0. > And even then if the task is recognized you could use a CmdSN of 0 > as the command is clearly active. The above is sufficient. The draft should state either (a) or (b) in its policy for initializing CmdSN on "retry" commands. Regards, Santosh From owner-ips@ece.cmu.edu Wed Jan 31 05:25:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id FAA08798 for ; Wed, 31 Jan 2001 05:25:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0V8V7G06940 for ips-outgoing; Wed, 31 Jan 2001 03:31:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0V8V5Z06936 for ; Wed, 31 Jan 2001 03:31:05 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 8B4911361 for ; Wed, 31 Jan 2001 00:31:04 -0800 (PST) Received: (from santoshr@localhost) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) id AAA15648 for ips@ece.cmu.edu; Wed, 31 Jan 2001 00:31:00 -0800 (PST) From: Santosh Rao Message-Id: <200101310831.AAA15648@hpcuhe.cup.hp.com> Subject: iSCSI : Holes in StatSN To: ips@ece.cmu.edu (ips) Date: Wed, 31 Jan 2001 00:30:59 -0800 (PST) X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian & All, The StatSN ACK is of the form "ExpStatSN acknowledges all Status upto the specified value", and hence, it cannot be sent until all prior command status' have been received. If StatSN 1 were yet to be received and StatSN 2 - 1000 were received thereafter, they cannot be acknowledged by the initiator since the current model does not allow an ExpStatSN ack of 2 - 1000, without also ACKing 1. Since StatSNs are assigned on a per-connection basis, this avoids holes in StatSN received at the initiator. [since TCP orders delivery to iSCSI within a connection]. However, this in itself is insufficient and holes can still occur in the StatSN sequence seen by the initiator, as explained below. Holes in StatSN sequence are seen by an initiator when it detects a digest error on the Status PDU [and discards that PDU] , thereby, dropping such Status'. In such cases, the ExpStatSN acknowledgement is not straight forward at the initiator and does involve some complexity to keep track of possible holes in StatSN. Further, such holes may never be filled, since the "retry" command only uses the same CmdSN while the target sends its response on a different StatSN. In the presence of StatSN holes, [and given the current model of ExpStatSN], initiators will need to score-board received StatSNs prior to sending ExpStatSN acknowledgements. A selective StatSN ack (i.e. ExpStatSN ACKs the specified StatSN) is simpler to implement on the initiator side and allows for quicker de-alloc of resources at the target end. This could be considered as either a replacement for the existing ExpStatSN model, or as a complement to it. (possibly indicated by a SACK bit in the outbound PDUs containing the ExpStatSN). Regards, Santosh ps : There is an earlier thread that dates 10/26/00 and is titled "Re: iSCSI Error Recovery", which proposed StatSN per connection as a solution to this problem, but the problem does not go away with just setting StatSNs per connection, since holes still occur on digest errors. -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Wed Jan 31 09:42:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id JAA15375 for ; Wed, 31 Jan 2001 09:42:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0VD3AI21789 for ips-outgoing; Wed, 31 Jan 2001 08:03:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0VD2NZ21771 for ; Wed, 31 Jan 2001 08:02:24 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id OAA87626 for ; Wed, 31 Jan 2001 14:02:17 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id OAA30218 for ; Wed, 31 Jan 2001 14:02:17 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E5.00479BFA ; Wed, 31 Jan 2001 14:02:09 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 31 Jan 2001 14:57:46 +0200 Subject: Re: iSCSI : Holes in StatSN Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I've also realized this and included a "sack" mechanism that kicks-in as soon as the initiator gets a hole in the status order. ExpstatSN is still needed to do a low cost bulk acknowledgement. With a data sequence we may want to use a similar mechanism to ask for a missed data block as soon as we see one of its successors or the status. Julo Santosh Rao on 31/01/2001 10:30:59 Please respond to Santosh Rao To: ips@ece.cmu.edu (ips) cc: Subject: iSCSI : Holes in StatSN Julian & All, The StatSN ACK is of the form "ExpStatSN acknowledges all Status upto the specified value", and hence, it cannot be sent until all prior command status' have been received. If StatSN 1 were yet to be received and StatSN 2 - 1000 were received thereafter, they cannot be acknowledged by the initiator since the current model does not allow an ExpStatSN ack of 2 - 1000, without also ACKing 1. Since StatSNs are assigned on a per-connection basis, this avoids holes in StatSN received at the initiator. [since TCP orders delivery to iSCSI within a connection]. However, this in itself is insufficient and holes can still occur in the StatSN sequence seen by the initiator, as explained below. Holes in StatSN sequence are seen by an initiator when it detects a digest error on the Status PDU [and discards that PDU] , thereby, dropping such Status'. In such cases, the ExpStatSN acknowledgement is not straight forward at the initiator and does involve some complexity to keep track of possible holes in StatSN. Further, such holes may never be filled, since the "retry" command only uses the same CmdSN while the target sends its response on a different StatSN. In the presence of StatSN holes, [and given the current model of ExpStatSN], initiators will need to score-board received StatSNs prior to sending ExpStatSN acknowledgements. A selective StatSN ack (i.e. ExpStatSN ACKs the specified StatSN) is simpler to implement on the initiator side and allows for quicker de-alloc of resources at the target end. This could be considered as either a replacement for the existing ExpStatSN model, or as a complement to it. (possibly indicated by a SACK bit in the outbound PDUs containing the ExpStatSN). Regards, Santosh ps : There is an earlier thread that dates 10/26/00 and is titled "Re: iSCSI Error Recovery", which proposed StatSN per connection as a solution to this problem, but the problem does not go away with just setting StatSNs per connection, since holes still occur on digest errors. -- ################################# Santosh Rao Software Design Engineer, HP, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################# From owner-ips@ece.cmu.edu Wed Jan 31 09:42:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id JAA15410 for ; Wed, 31 Jan 2001 09:42:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0VCOA420836 for ips-outgoing; Wed, 31 Jan 2001 07:24:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0VCNNZ20817 for ; Wed, 31 Jan 2001 07:23:23 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id NAA29080 for ; Wed, 31 Jan 2001 13:23:15 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id NAA24086 for ; Wed, 31 Jan 2001 13:23:15 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E5.00440859 ; Wed, 31 Jan 2001 13:23:05 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 31 Jan 2001 12:11:30 +0200 Mime-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=DwvhFtsT5Ob1dZm4pxiT7mm10HH5PNhUfCds6FCa7J1Pn81xW8GzkGOQ" Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=DwvhFtsT5Ob1dZm4pxiT7mm10HH5PNhUfCds6FCa7J1Pn81xW8GzkGOQ Content-type: text/plain; charset=us-ascii Content-Disposition: inline Matt Wakeley pointed out to me that the formats I sent on the new outline have a type. (I left in the length field). Here are the Command and Response BSH for 04: (See attached file: Cmd-and-response.txt) --0__=DwvhFtsT5Ob1dZm4pxiT7mm10HH5PNhUfCds6FCa7J1Pn81xW8GzkGOQ Content-type: application/octet-stream; name="Cmd-and-response.txt" Content-Disposition: attachment; filename="Cmd-and-response.txt" Content-Description: Text - character set unknown Content-Transfer-Encoding: base64 Mi4zIFNDU0kgQ29tbWFuZCANCg0KQnl0ZSAvICAgIDAgICAgICAgfCAgICAgICAxICAgICAgIHwg ICAgICAgMiAgICAgICB8ICAgICAgIDMgICAgICAgfA0KICAgLyAgICAgICAgICAgICAgfCAgICAg ICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfA0KICB8NyA2IDUgNCAz IDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfA0K ICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKw0KIDB8IDB4MDEgICAgICAgICAgfFh8UnxXfDAgMHxBVFRSIHwgUmVzZXJ2ZWQg ICAgICB8IENtZFJOIG9yIFJzdmQgfA0KICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KIDR8IExvZ2ljYWwgVW5pdCBOdW1i ZXIgKExVTikgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICArICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg Kw0KIDh8ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgfA0KICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KMTJ8IEluaXRpYXRvciBUYXNrIFRhZyAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICArLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KMTZ8IEV4 cGVjdGVkIERhdGEgVHJhbnNmZXIgTGVuZ3RoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgfA0KICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKw0KMjB8IENtZFNOICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KMjR8IEV4cFN0YXRTTiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAr LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKw0KMjgvIFNDU0kgQ29tbWFuZCBEZXNjcmlwdG9yIEJsb2NrIChDREIpICAgICAgICAg ICAgICAgICAgICAgICAgICAgLw0KICsvICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0KICArLS0tLS0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KNDQNCg0KDQoyLjMu MSBGbGFncyAmIFRhc2sgQXR0cmlidXRlcw0KDQpUaGUgZmxhZ3MgZmllbGQgZm9yIGEgU0NTSSBD b21tYW5kIGlzOg0KDQoNCmI3ICAgUmV0cnkgKFgpIA0KYjYgICAoUikgc2V0IHRvIDEgd2hlbiBp bnB1dCBkYXRhIGlzIGV4cGVjdGVkICANCmI1ICAgKFcpIHNldCB0byAxIHdoZW4gb3V0cHV0IGRh dGEgaXMgZXhwZWN0ZWQNCmIzLTQgUmVzZXJ2ZWQgKE1VU1QgYmUgMCkNCmIwLTIgdXNlZCB0byBp bmRpY2F0ZSBUYXNrIEF0dHJpYnV0ZXMgDQoNClRoZSBUYXNrIEF0dHJpYnV0ZXMgKEFUVFIpIGNh biBoYXZlIG9uZSBvZiB0aGUgZm9sbG93aW5nIGludGVnZXIgdmFsdWVzIChzZWUgW1NBTTJdIGZv ciBkZXRhaWxzKToNCg0KMCAgICBVbnRhZ2dlZA0KMSAgICBTaW1wbGUNCjIgICAgT3JkZXJlZA0K MyAgICBIZWFkIG9mIFF1ZXVlDQo0ICAgIEFDQQ0KDQoyLjMuMiBDbWRSTiANCg0KU0NTSSBjb21t YW5kIHJlZmVyZW5jZSBudW1iZXIgLSBpZiBwcmVzZW50IGluIHRoZSBTQ1NJIGV4ZWN1dGUgYXJn dW1lbnRzDQoNCjIuMy4zIENtZFNOIC0gQ29tbWFuZCBTZXF1ZW5jZSBOdW1iZXINCg0KRW5hYmxl cyBvcmRlcmVkIGRlbGl2ZXJ5IGFjcm9zcyBtdWx0aXBsZSBjb25uZWN0aW9ucyBpbiBhIHNpbmds ZSBzZXNzaW9uLg0KDQoyLjMuNCBFeHBTdGF0U04gLSBFeHBlY3RlZCBTdGF0dXMgU2VxdWVuY2Ug TnVtYmVyIA0KDQpDb21tYW5kIHJlc3BvbnNlcyB1cCB0byBFeHBTdGF0U04tMSAobW9kIDIqKjMy KSBoYXZlIGJlZW4gcmVjZWl2ZWQgKGFja25vd2xlZGdlcyBzdGF0dXMpIG9uIHRoZSBjb25uZWN0 aW9uLiAgDQoNCjIuMy41IEV4cGVjdGVkIERhdGEgVHJhbnNmZXIgTGVuZ3RoDQoNCkZvciB1bmlk aXJlY3Rpb25hbCBvcGVyYXRpb25zLCB0aGUgRXhwZWN0ZWQgRGF0YSBUcmFuc2ZlciBMZW5ndGgg ZmllbGQgc3RhdGVzIHRoZSBudW1iZXIgb2YgYnl0ZXMgb2YgZGF0YSBpbnZvbHZlZCBpbiB0aGlz IFNDU0kgb3BlcmF0aW9uLiAgRm9yIGEgV1JJVEUgb3BlcmF0aW9uLCB0aGUgaW5pdGlhdG9yIHVz ZXMgdGhpcyBmaWVsZCB0byBzcGVjaWZ5IHRoZSBudW1iZXIgb2YgYnl0ZXMgb2YgZGF0YSBpdCBl eHBlY3RzIHRvIHRyYW5zZmVyIGZvciB0aGlzIG9wZXJhdGlvbi4gIEZvciBhIFJFQUQgb3BlcmF0 aW9uLCB0aGUgaW5pdGlhdG9yIHVzZXMgdGhpcyBmaWVsZCB0byBzcGVjaWZ5IHRoZSBudW1iZXIg b2YgYnl0ZXMgb2YgZGF0YSBpdCBleHBlY3RzIHRoZSB0YXJnZXQgdG8gdHJhbnNmZXIgdG8gdGhl IGluaXRpYXRvci4gIEl0IGNvcnJlc3BvbmRzIHRvIHRoZSBTQU0tMiBieXRlIGNvdW50Lg0KDQpJ ZiB0aGUgRXhwZWN0ZWQgRGF0YSB0cmFuc2ZlciBMZW5ndGggZm9yIGEgV1JJVEUgYW5kIHRoZSBs ZW5ndGggb2YgaW1tZWRpYXRlIGRhdGEgcGFydCB0aGF0IGZvbGxvd3MgdGhlIGNvbW1hbmQgKGlm IGFueSkgYXJlIHRoZSBzYW1lIHRoZW4gbm8gbW9yZSBkYXRhIFBEVXMgYXJlIGV4cGVjdGVkIHRv IGZvbGxvdy4NCg0KSWYgbm8gZGF0YSB3aWxsIGJlIHRyYW5zZmVycmVkIGluIFNDU0kgRGF0YSBw YWNrZXRzIGZvciB0aGlzIFNDU0kgb3BlcmF0aW9uLCB0aGlzIGZpZWxkIHNob3VsZCBiZSBzZXQg dG8gemVyby4NCg0KRm9yIGJpLWRpcmVjdGlvbmFsIG9wZXJhdGlvbnMsIHRoaXMgZmllbGQgc3Rh dGVzIHRoZSBudW1iZXIgb2YgZGF0YSBieXRlcyBpbnZvbHZlZCBpbiB0aGUgb3V0Ym91bmQgdHJh bnNmZXIuIEZvciBiaS1kaXJlY3Rpb25hbCBvcGVyYXRpb25zLCBhbiBhZGRpdGlvbmFsIGhlYWRl ciBzZWdtZW50IE1VU1QgYmUgcHJlc2VudCBpbiB0aGUgaGVhZGVyIHNlcXVlbmNlIGluZGljYXRp bmcgdGhlIEV4cGVjdGVkIEJpLWRpcmVjdGlvbmFsIFJlYWQgRGF0YSBMZW5ndGguICBJZiB0aGlz IGFkZGl0aW9uYWwgaGVhZGVyIHNlZ21lbnQgaXMgYWJzZW50IHRoZSBFeHBlY3RlZCBCaS1kaXJl Y3Rpb25hbCBSZWFkIERhdGEgTGVuZ3RoIE1VU1QgYmUgYXNzdW1lZCAwLg0KDQpVcG9uIGNvbXBs ZXRpb24gb2YgYSBkYXRhIHRyYW5zZmVyLCB0aGUgdGFyZ2V0IHdpbGwgaW5mb3JtIHRoZSBpbml0 aWF0b3Igb2YgaG93IG1hbnkgYnl0ZXMgd2VyZSBhY3R1YWxseSBwcm9jZXNzZWQgKHNlbnQgb3Ig cmVjZWl2ZWQpIGJ5IHRoZSB0YXJnZXQuICBUaGlzIHdpbGwgYmUgZG9uZSB0aHJvdWdoIHJlc2lk dWFsIGNvdW50cy4NCg0KMi4zLjYgQ0RCIC0gU0NTSSBDb21tYW5kIERlc2NyaXB0b3IgQmxvY2sN Cg0KVGhlcmUgYXJlIDE2IGJ5dGVzIGluIHRoZSBDREIgZmllbGQgdG8gYWNjb21tb2RhdGUgdGhl IGNvbW1vbmx5IHVzZWQgQ0RCLiAgV2hlbmV2ZXIgbGFyZ2VyIENEQnMgYXJlIHVzZWQsIHRoZSBD REIgc3BpbGxvdmVyIE1BWSBleHRlbmQgYmV5b25kIHRoZSA0OC1ieXRlIGhlYWRlci4NCg0KMi4z LjcgQ29tbWFuZC1EYXRhIA0KDQpTb21lIFNDU0kgY29tbWFuZHMgcmVxdWlyZSBhZGRpdGlvbmFs IHBhcmFtZXRlciBkYXRhIHRvIGFjY29tcGFueSB0aGUgU0NTSSBjb21tYW5kLiBUaGlzIGRhdGEg bWF5IGJlIHBsYWNlZCBiZXlvbmQgdGhlIGJvdW5kYXJ5IG9mIHRoZSBpU0NTSSBoZWFkZXIgKGEg ZGF0YSBzZWdtZW50KS4gIEFsdGVybmF0aXZlbHksIHVzZXIgZGF0YSAoYXMgZnJvbSBhIFdSSVRF IG9wZXJhdGlvbikgY2FuIGJlIHBsYWNlZCBpbiB0aGUgc2FtZSBQRFUgKGJvdGggY2FzZXMgcmVm ZXJyZWQgdG8gYXMgaW1tZWRpYXRlIGRhdGEpLg0KDQogDQoNCg0KMi40IFNDU0kgUmVzcG9uc2Ug DQoNCkJ5dGUgLyAgICAwICAgICAgIHwgICAgICAgMSAgICAgICB8ICAgICAgIDIgICAgICAgfCAg ICAgICAzICAgICAgIHwNCiAgIC8gICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAgICAg ICAgICAgICAgfCAgICAgICAgICAgICAgIHwNCiAgfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMg MiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHwNCiAgKy0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAwfCAw eDgxICAgICAgICAgIHxSc3ZkIHxTfG98dXxPfFV8IFJlc2VydmVkICgwKSAgICAgICAgICAgICAg ICAgIHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSsNCiA0fCBSZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgKyAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICsNCiA4fCAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAg Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSsNCjEyfCBJbml0aWF0b3IgVGFzayBUYWcgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCjE2fCBCYXNpYyBSZXNpZHVhbCBDb3Vu dCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsN CjIwfCBTdGF0U04gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCjI0fCBFeHBDbWRTTiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCjI4fCBNYXhD bWRTTiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLSsNCjMyfFN0YXR1cy9SZXNwb25zZXwgUmVzZXJ2ZWQgKDApICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCjM2fCBTUkxlbmd0aCAgICAg ICAgICAgICAgICAgICAgICB8IFJlc2VydmVkICgwKSAgICAgICAgICAgICAgICAgIHwNCiAgKy0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSsNCjQwfCBCaWRpLVJlYWQgUmVzaWR1YWwgQ291bnQgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHwNCiAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCjQ0LyBTZW5zZSBEYXRhIChvcHRpb25hbCkg b3IgUmVzcG9uc2UgRGF0YSAgICAgICAgICAgICAgICAgICAgICAgIC8NCiArLyAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAg Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSsNCg0KMi40LjEgQnl0ZSAxIC0gRmxhZ3MNCg0KYjAgICAoVSkgc2V0IGZvciBSZXNp ZHVhbCBVbmRlcmZsb3cuIEluIHRoaXMgY2FzZSwgdGhlIEJhc2ljIFJlc2lkdWFsIENvdW50IGlu ZGljYXRlcyBob3cgbWFueSBieXRlcyB3ZXJlIG5vdCB0cmFuc2ZlcnJlZCBvdXQgb2YgdGhvc2Ug ZXhwZWN0ZWQgdG8gYmUgdHJhbnNmZXJyZWQuDQpiMSAgIChPKSBzZXQgZm9yIFJlc2lkdWFsIE92 ZXJmbG93LiBJbiB0aGlzIGNhc2UsIHRoZSBCYXNpYyBSZXNpZHVhbCBDb3VudCBpbmRpY2F0ZXMg aG93IG1hbnkgYnl0ZXMgY291bGQgbm90IGJlIHRyYW5zZmVycmVkIGJlY2F1c2UgdGhlIGluaXRp YXRvcidzIEV4cGVjdGVkIERhdGEgVHJhbnNmZXIgTGVuZ3RoIHdhcyB0b28gc21hbGwuDQpiMiAg ICh1KSBzYW1lIGFzIGIwIGJ1dCBmb3IgdGhlIHJlYWQtcGFydCBvZiBhIGJpLWRpcmVjdGlvbmFs IG9wZXJhdGlvbiAgICAgDQpiMyAgIChvKSBzYW1lIGFzIGIxIGJ1dCBmb3IgdGhlIHJlYWQtcGFy dCBvZiBhIGJpLWRpcmVjdGlvbmFsIG9wZXJhdGlvbg0KYjQgICAoUykgU3RhdHVzLVJlc3BvbnNl IHNlbGVjdG9yIC0gaWYgMCB0aGUgcmVzcG9uc2UgY29udGFpbnMgYSB2YWxpZCBzdGF0dXMgZWxz ZSBhIHZhbGlkIFJlc3BvbnNlICAgICANCmIzLTcgbm90IHVzZWQgKFNIT1VMRCBiZSBzZXQgdG8g MCkNCg0KQml0cyBPIGFuZCBVIGFyZSBtdXR1YWxseSBleGNsdXNpdmUgYW5kIHNvIGFyZSBiaXRz IG8gYW5kIHUuDQoNCjIuNC4yIEJhc2ljIFJlc2lkdWFsIENvdW50IA0KDQpUaGUgQmFzaWMgUmVz aWR1YWwgQ291bnQgZmllbGQgaXMgdmFsaWQgb25seSBpbiBjYXNlIGVpdGhlciB0aGUgVSBiaXQg b3IgdGhlIE8gYml0IGlzIHNldC4gSWYgbmVpdGhlciBiaXQgaXMgc2V0LCB0aGUgQmFzaWMgUmVz aWR1YWwgQ291bnQgZmllbGQgU0hPVUxEIGJlIHplcm8uICBJZiB0aGUgVSBiaXQgaXMgc2V0LCB0 aGUgQmFzaWMgUmVzaWR1YWwgQ291bnQgaW5kaWNhdGVzIGhvdyBtYW55IGJ5dGVzIHdlcmUgbm90 IHRyYW5zZmVycmVkIG91dCBvZiB0aG9zZSBleHBlY3RlZCB0byBiZSB0cmFuc2ZlcnJlZC4gIElm IHRoZSBPIGJpdCBpcyBzZXQsIHRoZSBCYXNpYyBSZXNpZHVhbCBDb3VudCBpbmRpY2F0ZXMgaG93 IG1hbnkgYnl0ZXMgY291bGQgbm90IGJlIHRyYW5zZmVycmVkIGJlY2F1c2UgdGhlIGluaXRpYXRv cidzIEV4cGVjdGVkIERhdGEgVHJhbnNmZXIgTGVuZ3RoIHdhcyB0b28gc21hbGwuDQoNCjIuNC4z IEJpZGktUmVhZCBSZXNpZHVhbCBDb3VudCANCg0KVGhlIEJpZGktUmVhZCBSZXNpZHVhbCBDb3Vu dCBmaWVsZCBpcyB2YWxpZCBvbmx5IGluIGNhc2UgZWl0aGVyIHRoZSB1IGJpdCBvciB0aGUgbyBi aXQgaXMgc2V0LiBJZiBuZWl0aGVyIGJpdCBpcyBzZXQsIHRoZSBCaWRpLVJlYWQgUmVzaWR1YWwg Q291bnQgZmllbGQgU0hPVUxEIGJlIHplcm8uICBJZiB0aGUgdSBiaXQgaXMgc2V0LCB0aGUgQmlk aS1SZWFkIFJlc2lkdWFsIENvdW50IGluZGljYXRlcyBob3cgbWFueSBieXRlcyB3ZXJlIG5vdCB0 cmFuc2ZlcnJlZCBpbiBvdXQgb2YgdGhvc2UgZXhwZWN0ZWQgdG8gYmUgdHJhbnNmZXJyZWQuICBJ ZiB0aGUgbyBiaXQgaXMgc2V0LCB0aGUgQmlkaS1SZWFkIFJlc2lkdWFsIENvdW50IGluZGljYXRl cyBob3cgbWFueSBieXRlcyBjb3VsZCBub3QgYmUgdHJhbnNmZXJyZWQgaW4gYmVjYXVzZSB0aGUg aW5pdGlhdG9yJ3MgRXhwZWN0ZWQgQmlkaS1SZWFkIFRyYW5zZmVyIExlbmd0aCB3YXMgdG9vIHNt YWxsLg0KDQoyLjQuNCBTdGF0dXMvUmVzcG9uc2UNCg0KVGhlIFN0YXR1cyBmaWVsZCBpcyB1c2Vk IHRvIHJlcG9ydCB0aGUgU0NTSSBzdGF0dXMgb2YgdGhlIGNvbW1hbmQgKGFzIHNwZWNpZmllZCBp biBbU0FNMl0pLiAgVGhlIFJlc3BvbnNlIGlzIHVzZWQgdG8gcmVwb3J0IGEgU2VydmljZSBSZXNw b25zZS4gVGhlIGV4YWN0IG1hcHBpbmcgb2YgdGhlIGlTQ1NJIHJlc3BvbnNlIGNvZGVzIHRvIFNB TSBzZXJ2aWNlIHJlc3BvbnNlIHN5bWJvbHMgaXMgb3V0c2lkZSB0aGUgc2NvcGUgb2YgdGhpcyBk b2N1bWVudC4NCg0KSWYgYSBTQ1NJIGRldmljZSBlcnJvciBpcyBkZXRlY3RlZCB3aGlsZSBkYXRh IGZyb20gdGhlIGluaXRpYXRvciBhcmUgc3RpbGwgZXhwZWN0ZWQgKHRoZSBjb21tYW5kIFBEVSBk aWQgbm90IGNvbnRhaW4gYWxsIHRoZSBkYXRhIGFuZCB0aGUgdGFyZ2V0IGhhcyBub3QgcmVjZWl2 ZWQgYSBEYXRhIFBEVSB3aXRoIHRoZSBmaW5hbCBiaXQgU2V0KSB0aGUgdGFyZ2V0IE1VU1Qgd2Fp dCB1bnRpbCByZWNlaXZpbmcgdGhlIGEgRGF0YSBQRFUgd2l0aCB0aGUgRiBiaXQgc2V0IGJlZm9y ZSAgc2VuZGluZyB0aGUgUmVzcG9uc2UgUERVLiANCg0KVmFsaWQgUmVzcG9uc2UgY29kZXMgYXJl Og0KDQoxIC0gVGFyZ2V0IEZhaWx1cmUNCjIgLSBEZWxpdmVyeSBTdWJzeXN0ZW0gRmFpbHVyZQ0K MyAtIEZvcm1hdCBFcnJvcg0KNCAtIFJldHJ5IFJlamVjdGVkDQoNCjIuNC41IFNSLWxlbmd0aCAN Cg0KTGVuZ3RoIG9mIHNlbnNlIGRhdGEgb3Igb2YgdGhlIHJlc3BvbnNlLg0KDQoyLjQuNiBTZW5z ZSBvciBSZXNwb25zZSBEYXRhDQoNCmlTQ1NJIHRhcmdldHMgTVVTVCBzdXBwb3J0IGFuZCBlbmFi bGUgYXV0b3NlbnNlLiAgSWYgdGhlIENvbW1hbmQgU3RhdHVzIHdhcyBDSEVDSyBDT05ESVRJT04g KDB4MDIpLCB0aGVuIHRoZSBTZW5zZSBEYXRhIGZpZWxkIHdpbGwgY29udGFpbiBzZW5zZSBkYXRh IGZvciB0aGUgZmFpbGVkIGNvbW1hbmQuICANCg0KRm9yIHNvbWUgaVNDU0kgcmVzcG9uc2VzIHRo ZSByZXNwb25zZSBmaWVsZCBNQVkgY29udGFpbiBzb21lIHJlc3BvbnNlIHJlbGF0ZWQgaW5mb3Jt YXRpb24uIEZvciBhIEZvcm1hdCBFcnJvciBSZXNwb25zZSBpdCBNVVNUIGNvbnRhaW4gdGhlIG9m ZnNldCBvZiB0aGUgZmlyc3Qgb2ZmZW5kaW5nIGJ5dGUgaW4gdGhlIGhlYWRlciByZWxhdGl2ZSB0 byB0aGUgc3RhcnQgb2YgdGhlIEJTSCBhcyBhIDIgYnl0ZSBpbnRlZ2VyLg0KDQoNCjIuNC43IFN0 YXRTTiAtIFN0YXR1cyBTZXF1ZW5jZSBOdW1iZXINCg0KU3RhdFNOIGlzIGEgU2VxdWVuY2UgTnVt YmVyIHRoYXQgdGhlIHRhcmdldCBpU0NTSSBsYXllciBnZW5lcmF0ZXMgcGVyIGNvbm5lY3Rpb24g YW5kIHRoYXQgaW4gdHVybiBlbmFibGVzIHRoZSBpbml0aWF0b3IgdG8gYWNrbm93bGVkZ2Ugc3Rh dHVzIHJlY2VwdGlvbi4gU3RhdFNOIGlzIGluY3JlbWVudGVkIGJ5IDEgZm9yIGV2ZXJ5IHJlc3Bv bnNlL3N0YXR1cyBzZW50IG9uIGEgY29ubmVjdGlvbi4NCg0KMi40LjggRXhwQ21kU04gLSBuZXh0 IGV4cGVjdGVkIENtZFNOIGZyb20gdGhpcyBpbml0aWF0b3INCg0KRXhwQ21kU04gaXMgYSBTZXF1 ZW5jZSBOdW1iZXIgdGhhdCB0aGUgdGFyZ2V0IGlTQ1NJIHJldHVybnMgdG8gdGhlIGluaXRpYXRv ciB0byBhY2tub3dsZWRnZSBjb21tYW5kIHJlY2VwdGlvbi4gSXQgaXMgdXNlZCB0byB1cGRhdGUg YSBsb2NhbCBjb3VudGVyIHdpdGggdGhlIHNhbWUgbmFtZS4NCg0KMi40LjkgTWF4Q21kU04gLSBt YXhpbXVtIENtZFNOIGFjY2VwdGFibGUgZnJvbSB0aGlzIGluaXRpYXRvcg0KDQpNYXhDbWRTTiBp cyBhIFNlcXVlbmNlIE51bWJlciB0aGF0IHRoZSB0YXJnZXQgaVNDU0kgcmV0dXJucyB0byB0aGUg aW5pdGlhdG9yIHRvIGluZGljYXRlIHRoZSBtYXhpbXVtIENtZFNOIHRoZSBpbml0aWF0b3IgY2Fu IHNlbmQuIEl0IGlzIHVzZWQgdG8gdXBkYXRlIGEgbG9jYWwgY291bnRlciB3aXRoIHRoZSBzYW1l IG5hbWUuDQoNCk1heENtZFNOIGFuZCBFeHBDbWRTTiBhcmUgcHJvY2Vzc2VkIGFzIGZvbGxvd3M6 DQoNCi1pZiB0aGUgUERVIE1heENtZFNOIGlzIGxlc3MgdGhhbiB0aGUgUERVIEV4cENtZFNOIChp biBTZXJpYWwgQXJpdGhtZXRpYyBTZW5zZSBhbmQgd2l0aCBhIGRpZmZlcmVuY2UgYm91bmRlZCBi eSAyKiozMS0xKSB0aGV5IGFyZSBib3RoIGlnbm9yZWQNCi1pZiB0aGUgUERVIE1heENtZFNOIGlz IGxlc3MgdGhhbiB0aGUgY3VycmVudCBNYXhDbWRTTiAoaW4gU2VyaWFsIEFyaXRobWV0aWMgU2Vu c2UgYW5kIHdpdGggYSBkaWZmZXJlbmNlIGJvdW5kZWQgYnkgMioqMzEtMSkgaXQgaXMgaWdub3Jl ZCBlbHNlIGl0IHVwZGF0ZXMgTWF4Q21kU04NCi1pZiB0aGUgUERVIEV4cENtZFNOIGlzIGxlc3Mg dGhhbiB0aGUgY3VycmVudCBFeHBDbWRTTiAoaW4gU2VyaWFsIEFyaXRobWV0aWMgU2Vuc2UgYW5k IHdpdGggYSBkaWZmZXJlbmNlIGJvdW5kZWQgYnkgMioqMzEtMSkgaXQgaXMgaWdub3JlZCBlbHNl IGl0IHVwZGF0ZXMgRXhwQ21kU04NCg0KVGhpcyBzZXF1ZW5jZSBpcyByZXF1aXJlZCBhcyB1cGRh dGVzIG1heSBhcnJpdmUgb3V0IG9mIG9yZGVyICh0aGV5IHRyYXZlbCBvbiBkaWZmZXJlbnQgVENQ IGNvbm5lY3Rpb25zKS4NCg0K --0__=DwvhFtsT5Ob1dZm4pxiT7mm10HH5PNhUfCds6FCa7J1Pn81xW8GzkGOQ-- From owner-ips@ece.cmu.edu Wed Jan 31 13:21:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA22521 for ; Wed, 31 Jan 2001 13:21:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0VFVDF26663 for ips-outgoing; Wed, 31 Jan 2001 10:31:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0VFUoZ26647 for ; Wed, 31 Jan 2001 10:30:50 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA112792; Wed, 31 Jan 2001 16:30:38 +0100 From: julian_satran@il.ibm.com Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id QAA137904; Wed, 31 Jan 2001 16:30:38 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12569E5.00552F9B ; Wed, 31 Jan 2001 16:30:27 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Black_David@emc.com cc: ips@ece.cmu.edu Message-ID: Date: Wed, 31 Jan 2001 17:26:04 +0200 Subject: RE: Comments on iSCSI -03 Mime-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=wkFYIdSB2YRdtjK0VK5qGrqggx4V5oGSkT7qKSPEWwp2MCdY7TmZ47cu" Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=wkFYIdSB2YRdtjK0VK5qGrqggx4V5oGSkT7qKSPEWwp2MCdY7TmZ47cu Content-type: text/plain; charset=us-ascii Content-Disposition: inline I did copy it to notepad and attach it. Notepad said something about unicode -:) (See attached file: binford.txt) --0__=wkFYIdSB2YRdtjK0VK5qGrqggx4V5oGSkT7qKSPEWwp2MCdY7TmZ47cu Content-type: application/octet-stream; name="binford.txt" Content-Disposition: attachment; filename="binford.txt" Content-Description: Text - character set unknown Content-Transfer-Encoding: base64 Q29tbWVudHMgaW4gdGV4dC4NCg0KVGhhbmtzLA0KSnVsbw0KDQpQbGVhc2UgcmVzcG9uZCB0byAi Q2hhcmxlcyBCaW5mb3JkIiA8Q2hhcmxlcy5CaW5mb3JkQEJsdWVTcHJ1Y2VOZXQuY29tPiANClRv OglKdWxpYW4gU2F0cmFuL0hhaWZhL0lCTUBJQk1JTA0KY2M6CSANClN1YmplY3Q6CUNvbW1lbnRz IG9uIGlTQ1NJIC0wMw0KDQoNCg0KDQoNCkp1bGlhbiwgIEJlbG93IGFyZSBzZXZlcmFsIGNvbW1l bnRzIG9uIGlTQ1NJLTAzLiAgST9tIGZhaXJseSBuZXcgdG8gdGhlDQppU0NTSSBkaXNjdXNzaW9u IChidXQgaGF2ZSBiZWVuIGludm9sdmVkIGluIFQxMC9UMTEgZm9yIHNldmVyYWwgeWVhcnMpIHNv IEkNCmFwb2xvZ2l6ZSBpZiBJP20gcmVoYXNoaW5nIGFscmVhZHkgbWFkZSBkZWNpc2lvbnMuICBJ P3ZlIHNlbnQgdGhpcyBkaXJlY3RseQ0KdG8geW91IGluc3RlYWQgb2YgdGhlIGdlbmVyYWwgcmVm bGVjdG9yIGJlY2F1c2UgbWFueSBvZiB0aGUgaXRlbXMgYXJlDQp0cml2aWFsIGVkaXRpbmcgaXNz dWVzLiAgUGxlYXNlIGZlZWwgZnJlZSB0byBjdXQgYW5kIHBhc3RlIHRvIHRoZSByZWZsZWN0b3IN CmFueSBpc3N1ZShzKSB5b3UgZmVlbCBqdXN0aWZ5IGEgd2lkZXIgZGlzY3Vzc2lvbi4gIChPZiBj b3Vyc2UgSSByZXNlcnZlIHRoZQ0KcmlnaHQgdG8gZG8gdGhlIHNhbWUgaWYgSSBzdHJvbmdseSBk aXNhZ3JlZSB3aXRoIHlvdXIgYW5zd2VyIDotKSApDQoNCkJlZm9yZSBJIGdldCBpbnRvIHNwZWNp ZmljcywgbGV0IG1lIHN0YXJ0IHdpdGggYW4gb3ZlcmFsbCBjb21tZW50LiAgQ29taW5nDQpmcm9t IFQxMC9UMTEgd29yaywgaXQgYm90aGVycyBtZSB0aGF0IG1hbnkgb2YgdGhlIFBEVSBkZXNjcmlw dGlvbnMgaW4NCnNlY3Rpb24gMiBhcmUgbm90IGNvbXBsZXRlLiAgSXQgc2VlbXMgdG8gbWUgYW4g YXR0aXR1ZGUgd2FzIHRha2VuIHRoYXQNCnVubGVzcyBhIGZpZWxkIGhhcyBuZXcgbWVhbmluZyBm b3IgdGhpcyBQRFUsIGl0IGRvZXNuP3QgbmVlZCBhbnkgdGV4dC4gIEkNCmRpc2FncmVlLiAgSSBi ZWxpZXZlIHRoYXQgZXZlcnkgZmllbGQgaW4gZXZlcnkgUERVIG5lZWRzIGEgZGVzY3JpcHRpb24u ICBJbg0KbWFueSBjYXNlcyB0aGF0IGRlc2NyaXB0aW9uIG1heSBvbmx5IHNwZWxsIG91dCB0aGUg YWNyb255bSBhbmQgdGhlbiBzYXkgP1NlZQ0KeC55PyBidXQgYXQgbGVhc3QgdGhlIGltcGxlbWVu dGVyIHVzaW5nIHRoZSBzcGVjIGZvciByZWZlcmVuY2UgY2FuIHF1aWNrbHkNCmdldCBhIHBvaW50 ZXIgdG8gdGhlIGF1dGhvcml0YXRpdmUgdGV4dCBvbiB0aGUgZ2l2ZW4gZmllbGQuDQoNClRoYW5r cyBmb3IgeW91ciB0aW1lLg0KQ2hhcmxlcyBCaW5mb3JkDQpCbHVlIFNwcnVjZSBOZXR3b3Jrcw0K DQoNCioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioNCltj Yi0wMV0gcGFnZXMgOCBhbmQgOQ0KSW4gdGhlIGZvbGxvd2luZyB0d28gcGxhY2VzID9DREI/IGlz IHJlZmVycmVkIHRvIGFzID9Db21tYW5kIERhdGEgQmxvY2s/Lg0KSXQ/cyByZWFsIGRlZmluaXRp b24gKGZyb20gU0FNKSBpcyBDb21tYW5kIERlc2NyaXB0b3IgQmxvY2suDQoqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqDQo8anM+IHdpbGwgY2hhbmdlIDwv anM+DQoxLiBPdmVydmlldw0KMS4xIFNDU0kgQ29uY2VwdHMNCi4gLiAuDQogICBDb21tYW5kIERh dGEgQmxvY2tzIChDREIpIGFyZSB0aGUgZGF0YSBzdHJ1Y3R1cmVzIHVzZWQgdG8gY29udGFpbiB0 aGUNCiAgIF5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl4NCiAgIGNvbW1hbmQgcGFyYW1ldGVycyB0 byBiZSBoYW5kZWQgYnkgYW4gaW5pdGlhdG9yIHRvIGEgdGFyZ2V0LiBUaGUgQ0RCDQogICBjb250 ZW50IGFuZCBzdHJ1Y3R1cmUgaXMgZGVmaW5lZCBieSBbU0FNXSBhbmQgZGV2aWNlLXR5cGUgc3Bl Y2lmaWMNCiAgIFNDU0kgc3RhbmRhcmRzLg0KDQoxLjIuMSBMYXllcnMgJiBTZXNzaW9ucw0KDQog ICBUaGUgZm9sbG93aW5nIGNvbmNlcHR1YWwgbGF5ZXJpbmcgbW9kZWwgaXMgdXNlZCBpbiB0aGlz IGRvY3VtZW50IHRvDQogICBzcGVjaWZ5IGluaXRpYXRvciBhbmQgdGFyZ2V0IGFjdGlvbnMgYW5k IGhvdyB0aG9zZSByZWxhdGUgdG8NCiAgIHRyYW5zbWl0dGVkIGFuZCByZWNlaXZlZCBQcm90b2Nv bCBEYXRhIFVuaXRzOg0KDQogICAgICAtdGhlIFNDU0kgbGF5ZXIgYnVpbGRzL3JlY2VpdmVzIFND U0kgQ0RCIChDb21tYW5kIERhdGEgQmxvY2tzKQ0KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgXl5eXl5eXl5eXl5eXl5eXl5eXg0KICAgICAgYW5kIHJlbGF5 cy9yZWNlaXZlcyB0aGVtIHdpdGggdGhlIHJlbWFpbmluZyBjb21tYW5kIGV4ZWN1dGUNCiAgICAg IHBhcmFtZXRlcnMgKGNmLiBTQU0tMikgdG8vZnJvbSB0aGUNCiAgICAgIC10aGUgaVNDU0kgbGF5 ZXIgdGhhdCBidWlsZHMvcmVjZWl2ZXMgaVNDU0kgUERVcyBhbmQNCiAgICAgIHJlbGF5cy9yZWNl aXZlcyB0aGVtIHRvL2Zyb20gLSBvbmUgb3IgbW9yZSBUQ1AgY29ubmVjdGlvbnMgdGhhdA0KICAg ICAgZm9ybSBhbiBpbml0aWF0b3ItdGFyZ2V0ICJzZXNzaW9uIi4NCg0KDQoqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqDQpbY2ItMDJdIHBhZ2UgMTENCkl0 IHNlZW1zIHRvIHJlYWQgYmV0dGVyIGlmIHlvdSBjaGFuZ2UgP3RhcmdldCBTQ1NJPyB0byA/U0NT SSB0YXJnZXQ/Lg0KKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKg0KDQoxLjIuMi4xIENvbW1hbmQgbnVtYmVyaW5nDQouIC4gLg0KICAgQ21kUk5zIGFyZSBz aWduaWZpY2FudCBvbmx5IGR1cmluZyBjb21tYW5kIGRlbGl2ZXJ5IHRvIHRoZSB0YXJnZXQuDQog ICBPbmNlIHRoZSBkZXZpY2Ugc2VydmluZyBwYXJ0IG9mIHRoZSB0YXJnZXQgU0NTSSBoYXMgcmVj ZWl2ZWQgYQ0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgXl5eXl5eXl5e Xl4NCiAgIGNvbW1hbmQsIENtZFJOIGNlYXNlcyB0byBiZSBzaWduaWZpY2FudC4gIER1cmluZyBj b21tYW5kIGRlbGl2ZXJ5IHRvDQogICB0aGUgdGFyZ2V0LCB0aGUgYWxsb2NhdGVkIG51bWJlcnMg YXJlIHVuaXF1ZSBzZXNzaW9uIHdpZGUuDQo8anM+d2lsbCBmaXg8L2pzPg0KDQoqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqDQpbY2ItMDNdIHBhZ2UgMTQN CkNoYW5nZSA/YW4/IHRvID9hbmQ/Lg0KKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKg0KDQoxLjIuNSBpU0NTSSBGdWxsIEZlYXR1cmUgUGhhc2UNCiAuIC4g Lg0KICAgdGhhdCB3YXMgdXNlZCB0byBkZWxpdmVyIHRoZSBTQ1NJIGNvbW1hbmQuICBJZiBhbiBp bml0aWF0b3IgaXNzdWVzIGENCiAgIFdSSVRFIGNvbW1hbmQsIHRoZSBpbml0aWF0b3IgbXVzdCBz ZW5kIHRoZSBkYXRhLCBpZiBhbnksIGZvciB0aGF0DQogICBjb21tYW5kIGFuZCB0aGUgdGFyZ2V0 IE1VU1QgcmV0dXJuIFIyVCwgaWYgYW55LCBhbiB0aGUgc3RhdHVzIG92ZXINCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIF5eXg0KICAgdGhlIHNhbWUg VENQIGNvbm5lY3Rpb24gdGhhdCB3YXMgdXNlZCB0byBkZWxpdmVyIHRoZSBTQ1NJIGNvbW1hbmQu DQoNCg0KPGpzPndpbGwgZml4PC9qcz4NCg0KKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKg0KW2NiLTA0XSBwYWdlIDIyDQpDaGFuZ2UgP1Jlc3BvbnNlIGJp dCAoYml0IDYpPyB0byA/UmVzcG9uc2UgYml0IChiaXQgNyk/Lg0KKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKg0KDQoyLjIuMSBPcGNvZGUNCg0KICAgVGhl IE9wY29kZSBpbmRpY2F0ZXMgd2hhdCB0eXBlIG9mIGlTQ1NJIFBEVSB0aGUgaGVhZGVyIGVuY2Fw c3VsYXRlcy4NCiAgIFRoZSBPcGNvZGUgaXMgZnVydGhlciBlbmNvZGVkIGFzIGZvbGxvd3M6DQoN CiAgICAgIGI3ICAgUmVzcG9uc2UNCiAgICAgIGI2LTAgT3BlcmF0aW9uDQoNCiAgIFRoZSBvcGNv ZGVzIGFyZSBkaXZpZGVkIGludG8gdHdvIGNhdGVnb3JpZXM6IGluaXRpYXRvciBvcGNvZGVzIGFu ZA0KICAgdGFyZ2V0IG9wY29kZXMuIEluaXRpYXRvciBvcGNvZGVzIGFyZSBpbiBQRFVzIHNlbnQg YnkgdGhlIGluaXRpYXRvcnMsDQogICBhbmQgdGFyZ2V0IG9wY29kZXMgYXJlIGluIFBEVXMgc2Vu dCBieSB0aGUgdGFyZ2V0LiBUaGUgaW5pdGlhdG9yIE1VU1QNCiAgIE5PVCBzZW5kIHRhcmdldCBv cGNvZGVzIGFuZCB0aGUgdGFyZ2V0IE1VU1QgTk9UIHNlbmQgaW5pdGlhdG9yDQogICBvcGNvZGVz LiAgVGFyZ2V0IG9wY29kZXMgYXJlIGFsc28gY2FsbGVkIHJlc3BvbnNlcyBhbmQgYXJlDQogICBk aXN0aW5ndWlzaGVkIGJ5IGhhdmluZyB0aGUgUmVzcG9uc2UgYml0IChiaXQgNikgc2V0IHRvIDEu DQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBeXl4NCjxq cz53aWxsIGZpeDwvanM+DQoNCg0KKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKg0KW2NiLTA1XSBwYWdlIDMwDQpBZ2FpbiwgST9tIGhhdmluZyBhIGhhcmQg dGltZSBmaW5kaW5nIGEgY29uY2lzZSBzZXQgb2YgcnVsZXMgb2YgaG93IHRoaXMNCnBhcmFtZXRl ciAoU3RhdFJOKSBpcyB1c2VkLiAgSXQgaXMgaW5jcmVtZW50ZWQgYnkgMSBmb3IgZXZlcnkgcmVz cG9uc2UsIGJ1dA0Kd2hhdCBpcyB0aGUgc3RhcnRpbmcgdmFsdWU7IDA/IDEsIGFueSBub24temVy bz8gIFdoYXQgY29uZGl0aW9ucyByZXNldCBpdCA/DQphbnk/ICBEb2VzIGlzIHJvbGwgb3ZlciBi YWNrIHRvIDAsIG9yIGRvZXMgaXQgZXhwbGljaXRseSBza2lwIDAgd2hlbiByb2xsaW5nDQpvdmVy IGZyb20gMHhmZmZmZmZmZj8NCioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioNCg0KMi40LjcgU3RhdFJOIC0gU3RhdHVzIFJlZmVyZW5jZSBOdW1iZXINCg0K ICAgU3RhdFJOIGlzIGEgcmVmZXJlbmNlIG51bWJlciB0aGF0IHRoZSB0YXJnZXQgaVNDU0kgbGF5 ZXIgZ2VuZXJhdGVzDQogICBwZXIgY29ubmVjdGlvbiBhbmQgdGhhdCBpbiB0dXJuIGVuYWJsZXMg dGhlIGluaXRpYXRvciB0byBhY2tub3dsZWRnZQ0KICAgc3RhdHVzIHJlY2VwdGlvbi4gU3RhdFJO IGlzIGluY3JlbWVudGVkIGJ5IDEgZm9yIGV2ZXJ5DQogICByZXNwb25zZS9zdGF0dXMgc2VudCBv biBhIGNvbm5lY3Rpb24uDQoNCjxqcz5UaGVyZSBpcyBhbiBJbml0U3RhdFNOIGZvciBldmVyeSBj b25uZWN0aW9uIGluIHRoZSBsb2dpbiByZXNwb25zZS4gVGhlIHZhbHVlIDAgaGFzIG5vIHNwZWNp YWwgc2lnbmlmaWNhbmNlIHNvIHlvdSB3cmFwIGFyb3VuZC4gQXMgdXN1YWxsIGlmIHRoZSBkaWZm ZXJlbmNlIGJldHdlZW4gZXhwZWN0ZWQgYW5kIHNlbnQgaXMgbGFyZ2VyIHRoYW4gMioqMzEtMSBh IHJlY292ZXJ5IGFjdGlvbiBNVVNUIGJlIHRha2VuLiA8L2pzPg0KKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKg0KW2NiLTA2XSBwYWdlIDMyDQpUaGlzIHNl Y3Rpb24gaW1wbGllcyBBQ0EgaGFuZGluZyBpcyBtYW5kYXRvcnkgaWYgQUUgcmVwb3J0aW5nIGlz IHN1cHBvcnRlZC4NClRoaXMgc2VlbXMgdG8gYmUgYW4gdW5uZWNlc3NhcnkgbGlua2luZyBvZiB0 d28gaW5kZXBlbmRlbnQgY29uY2VwdHMuDQoNCkF0IGxlYXN0IHBhcnQgb2Ygd2hhdCBJIHRoaW5r IGlzIGJlaW5nIHNvbHZlZCB3aXRoIHRoaXMgYXV0byBBQ0EgcmVxdWlyZW1lbnQNCmlzIGhhbmRs ZWQgYmV0dGVyIGJ5IHRoZSBuZXcgVGFzayBBYm9ydGVkIFN0YXR1cyBkZXNjcmliZWQgaW4gU0FN LTIuDQooP2JldHRlcj8gaW4gbXkgYmlhc2VkIG9waW5pb24gYW55d2F5ID8gSSB3YXMgdGhlIGF1 dGhvciBvZiB0aGUgVGFzayBBYm9ydGVkDQpTdGF0dXMgcHJvcG9zYWwuKSAgV2FzIHV0aWxpemlu ZyBUYXNrIEFib3J0ZWQgY29uc2lkZXJlZCBhbmQgcmVqZWN0ZWQsIG9yDQp3YXMgaXQgbm90IGV2 ZW4ga25vd24vdW5kZXJzdG9vZCBzaW5jZSBpdCBpcyBzbyBuZXc/DQoqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqDQoNCiAgIEZvciB0aGUgPENsZWFyIFRh c2sgU2V0PiwgaWYgU0NTSSBjb250cm9sIG1vZGUgZW5hYmxlcyBBRSByZXBvcnRpbmcsDQogICB0 aGUgdGFyZ2V0IE1VU1Qgc2VuZCBhbiBBc3luY2hyb25vdXMgRXZlbnQgdG8gYWxsIG90aGVyIGF0 dGFjaGVkDQogICBpbml0aWF0b3JzIHRvIGluZm9ybSB0aGVtIHRoYXQgYWxsIHBlbmRpbmcgdGFz a3MgYXJlIGNhbmNlbGxlZCBhbmQNCiAgIHRoZW4gZW50ZXIgdGhlIEFDQSBzdGF0ZSBmb3IgYW55 IGluaXRpYXRvciBmb3Igd2hpY2ggaXQgaGFkIHBlbmRpbmcNCiAgICAgICAgXl5eXl5eXl5eXl5e Xl5eXl5eXg0KICAgdGFza3MuDQoNCiAgIEZvciB0aGUgPFRhcmdldCBXYXJtIFJlc2V0PiBhbmQg PFRhcmdldCBDb2xkIFJlc2V0PiBmdW5jdGlvbnMsIHRoZQ0KICAgdGFyZ2V0IGNhbmNlbHMgYWxs IHBlbmRpbmcgb3BlcmF0aW9ucyBhbmQgYXJlIGJvdGggZXF1aXZhbGVudCB0byB0aGUNCiAgIFRh cmdldCBSZXNldCBhcyBzcGVjaWZpZWQgYnkgU0FNLTIuICBQcm92aWRlZCB0aGF0IFNDU0kgY29u dHJvbCBtb2RlDQogICBlbmFibGVzIEFFIHJlcG9ydGluZywgdGhlIHRhcmdldCBNVVNUIHNlbmQg YW4gQXN5bmNocm9ub3VzIEV2ZW50IHRvDQogICBhbGwgYXR0YWNoZWQgaW5pdGlhdG9ycyBub3Rp ZnlpbmcgdGhlbSB0aGF0IHRoZSB0YXJnZXQgaXMgYmVpbmcNCiAgIHJlc2V0Lg0KDQogICBJbiBh ZGRpdGlvbiwgZm9yIHRoZSA8VGFyZ2V0IFdhcm0gUmVzZXQ+IHRoZSB0YXJnZXQgd2lsbCBlbnRl ciB0aGUNCiAgIEFDQSBzdGF0ZSBvbiBhbGwgc2Vzc2lvbnMgYW5kIGFsbCBMVXMgb24gd2hpY2gg YW4gQUUgd2FzIHNlbnQuDQogICBeXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5e DQo8anM+IFdlIGhlYXJkIGFib3V0IHRhc2sgYWJvcnRlZCBidXQgaGFkIGxpdHRsZSB1bmRlcnN0 YW5kaW5nIGFib3V0IGhvdyB0byB1c2UgaXQuDQpJZiBsaWtlIEFDQSBpdCB3aWxsIHN0YXkgb24g dW50aWxsIGNsZWFyZWQgYW5kIHdpbGwgdGh1cyBzZXJ2ZSB1cyB0byBkcm9wIGFsbCBjb21tYW5k cyBpbiBmbGlnaHQgdGhlIHdlIGNvdWxkIHVzZSBpdC4gSWYgaXQgZG9lcyBub3Qgd2Ugd2lsbCBo YXZlIHRvIHN0YXkgd2l0aCBBQ0EuIEFnYWluIHRoZSBpc3N1ZSB3aGVyZSB0aGUgY29tbWFuZHMg aW4gZmxpZ2h0IDwvanM+DQoNCg0KKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKg0KW2NiLTA3XSBwYWdlIDQ2DQpOZWVkIHRvIHJld29yZCBzZW50ZW5jZSA/ IHRoZSB3b3JkID9idXQ/IGRvZXNuP3QgZml0Lg0KKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKg0KDQoyLjExLjIgVmVyc2lvbi1hY3RpdmUvbG93ZXN0DQoN CiAgIEluZGljYXRlcyB0aGUgdmVyc2lvbiBzdXBwb3J0ZWQgKHRoZSBoaWdoZXN0IHN1cHBvcnRl ZCBieSB0aGUgdGFyZ2V0DQogICBhbmQgaW5pdGlhdG9yKS4gSWYgdGhlIHRhcmdldCBpcyBub3Qg c3VwcG9ydGluZyBhIHZlcnNpb24gd2l0aGluIHRoZQ0KDQpTYXRyYW4sIEouICAgICAgICAgICBT dGFuZGFyZHMtVHJhY2ssIEF1Z3VzdCAyMDAxICAgICAgICAgICAgICAgICAgIDQ1DQoNCiAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgaVNDU0kgICAgICAgICAgICAgICAgSmFudWFyeSAx MCwgMjAwMA0KDQogICByYW5nZSBvZiB0aGUgaW5pdGlhdG9yIGl0IHdpbGwgcmVqZWN0IHRoZSBs b2dpbiBidXQgYW5kIHRoaXMgZmllbGQNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIF5eXl4NCiAgIHdpbGwgaW5kaWNhdGUgdGhlIGxvd2VzdCB2ZXJz aW9uIHN1cHBvcnRlZCBieSB0aGUgdGFyZ2V0Lg0KDQo8anM+d2lsbCBmaXg8L2pzPg0KDQoqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqDQpbY2ItMDhdIHBh Z2UgNTANClNlbnRlbmNlIG9yZ2FuaXphdGlvbjogIEluIHRoZSBkZXNjcmlwdGlvbiBvZiB0aGUg UCBiaXQsIHRoZSBsYXN0IHNlbnRlbmNlDQp0aGUgP0lGPyBpcyB0aGUgZmluYWwgcGhyYXNlLCBt YWtpbmcgaXQgaGFyZCB0byB1bmRlcnN0YW5kIG9uIHRoZSBmaXJzdA0KcmVhZGluZy4gIElmIHRo ZSBzZW50ZW5jZSBpcyB0dXJuZWQgYXJvdW5kIChJRiAuLi4gVEhFTiBzdHlsZSkgSSBiZWxpZXZl IGl0DQppcyBtdWNoIGNsZWFyZXIuDQoqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqDQoNCjIuMTMuMSBQIGJpdA0KDQogICBBIHRhcmdldCBtYXkgaXNzdWUg YSBOT1AtSW4gYnkgaXRzIG93biB0byB0ZXN0IGNvbm5lY3Rpb24gYW5kIHRoZQ0KICAgc3RhdGUg b2YgdGhlIGluaXRpYXRvci4gSW4gdGhpcyBjYXNlIHRoZSBJbml0aWF0b3IgVGFzayBUYWcgTVVT VCBiZSAwDQogICBhbmQgdGhlIFRhcmdldCBUYWcgTVVTVCBiZSBzZXQgKG5vdCB4J2ZmZmZmZmZm Jykgb25seSBpZiB0aGUgUCBiaXQgaXMNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBeXl5eXl5eXl5eXl5eXl5eXl5eXg0KICAgMS4NCiAgIF5eDQo8 anM+d2lsbCBmaXg8L2pzPg0KDQoqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqDQpbY2ItMDldIHBhZ2UgNTcNClR3byBwcm9ibGVtcyB3aXRoIHRoZSBmb2xs b3dpbmcgbGlzdDsgNSAmIDYgc2hvdWxkIGJlIDQgJiA1LCBhbmQgaXRlbSBudW1iZXINCjMgaXMg bm90IGluIFNBTS0yIHIxNSAodGhlIGxhdGVzdCB2ZXJzaW9uIEkgY291bGQgZmluZCkuICBXYXMg c29tZXRoaW5nDQpyZWNlbnRseSBhZGRlZD8gIEFnYWluLCBJIGJlbGlldmUgdGhlIG5ldyBUYXNr IEFib3J0ZWQgU3RhdHVzIHNvbHZlcyB0aGUNCnByb2JsZW0uDQoqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqDQoNCjIuMTcuMiBTQ1NJIEV2ZW50IEluZGlj YXRvcg0KDQogICBUaGUgZm9sbG93aW5nIHZhbHVlcyBhcmUgZGVmaW5lZC4gIChTZWUgW1NBTTJd IGZvciBkZXRhaWxzKToNCg0KICAgICAgMSAgICBBbiBlcnJvciBjb25kaXRpb24gd2FzIGVuY291 bnRlcmVkIGFmdGVyIGNvbW1hbmQNCiAgICAgIGNvbXBsZXRpb24uDQogICAgICAyICAgIEEgbmV3 bHkgaW5pdGlhbGl6ZWQgZGV2aWNlIGlzIGF2YWlsYWJsZSB0byB0aGlzIGluaXRpYXRvci4NCiAg ICAgIDMgICAgQWxsIFRhc2sgU2V0cyBhcmUgYmVpbmcgUmVzZXQgYnkgYW5vdGhlciBJbml0aWF0 b3INCiAgICAgIDUgICAgU29tZSBvdGhlciB0eXBlIG9mIHVuaXQgYXR0ZW50aW9uIGNvbmRpdGlv biBoYXMgb2NjdXJyZWQuDQogICAgICA2ICAgIEFuIGFzeW5jaHJvbm91cyBldmVudCBoYXMgb2Nj dXJyZWQuDQoNCg0KPGpzPm51bWJlcmluZyB3aWxsIGJlIGZpeGVkLiBJIHdpbGwgYmUgZ2xhZCB0 byBjaGFuZ2UvZWxsaW1pbmF0ZSAzIHRvIHRhc2sgYWJvcnRlZCBpZiBJIGdldCBhIGdvb2QgcG9p bnRlciBhbmQgaXQgaXMgZ29vZCBmb3IgY29tbWFuZHMgaW4gZmxpZ2h0IC0gbWVhbndoaWxlIHdv dWxkIGl0IGJlIGZhaXIgdG8gc2F5IHRoYXQgaXQgaXMgc3Vic3VtZWQgYnkgNiA8L2pzPg0KDQoN CioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioNCltjYi0x MF0gcGFnZSA1OQ0KQXQgdGhlIGVuZCBvZiBzZWN0aW9uIDIsIGFuZCB0aGVyZSBpcyBubyBkZXNj cmlwdGlvbiBvZiBNYXJrZXJzLiAgSSBiZWxpZXZlDQp0aGUgc3RhbmRhcmQgbmVlZHMgYSBmb3Jt YXQgZGlhZ3JhbSBqdXN0IGxpa2UgYW55IG9mIHRoZSBvdGhlciBQRFVzID8gbWF5YmUNCm5vdCBp biBzZWN0aW9uIDIsIGJ1dCBzb21ld2hlcmUuDQoqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqDQo8anM+IG1hcmtlcnMgd2lsbCBtb3ZlIHRvIGFuIGFwcGVu ZGl4IGFuZCBJJ2xsIGFkZCBhIGRlc2NyaXB0b3IgPC9qcz4NCioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioNCltjYi0xMV0gcGFnZSA2MA0KV2h5IHJlZGVm aW5lIHRoZSBtb2RlIHBhZ2U/Pz8gIFRoZSBNYXggQnVyc3QgU2l6ZSBhZCBGaXJzdCBCdXJzdCBT aXplIGhhdmUNCmJlZW4gZGVmaW5lZCBhcyA1MTIgYnl0ZSBjaHVua3MgZm9yIGEgbG9uZyB0aW1l LiAgQSBTQ1NJIHRhcmdldCBkb2Vzbj90IHdhbnQNCnRvIGhhdmUgdG8gdHJhbnNsYXRlIGZpZWxk cyBiYXNlZCBvbiB0aGUgcGFydGljdWxhciB0cmFuc3BvcnQuICBUaGUgNTEyDQpkZWZpbml0aW9u IGdpdmUgYSByYW5nZSBvZiB1cCB0byBqdXN0IHVuZGVyIDMyTUIgPyBzZWVtcyBsaWtlIGEgYmln IGVub3VnaA0KP2ZpcnN0IGJ1cnN0PyB0byBtZS4NCioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioNCg0KMy4xLjIgTWF4aW11bSBCdXJzdCBTaXplIGZpZWxk ICgxNiBiaXQpDQoNCiAgIFRoaXMgZmllbGQgaXMgdXNlZCBieSBpU0NTSSB0byBkZWZpbmUgdGhl IG1heGltdW0gZGF0YSBwYXlsb2FkIGluDQogICBpU0NTSSBkYXRhIFBEVXMgb3IgYXMgaW1tZWRp YXRlIGRhdGEgaW4gY29tbWFuZCBQRFVzIGluIHVuaXRzIG9mIDQwOTYNCiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBeXl5eXl5eXl5eXl5e Xg0KICAgYnl0ZXMuIFRoaXMgdmFsdWUgY2FuIGFsc28gYmUgc2V0IGJ5IGEgdGV4dC1tb2RlIGtl eTp2YWx1ZSBwYWlyDQogICAoRGF0YVBEVUxlbmd0aCkuDQoNCjMuMS4zIEZpcnN0IEJ1cnN0IFNp emUgZmllbGQgKDE2IGJpdCkNCg0KICAgVGhpcyBmaWVsZCBpcyB1c2VkIGJ5IGlTQ1NJIHRvIGRl ZmluZSB0aGUgbWF4aW11bSBvZiB1bnNvbGljaXRlZCBkYXRhDQogICBhbiBpU0NTSSBpbml0aWF0 b3IgaXMgYWxsb3dlZCB0byBzZW5kIHRvIHRoZSB0YXJnZXQgaW4gdW5pdHMgb2YgNDA5Ng0KICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIF5e Xl5eXl5eXl5eXl4NCiAgIGJ5dGVzLiBUaGlzIHZhbHVlIGNhbiBhbHNvIGJlIHNldCBieSBhIHRl eHQtbW9kZSBrZXk6dmFsdWUgcGFpcg0KICAgKEluaXRpYWxCdXJzdExlbmd0aCkuDQo8anM+IFdl IHdoZXJlICB0b2xkIHRoYXQgdGhvc2UgZmllbGRzIGFyZSBwcm90b2NvbCBzcGVjaWZpYyBhbmQg d2UgaGF2ZSB0byBkZWZpbmUgdGhlaXIgdmFsdWVzLiAgSWYgYSBtdWx0aXBsZSBvZiA1MTIgaXMg bW9yZSBhIGJldHRlciBmaXQgZm9yIFNDU0kgSSB3aWxsIGNoYW5nZSBpdCA8L2pzPg0KDQoNCkNo YXJsZXMgQmluZm9yZA0KQmx1ZSBTcHJ1Y2UgTmV0d29ya3MNCm9mZmljZS9jZWxsOiAoMzE2KSAy MTAtNjQwNA0KZS1mYXg6ICg1MDkpIDc1Ni00NDI1DQoNCg0K --0__=wkFYIdSB2YRdtjK0VK5qGrqggx4V5oGSkT7qKSPEWwp2MCdY7TmZ47cu-- From owner-ips@ece.cmu.edu Wed Jan 31 15:39:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA25355 for ; Wed, 31 Jan 2001 15:39:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0VIgHE04749 for ips-outgoing; Wed, 31 Jan 2001 13:42:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0VIfNZ04722 for ; Wed, 31 Jan 2001 13:41:23 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Wed, 31 Jan 2001 10:41:12 -0800 Message-ID: From: Joshua Tseng To: ips@ece.cmu.edu Subject: iSCSI: Text Commands Date: Wed, 31 Jan 2001 10:41:11 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, I think a statement needs to be added that the iSCSI initiator MUST support all of the text commands listed in Appendix C. Currently, it is unclear whether a target can respond with key:value pairs that were not solicited in the original text command. The spec should explicitly say the target can do so, and require the initiator to recognize all text keys. For example, if the target wants to specify a FirstBurst of less than 65535 (BTW, why is the the default value for FirstBurst so high???), it should be able to specify its own value for FirstBurst, even if the "FirstBurst:" text key was not listed in the original text command. Josh From owner-ips@ece.cmu.edu Wed Jan 31 15:41:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA25406 for ; Wed, 31 Jan 2001 15:41:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0VIIHl03678 for ips-outgoing; Wed, 31 Jan 2001 13:18:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0VIIBZ03666 for ; Wed, 31 Jan 2001 13:18:11 -0500 (EST) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id 1D8D627A8 for ; Wed, 31 Jan 2001 10:18:09 -0800 (PST) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id KAA23671; Wed, 31 Jan 2001 10:18:03 -0800 (PST) Message-ID: <3A785725.4B9ACFC3@hp.com> Date: Wed, 31 Jan 2001 10:19:17 -0800 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI : Digest Error recovery causes data corruption Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hello, I would invite you to revisit the Mike Krause mail about "iSCSI Data Integrity - Digests". If Alternative 1 or 2 is adopted, when a digest error (CRC) occurs, the segment could be discarded, and TCP will retransmit. For software implementation with an hardware assist to calculate the CRC, (same kind of the one that exists now to calculate the checksum now), if a CRC error is detected it could be assimilated as a checksum error. Hence there is no need to overload the iSCSI protocol to deal with digest errors, because the iSCSI layer could not see them. Why to do complicate when simple an efficient can be achieved? Regards, Pierre From owner-ips@ece.cmu.edu Wed Jan 31 15:42:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA25420 for ; Wed, 31 Jan 2001 15:42:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0VIwGC05489 for ips-outgoing; Wed, 31 Jan 2001 13:58:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1t.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0VIw8Z05482 for ; Wed, 31 Jan 2001 13:58:08 -0500 (EST) Received: from msgrel1.and.agilent.com (msgrel1.and.agilent.com [130.30.33.104]) by msgbas1t.cos.agilent.com (Postfix) with ESMTP id 2B1A8459 for ; Wed, 31 Jan 2001 11:58:07 -0700 (MST) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by msgrel1.and.agilent.com (Postfix) with ESMTP id 3F434134 for ; Wed, 31 Jan 2001 13:58:05 -0500 (EST) Received: from agilent.com (matt5670.rose.agilent.com [156.140.234.148]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id KAA18883 for ; Wed, 31 Jan 2001 10:58:02 -0800 (PST) Message-ID: <3A7860C3.E0764D73@agilent.com> Date: Wed, 31 Jan 2001 11:00:19 -0800 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.76 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI : Digest Error recovery causes data corruption References: <3A785725.4B9ACFC3@hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit At first glance, the idea is nice. However, options 1 and 2 require changes to the implementation and API of TCP to provide the framing. As you (should) know, we are not allowed to change TCP. Another problem is that your proposal to drop segments that have digest errors violates the network layering. TCP will deliver the PDUs to iSCSI. iSCSI is the layer that checks the CRC. When TCP delivers the data to iSCSI, it has already acknowledged the segment, so it can't be discarded and retransmitted. There is no way for iSCSI to tell TCP to accept or drop a segment. [all of the above assumes existing TCP definition/implementation. Anything is possible if one is allowed to change things] -Matt Pierre Labat wrote: > > Hello, > > I would invite you to revisit the Mike Krause mail about > "iSCSI Data Integrity - Digests". > > If Alternative 1 or 2 is adopted, when a digest error (CRC) > occurs, the segment could be discarded, and TCP will > retransmit. For software implementation with an hardware > assist to calculate the CRC, (same kind of the one that exists > now to calculate the checksum now), if a CRC error is detected > it could be assimilated as a checksum error. > Hence there is no need to overload the iSCSI protocol > to deal with digest errors, because the iSCSI layer could not > see them. > > Why to do complicate when simple an efficient can be achieved? > > Regards, > > Pierre From owner-ips@ece.cmu.edu Wed Jan 31 16:19:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA25825 for ; Wed, 31 Jan 2001 16:19:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0VJLOY06648 for ips-outgoing; Wed, 31 Jan 2001 14:21:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0VJKGZ06574 for ; Wed, 31 Jan 2001 14:20:17 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0VKUi063794; Wed, 31 Jan 2001 12:30:44 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: , Subject: RE: iSCSI : Holes in StatSN Date: Wed, 31 Jan 2001 11:19:02 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, If there is a complete implementation of client and server sequencing without holes due to zero being a special case, then a sack mechanism should suffice in allowing the removal of the 'retry' mechanism and muddling with SCSI. You would then avoid impacting SCSI with errors induced by the iSCSI transport. The interesting aspect of server or target side implementation of a selective acknowledgement would be with the freedom allowed for the connection the sack is sent. As part of that selective acknowledgement, a digest error detected would help avoid the reliance on a timeout. Doug > I've also realized this and included a "sack" mechanism that kicks-in as > soon as the initiator gets a hole in the status order. ExpstatSN is still > needed to do a low cost bulk acknowledgement. > > With a data sequence we may want to use a similar mechanism to ask for a > missed data block as soon as we see one of its successors or the status. > > Julo > > Santosh Rao on 31/01/2001 10:30:59 > Subject: iSCSI : Holes in StatSN > > Julian & All, > > The StatSN ACK is of the form "ExpStatSN acknowledges all Status > upto the specified value", and hence, it cannot be sent until > all prior command status' have been received. > > If StatSN 1 were yet to be received and StatSN 2 - 1000 were received > thereafter, they cannot be acknowledged by the initiator since > the current model does not allow an ExpStatSN ack of 2 - 1000, > without also ACKing 1. > > Since StatSNs are assigned on a per-connection basis, this avoids > holes in StatSN received at the initiator. [since TCP orders > delivery to iSCSI within a connection]. However, this in itself > is insufficient and holes can still occur in the StatSN > sequence seen by the initiator, as explained below. > > Holes in StatSN sequence are seen by an initiator when > it detects a digest error on the Status PDU > [and discards that PDU] , thereby, dropping such > Status'. > > In such cases, the ExpStatSN acknowledgement is not straight > forward at the initiator and does involve some complexity to > keep track of possible holes in StatSN. Further, such holes > may never be filled, since the "retry" command only uses the > same CmdSN while the target sends its response on a different > StatSN. > > In the presence of StatSN holes, [and given the current model > of ExpStatSN], initiators will need to score-board received > StatSNs prior to sending ExpStatSN acknowledgements. > > A selective StatSN ack (i.e. ExpStatSN ACKs the specified StatSN) > is simpler to implement on the initiator side and allows for > quicker de-alloc of resources at the target end. > This could be considered as either a replacement for the existing > ExpStatSN model, or as a complement to it. (possibly indicated > by a SACK bit in the outbound PDUs containing the ExpStatSN). > > Regards, > Santosh > > ps : > There is an earlier thread that dates 10/26/00 > and is titled "Re: iSCSI Error Recovery", which proposed StatSN per > connection as a solution to this problem, but the problem does not > go away with just setting StatSNs per connection, since holes still > occur on digest errors. > > -- > ################################# > Santosh Rao > Software Design Engineer, > HP, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################# > > > > From owner-ips@ece.cmu.edu Wed Jan 31 17:17:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA26658 for ; Wed, 31 Jan 2001 17:17:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0VKJMV09176 for ips-outgoing; Wed, 31 Jan 2001 15:19:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0VKIHZ09130 for ; Wed, 31 Jan 2001 15:18:17 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0VLRe063836; Wed, 31 Jan 2001 13:27:44 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Pierre Labat" , Subject: RE: iSCSI : Digest Error recovery causes data corruption Date: Wed, 31 Jan 2001 12:15:58 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <3A785725.4B9ACFC3@hp.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Pierre, A digest error is not seen by TCP. TCP knows nothing about the content of the byte stream and depends only on the TCP 16-bit checksum. As such, TCP will not retransmit. There are two aspects to the iSCSI transport layer added to TCP. It should allow a mechanism to deal with transport related errors independent of SCSI and it allows multiple TCP connections to be shared as a single connection. As the client or initiator share a single sequence count, the handshake and selective acknowledgement is effectively independent of any connection. With an assumption the server or target response does not require strict ordering over multiple connections, each server connection enjoys an independent sequence. It should be through these sequences digest or header errors are recovered. The iSCSI header should be seen separate from SCSI. Presently this is not the case and the cause of complexity in my opinion. Doug > Hello, > > > I would invite you to revisit the Mike Krause mail about > "iSCSI Data Integrity - Digests". > > If Alternative 1 or 2 is adopted, when a digest error (CRC) > occurs, the segment could be discarded, and TCP will > retransmit. For software implementation with an hardware > assist to calculate the CRC, (same kind of the one that exists > now to calculate the checksum now), if a CRC error is detected > it could be assimilated as a checksum error. > Hence there is no need to overload the iSCSI protocol > to deal with digest errors, because the iSCSI layer could not > see them. > > Why to do complicate when simple an efficient can be achieved? > > Regards, > > Pierre > > From owner-ips@ece.cmu.edu Wed Jan 31 17:20:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA26705 for ; Wed, 31 Jan 2001 17:20:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f0VJLMW06643 for ips-outgoing; Wed, 31 Jan 2001 14:21:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sanlight.org (adsl-63-202-160-80.dsl.snfc21.pacbell.net [63.202.160.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f0VJKsZ06621 for ; Wed, 31 Jan 2001 14:20:54 -0500 (EST) Received: from ljoy ([10.0.0.18]) by gateway.sanlight.org (8.11.0/8.11.0) with SMTP id f0VKUg063791; Wed, 31 Jan 2001 12:30:42 -0800 (PST) (envelope-from dotis@sanlight.net) From: "Douglas Otis" To: "Sean Quinlan" , "Glen Turner" Cc: "IPS Reflector" Subject: RE: iSCSI Data Integrity - Digests Date: Wed, 31 Jan 2001 11:19:00 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <3A778275.9B41AEDD@research.bell-labs.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Sean, > I am certainly no expert in checksums, but I believe the adler32 checksum > used in zip (rfc1950) and SCTP (rfc2960) is claimed to be as good > as a 32 bit > CRC and it is more friendly to software implementation on a 32 bit > machine, though given that it uses the mod operator it is more difficult > than CRC to implement in hardware. Secure hashes such as md5 are also > fairly reasonable in software with the added bonus that they can resist > tampering - these are also probably difficult to do in hardware. I am not sure I understand the difficulty imposed by adler32 with respect to hardware. Optimal assembly code looks different with about three instructions per byte. Here is the example implementation of adler32 from Randall Stewarts reference code (edited): /* SCTP reference Implementation Copyright (C) 1999 Cisco And Motorola */ ui32 update_adler32 (ui32 adler, ui8 *buf, ui32 len) { ui32 s1 = adler & 0xffff; ui32 s2 = (adler >> 16) & 0xffff; i32 n; for (n = 0; n < len; n++, buf++) { s1 = (s1 + *buf); if (s1 >= BASE) s1 -= BASE; s2 = (s2 + s1); if (s2 >= BASE) s2 -= BASE; } return (s2 << 16) + s1; } Compliments to Mark Adler (mailto:madler@alumni.caltech.edu). Doug From owner-ips@ece.cmu.edu Wed Jan 31 21:18:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA00750 for ; Wed, 31 Jan 2001 21:18:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f110tTa21857 for ips-outgoing; Wed, 31 Jan 2001 19:55:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f110svZ21810 for ; Wed, 31 Jan 2001 19:54:57 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel1.hp.com (Postfix) with ESMTP id 3A1721B5B for ; Wed, 31 Jan 2001 16:54:46 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id QAA26630 for ; Wed, 31 Jan 2001 16:54:41 -0800 (PST) Message-ID: <3A78B474.4989B99B@cup.hp.com> Date: Wed, 31 Jan 2001 16:57:25 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 Cc: ips@ece.cmu.edu Subject: Re: iSCSI : Holes in StatSN References: Content-Type: multipart/mixed; boundary="------------EEE7707FF8A9E5D426E2B0F9" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------EEE7707FF8A9E5D426E2B0F9 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Julian, Once the selective StatSN ACK mechanism kicks in, how is the initiator to revert to the bulk StatSN ACK ? (i.e. when/how does an initiator realize that the hole is filled ?) Or, does it only use selective StatSN ACK from there on ? The difficulty lies in the fact that a hole created will never be filled since "retry" will result in target sending back a subsequent Status PDU with a different StatSN. However, the initiator does not know when to safely claim that the hole is filled (by sending a bulk StatSN ACK), since there is no way to detect this. Regards, Santosh julian_satran@il.ibm.com wrote: > I've also realized this and included a "sack" mechanism that kicks-in as > soon as the initiator gets a hole in the status order. ExpstatSN is still > needed to do a low cost bulk acknowledgement. > > With a data sequence we may want to use a similar mechanism to ask for a > missed data block as soon as we see one of its successors or the status. > > Julo > > Santosh Rao on 31/01/2001 10:30:59 > > Please respond to Santosh Rao > > To: ips@ece.cmu.edu (ips) > cc: > Subject: iSCSI : Holes in StatSN > > Julian & All, > > The StatSN ACK is of the form "ExpStatSN acknowledges all Status > upto the specified value", and hence, it cannot be sent until > all prior command status' have been received. > > If StatSN 1 were yet to be received and StatSN 2 - 1000 were received > thereafter, they cannot be acknowledged by the initiator since > the current model does not allow an ExpStatSN ack of 2 - 1000, > without also ACKing 1. > > Since StatSNs are assigned on a per-connection basis, this avoids > holes in StatSN received at the initiator. [since TCP orders > delivery to iSCSI within a connection]. However, this in itself > is insufficient and holes can still occur in the StatSN > sequence seen by the initiator, as explained below. > > Holes in StatSN sequence are seen by an initiator when > it detects a digest error on the Status PDU > [and discards that PDU] , thereby, dropping such > Status'. > > In such cases, the ExpStatSN acknowledgement is not straight > forward at the initiator and does involve some complexity to > keep track of possible holes in StatSN. Further, such holes > may never be filled, since the "retry" command only uses the > same CmdSN while the target sends its response on a different > StatSN. > > In the presence of StatSN holes, [and given the current model > of ExpStatSN], initiators will need to score-board received > StatSNs prior to sending ExpStatSN acknowledgements. > > A selective StatSN ack (i.e. ExpStatSN ACKs the specified StatSN) > is simpler to implement on the initiator side and allows for > quicker de-alloc of resources at the target end. > This could be considered as either a replacement for the existing > ExpStatSN model, or as a complement to it. (possibly indicated > by a SACK bit in the outbound PDUs containing the ExpStatSN). > > Regards, > Santosh > > ps : > There is an earlier thread that dates 10/26/00 > and is titled "Re: iSCSI Error Recovery", which proposed StatSN per > connection as a solution to this problem, but the problem does not > go away with just setting StatSNs per connection, since holes still > occur on digest errors. > > -- > ################################# > Santosh Rao > Software Design Engineer, > HP, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################# --------------EEE7707FF8A9E5D426E2B0F9 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------EEE7707FF8A9E5D426E2B0F9-- From owner-ips@ece.cmu.edu Wed Jan 31 21:18:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA00769 for ; Wed, 31 Jan 2001 21:18:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f1115Su22310 for ips-outgoing; Wed, 31 Jan 2001 20:05:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f11156Z22276 for ; Wed, 31 Jan 2001 20:05:06 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Wed, 31 Jan 2001 20:05:28 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07041014C5@corpmx9.isus.emc.com> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: RE: iSCSI Data Integrity - Digests Date: Wed, 31 Jan 2001 20:04:58 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > I am not sure I understand the difficulty imposed by adler32 with respect to > hardware. Optimal assembly code looks different with about three > instructions per byte. I believe the concern is about how fast a hot-rodded ASIC can go. The arithmetic involved in CRCs doesn't have to cope with carry/borrow propagation in contrast to the arithmetic involved in the Adler-32 modulus. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Wed Jan 31 21:18:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA00790 for ; Wed, 31 Jan 2001 21:18:56 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f110CQj19938 for ips-outgoing; Wed, 31 Jan 2001 19:12:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f110C9Z19923 for ; Wed, 31 Jan 2001 19:12:10 -0500 (EST) Received: by smtp.nishansystems.com with Internet Mail Service (5.5.2448.0) id ; Wed, 31 Jan 2001 16:12:03 -0800 Message-ID: From: Joshua Tseng To: Raghavendra Rao , ips@ece.cmu.edu Subject: RE: iSCSI : Login Phase Problems Date: Wed, 31 Jan 2001 16:12:00 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk JP, > What is being achieved by preparing the target to expect a > text command after > login ? > > It is the responsibility of the initiator to negotiate before > getting into > serious stuff - If it does serious stuff without > negotiations, or before > negotiations the defaults will take effect. > > I don't see a reason to change this. > > -JP > If this is the case, then we need a statement in the spec that says the target MUST support all defaults that are defined in Appendix C, as it might not be given an opportunity to change the defaults. Are you sure this is how we want to proceed? Wouldn't it be wise to support targets that need to change the defaults? For example, what of an iSCSI proxy target on an iSCSI-FC gateway that wants to use 16-bit task tags? Josh From owner-ips@ece.cmu.edu Wed Jan 31 22:35:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA04041 for ; Wed, 31 Jan 2001 22:35:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f111pUi24315 for ips-outgoing; Wed, 31 Jan 2001 20:51:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f111olZ24289 for ; Wed, 31 Jan 2001 20:50:48 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 0FD57592 for ; Wed, 31 Jan 2001 17:50:37 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA00510 for ; Wed, 31 Jan 2001 17:50:30 -0800 (PST) Message-ID: <3A78C189.4928526A@cup.hp.com> Date: Wed, 31 Jan 2001 17:53:13 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI : Holes in StatSN References: Content-Type: multipart/mixed; boundary="------------8F2D1551422435C886614B6F" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------8F2D1551422435C886614B6F Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit julian_satran@il.ibm.com wrote: > With a data sequence we may want to use a similar mechanism to ask for a > missed data block as soon as we see one of its successors or the status. Julian, The missing data PDU is missing due to either a header format error, header digest error or data digest error. [in all other cases, TCP ensures reliable delivery]. In 2 of the above 3 cases, [header format error & header digest error] the initiator CANNOT do a safe interpretation of the PDU header. Without interpreting the PDU header, the initiator does NOT get the Initiator Task Tag. Any request to re-send a particular data PDU MUST be qualified by : I.T.T + missing_DataSN [+ T.T.T + CmdSN, optionally]. Since I.T.T. cannot be reliably determined in 2 of the 3 cases, such a re-send request cannot be reliably achieved. The alternate proposal that was made should be considered in its place, which was to : - dis-allow overlapped data xfer's - initiators do a count check - a command level retry is performed at the iSCSI layer on detecting an underrun [due to a missing PDU]. On several ocassions, requests from different people have been made on this list to dis-allow overlapped data xfers. Can a WG consensus be sought on this issue to see if the benefits of allowing overlapped data xfer's offset its complexities and justify its support ? Regards, Santosh --------------8F2D1551422435C886614B6F Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------8F2D1551422435C886614B6F-- From owner-ips@ece.cmu.edu Wed Jan 31 22:36:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA04052 for ; Wed, 31 Jan 2001 22:36:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id f111ZUG23574 for ips-outgoing; Wed, 31 Jan 2001 20:35:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id f111YdZ23523 for ; Wed, 31 Jan 2001 20:34:40 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel3.hp.com (Postfix) with ESMTP id 240E7C7D; Wed, 31 Jan 2001 17:34:39 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA29205; Wed, 31 Jan 2001 17:34:34 -0800 (PST) Message-ID: <3A78BDCE.63FE5E23@cup.hp.com> Date: Wed, 31 Jan 2001 17:37:18 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.7 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng Cc: "'Ips@Ece. Cmu. Edu'" Subject: Re: iSCSI: CmdSN and Retry References: <001101c08b46$0582be80$8f406620@yp_portable.advansys.com> Content-Type: multipart/mixed; boundary="------------C2733CD7492453AD8A52E512" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------C2733CD7492453AD8A52E512 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit YP, Further discussion below. Regards, Santosh Y P Cheng wrote: > A few points herein would help reduce the confusion of ordered delivery > using CmdSN and error retry. > > If the initiator sends both commands A and B to a target and requires B to > follow A. After completing A with an error condition, nothing prevents the > target to start B immediately. Therefore, what do we accomplishing by > retrying A? The goal of the retry is to complete A. Not having a policy of retries will result in flaky I/Os with failures being propagated to the user application. Almost every storage stack does have SOME form of retries on failures, unless the failure is clearly a fatal problem and retry is known not to help. If your concern is regarding the re-ordering of execution caused by the retry, then, specific scenarios need to be discussed : a) Failure of A at the target [if A is part of an ordered set of commands being processed], should be accompanied by the target raising a CHECK CONDITION and entering a state of ACA. This will then involve complex initiator recovery to restore the order of I/Os being executed. None of this behaviour exists in most/all SCSI stacks today , and that is one of the reasons ordering is enforced by the layer above SCSI ULP. b) If the failure occurred while initiator was parsing the response sent to A [ex: format error, digest error], this is NOT a problem IF the target implements data/status recovery. IOW, such a problem will not result in re-ordering of execution since the original execution completed in the order desired, and the "retry" only returns data from the iSCSI layer's buffers. However, if the failure was detected at the initiator AND the target does NOT implement data/status recovery, then, the ordering is truly messed up. > In every SCSI implementation I know of, initiator is always > ultimately responsible of understanding the ordered execution as well as > retry. When you say initiator, do you mean the SCSI transport layer or the SCSI ULP or the layer above the ULP ? Ordered execution is typically managed by the layer above the SCSI ULP. The retry policy is handled by the SCSI ULP and could be complemented by the scsi transport if it does internal retries. > Therefore, the retry corner cases pointed by Santosh simply will not > exist with properly behaved initiator, unless an iSCSI initiator will behave > differently. Not sure which corner cases you are referring to. (Several have been pointed out.) Can you please state specific scenarios to illustrate which corner case is not likely to occur ? > Since TCP ensures ordered delivery, for an iSCSI session with > multiple TCP connections, all we need to do is to ensure the sequentiality > of CmdSN from multiple connections. Ideally, if all the traffic were simple task tag based traffic, even the above would not be necessary. If the traffic DID contain ordered tags I/Os [and the appln may be assuming end-to-end ordering], the above by itself is not sufficient. If the ordering did get messed up due to a failure either at the target or initiator, then, complex recovery schemes need to be resorted to, and in some cases, that may not help either. > I am not aware of any SCSI target allocates resource for status phase. Once > the status is sent, all resources are released. If the initiator times out > the status, it retries the whole command. What is the defn. of "sent" ? The target cannot release its resources until TCP ACK for all the octets of the Status PDU have been received. Since this mapping b/n TCP ACKs to an entire Status PDU is difficult to achieve, it must hold onto its resources until StatSN ACK is received. > A header digest error is same as a missing PDU except it is detect by iSCSI, > not TCP. Because TCP has delivered the segment, it is possible for the > receiver to quickly notify the sender to resend the erroneous header. How ? The initiator cannot request a re-send without having to specify I.T.T & DataSN in the request to re-send [+ optionally, CmdSN & T.T.T.] . With a header digest error, the I.T.T. in the PDU itself is un-trustworthy. Recovery in such cases cannot safely be done at a PDU level and must be performed at a command level. > In conclusion, if CmdSN is enforced, a target must take the TCP transport > delivery sequentially whether there is one or more TCP connections because > the missing one could just be an ordered-queue or head-of-queue. For a > header digest error, a target can't proceed until it gets the missing header > as long as CmdSN is non-zero. The above is obvious, since the missing CmdSN causes a hole in the CmdSN causing the target to stall all further processing of CmdSNs behind the missing one. > Since a target will always move to next > command as soon as it completes one with or without error, all application > software and initiators should know better not to send a SCSI request which > depends on the success of a previous request. This is correct, in the case where the target does NOT implement data/status recovery. i.e. Assume ordering is required, 2 commands , say, 1 & 2, executed in order at the target. Now, if 1 encountered a digest error or format error at the initiator, and was re-sent with the "retry" bit AND the target were to NOT implement data/status recovery, it would result in target executing 1, 2, 1. This may be a problem and canot be addressed, unless iSCSI mandates data/status recovery. --------------C2733CD7492453AD8A52E512 Content-Type: text/x-vcard; charset=us-ascii; name="santoshr.vcf" Content-Description: Card for Santosh Rao Content-Disposition: attachment; filename="santoshr.vcf" Content-Transfer-Encoding: 7bit begin:vcard n:Rao;Santosh tel;work:408-447-3751 x-mozilla-html:FALSE org:Hewlett Packard, Cupertino.;SISL adr:;;19420, Homestead Road, M\S 43LN, ;Cupertino.;CA.;95014.;USA. version:2.1 email;internet:santoshr@cup.hp.com title:Software Design Engineer x-mozilla-cpt:;21088 fn:Santosh Rao end:vcard --------------C2733CD7492453AD8A52E512--